DPDK-dev Archive on lore.kernel.org
 help / color / mirror / Atom feed
* Re: [PATCH 15/18] net/ixgbe: parse flow director filter
From: Ferruh Yigit @ 2016-12-22 10:44 UTC (permalink / raw)
  To: Zhao1, Wei, dev@dpdk.org; +Cc: Lu, Wenzhuo, Adrien Mazarguil
In-Reply-To: <A2573D2ACFCADC41BB3BE09C6DE313CA0200DDDD@PGSMSX103.gar.corp.intel.com>

On 12/22/2016 9:19 AM, Zhao1, Wei wrote:
> Hi, Yigit
> 
>> -----Original Message-----
>> From: Yigit, Ferruh
>> Sent: Wednesday, December 21, 2016 1:01 AM
>> To: Zhao1, Wei <wei.zhao1@intel.com>; dev@dpdk.org
>> Cc: Lu, Wenzhuo <wenzhuo.lu@intel.com>
>> Subject: Re: [dpdk-dev] [PATCH 15/18] net/ixgbe: parse flow director filter
>>
>> On 12/2/2016 10:43 AM, Wei Zhao wrote:
>>> From: wei zhao1 <wei.zhao1@intel.com>
>>>
>>> check if the rule is a flow director rule, and get the flow director info.
>>>
>>> Signed-off-by: wei zhao1 <wei.zhao1@intel.com>
>>> Signed-off-by: Wenzhuo Lu <wenzhuo.lu@intel.com>
>>> ---
>>
>> <...>
>>
>>> +	PATTERN_SKIP_VOID(rule, struct ixgbe_fdir_rule,
>>> +			  RTE_FLOW_ERROR_TYPE_ITEM_NUM);
>>> +	if (item->type != RTE_FLOW_ITEM_TYPE_ETH &&
>>> +	    item->type != RTE_FLOW_ITEM_TYPE_IPV4 &&
>>> +	    item->type != RTE_FLOW_ITEM_TYPE_IPV6 &&
>>> +	    item->type != RTE_FLOW_ITEM_TYPE_UDP &&
>>> +	    item->type != RTE_FLOW_ITEM_TYPE_VXLAN &&
>>> +	    item->type != RTE_FLOW_ITEM_TYPE_NVGRE) {
>>
>> This gives build error [1], there are a few more same usage:
>>
>> .../drivers/net/ixgbe/ixgbe_ethdev.c:9238:17: error: comparison of constant
>> 241 with expression of type 'const enum rte_flow_item_type' is always true
>> [-Werror,-Wtautological-constant-out-of-range-compare]
>>             item->type != RTE_FLOW_ITEM_TYPE_NVGRE) {
>>
>>
>>
> 
> Ok, I will add two type definition RTE_FLOW_ITEM_TYPE_NVGRE and RTE_FLOW_ITEM_TYPE_E_TAG  into const enum rte_flow_item_type to eliminate this problem.
> Thank you.
> 

CC: Adrien Mazarguil <adrien.mazarguil@6wind.com>

Yes, that is what right thing to do, since rte_flow patchset not merged
yet, perhaps Adrien may want to include this as next version of his
patchset?

What do you think Adrien?

Thanks,
ferruh

^ permalink raw reply

* Re: [PATCH 1/2] eth: get rid of goto's in rte_eth_dev_detach
From: Thomas Monjalon @ 2016-12-22 10:33 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: dev, Ferruh Yigit
In-Reply-To: <4179dac2-f433-ca40-2f4f-235abab77c6c@intel.com>

2016-12-08 10:25, Ferruh Yigit:
> On 12/8/2016 1:47 AM, Stephen Hemminger wrote:
> > --- a/lib/librte_ether/rte_ethdev.c
> > +++ b/lib/librte_ether/rte_ethdev.c
> > @@ -466,27 +466,20 @@ rte_eth_dev_attach(const char *devargs, uint8_t *port_id)
> >  int
> >  rte_eth_dev_detach(uint8_t port_id, char *name)
> >  {
> > -	int ret = -1;
> > +	int ret;
> >  
> > -	if (name == NULL) {
> > -		ret = -EINVAL;
> > -		goto err;
> > -	}
> > +	if (name == NULL)
> > +		return -EINVAL;
> >  
> >  	/* FIXME: move this to eal, once device flags are relocated there */
> > -	if (rte_eth_dev_is_detachable(port_id))
> > -		goto err;
> > +	ret = rte_eth_dev_is_detachable(port_id);
> > +	if (ret < 0)
> 
> rte_eth_dev_is_detachable() can return 1 to indicate device is not
> detachable.
> 
> > +		return  ret;

It should be

ret = rte_eth_dev_is_detachable(port_id);
if (ret)
	return -1;

^ permalink raw reply

* [PATCH v2 2/2] app/test: add SGL tests to cryptodev QAT suite
From: Arek Kusztal @ 2016-12-22 10:15 UTC (permalink / raw)
  To: dev
  Cc: fiona.trahe, pablo.de.lara.guarch, john.griffin, deepak.k.jain,
	Arek Kusztal
In-Reply-To: <1482401702-11039-1-git-send-email-arkadiuszx.kusztal@intel.com>

This commit adds GCM tests to use within scatter-gather list.
Test use direct chained mbufs created based on the input parameter
for max size for in place operations and out of place operations.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
 app/test/test_cryptodev.c                  | 356 +++++++++++++
 app/test/test_cryptodev_gcm_test_vectors.h | 823 ++++++++++++++++++++++++++++-
 2 files changed, 1176 insertions(+), 3 deletions(-)

diff --git a/app/test/test_cryptodev.c b/app/test/test_cryptodev.c
index 23e56ec..ae55657 100644
--- a/app/test/test_cryptodev.c
+++ b/app/test/test_cryptodev.c
@@ -5940,6 +5940,356 @@ test_authenticated_decryption_fail_when_corruption(
 }
 
 static int
+create_gcm_operation_SGL(enum rte_crypto_cipher_operation op,
+		const struct gcm_test_data *tdata,
+		void *digest_mem, uint64_t digest_phys)
+{
+	struct crypto_testsuite_params *ts_params = &testsuite_params;
+	struct crypto_unittest_params *ut_params = &unittest_params;
+
+	const unsigned int auth_tag_len = tdata->auth_tag.len;
+	const unsigned int iv_len = tdata->iv.len;
+	const unsigned int aad_len = tdata->aad.len;
+
+	unsigned int iv_pad_len = 0, aad_buffer_len = 0;
+
+	/* Generate Crypto op data structure */
+	ut_params->op = rte_crypto_op_alloc(ts_params->op_mpool,
+			RTE_CRYPTO_OP_TYPE_SYMMETRIC);
+	TEST_ASSERT_NOT_NULL(ut_params->op,
+		"Failed to allocate symmetric crypto operation struct");
+
+	struct rte_crypto_sym_op *sym_op = ut_params->op->sym;
+
+	sym_op->auth.digest.data = digest_mem;
+
+	TEST_ASSERT_NOT_NULL(sym_op->auth.digest.data,
+			"no room to append digest");
+
+	sym_op->auth.digest.phys_addr = digest_phys;
+	sym_op->auth.digest.length = auth_tag_len;
+
+	if (op == RTE_CRYPTO_CIPHER_OP_DECRYPT) {
+		rte_memcpy(sym_op->auth.digest.data, tdata->auth_tag.data,
+				auth_tag_len);
+		TEST_HEXDUMP(stdout, "digest:",
+				sym_op->auth.digest.data,
+				sym_op->auth.digest.length);
+	}
+
+	iv_pad_len = RTE_ALIGN_CEIL(iv_len, 16);
+
+	sym_op->cipher.iv.data = (uint8_t *)rte_pktmbuf_prepend(
+			ut_params->ibuf, iv_pad_len);
+
+	TEST_ASSERT_NOT_NULL(sym_op->cipher.iv.data,
+			"no room to prepend iv");
+
+	memset(sym_op->cipher.iv.data, 0, iv_pad_len);
+	sym_op->cipher.iv.phys_addr = rte_pktmbuf_mtophys(ut_params->ibuf);
+	sym_op->cipher.iv.length = iv_len;
+
+	rte_memcpy(sym_op->cipher.iv.data, tdata->iv.data, iv_pad_len);
+
+	aad_buffer_len = ALIGN_POW2_ROUNDUP(aad_len, 16);
+
+	sym_op->auth.aad.data = (uint8_t *)rte_pktmbuf_prepend(
+			ut_params->ibuf, aad_buffer_len);
+	TEST_ASSERT_NOT_NULL(sym_op->auth.aad.data,
+			"no room to prepend aad");
+	sym_op->auth.aad.phys_addr = rte_pktmbuf_mtophys(
+			ut_params->ibuf);
+	sym_op->auth.aad.length = aad_len;
+
+	memset(sym_op->auth.aad.data, 0, aad_buffer_len);
+	rte_memcpy(sym_op->auth.aad.data, tdata->aad.data, aad_len);
+
+	TEST_HEXDUMP(stdout, "iv:", sym_op->cipher.iv.data, iv_pad_len);
+	TEST_HEXDUMP(stdout, "aad:",
+			sym_op->auth.aad.data, aad_len);
+
+	sym_op->cipher.data.length = tdata->plaintext.len;
+	sym_op->cipher.data.offset = aad_buffer_len + iv_pad_len;
+
+	sym_op->auth.data.offset = aad_buffer_len + iv_pad_len;
+	sym_op->auth.data.length = tdata->plaintext.len;
+
+	return 0;
+}
+
+#define SGL_MAX_NO	16
+
+static int
+test_AES_GCM_authenticated_encryption_SGL(const struct gcm_test_data *tdata,
+		const int oop, uint32_t fragsz, uint32_t fragsz_oop)
+{
+	struct crypto_testsuite_params *ts_params = &testsuite_params;
+	struct crypto_unittest_params *ut_params = &unittest_params;
+	struct rte_mbuf *buf, *buf_oop = NULL, *buf_last_oop = NULL;
+	int retval;
+	int to_trn = 0;
+	int to_trn_tbl[SGL_MAX_NO];
+	int segs = 1;
+	unsigned int trn_data = 0;
+	uint8_t *plaintext, *ciphertext, *auth_tag;
+
+	if (fragsz > tdata->plaintext.len)
+		fragsz = tdata->plaintext.len;
+
+	uint16_t plaintext_len = fragsz;
+	uint16_t frag_size_oop = fragsz_oop ? fragsz_oop : fragsz;
+
+	if (fragsz_oop > tdata->plaintext.len)
+		frag_size_oop = tdata->plaintext.len;
+
+	int ecx = 0;
+	void *digest_mem = NULL;
+
+	uint32_t prepend_len = ALIGN_POW2_ROUNDUP(tdata->iv.len, 16)
+			+ tdata->aad.len;
+
+	if (tdata->plaintext.len % fragsz != 0) {
+		if (tdata->plaintext.len / fragsz + 1 > SGL_MAX_NO)
+			return 1;
+	}	else {
+		if (tdata->plaintext.len / fragsz > SGL_MAX_NO)
+			return 1;
+	}
+
+	/*
+	 * For out-op-place we need to alloc another mbuf
+	 */
+	if (oop) {
+		ut_params->obuf = rte_pktmbuf_alloc(ts_params->mbuf_pool);
+		rte_pktmbuf_append(ut_params->obuf,
+				frag_size_oop + prepend_len);
+		buf_oop = ut_params->obuf;
+	}
+
+	/* Create GCM session */
+	retval = create_gcm_session(ts_params->valid_devs[0],
+			RTE_CRYPTO_CIPHER_OP_ENCRYPT,
+			tdata->key.data, tdata->key.len,
+			tdata->aad.len, tdata->auth_tag.len,
+			RTE_CRYPTO_AUTH_OP_GENERATE);
+	if (retval < 0)
+		return retval;
+
+	ut_params->ibuf = rte_pktmbuf_alloc(ts_params->mbuf_pool);
+
+	/* clear mbuf payload */
+	memset(rte_pktmbuf_mtod(ut_params->ibuf, uint8_t *), 0,
+			rte_pktmbuf_tailroom(ut_params->ibuf));
+
+	plaintext = (uint8_t *)rte_pktmbuf_append(ut_params->ibuf,
+			plaintext_len);
+
+	memcpy(plaintext, tdata->plaintext.data, plaintext_len);
+
+	trn_data += plaintext_len;
+
+	buf = ut_params->ibuf;
+
+	/*
+	 * Loop until no more fragments
+	 */
+
+	while (trn_data < tdata->plaintext.len) {
+		++segs;
+		to_trn = (tdata->plaintext.len - trn_data < fragsz) ?
+				(tdata->plaintext.len - trn_data) : fragsz;
+
+		to_trn_tbl[ecx++] = to_trn;
+
+		buf->next = rte_pktmbuf_alloc(ts_params->mbuf_pool);
+		buf = buf->next;
+
+		memset(rte_pktmbuf_mtod(buf, uint8_t *), 0,
+				rte_pktmbuf_tailroom(buf));
+
+		/* OOP */
+		if (oop && !fragsz_oop) {
+			buf_last_oop = buf_oop->next =
+					rte_pktmbuf_alloc(ts_params->mbuf_pool);
+			buf_oop = buf_oop->next;
+			memset(rte_pktmbuf_mtod(buf_oop, uint8_t *),
+					0, rte_pktmbuf_tailroom(buf_oop));
+			rte_pktmbuf_append(buf_oop, to_trn);
+		}
+
+		plaintext = (uint8_t *)rte_pktmbuf_append(buf,
+				to_trn);
+
+		memcpy(plaintext, tdata->plaintext.data + trn_data,
+				to_trn);
+		trn_data += to_trn;
+		if (trn_data  == tdata->plaintext.len) {
+			if (oop) {
+				if (!fragsz_oop)
+					digest_mem = rte_pktmbuf_append(buf_oop,
+						tdata->auth_tag.len);
+			} else
+				digest_mem = (uint8_t *)rte_pktmbuf_append(buf,
+					tdata->auth_tag.len);
+		}
+	}
+
+	uint64_t digest_phys = 0;
+
+	ut_params->ibuf->nb_segs = segs;
+
+	segs = 1;
+	if (fragsz_oop && oop) {
+		to_trn = 0;
+		ecx = 0;
+
+		if (frag_size_oop == tdata->plaintext.len) {
+			digest_mem = rte_pktmbuf_append(ut_params->obuf,
+				tdata->auth_tag.len);
+
+			digest_phys = rte_pktmbuf_mtophys_offset(
+					ut_params->obuf,
+					tdata->plaintext.len + prepend_len);
+		}
+
+		trn_data = frag_size_oop;
+		while (trn_data < tdata->plaintext.len) {
+			++segs;
+			to_trn =
+				(tdata->plaintext.len - trn_data <
+						frag_size_oop) ?
+				(tdata->plaintext.len - trn_data) :
+						frag_size_oop;
+
+			to_trn_tbl[ecx++] = to_trn;
+
+			buf_last_oop = buf_oop->next =
+					rte_pktmbuf_alloc(ts_params->mbuf_pool);
+			buf_oop = buf_oop->next;
+			memset(rte_pktmbuf_mtod(buf_oop, uint8_t *),
+					0, rte_pktmbuf_tailroom(buf_oop));
+			rte_pktmbuf_append(buf_oop, to_trn);
+
+			trn_data += to_trn;
+
+			if (trn_data  == tdata->plaintext.len) {
+				digest_mem = rte_pktmbuf_append(buf_oop,
+					tdata->auth_tag.len);
+			}
+		}
+
+		ut_params->obuf->nb_segs = segs;
+	}
+
+	/*
+	 * Place digest at the end of the last buffer
+	 */
+	if (!digest_phys)
+		digest_phys = rte_pktmbuf_mtophys(buf) + to_trn;
+	if (oop && buf_last_oop)
+		digest_phys = rte_pktmbuf_mtophys(buf_last_oop) + to_trn;
+
+	if (!digest_mem && !oop) {
+		digest_mem = (uint8_t *)rte_pktmbuf_append(ut_params->ibuf,
+				+ tdata->auth_tag.len);
+		digest_phys = rte_pktmbuf_mtophys_offset(ut_params->ibuf,
+				tdata->plaintext.len);
+	}
+
+	/* Create GCM opertaion */
+	retval = create_gcm_operation_SGL(RTE_CRYPTO_CIPHER_OP_ENCRYPT,
+			tdata, digest_mem, digest_phys);
+
+	if (retval < 0)
+		return retval;
+
+	rte_crypto_op_attach_sym_session(ut_params->op, ut_params->sess);
+
+	ut_params->op->sym->m_src = ut_params->ibuf;
+	if (oop)
+		ut_params->op->sym->m_dst = ut_params->obuf;
+
+	/* Process crypto operation */
+	TEST_ASSERT_NOT_NULL(process_crypto_request(ts_params->valid_devs[0],
+			ut_params->op), "failed to process sym crypto op");
+
+	TEST_ASSERT_EQUAL(ut_params->op->status, RTE_CRYPTO_OP_STATUS_SUCCESS,
+			"crypto op processing failed");
+
+
+	ciphertext = rte_pktmbuf_mtod_offset(ut_params->op->sym->m_src,
+			uint8_t *, prepend_len);
+	if (oop) {
+		ciphertext = rte_pktmbuf_mtod_offset(ut_params->op->sym->m_dst,
+				uint8_t *, prepend_len);
+	}
+
+	if (fragsz_oop)
+		fragsz = fragsz_oop;
+
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(
+			ciphertext,
+			tdata->ciphertext.data,
+			fragsz,
+			"GCM Ciphertext data not as expected");
+
+	buf = ut_params->op->sym->m_src->next;
+	if (oop)
+		buf = ut_params->op->sym->m_dst->next;
+
+	unsigned int off = fragsz;
+
+	ecx = 0;
+	while (buf) {
+		ciphertext = rte_pktmbuf_mtod(buf,
+				uint8_t *);
+
+		TEST_ASSERT_BUFFERS_ARE_EQUAL(
+				ciphertext,
+				tdata->ciphertext.data + off,
+				to_trn_tbl[ecx],
+				"GCM Ciphertext data not as expected");
+
+		off += to_trn_tbl[ecx++];
+		buf = buf->next;
+	}
+
+	auth_tag = digest_mem;
+	TEST_ASSERT_BUFFERS_ARE_EQUAL(
+			auth_tag,
+			tdata->auth_tag.data,
+			tdata->auth_tag.len,
+			"GCM Generated auth tag not as expected");
+
+	return 0;
+}
+
+#define IN_PLACE	0
+#define OUT_OF_PLACE	1
+
+static int
+test_AES_GCM_auth_encrypt_SGL_out_of_place_400B_400B(void)
+{
+	return test_AES_GCM_authenticated_encryption_SGL(
+			&gcm_test_case_SGL_1, OUT_OF_PLACE, 400, 400);
+}
+
+static int
+test_AES_GCM_auth_encrypt_SGL_out_of_place_1500B_2000B(void)
+{
+	return test_AES_GCM_authenticated_encryption_SGL(
+			&gcm_test_case_SGL_1, OUT_OF_PLACE, 1500, 2000);
+}
+
+static int
+test_AES_GCM_auth_encrypt_SGL_in_place_1500B(void)
+{
+
+	return test_AES_GCM_authenticated_encryption_SGL(
+			&gcm_test_case_SGL_1, IN_PLACE, 1500, 0);
+}
+
+static int
 test_authentication_verify_fail_when_data_corrupted(
 		struct crypto_testsuite_params *ts_params,
 		struct crypto_unittest_params *ut_params,
@@ -6075,6 +6425,12 @@ static struct unit_test_suite cryptodev_qat_testsuite  = {
 
 		/** AES GCM Authenticated Encryption */
 		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_AES_GCM_auth_encrypt_SGL_in_place_1500B),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_AES_GCM_auth_encrypt_SGL_out_of_place_400B_400B),
+		TEST_CASE_ST(ut_setup, ut_teardown,
+			test_AES_GCM_auth_encrypt_SGL_out_of_place_1500B_2000B),
+		TEST_CASE_ST(ut_setup, ut_teardown,
 			test_mb_AES_GCM_authenticated_encryption_test_case_1),
 		TEST_CASE_ST(ut_setup, ut_teardown,
 			test_mb_AES_GCM_authenticated_encryption_test_case_2),
diff --git a/app/test/test_cryptodev_gcm_test_vectors.h b/app/test/test_cryptodev_gcm_test_vectors.h
index b404242..df984fc 100644
--- a/app/test/test_cryptodev_gcm_test_vectors.h
+++ b/app/test/test_cryptodev_gcm_test_vectors.h
@@ -35,6 +35,7 @@
 
 #define GMAC_LARGE_PLAINTEXT_LENGTH		65376
 
+
 struct gcm_test_data {
 	struct {
 		uint8_t data[64];
@@ -52,12 +53,12 @@ struct gcm_test_data {
 	} aad;
 
 	struct {
-		uint8_t data[1024];
+		uint8_t data[8096];
 		unsigned len;
 	} plaintext;
 
 	struct {
-		uint8_t data[1024];
+		uint8_t data[8096];
 		unsigned len;
 	} ciphertext;
 
@@ -65,7 +66,6 @@ struct gcm_test_data {
 		uint8_t data[16];
 		unsigned len;
 	} auth_tag;
-
 };
 
 struct gmac_test_data {
@@ -1235,4 +1235,821 @@ static const struct gmac_test_data gmac_test_case_4 = {
 	}
 };
 
+static const struct gcm_test_data gcm_test_case_SGL_1 = {
+	.key = {
+		.data = {
+			0xfe, 0xff, 0xe9, 0x92, 0x86, 0x65, 0x73, 0x1c,
+			0x6d, 0x6a, 0x8f, 0x94, 0x67, 0x30, 0x83, 0x08 },
+		.len = 16
+	},
+	.iv = {
+		.data = {
+			0xca, 0xfe, 0xba, 0xbe, 0xfa, 0xce, 0xdb, 0xad,
+			0xde, 0xca, 0xf8, 0x88 },
+		.len = 12
+	},
+	.aad = {
+		.data = { 0 },
+		.len = 0
+	},
+	.plaintext = {
+		.data = {
+			0xd9, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9a,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x55,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0xd9, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9a,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x55,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0xd9, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9a,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x55,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0xd9, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9a,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x55,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0xd9, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9a,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x55,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0xd9, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9a,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x55,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0xd9, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9a,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x55,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0xd9, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9a,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x55,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0xd9, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9a,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x55,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0xba, 0x63, 0x7b, 0x39, 0x1a, 0xaf, 0xd2, 0x54,
+			0xd7, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9c,
+			0xd8, 0x31, 0x32, 0x25, 0xf8, 0x84, 0x06, 0xe5,
+			0xa5, 0x59, 0x09, 0xc5, 0xaf, 0xf5, 0x26, 0x9b,
+			0x86, 0xa7, 0xa9, 0x53, 0x15, 0x34, 0xf7, 0xda,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+			0x2e, 0x4c, 0x30, 0x3d, 0x8a, 0x31, 0x8a, 0x72,
+			0x1c, 0x3c, 0x0c, 0x95, 0x95, 0x68, 0x09, 0x53,
+			0x2f, 0xcf, 0x0e, 0x24, 0x49, 0xa6, 0xb5, 0x25,
+			0xb1, 0x6a, 0xed, 0xf5, 0xaa, 0x0d, 0xe6, 0x57,
+
+		},
+		.len = 3120
+	},
+		.ciphertext = {
+			.data = {
+			0x42, 0x83, 0x1E, 0xC2, 0x21, 0x77, 0x74, 0x24,
+			0x4B, 0x72, 0x21, 0xB7, 0x84, 0xD0, 0xD4, 0x9C,
+			0xE3, 0xAA, 0x21, 0x2F, 0x2C, 0x02, 0xA4, 0xE0,
+			0x35, 0xC1, 0x7E, 0x23, 0x29, 0xAC, 0xA1, 0x2E,
+			0x21, 0xD5, 0x14, 0xB2, 0x54, 0x66, 0x93, 0x1C,
+			0x7D, 0x8F, 0x6A, 0x5A, 0xAC, 0x84, 0xAA, 0x05,
+			0x1B, 0xA3, 0x0B, 0x39, 0x6A, 0x0A, 0xAC, 0x97,
+			0x3D, 0x58, 0xE0, 0x91, 0x47, 0x3F, 0x59, 0x85,
+			0x05, 0x99, 0x55, 0xE1, 0x36, 0x76, 0xB7, 0x14,
+			0x1D, 0xF0, 0xF6, 0x8C, 0x65, 0xD5, 0xAD, 0xFA,
+			0x90, 0x7F, 0x5D, 0xA2, 0xD6, 0xFD, 0xD0, 0xE5,
+			0x0D, 0x9B, 0x68, 0x21, 0x49, 0x42, 0x6E, 0x13,
+			0xEC, 0x22, 0x50, 0x2A, 0x30, 0x47, 0x49, 0xA1,
+			0x7F, 0xC3, 0x09, 0xE0, 0x56, 0x91, 0xC4, 0x54,
+			0x70, 0xD7, 0x19, 0x40, 0xCA, 0x6B, 0x65, 0x27,
+			0x3E, 0xE9, 0xD1, 0x0F, 0x1C, 0xB5, 0x45, 0x0C,
+			0x29, 0xE7, 0xCF, 0x94, 0x10, 0xBF, 0xA2, 0xFA,
+			0x86, 0x20, 0x3F, 0x6E, 0xE9, 0x95, 0x03, 0x5C,
+			0x46, 0x11, 0x75, 0xD5, 0x37, 0x71, 0x7F, 0xE0,
+			0xBC, 0x9F, 0xC8, 0xE9, 0xB1, 0x08, 0x2C, 0x59,
+			0x6E, 0x51, 0x4A, 0x83, 0x38, 0xC1, 0xED, 0xE2,
+			0x2E, 0x88, 0x90, 0xA5, 0x7D, 0xA4, 0x93, 0x9A,
+			0x30, 0xD6, 0x96, 0x34, 0x0F, 0xC4, 0xD1, 0x7E,
+			0xC9, 0x8F, 0xC5, 0xBB, 0x80, 0x50, 0x85, 0x73,
+			0x8B, 0x7C, 0x0A, 0xDA, 0xD3, 0x37, 0x1C, 0x8B,
+			0x1E, 0xAE, 0x29, 0x54, 0x05, 0x53, 0x48, 0xE5,
+			0x94, 0xF1, 0xC5, 0x1A, 0x60, 0xDC, 0x61, 0x43,
+			0xCD, 0x45, 0x4C, 0x6B, 0x95, 0xAD, 0x52, 0xE0,
+			0x9E, 0xD1, 0x4E, 0xCC, 0x03, 0x27, 0x50, 0xD4,
+			0xEB, 0xBD, 0x71, 0xA6, 0xD0, 0x2B, 0x23, 0xC0,
+			0x9E, 0x5F, 0x34, 0xFD, 0xDE, 0xC1, 0x43, 0x35,
+			0x77, 0xFB, 0xFD, 0xDF, 0xA0, 0x28, 0x42, 0x3B,
+			0x0F, 0x2D, 0x31, 0xB4, 0x7A, 0xA8, 0x2F, 0xDF,
+			0x58, 0xB5, 0x00, 0x19, 0x8D, 0xEB, 0x2C, 0xBB,
+			0xAE, 0xAD, 0x74, 0x7F, 0x25, 0xAA, 0x24, 0x3E,
+			0xCD, 0x89, 0x5E, 0x05, 0xD3, 0xBA, 0x0E, 0x9A,
+			0x34, 0x7B, 0xE0, 0x11, 0xD2, 0xBA, 0x5A, 0x51,
+			0xB4, 0x0D, 0xEE, 0x61, 0x73, 0xFC, 0xD2, 0x01,
+			0x2D, 0x52, 0x3E, 0x37, 0x55, 0x3F, 0x58, 0xA8,
+			0x1C, 0x8F, 0x1D, 0xD6, 0x3C, 0x39, 0x06, 0x18,
+			0x65, 0x60, 0x55, 0x19, 0xAD, 0x1E, 0x78, 0xE9,
+			0xF7, 0xF5, 0xFC, 0xCD, 0x5F, 0xF1, 0x34, 0x0C,
+			0xA6, 0xFD, 0x1E, 0x9E, 0xB3, 0xCE, 0x2E, 0x10,
+			0xFB, 0x98, 0xDD, 0x0E, 0x09, 0x5D, 0x4E, 0x58,
+			0x75, 0x9A, 0x54, 0x74, 0xFB, 0x40, 0x76, 0x55,
+			0x0E, 0x3E, 0xA4, 0xCE, 0x56, 0xA5, 0xE0, 0x53,
+			0xB7, 0xAD, 0x36, 0x99, 0x6E, 0xCD, 0xC2, 0x90,
+			0x6E, 0xEA, 0xBC, 0x21, 0xAC, 0x31, 0xFF, 0x2B,
+			0x00, 0xA7, 0x5E, 0xC1, 0x7A, 0xF1, 0xAB, 0x24,
+			0xA3, 0x40, 0x0B, 0xEB, 0x16, 0x62, 0x35, 0x1E,
+			0xE9, 0xA5, 0xD3, 0x7E, 0xAA, 0x7E, 0x28, 0xA8,
+			0x3F, 0xD8, 0x0A, 0x04, 0x12, 0x0F, 0xFF, 0x68,
+			0x10, 0x85, 0x22, 0xD6, 0x05, 0x6A, 0x3A, 0xCB,
+			0xC0, 0xCF, 0x8C, 0x20, 0xF0, 0x34, 0x32, 0xAA,
+			0x76, 0x93, 0xE2, 0x23, 0x4F, 0xF2, 0xE6, 0x84,
+			0x3B, 0xD4, 0xF3, 0x5D, 0xF3, 0x17, 0xEE, 0x27,
+			0x67, 0xC3, 0x01, 0x6F, 0x32, 0xDE, 0xF6, 0xF6,
+			0x87, 0xE9, 0x82, 0xEF, 0x1F, 0xA1, 0xE2, 0x68,
+			0xF8, 0x5D, 0x49, 0x92, 0x47, 0x01, 0x75, 0x87,
+			0x52, 0xD3, 0x54, 0xAE, 0x3B, 0xB7, 0xB2, 0x07,
+			0x0F, 0x62, 0x7B, 0xF7, 0x50, 0x97, 0x9A, 0x4A,
+			0x98, 0x65, 0x23, 0xA3, 0x5D, 0x76, 0x0A, 0x9C,
+			0x6C, 0xE7, 0x89, 0xAD, 0x86, 0x70, 0xE7, 0x16,
+			0x5F, 0x2F, 0x2E, 0x97, 0x29, 0x31, 0xF0, 0x60,
+			0x33, 0x2C, 0xD7, 0xAA, 0xD6, 0xF0, 0x50, 0xB8,
+			0xBD, 0x29, 0xA8, 0xA9, 0xAC, 0x5E, 0x0A, 0x3A,
+			0x59, 0x34, 0x9A, 0x92, 0x25, 0x71, 0xB3, 0x16,
+			0xC5, 0xD3, 0xA4, 0x15, 0x75, 0x9A, 0xB5, 0x78,
+			0x6E, 0xCF, 0xAF, 0xC0, 0x39, 0x28, 0x44, 0x21,
+			0xBB, 0xE8, 0x32, 0xAB, 0xCB, 0xF8, 0x4B, 0xE7,
+			0x63, 0x9C, 0x56, 0xE7, 0xB2, 0xD6, 0x23, 0x17,
+			0xDE, 0x92, 0xE9, 0x22, 0xC3, 0x36, 0xA5, 0xAC,
+			0xA9, 0x98, 0x34, 0xAA, 0xFB, 0x03, 0x33, 0x33,
+			0xBE, 0xD8, 0x22, 0x7F, 0xFA, 0x34, 0xA0, 0x35,
+			0xC8, 0xA0, 0xDC, 0x35, 0x82, 0x06, 0x58, 0xE6,
+			0xBF, 0x7C, 0x4F, 0x63, 0x5D, 0x62, 0x64, 0x67,
+			0x0D, 0x07, 0x7F, 0x24, 0x4A, 0x23, 0xBC, 0x35,
+			0xE0, 0x92, 0x6F, 0x51, 0xE7, 0x25, 0x97, 0xB9,
+			0x14, 0x35, 0x2B, 0x48, 0xAC, 0x6F, 0x54, 0xDF,
+			0xF2, 0xB4, 0xB0, 0xE0, 0xD3, 0x28, 0x0D, 0x67,
+			0x48, 0x28, 0x0A, 0x16, 0x9C, 0x87, 0x73, 0xB7,
+			0x9C, 0x2B, 0xB5, 0x43, 0xC9, 0x46, 0xB9, 0x19,
+			0x01, 0xAA, 0xDE, 0x75, 0xA6, 0x0F, 0xB5, 0x72,
+			0x6A, 0x51, 0xE3, 0xAC, 0xE0, 0xF6, 0x96, 0x13,
+			0xBB, 0xC7, 0x08, 0x13, 0x9E, 0x47, 0xAA, 0xF5,
+			0x9E, 0x69, 0xAC, 0x95, 0x29, 0xFE, 0xFF, 0x99,
+			0xB2, 0x52, 0x72, 0x45, 0xF2, 0x07, 0xEB, 0x3C,
+			0x0F, 0x75, 0x29, 0x73, 0x0D, 0x77, 0x58, 0x83,
+			0xCB, 0xDD, 0xE7, 0x68, 0x1C, 0xE3, 0xD1, 0xA4,
+			0x5D, 0xD1, 0xAB, 0xB4, 0x5A, 0x3F, 0x27, 0x66,
+			0xDA, 0xB4, 0x81, 0x65, 0xCE, 0x1A, 0x9A, 0x7D,
+			0xC7, 0xB6, 0x31, 0xDE, 0x83, 0xC2, 0x7C, 0xF8,
+			0xD3, 0xC7, 0x97, 0x28, 0x50, 0xF2, 0x95, 0xFC,
+			0xA7, 0xB2, 0xA6, 0x46, 0xEF, 0x10, 0xD2, 0x38,
+			0x93, 0x14, 0x8D, 0xA7, 0x09, 0x17, 0x42, 0x7A,
+			0x85, 0xB9, 0x42, 0x71, 0x2A, 0x51, 0x9B, 0x66,
+			0x71, 0x12, 0x57, 0xB7, 0xBD, 0x26, 0xB7, 0x91,
+			0xF8, 0x84, 0x44, 0x35, 0xAD, 0x6F, 0xCB, 0xD7,
+			0xFC, 0xA1, 0x28, 0x77, 0x09, 0x5B, 0x6D, 0x52,
+			0x43, 0xA1, 0xE2, 0x0A, 0x7E, 0x5A, 0x84, 0x45,
+			0x20, 0xDE, 0xA5, 0x73, 0x1D, 0x37, 0x6E, 0xD8,
+			0x7A, 0x0D, 0x91, 0xBE, 0xF4, 0xB3, 0x89, 0xE9,
+			0x1F, 0x1E, 0xF6, 0xD5, 0x37, 0xB4, 0x3C, 0x1D,
+			0xBE, 0x0D, 0x5B, 0x01, 0xB0, 0x8B, 0xCE, 0x3E,
+			0x6D, 0x8B, 0x99, 0x9A, 0xC5, 0xAE, 0xFE, 0xA9,
+			0x78, 0x34, 0x20, 0xA7, 0x6C, 0x7D, 0x46, 0x72,
+			0x37, 0xAF, 0xFD, 0x17, 0x59, 0xED, 0x83, 0x5B,
+			0xEB, 0x6E, 0x4A, 0xF1, 0xE6, 0x0D, 0x44, 0x92,
+			0x65, 0x8E, 0x97, 0xD6, 0x83, 0x6E, 0x97, 0xCA,
+			0x4C, 0x0A, 0xCE, 0x32, 0x2A, 0xAD, 0x22, 0x73,
+			0xCB, 0xCB, 0xC3, 0x55, 0x08, 0x63, 0x23, 0xC2,
+			0x31, 0x24, 0x90, 0x54, 0x99, 0xB2, 0x8C, 0xC7,
+			0x8A, 0xB6, 0xFF, 0xC2, 0x75, 0xB1, 0xD9, 0x3D,
+			0x95, 0xDC, 0xB6, 0xCF, 0x11, 0x74, 0x06, 0x54,
+			0x03, 0xE3, 0x9B, 0x49, 0xE4, 0xF2, 0x73, 0x04,
+			0xF7, 0xDC, 0x71, 0xD7, 0xFA, 0x3C, 0xD2, 0x61,
+			0x77, 0x61, 0xB3, 0xDB, 0x6B, 0xCE, 0xCA, 0xFF,
+			0xF0, 0xAD, 0xBC, 0x94, 0xC8, 0xF8, 0xD5, 0xF4,
+			0x38, 0xA3, 0x61, 0xAA, 0x8C, 0x96, 0xEE, 0x56,
+			0xAC, 0xB4, 0x42, 0xBA, 0x1A, 0xE1, 0x70, 0x98,
+			0x1F, 0x9A, 0x6F, 0x98, 0xB9, 0x13, 0x46, 0xAB,
+			0x0B, 0xCD, 0xA3, 0x7B, 0x0C, 0xCB, 0x8F, 0x72,
+			0x23, 0xCF, 0x9E, 0xD8, 0xBB, 0x3F, 0x32, 0x27,
+			0x54, 0xB8, 0x60, 0x64, 0x83, 0xAE, 0x22, 0xD1,
+			0x6A, 0xC9, 0xF8, 0x13, 0xC4, 0xE4, 0xFF, 0x97,
+			0xD8, 0x92, 0xA3, 0xD1, 0xD4, 0x86, 0xD7, 0xC3,
+			0xBB, 0x40, 0xA2, 0x45, 0x78, 0xB1, 0xDB, 0x80,
+			0xC6, 0x8D, 0x0A, 0xF0, 0xC3, 0xC2, 0xE3, 0x48,
+			0xA1, 0x05, 0xC2, 0x32, 0xC8, 0x6C, 0x50, 0xA8,
+			0x06, 0x58, 0xBE, 0x6C, 0x7D, 0x22, 0xD6, 0x0D,
+			0x74, 0x40, 0xCE, 0xD6, 0x64, 0xD6, 0x47, 0xD0,
+			0xBF, 0xF1, 0x5C, 0x54, 0xF9, 0x06, 0x3F, 0x3D,
+			0x86, 0xBA, 0xF2, 0x0F, 0x5E, 0x2C, 0x01, 0xCC,
+			0xD9, 0xC7, 0xB1, 0x4A, 0xB3, 0xD7, 0x26, 0xCC,
+			0xC3, 0x7A, 0x74, 0x2C, 0xE1, 0x22, 0x65, 0xA0,
+			0x5B, 0xCA, 0xF4, 0xE1, 0x7D, 0xE1, 0x56, 0xFD,
+			0x95, 0x10, 0xC6, 0xA1, 0x4A, 0xE8, 0x6B, 0x34,
+			0x4E, 0x71, 0x60, 0x77, 0x0F, 0x03, 0xDD, 0xFE,
+			0xC8, 0x59, 0x54, 0x6C, 0xD4, 0x4A, 0x55, 0x24,
+			0x35, 0x21, 0x60, 0x73, 0xDF, 0x6F, 0xE7, 0x3C,
+			0xC2, 0xF0, 0xDA, 0xA9, 0xE5, 0x8C, 0xAC, 0xB6,
+			0xFD, 0x2E, 0xF7, 0xA0, 0x18, 0xA7, 0x55, 0x47,
+			0xD1, 0xCB, 0x9E, 0xAA, 0x58, 0x54, 0x3B, 0x37,
+			0x18, 0xB5, 0xC1, 0xBB, 0x41, 0x59, 0xE4, 0x29,
+			0x44, 0x13, 0x90, 0x6A, 0xF7, 0xD1, 0xB3, 0x71,
+			0xB6, 0x6E, 0xF6, 0x5D, 0x2E, 0x0E, 0x6C, 0x4C,
+			0x7B, 0xF7, 0xB6, 0x21, 0xD4, 0xFC, 0x47, 0x8C,
+			0x9B, 0x0A, 0x90, 0xAC, 0x11, 0x52, 0x86, 0x07,
+			0x24, 0xDA, 0xA9, 0x49, 0x50, 0xD9, 0xDC, 0xE2,
+			0x19, 0x87, 0x73, 0x88, 0xC3, 0xE4, 0xED, 0xC9,
+			0x1C, 0xA8, 0x7E, 0x39, 0x48, 0x91, 0x10, 0xAB,
+			0xFC, 0x3C, 0x1E, 0xEE, 0x08, 0xA1, 0xB9, 0xB4,
+			0xF4, 0xA9, 0x8D, 0xD0, 0x84, 0x7C, 0x8E, 0x54,
+			0xEF, 0x05, 0xC3, 0x2A, 0x0B, 0x8D, 0x3C, 0x71,
+			0xE7, 0x37, 0x27, 0x16, 0x07, 0xA2, 0x8F, 0x7A,
+			0x86, 0x05, 0x56, 0xA3, 0xB2, 0x75, 0xC5, 0x2C,
+			0xD4, 0x52, 0x60, 0x68, 0xA6, 0x6A, 0x48, 0xB6,
+			0x92, 0x50, 0xEC, 0x22, 0xAD, 0x01, 0x75, 0x57,
+			0xAF, 0xDF, 0x0F, 0x36, 0x93, 0x59, 0xF9, 0xE3,
+			0xA1, 0x41, 0x3B, 0x60, 0xB3, 0x13, 0x12, 0x50,
+			0x4B, 0x18, 0x20, 0xB9, 0x7B, 0x88, 0x27, 0x81,
+			0xB1, 0xDA, 0xCA, 0x6F, 0x63, 0x95, 0x40, 0xA1,
+			0x42, 0xE2, 0x14, 0xB8, 0x2B, 0x10, 0xB9, 0xDA,
+			0xE7, 0x30, 0x91, 0x13, 0x52, 0xC9, 0xA3, 0x5C,
+			0xD7, 0xBB, 0x39, 0x8F, 0x9A, 0xB8, 0xC5, 0xAF,
+			0xC6, 0x3E, 0x65, 0x90, 0x91, 0x8C, 0x9F, 0xDD,
+			0x84, 0xFB, 0xAD, 0x72, 0x4D, 0xD1, 0x42, 0xAD,
+			0x0A, 0x1B, 0x3A, 0xC6, 0x06, 0x03, 0x19, 0xCB,
+			0x31, 0x8C, 0x18, 0xD4, 0xEE, 0x90, 0x94, 0x3C,
+			0x44, 0xDC, 0xFB, 0x78, 0x5C, 0xB5, 0xE3, 0x2F,
+			0x89, 0x74, 0x0E, 0x28, 0x9C, 0xE4, 0xB4, 0xD2,
+			0xE3, 0x5A, 0x32, 0xF9, 0xC0, 0x81, 0x6A, 0x38,
+			0xC2, 0xCF, 0xD8, 0xD9, 0x3E, 0xAD, 0xF9, 0xB1,
+			0xA2, 0x55, 0x64, 0x1E, 0xEC, 0xF5, 0x0D, 0xB1,
+			0x8D, 0x07, 0x4E, 0xE5, 0x59, 0xE1, 0xE7, 0xFE,
+			0x4C, 0xCF, 0x11, 0xF8, 0x27, 0xC2, 0x29, 0xE2,
+			0xAF, 0x74, 0xAA, 0x53, 0x81, 0xD2, 0xFD, 0x5A,
+			0xF1, 0xEB, 0x96, 0x2C, 0x3E, 0x9B, 0xC2, 0x74,
+			0xFB, 0x65, 0x08, 0xA2, 0x63, 0xD3, 0xC5, 0x51,
+			0xAF, 0x19, 0x8B, 0x34, 0x8B, 0x7D, 0xB7, 0x97,
+			0x55, 0x97, 0x6D, 0x01, 0x5D, 0x98, 0xAA, 0x67,
+			0x11, 0xBD, 0xC2, 0x99, 0x2F, 0xB4, 0xCA, 0x04,
+			0x36, 0xF0, 0xB1, 0xA0, 0xBD, 0xA3, 0x4F, 0x4F,
+			0xB6, 0x7B, 0xF5, 0x1E, 0x38, 0x87, 0xC2, 0x38,
+			0x99, 0x5C, 0xE9, 0x2D, 0xDF, 0xAF, 0x5A, 0xF3,
+			0x7A, 0x17, 0x70, 0x35, 0xEC, 0xD5, 0x19, 0xF7,
+			0xB0, 0x21, 0x1E, 0x77, 0x30, 0x23, 0x54, 0x26,
+			0x61, 0x4E, 0xB9, 0x02, 0xDE, 0xF4, 0x86, 0x93,
+			0x47, 0x28, 0x43, 0x47, 0xB0, 0x56, 0xDC, 0x84,
+			0x3E, 0x6A, 0x6B, 0xEA, 0x4D, 0x63, 0xFE, 0x56,
+			0x5E, 0xF7, 0x6B, 0x1E, 0x5B, 0x63, 0xF1, 0x07,
+			0x20, 0x2E, 0x9B, 0xEE, 0xDC, 0x70, 0x5E, 0x36,
+			0x59, 0xE3, 0x3D, 0xA6, 0x0E, 0x50, 0x71, 0x06,
+			0xDD, 0x8B, 0x3C, 0xF7, 0xEC, 0x3C, 0x7A, 0x08,
+			0x8D, 0x4E, 0x6A, 0x08, 0xB0, 0xEE, 0x50, 0xE0,
+			0xF9, 0x0E, 0x40, 0xC0, 0x11, 0xBF, 0x8A, 0x17,
+			0x63, 0x9D, 0x59, 0x14, 0x0E, 0x25, 0x94, 0x09,
+			0xE6, 0x34, 0xEC, 0x0F, 0xE4, 0x7C, 0x59, 0xCD,
+			0x99, 0x85, 0x8E, 0x0F, 0xA1, 0x9E, 0x84, 0xBC,
+			0x13, 0x20, 0x5F, 0x56, 0x26, 0x10, 0x1A, 0x77,
+			0x77, 0x7B, 0x4B, 0x68, 0x13, 0x8A, 0x2C, 0xA5,
+			0x01, 0xBF, 0xAD, 0xF2, 0x2C, 0xD9, 0x4B, 0x24,
+			0x4C, 0xF5, 0x96, 0x4E, 0xD8, 0xE8, 0x98, 0xA8,
+			0x9C, 0x63, 0x2F, 0xC3, 0x26, 0xC7, 0x74, 0x83,
+			0x05, 0xED, 0x67, 0x02, 0x85, 0xAD, 0x1D, 0x0E,
+			0xA9, 0xD6, 0xE1, 0xC7, 0x39, 0xA0, 0x6E, 0x72,
+			0xCE, 0x56, 0x6C, 0xB8, 0x4A, 0xDE, 0x11, 0xA2,
+			0xBF, 0xC1, 0x84, 0x98, 0x8F, 0xCA, 0x79, 0x75,
+			0xC4, 0x9F, 0x45, 0x16, 0xBC, 0xB1, 0xF4, 0x03,
+			0x76, 0x6E, 0xD5, 0x46, 0x60, 0xD7, 0x1D, 0xF6,
+			0xD9, 0xBF, 0xF8, 0x71, 0xEB, 0x09, 0x33, 0x56,
+			0xE6, 0xEC, 0x72, 0xC8, 0xB3, 0x47, 0x14, 0x2C,
+			0x24, 0xA1, 0x1F, 0x16, 0xBE, 0x77, 0xFA, 0x9F,
+			0x6B, 0x83, 0x05, 0x03, 0x4D, 0x6F, 0xC9, 0x76,
+			0x69, 0x8D, 0xD7, 0x91, 0x26, 0x2B, 0x1C, 0x84,
+			0xF2, 0x2B, 0x23, 0xA6, 0xFF, 0x7B, 0xEE, 0xCC,
+			0x4E, 0x03, 0x8A, 0x80, 0x9E, 0x88, 0x96, 0xC3,
+			0x7A, 0x3E, 0x1B, 0xAC, 0x40, 0x84, 0xD1, 0x64,
+			0x89, 0x5F, 0xE3, 0x41, 0x89, 0x77, 0x4B, 0x28,
+			0x83, 0xCA, 0x78, 0x4F, 0x36, 0xC8, 0xCE, 0x53,
+			0x75, 0x39, 0x3A, 0x58, 0x92, 0x91, 0xF5, 0xA7,
+			0x6A, 0xD0, 0xB2, 0xBB, 0xFC, 0x8E, 0x3B, 0xFC,
+			0x83, 0x67, 0x42, 0xAA, 0x18, 0x51, 0x48, 0xD4,
+			0xC4, 0x85, 0x60, 0xA4, 0x2D, 0xD4, 0x4E, 0xA1,
+			0xF0, 0xB6, 0x41, 0x98, 0x6F, 0x84, 0xDE, 0x0C,
+			0x03, 0x8D, 0x83, 0x4A, 0x71, 0xBB, 0x32, 0x8B,
+			0x83, 0xF7, 0xD8, 0x08, 0x05, 0xA4, 0x48, 0xFE,
+			0xCA, 0xBB, 0x21, 0xA8, 0xBA, 0x2A, 0xD2, 0x65,
+			0x4E, 0xEF, 0xA1, 0x8F, 0x01, 0x09, 0xC6, 0x8C,
+			0xE5, 0x35, 0x32, 0xBB, 0x19, 0x15, 0xAB, 0x7A,
+			0xFD, 0x29, 0x76, 0xF9, 0xD1, 0xC5, 0x3E, 0xFD,
+			0x7A, 0x74, 0xBC, 0x41, 0x4F, 0x2C, 0x79, 0x6F,
+			0x45, 0x4E, 0xFD, 0x88, 0x49, 0x9A, 0x90, 0x6F,
+			0x65, 0x00, 0xC8, 0x08, 0xB8, 0x3B, 0x40, 0x06,
+			0x9A, 0x98, 0x5B, 0x6A, 0xD3, 0x5E, 0x32, 0x0E,
+			0xB0, 0x21, 0xE6, 0x2D, 0xEF, 0x7B, 0x99, 0x1B,
+			0xAF, 0x96, 0x20, 0x12, 0xE9, 0x31, 0xDA, 0x20,
+			0xB0, 0x27, 0x99, 0xC7, 0x14, 0x56, 0x3A, 0x08,
+			0x46, 0xA4, 0xB2, 0x0C, 0x6C, 0x1F, 0x1B, 0xAF,
+			0x9F, 0x90, 0x03, 0xBB, 0x03, 0xE0, 0x20, 0xE9,
+			0x45, 0x33, 0xA0, 0x3E, 0x01, 0x2C, 0xA7, 0x4A,
+			0xCC, 0xC6, 0xF5, 0xA3, 0x35, 0x0D, 0xE1, 0x5E,
+			0x90, 0x0B, 0xAC, 0x9A, 0x05, 0x79, 0xB2, 0x90,
+			0x39, 0xEE, 0xC8, 0x20, 0x55, 0xB3, 0x71, 0x46,
+			0xAC, 0x92, 0x42, 0x85, 0xD5, 0x12, 0x03, 0x8D,
+			0xBC, 0x82, 0xE7, 0x5A, 0x6E, 0x2E, 0x2C, 0xC0,
+			0xB6, 0x44, 0xF8, 0xBB, 0x5F, 0x7A, 0x42, 0x86,
+			0x28, 0xF0, 0x9B, 0xF9, 0x17, 0xDD, 0x35, 0x2F,
+			0x56, 0xE4, 0x63, 0xFF, 0xEC, 0x87, 0xC5, 0x53,
+			0xBF, 0x64, 0xB2, 0xDA, 0xDE, 0xC1, 0x6C, 0x85,
+			0x82, 0x51, 0x40, 0x41, 0xC9, 0x7A, 0x0A, 0xB8,
+			0xB2, 0x75, 0x03, 0x88, 0x22, 0x6D, 0x76, 0x6E,
+			0x2D, 0x2B, 0x73, 0xCB, 0x48, 0xC4, 0xED, 0xE0,
+			0x96, 0xFA, 0x36, 0x9F, 0x99, 0xC7, 0x97, 0xDE,
+			0x6D, 0xFC, 0x69, 0x86, 0x57, 0x5F, 0xB9, 0x93,
+			0x78, 0x5C, 0x07, 0x64, 0x61, 0xD0, 0x41, 0x14,
+			0x32, 0xED, 0xC0, 0xE4, 0xAC, 0xFC, 0x10, 0x0D,
+			0xAF, 0xEE, 0xDA, 0xB3, 0x6D, 0xB8, 0x7C, 0x10,
+			0xD5, 0x3B, 0x88, 0xE1, 0x15, 0xE1, 0xA4, 0x27,
+			0xFE, 0xEE, 0x0A, 0xC8, 0x95, 0xCF, 0xCA, 0x99,
+			0x98, 0x1D, 0xF3, 0x0E, 0xB8, 0x03, 0xD5, 0x51,
+			0x4B, 0x56, 0xB9, 0x07, 0x85, 0x58, 0x17, 0x51,
+			0x16, 0xC4, 0x86, 0xBB, 0xD3, 0x50, 0x01, 0x0E,
+			0x7B, 0x9C, 0xEF, 0xF0, 0x28, 0x4A, 0xD7, 0x3D,
+			0x1E, 0x3A, 0xBB, 0xCF, 0x2C, 0x90, 0x12, 0x2A,
+			0xB3, 0x90, 0x72, 0xE3, 0x93, 0x81, 0xE8, 0xA4,
+			0xEF, 0x8F, 0xD9, 0x45, 0x4F, 0xB1, 0xD0, 0x21,
+			0xDA, 0x20, 0x5C, 0xE9, 0x41, 0x41, 0x4E, 0x48,
+			0x95, 0x4D, 0x5A, 0xB3, 0xE5, 0x8B, 0xFC, 0xDE,
+			0xB9, 0x7B, 0x93, 0xBE, 0xA2, 0x74, 0x1B, 0xFA,
+			0xED, 0xCC, 0x0E, 0xDD, 0x96, 0x13, 0x2C, 0xAC,
+			0xDE, 0x2B, 0x2D, 0x8A, 0x30, 0x5A, 0xB8, 0x4B,
+			0x08, 0x2C, 0x74, 0xF7, 0xB4, 0x45, 0xD3, 0xA5,
+			0x62, 0x87, 0xCA, 0x16, 0xEB, 0x49, 0x46, 0x0C,
+			0x87, 0x7F, 0x11, 0x1D, 0x22, 0x66, 0x0A, 0x38,
+			0x90, 0x3A, 0x31, 0x38, 0x73, 0xB2, 0xD5, 0x5E,
+			0x06, 0xC4, 0x1E, 0x3D, 0xB7, 0x52, 0xB8, 0xE5,
+			0xC0, 0xF9, 0x72, 0xBC, 0x7A, 0x8A, 0xD3, 0xB4,
+			0x1D, 0xA9, 0x93, 0x3B, 0x7E, 0xFF, 0x8E, 0xA0,
+			0x96, 0x52, 0xE9, 0x9E, 0x60, 0x4C, 0x02, 0x90,
+			0xE5, 0x46, 0x92, 0xB3, 0xB8, 0x24, 0xE9, 0xD0,
+			0xCE, 0xD3, 0x0B, 0xCD, 0x8B, 0xE8, 0x72, 0xEA,
+			0x6E, 0xBF, 0x2B, 0x99, 0x6F, 0xC0, 0x65, 0xE8,
+			0x92, 0x30, 0x03, 0x28, 0xA9, 0xB0, 0xA7, 0x03,
+			0x92, 0x2C, 0xC8, 0x38, 0x8C, 0x38, 0x56, 0xEE,
+			0xDB, 0x39, 0xBD, 0x7E, 0xE9, 0x8D, 0xDB, 0xC1,
+			0xD5, 0x71, 0xC7, 0x84, 0xF3, 0xB2, 0x23, 0x22,
+			0xB5, 0x98, 0xB3, 0x36, 0xF1, 0xC4, 0xB1, 0xA4,
+			0xF2, 0x84, 0x24, 0xE5, 0x97, 0x48, 0x34, 0x43,
+			0xEF, 0xD9, 0xF4, 0x10, 0xE4, 0x13, 0xEE, 0x6C,
+			0xE7, 0x5D, 0x9B, 0xBA, 0x35, 0xF5, 0x7D, 0xE5,
+			0xBF, 0x8A, 0xCC, 0x3D, 0x28, 0xCF, 0xE8, 0x90,
+			0xE3, 0xCF, 0x01, 0x69, 0xD7, 0xC0, 0xD2, 0x2C,
+			0xC2, 0x9B, 0x89, 0xF2, 0xA9, 0x83, 0xA2, 0xA9,
+			0x12, 0xAA, 0x56, 0xD8, 0xCB, 0xA5, 0x8B, 0x0A,
+			0x03, 0xC1, 0xE1, 0x8E, 0x02, 0x36, 0x3D, 0x8F,
+			0x58, 0x4D, 0xEB, 0x93, 0x91, 0xC6, 0xE7, 0x22,
+			0xCE, 0xA8, 0x02, 0xD2, 0x82, 0x0D, 0x43, 0x4D,
+			0x4E, 0x11, 0xF8, 0x7B, 0x45, 0xD0, 0x23, 0xF7,
+			0x14, 0x35, 0x16, 0xA4, 0x0B, 0xAD, 0xFE, 0xE2,
+			0x2B, 0xFD, 0xF7, 0x17, 0xA9, 0x93, 0x77, 0x82,
+			0x45, 0x6E, 0x51, 0x1F, 0x5C, 0x2C, 0x5F, 0xFF,
+			0x1A, 0xA3, 0x0E, 0x29, 0xA5, 0x1D, 0xFD, 0x0E,
+			0xDD, 0x14, 0xF6, 0x69, 0x20, 0x15, 0xFD, 0xBB,
+			0xF8, 0xAF, 0x3D, 0xF3, 0xCC, 0xB8, 0x7E, 0x64,
+			0xED, 0x99, 0xF3, 0x1D, 0xFC, 0x96, 0xA2, 0x0A,
+			0x9C, 0xC2, 0x9B, 0xD7, 0x03, 0xA6, 0x79, 0x3B,
+			0x16, 0x0C, 0x6C, 0x5C, 0x2B, 0x61, 0x0E, 0x48,
+			0x96, 0x5C, 0x46, 0x7F, 0xC3, 0xCD, 0x3C, 0x10,
+			0x30, 0x8F, 0xC4, 0xB5, 0x92, 0x46, 0x1C, 0xDF,
+			0x10, 0xEE, 0x43, 0x27, 0x42, 0x70, 0xD2, 0xC4,
+			0x5E, 0x77, 0x78, 0x0E, 0x0E, 0xC3, 0x8B, 0x72,
+			0xA0, 0xFC, 0x4C, 0x0F, 0x5D, 0xBE, 0xBE, 0x07,
+			0x5B, 0x53, 0x38, 0xC8, 0x96, 0x82, 0x2D, 0x2D,
+			0x8E, 0xA8, 0x6C, 0x68, 0x34, 0x42, 0x31, 0x90,
+			0xD6, 0x4D, 0x29, 0xA9, 0x90, 0x95, 0x19, 0xD6,
+			0x8F, 0x2F, 0xF4, 0xD3, 0x71, 0x21, 0xB7, 0x7D,
+			0x51, 0xA6, 0x15, 0xE5, 0xDA, 0x08, 0x6A, 0x23,
+			0xDE, 0x6C, 0xBA, 0xCF, 0x84, 0xF1, 0x47, 0x25,
+			0x4A, 0xF1, 0x2F, 0x24, 0xED, 0x3B, 0xED, 0xF0,
+			0xA7, 0x48, 0xAE, 0x58, 0x7F, 0x0B, 0x3B, 0x78,
+			0xCE, 0x94, 0x32, 0x82, 0x63, 0x22, 0x67, 0xAA,
+			0x45, 0x37, 0xCC, 0x43, 0xD5, 0x10, 0x59, 0x5B,
+			0x09, 0xC6, 0x1C, 0x32, 0xCD, 0x19, 0xA2, 0x3C,
+			0x2B, 0x84, 0x03, 0xD5, 0x97, 0x20, 0xE7, 0xFB,
+			0x2D, 0x0A, 0x3C, 0x5C, 0xFD, 0x39, 0x9C, 0xDE,
+			0x02, 0x3D, 0xC7, 0xDD, 0x51, 0xDE, 0x99, 0xB3,
+			0x65, 0x00, 0x60, 0xCF, 0xAE, 0xCD, 0xE2, 0x83,
+			0xD5, 0x36, 0x2C, 0x89, 0x28, 0x6D, 0xC3, 0x6A,
+			0x80, 0xCD, 0x1A, 0xC3, 0x75, 0x11, 0x7E, 0x65,
+			0x2A, 0x44, 0x9D, 0xB5, 0x12, 0x2A, 0x78, 0xD0,
+			0x4D, 0xF8, 0x5E, 0xBF, 0xEC, 0x6B, 0x60, 0xD2,
+			0x89, 0x92, 0x5E, 0x17, 0xDA, 0x33, 0x83, 0xDB,
+			0xED, 0xF4, 0x5E, 0x82, 0xE9, 0x04, 0xD7, 0xE0,
+			0xA4, 0x1B, 0xFE, 0x32, 0x93, 0x05, 0x2C, 0xCF,
+			0xA2, 0xAE, 0x83, 0xCA, 0x2F, 0x5E, 0x47, 0x1C,
+			0x85, 0x0D, 0x01, 0xE5, 0x44, 0x3D, 0xE4, 0x58,
+			0x8E, 0xC0, 0x46, 0x05, 0x95, 0xBE, 0x59, 0xED,
+			0x0F, 0x7B, 0xA1, 0xF7, 0xDB, 0x2C, 0x79, 0x86,
+			0xE9, 0x54, 0x98, 0xA6, 0x2A, 0xD0, 0xFE, 0xC9,
+			0x59, 0x1D, 0x31, 0xC6, 0x27, 0x83, 0x2C, 0x12,
+			0x9C, 0xE1, 0x43, 0x3C, 0xEC, 0x65, 0x3B, 0xEF,
+			0xFD, 0x92, 0xBC, 0x0E, 0x38, 0xBA, 0x56, 0x1C,
+			0xC0, 0x81, 0x9E, 0xBE, 0x76, 0x59, 0x88, 0xA4,
+			0x0C, 0x6B, 0xD9, 0x7C, 0xD6, 0x8C, 0x32, 0xCD,
+			0x3F, 0xB6, 0xEF, 0xBF, 0xA6, 0xC7, 0xC9, 0xD3,
+			0x02, 0xB0, 0x3B, 0xFF, 0xFC, 0x4A, 0x97, 0x14,
+			0xFF, 0xF2, 0x48, 0xFE, 0x1B, 0xCE, 0x7D, 0x24,
+			0xA1, 0xD6, 0x03, 0xB0, 0x2F, 0xAA, 0xF7, 0x71,
+			0xC9, 0x0E, 0xCB, 0x57, 0xBA, 0xEF, 0xB5, 0x65,
+			0xE1, 0x44, 0xE4, 0x6A, 0xEB, 0xE8, 0x2B, 0x8F,
+			0x06, 0x23, 0x7A, 0xA9, 0x70, 0xAE, 0x48, 0x65,
+			0x94, 0xEE, 0xA5, 0x94, 0x78, 0x7D, 0x09, 0xF8,
+			0xB5, 0x4D, 0x64, 0x67, 0x10, 0x16, 0xA2, 0xFC,
+			0x49, 0x93, 0x76, 0x71, 0xED, 0x56, 0x25, 0xB5,
+			0x87, 0xE8, 0x84, 0x16, 0x55, 0xE1, 0x1E, 0x34,
+			0xE3, 0xB2, 0x49, 0x8F, 0xDC, 0xDA, 0xC3, 0x17,
+			0x82, 0x0E, 0x19, 0xD7, 0xE0, 0x09, 0xD7, 0xD9,
+			0x59, 0x6B, 0x55, 0x60, 0x1C, 0x1B, 0x02, 0xE8,
+			0xD1, 0x90, 0xF6, 0x3E, 0x94, 0x4A, 0x12, 0x0C,
+			0xBB, 0x69, 0xFD, 0x7C, 0xA0, 0xDD, 0x5F, 0x93,
+			0x9F, 0xFE, 0x2E, 0x79, 0xDB, 0xBE, 0x6F, 0x85,
+			0xAD, 0x9B, 0xDE, 0xAA, 0x10, 0xCA, 0xDB, 0xF2,
+			0xF9, 0xD0, 0x54, 0x15, 0x00, 0xF0, 0x6F, 0x86,
+			0x16, 0xF6, 0xA8, 0xA4, 0x08, 0x7B, 0x50, 0xF1,
+			0x35, 0xAC, 0xB6, 0xBB, 0x8B, 0xA0, 0x86, 0x3B,
+			0x3B, 0xDA, 0x9F, 0x89, 0xB5, 0x9C, 0x44, 0x41,
+			0x6A, 0xFD, 0x8A, 0x79, 0xA0, 0xFB, 0x7D, 0x1B,
+			0xE8, 0xC4, 0xA7, 0x3F, 0x66, 0x97, 0xA9, 0xF8,
+			0xEA, 0x0C, 0x30, 0x81, 0x63, 0xE4, 0xE3, 0x84,
+			0x62, 0xC5, 0x19, 0xFB, 0x00, 0xD6, 0x72, 0xE6,
+			0xC9, 0x6C, 0xDB, 0xEB, 0xF3, 0x6F, 0xDB, 0xE7,
+			0x00, 0x53, 0xCE, 0x1D, 0xE5, 0xF5, 0x53, 0x18,
+			0xE5, 0xAA, 0xDA, 0x90, 0x7B, 0xCB, 0x2B, 0x74,
+			0xED, 0x70, 0xFE, 0x90, 0xA8, 0xC8, 0x80, 0x2B,
+			0x93, 0x08, 0xDB, 0x6A, 0x0F, 0x3D, 0xA1, 0xFA,
+			0xB6, 0x63, 0x18, 0xF8, 0x43, 0x68, 0x00, 0xD0,
+			0x7A, 0x97, 0xCD, 0x5B, 0xB2, 0x84, 0x90, 0x06,
+			0xB9, 0x81, 0xC5, 0x81, 0x05, 0x55, 0x8C, 0xC4,
+			0x03, 0x89, 0xF5, 0x63, 0x87, 0x39, 0xEC, 0xD6,
+			0x89, 0x01, 0xE7, 0x1C, 0x4C, 0xDF, 0x5D, 0x65,
+			0xFE, 0x4B, 0x91, 0x04, 0x5B, 0x0E, 0x03, 0x38,
+			0x2F, 0x21, 0xA8, 0x36, 0x58, 0x93, 0xAD, 0x1F,
+			0xEB, 0xC3, 0x91, 0x90, 0x9B, 0x95, 0xCD, 0x53,
+			0x81, 0xAA, 0xA9, 0x48, 0x4D, 0x2B, 0x22, 0xC7,
+			0xBE, 0x1B, 0x38, 0x21, 0xA1, 0xFE, 0x23, 0xB4,
+			0xAC, 0x66, 0x92, 0x9E, 0xF2, 0x27, 0xDC, 0x23,
+			0x70, 0x6E, 0xBA, 0xF9, 0xED, 0x3B, 0xCE, 0x63,
+			0xAD, 0x68, 0xF2, 0x80, 0xFA, 0x1B, 0x14, 0xB5,
+			0xB4, 0x07, 0xE3, 0x5A, 0x81, 0x74, 0xE1, 0xF2,
+			},
+		.len = 3120
+	},
+	.auth_tag = {
+		.data = {
+			0xEA, 0xE9, 0x10, 0xB6, 0xB7, 0xAB, 0xEA, 0x90,
+			0x8A, 0xD5, 0x63, 0x88, 0xDB, 0x2B, 0x8F, 0x23,
+		},
+		.len = 16
+	}
+};
+
 #endif /* TEST_CRYPTODEV_GCM_TEST_VECTORS_H_ */
-- 
2.1.0

^ permalink raw reply related

* [PATCH v2 1/2] crypto/qat: add SGL capability to Intel QuickAssist driver
From: Arek Kusztal @ 2016-12-22 10:15 UTC (permalink / raw)
  To: dev
  Cc: fiona.trahe, pablo.de.lara.guarch, john.griffin, deepak.k.jain,
	Arek Kusztal
In-Reply-To: <1482401702-11039-1-git-send-email-arkadiuszx.kusztal@intel.com>

This commit adds scatter-gather list capability to Intel QuickAssist
Technology driver.

Signed-off-by: Arek Kusztal <arkadiuszx.kusztal@intel.com>
---
 doc/guides/rel_notes/release_17_02.rst |   2 +
 drivers/crypto/qat/qat_adf/qat_algs.h  |  14 +++-
 drivers/crypto/qat/qat_crypto.c        | 130 +++++++++++++++++++++++++++++++--
 drivers/crypto/qat/qat_crypto.h        |   3 +
 drivers/crypto/qat/qat_qp.c            |  55 +++++++++++++-
 5 files changed, 195 insertions(+), 9 deletions(-)

diff --git a/doc/guides/rel_notes/release_17_02.rst b/doc/guides/rel_notes/release_17_02.rst
index f3e7bb6..7eaf98e 100644
--- a/doc/guides/rel_notes/release_17_02.rst
+++ b/doc/guides/rel_notes/release_17_02.rst
@@ -44,6 +44,8 @@ New Features
 
   * DES algorithm.
 
+  * Scatter-gather list (SGL) support.
+
 
 Resolved Issues
 ---------------
diff --git a/drivers/crypto/qat/qat_adf/qat_algs.h b/drivers/crypto/qat/qat_adf/qat_algs.h
index 5409e1e..e01b9d7 100644
--- a/drivers/crypto/qat/qat_adf/qat_algs.h
+++ b/drivers/crypto/qat/qat_adf/qat_algs.h
@@ -47,6 +47,7 @@
 #ifndef _ICP_QAT_ALGS_H_
 #define _ICP_QAT_ALGS_H_
 #include <rte_memory.h>
+#include <rte_crypto.h>
 #include "icp_qat_hw.h"
 #include "icp_qat_fw.h"
 #include "icp_qat_fw_la.h"
@@ -79,13 +80,24 @@ struct qat_alg_buf {
 	uint64_t addr;
 } __rte_packed;
 
+/*
+ * Maximum number of SGL entries
+ */
+#define QAT_SGL_MAX_NUMBER	16
+
 struct qat_alg_buf_list {
 	uint64_t resrvd;
 	uint32_t num_bufs;
 	uint32_t num_mapped_bufs;
-	struct qat_alg_buf bufers[];
+	struct qat_alg_buf bufers[QAT_SGL_MAX_NUMBER];
 } __rte_packed __rte_cache_aligned;
 
+struct qat_crypto_op_cookie {
+	struct qat_alg_buf_list qat_sgl_list;
+	struct qat_alg_buf_list qat_sgl_list_oop;
+	phys_addr_t cookie_phys_addr;
+};
+
 /* Common content descriptor */
 struct qat_alg_cd {
 	struct icp_qat_hw_cipher_algo_blk cipher;
diff --git a/drivers/crypto/qat/qat_crypto.c b/drivers/crypto/qat/qat_crypto.c
index fa78c60..27e40fd 100644
--- a/drivers/crypto/qat/qat_crypto.c
+++ b/drivers/crypto/qat/qat_crypto.c
@@ -67,6 +67,10 @@
 
 #define BYTE_LENGTH    8
 
+#define SGL_SECOND_COOKIE_ADDR(arg, cast)	((cast)(arg) \
+				+ offsetof(struct qat_crypto_op_cookie, \
+					qat_sgl_list_oop))
+
 static const struct rte_cryptodev_capabilities qat_pmd_capabilities[] = {
 	{	/* SHA1 HMAC */
 		.op = RTE_CRYPTO_OP_TYPE_SYMMETRIC,
@@ -523,7 +527,8 @@ static inline uint32_t
 adf_modulo(uint32_t data, uint32_t shift);
 
 static inline int
-qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg);
+qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg,
+		struct qat_crypto_op_cookie *qat_op_cookie);
 
 void qat_crypto_sym_clear_session(struct rte_cryptodev *dev,
 		void *session)
@@ -900,9 +905,17 @@ qat_pmd_enqueue_op_burst(void *qp, struct rte_crypto_op **ops,
 	}
 
 	while (nb_ops_sent != nb_ops_possible) {
-		ret = qat_write_hw_desc_entry(*cur_op, base_addr + tail);
+
+		ret = qat_write_hw_desc_entry(*cur_op, base_addr + tail,
+				tmp_qp->sgl_cookies[tail / queue->msg_size]);
 		if (ret != 0) {
 			tmp_qp->stats.enqueue_err_count++;
+			/*
+			 * This message cannot be enqueued,
+			 * decrease number of ops that wasnt sent
+			 */
+			rte_atomic16_sub(&tmp_qp->inflights16,
+					nb_ops_possible - nb_ops_sent);
 			if (nb_ops_sent == 0)
 				return 0;
 			goto kick_tail;
@@ -911,6 +924,7 @@ qat_pmd_enqueue_op_burst(void *qp, struct rte_crypto_op **ops,
 		tail = adf_modulo(tail + queue->msg_size, queue->modulo);
 		nb_ops_sent++;
 		cur_op++;
+
 	}
 kick_tail:
 	WRITE_CSR_RING_TAIL(tmp_qp->mmap_bar_addr, queue->hw_bundle_number,
@@ -936,12 +950,13 @@ qat_pmd_dequeue_op_burst(void *qp, struct rte_crypto_op **ops,
 
 	while (*(uint32_t *)resp_msg != ADF_RING_EMPTY_SIG &&
 			msg_counter != nb_ops) {
+
 		rx_op = (struct rte_crypto_op *)(uintptr_t)
 				(resp_msg->opaque_data);
 
 #ifdef RTE_LIBRTE_PMD_QAT_DEBUG_RX
 		rte_hexdump(stdout, "qat_response:", (uint8_t *)resp_msg,
-				sizeof(struct icp_qat_fw_comn_resp));
+			sizeof(struct icp_qat_fw_comn_resp));
 #endif
 		if (ICP_QAT_FW_COMN_STATUS_FLAG_OK !=
 				ICP_QAT_FW_COMN_RESP_CRYPTO_STAT_GET(
@@ -950,7 +965,9 @@ qat_pmd_dequeue_op_burst(void *qp, struct rte_crypto_op **ops,
 		} else {
 			rx_op->status = RTE_CRYPTO_OP_STATUS_SUCCESS;
 		}
+
 		*(uint32_t *)resp_msg = ADF_RING_EMPTY_SIG;
+
 		queue->head = adf_modulo(queue->head +
 				queue->msg_size,
 				ADF_RING_SIZE_MODULO(queue->queue_size));
@@ -972,8 +989,61 @@ qat_pmd_dequeue_op_burst(void *qp, struct rte_crypto_op **ops,
 }
 
 static inline int
-qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg)
+qat_sgl_fill_array(struct rte_mbuf *buf, uint64_t buff_start,
+		void *sgl_cookie, uint32_t data_len)
+{
+	int nr = 1;
+	struct qat_alg_buf_list *list = sgl_cookie;
+
+	uint32_t buf_len = rte_pktmbuf_mtophys(buf) -
+			buff_start + rte_pktmbuf_data_len(buf);
+
+	list->bufers[0].addr = buff_start;
+	list->bufers[0].resrvd = 0;
+	list->bufers[0].len = buf_len;
+
+	if (data_len < buf_len) {
+		list->num_bufs = nr;
+		list->bufers[0].len = rte_pktmbuf_mtophys(buf) -
+			buff_start + data_len;
+		return 0;
+	}
+
+	buf = buf->next;
+	while (buf) {
+		if (unlikely(nr == QAT_SGL_MAX_NUMBER)) {
+			PMD_DRV_LOG(ERR, "QAT PMD exceeded size of QAT SGL"
+					" entry(%u)",
+					QAT_SGL_MAX_NUMBER);
+			return -EINVAL;
+		}
+
+		list->bufers[nr].len = rte_pktmbuf_data_len(buf);
+		list->bufers[nr].resrvd = 0;
+		list->bufers[nr].addr = rte_pktmbuf_mtophys(buf);
+
+		buf_len += list->bufers[nr].len;
+		buf = buf->next;
+
+		if (buf_len > data_len) {
+			list->bufers[nr].len -=
+				buf_len - data_len;
+			buf = NULL;
+		}
+
+		++nr;
+	}
+
+	list->num_bufs = nr;
+
+	return 0;
+}
+
+static inline int
+qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg,
+		struct qat_crypto_op_cookie *qat_op_cookie)
 {
+	int ret = 0;
 	struct qat_session *ctx;
 	struct icp_qat_fw_la_cipher_req_params *cipher_param;
 	struct icp_qat_fw_la_auth_req_params *auth_param;
@@ -983,6 +1053,7 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg)
 	uint32_t auth_len = 0, auth_ofs = 0;
 	uint32_t min_ofs = 0;
 	uint64_t src_buf_start = 0, dst_buf_start = 0;
+	uint8_t do_sgl = 0;
 
 
 #ifdef RTE_LIBRTE_PMD_QAT_DEBUG_TX
@@ -1100,10 +1171,16 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg)
 
 	}
 
+	if (op->sym->m_src->next || (op->sym->m_dst && op->sym->m_dst->next))
+		do_sgl = 1;
+
 	/* adjust for chain case */
 	if (do_cipher && do_auth)
 		min_ofs = cipher_ofs < auth_ofs ? cipher_ofs : auth_ofs;
 
+	if (unlikely(min_ofs >= rte_pktmbuf_data_len(op->sym->m_src) && do_sgl))
+		min_ofs = 0;
+
 	if (unlikely(op->sym->m_dst != NULL)) {
 		/* Out-of-place operation (OOP)
 		 * Don't align DMA start. DMA the minimum data-set
@@ -1113,6 +1190,7 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg)
 			rte_pktmbuf_mtophys_offset(op->sym->m_src, min_ofs);
 		dst_buf_start =
 			rte_pktmbuf_mtophys_offset(op->sym->m_dst, min_ofs);
+
 	} else {
 		/* In-place operation
 		 * Start DMA at nearest aligned address below min_ofs
@@ -1158,8 +1236,47 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg)
 		(cipher_param->cipher_offset + cipher_param->cipher_length)
 		: (auth_param->auth_off + auth_param->auth_len);
 
-	qat_req->comn_mid.src_data_addr = src_buf_start;
-	qat_req->comn_mid.dest_data_addr = dst_buf_start;
+	if (do_sgl) {
+
+		ICP_QAT_FW_COMN_PTR_TYPE_SET(qat_req->comn_hdr.comn_req_flags,
+				QAT_COMN_PTR_TYPE_SGL);
+		ret = qat_sgl_fill_array(op->sym->m_src, src_buf_start,
+				&qat_op_cookie->qat_sgl_list,
+				qat_req->comn_mid.src_length);
+		if (ret) {
+			PMD_DRV_LOG(ERR, "QAT PMD Cannot fill sgl array");
+			return ret;
+		}
+
+		if (likely(op->sym->m_dst == NULL))
+			qat_req->comn_mid.dest_data_addr =
+				qat_req->comn_mid.src_data_addr =
+				qat_op_cookie->cookie_phys_addr;
+		else {
+			ret = qat_sgl_fill_array(op->sym->m_dst,
+					dst_buf_start,
+					&qat_op_cookie->qat_sgl_list_oop,
+						qat_req->comn_mid.dst_length);
+
+			if (ret) {
+				PMD_DRV_LOG(ERR, "QAT PMD Cannot "
+						"fill sgl array");
+				return ret;
+			}
+
+			qat_req->comn_mid.src_data_addr =
+				qat_op_cookie->cookie_phys_addr;
+
+			qat_req->comn_mid.dest_data_addr =
+				SGL_SECOND_COOKIE_ADDR(
+						qat_op_cookie->cookie_phys_addr,
+						phys_addr_t);
+		}
+
+	} else {
+		qat_req->comn_mid.src_data_addr = src_buf_start;
+		qat_req->comn_mid.dest_data_addr = dst_buf_start;
+	}
 
 	if (ctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_GALOIS_128 ||
 			ctx->qat_hash_alg == ICP_QAT_HW_AUTH_ALGO_GALOIS_64) {
@@ -1191,7 +1308,6 @@ qat_write_hw_desc_entry(struct rte_crypto_op *op, uint8_t *out_msg)
 		}
 	}
 
-
 #ifdef RTE_LIBRTE_PMD_QAT_DEBUG_TX
 	rte_hexdump(stdout, "qat_req:", qat_req,
 			sizeof(struct icp_qat_fw_la_bulk_req));
diff --git a/drivers/crypto/qat/qat_crypto.h b/drivers/crypto/qat/qat_crypto.h
index 6b84488..ef90281 100644
--- a/drivers/crypto/qat/qat_crypto.h
+++ b/drivers/crypto/qat/qat_crypto.h
@@ -69,6 +69,9 @@ struct qat_qp {
 	struct	qat_queue	tx_q;
 	struct	qat_queue	rx_q;
 	struct	rte_cryptodev_stats stats;
+	struct rte_mempool *sgl_pool;
+	void **sgl_cookies;
+	uint32_t nb_descriptors;
 } __rte_cache_aligned;
 
 /** private data structure for each QAT device */
diff --git a/drivers/crypto/qat/qat_qp.c b/drivers/crypto/qat/qat_qp.c
index 2e7188b..331ff31 100644
--- a/drivers/crypto/qat/qat_qp.c
+++ b/drivers/crypto/qat/qat_qp.c
@@ -41,6 +41,7 @@
 
 #include "qat_logs.h"
 #include "qat_crypto.h"
+#include "qat_algs.h"
 #include "adf_transport_access_macros.h"
 
 #define ADF_MAX_SYM_DESC			4096
@@ -136,6 +137,8 @@ int qat_crypto_sym_qp_setup(struct rte_cryptodev *dev, uint16_t queue_pair_id,
 {
 	struct qat_qp *qp;
 	int ret;
+	char sgl_pool_name[RTE_RING_NAMESIZE];
+	uint32_t i;
 
 	PMD_INIT_FUNC_TRACE();
 
@@ -166,7 +169,6 @@ int qat_crypto_sym_qp_setup(struct rte_cryptodev *dev, uint16_t queue_pair_id,
 				queue_pair_id);
 		return -EINVAL;
 	}
-
 	/* Allocate the queue pair data structure. */
 	qp = rte_zmalloc("qat PMD qp metadata",
 			sizeof(*qp), RTE_CACHE_LINE_SIZE);
@@ -174,6 +176,11 @@ int qat_crypto_sym_qp_setup(struct rte_cryptodev *dev, uint16_t queue_pair_id,
 		PMD_DRV_LOG(ERR, "Failed to alloc mem for qp struct");
 		return -ENOMEM;
 	}
+	qp->nb_descriptors = qp_conf->nb_descriptors;
+	qp->sgl_cookies = rte_zmalloc("qat PMD SGL cookie pointer",
+			qp_conf->nb_descriptors * sizeof(*qp->sgl_cookies),
+			RTE_CACHE_LINE_SIZE);
+
 	qp->mmap_bar_addr = dev->pci_dev->mem_resource[0].addr;
 	rte_atomic16_init(&qp->inflights16);
 
@@ -191,9 +198,45 @@ int qat_crypto_sym_qp_setup(struct rte_cryptodev *dev, uint16_t queue_pair_id,
 		qat_queue_delete(&(qp->tx_q));
 		goto create_err;
 	}
+
 	adf_configure_queues(qp);
 	adf_queue_arb_enable(&qp->tx_q, qp->mmap_bar_addr);
+
+	snprintf(sgl_pool_name, RTE_RING_NAMESIZE, "%s_qp_sgl_%d_%hu",
+		dev->driver->pci_drv.driver.name, dev->data->dev_id,
+		queue_pair_id);
+
+	qp->sgl_pool = rte_mempool_lookup(sgl_pool_name);
+
+	if (qp->sgl_pool == NULL)
+		qp->sgl_pool = rte_mempool_create(sgl_pool_name,
+				qp->nb_descriptors,
+				sizeof(struct qat_crypto_op_cookie), 64, 0,
+				NULL, NULL, NULL, NULL, socket_id,
+				0);
+	if (!qp->sgl_pool) {
+		PMD_DRV_LOG(ERR, "QAT PMD Cannot create"
+				" sgl mempool");
+		goto create_err;
+	}
+
+	for (i = 0; i < qp->nb_descriptors; i++) {
+		if (rte_mempool_get(qp->sgl_pool, &qp->sgl_cookies[i])) {
+			PMD_DRV_LOG(ERR, "QAT PMD Cannot get sgl_cookie");
+			return -EFAULT;
+		}
+
+		struct qat_crypto_op_cookie *sql_cookie =
+				qp->sgl_cookies[i];
+
+		sql_cookie->cookie_phys_addr =
+				rte_mempool_virt2phy(qp->sgl_pool,
+				sql_cookie) +
+				offsetof(struct qat_crypto_op_cookie,
+				qat_sgl_list);
+	}
 	dev->data->queue_pairs[queue_pair_id] = qp;
+
 	return 0;
 
 create_err:
@@ -206,6 +249,8 @@ int qat_crypto_sym_qp_release(struct rte_cryptodev *dev, uint16_t queue_pair_id)
 	struct qat_qp *qp =
 			(struct qat_qp *)dev->data->queue_pairs[queue_pair_id];
 
+	uint32_t i;
+
 	PMD_INIT_FUNC_TRACE();
 	if (qp == NULL) {
 		PMD_DRV_LOG(DEBUG, "qp already freed");
@@ -221,6 +266,14 @@ int qat_crypto_sym_qp_release(struct rte_cryptodev *dev, uint16_t queue_pair_id)
 	}
 
 	adf_queue_arb_disable(&(qp->tx_q), qp->mmap_bar_addr);
+
+	for (i = 0; i < qp->nb_descriptors; i++)
+		rte_mempool_put(qp->sgl_pool, qp->sgl_cookies[i]);
+
+	if (qp->sgl_pool)
+		rte_mempool_free(qp->sgl_pool);
+
+	rte_free(qp->sgl_cookies);
 	rte_free(qp);
 	dev->data->queue_pairs[queue_pair_id] = NULL;
 	return 0;
-- 
2.1.0

^ permalink raw reply related

* [PATCH v2 0/2] Add scatter-gather list capability to Intel(R) QuickAssist Technology driver
From: Arek Kusztal @ 2016-12-22 10:15 UTC (permalink / raw)
  To: dev
  Cc: fiona.trahe, pablo.de.lara.guarch, john.griffin, deepak.k.jain,
	Arek Kusztal

This patchset adds scatter-gather list (SGL) capability to Intel(R) QuickAssist Technology driver
and corresponding tests to QAT cryptodev test suite.
 
v2:
- removed _priv member from rte_crypto_op
- added cookie pool to qat_qp for sgl array entries 

Arek Kusztal (2):
  crypto/qat: add SGL capability to Intel QuickAssist driver
  app/test: add SGL tests to cryptodev QAT suite

 app/test/test_cryptodev.c                  | 356 +++++++++++++
 app/test/test_cryptodev_gcm_test_vectors.h | 823 ++++++++++++++++++++++++++++-
 doc/guides/rel_notes/release_17_02.rst     |   2 +
 drivers/crypto/qat/qat_adf/qat_algs.h      |  14 +-
 drivers/crypto/qat/qat_crypto.c            | 130 ++++-
 drivers/crypto/qat/qat_crypto.h            |   3 +
 drivers/crypto/qat/qat_qp.c                |  55 +-
 7 files changed, 1371 insertions(+), 12 deletions(-)

-- 
2.1.0

^ permalink raw reply

* Re: [PATCH 01/18] net/ixgbe: store SYN filter
From: Zhao1, Wei @ 2016-12-22  9:48 UTC (permalink / raw)
  To: Yigit, Ferruh, dev@dpdk.org; +Cc: Lu, Wenzhuo
In-Reply-To: <f7eb54bf-f834-097e-cb84-6ee067bfae29@intel.com>

Hi, Yigit

> -----Original Message-----
> From: Yigit, Ferruh
> Sent: Wednesday, December 21, 2016 12:56 AM
> To: Zhao1, Wei <wei.zhao1@intel.com>; dev@dpdk.org
> Cc: Lu, Wenzhuo <wenzhuo.lu@intel.com>
> Subject: Re: [dpdk-dev] [PATCH 01/18] net/ixgbe: store SYN filter
> 
> On 12/2/2016 10:42 AM, Wei Zhao wrote:
> > From: wei zhao1 <wei.zhao1@intel.com>
> >
> > Add support for storing SYN filter in SW.
> 
> Do you think does it makes more clear to refer as TCP SYN filter? Or SYN filter
> is clear enough?
> 

Ok, I will change to " TCP SYN filter " to make it more clear

> >
> > Signed-off-by: Wenzhuo Lu <wenzhuo.lu@intel.com>
> > Signed-off-by: wei zhao1 <wei.zhao1@intel.com>
> 
> Can you please update sign-off to your actual name?
>

Ok, I will change to " Signed-off-by: Wei Zhao <wei.zhao1@intel.com>"

> > ---
> >  drivers/net/ixgbe/ixgbe_ethdev.c | 12 ++++++++++--
> > drivers/net/ixgbe/ixgbe_ethdev.h |  2 ++
> >  2 files changed, 12 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
> > b/drivers/net/ixgbe/ixgbe_ethdev.c
> > index edc9b22..7f10cca 100644
> > --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> > +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> > @@ -1287,6 +1287,8 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev)
> >  	memset(filter_info->fivetuple_mask, 0,
> >  	       sizeof(uint32_t) * IXGBE_5TUPLE_ARRAY_SIZE);
> >
> > +	/* initialize SYN filter */
> > +	filter_info->syn_info = 0;
> 
> can it be an option to memset all filter_info? (and of course move list init
> after memset)
> 

Maybe, change to the following code?

memset(filter_info, 0, sizeof(struct ixgbe_filter_info)); 
TAILQ_INIT(&filter_info->fivetuple_list);

But that wiil mix /* initialize ether type filter */ and /* initialize 5tuple filter list */ two process together,
Because  struct ixgbe_filter_info store two type info of ether and 5tuple.
So, not to change ?

struct ixgbe_filter_info {
	uint8_t ethertype_mask;  /* Bit mask for every used ethertype filter */
	/* store used ethertype filters*/
	struct ixgbe_ethertype_filter ethertype_filters[IXGBE_MAX_ETQF_FILTERS];
	/* Bit mask for every used 5tuple filter */
	uint32_t fivetuple_mask[IXGBE_5TUPLE_ARRAY_SIZE];
	struct ixgbe_5tuple_filter_list fivetuple_list;
	/* store the SYN filter info */
	uint32_t syn_info;
};


> >  	return 0;
> >  }
> >
> > @@ -5509,15 +5511,19 @@ ixgbe_syn_filter_set(struct rte_eth_dev *dev,
> >  			bool add)
> >  {
> >  	struct ixgbe_hw *hw =
> > IXGBE_DEV_PRIVATE_TO_HW(dev->data->dev_private);
> > +	struct ixgbe_filter_info *filter_info =
> > +		IXGBE_DEV_PRIVATE_TO_FILTER_INFO(dev->data-
> >dev_private);
> > +	uint32_t syn_info;
> >  	uint32_t synqf;
> >
> >  	if (filter->queue >= IXGBE_MAX_RX_QUEUE_NUM)
> >  		return -EINVAL;
> >
> > +	syn_info = filter_info->syn_info;
> >  	synqf = IXGBE_READ_REG(hw, IXGBE_SYNQF);
> >
> >  	if (add) {
> > -		if (synqf & IXGBE_SYN_FILTER_ENABLE)
> > +		if (syn_info & IXGBE_SYN_FILTER_ENABLE)
> 
> If these checks will be done on syn_info, shouldn't syn_info be assigned to
> synqf before this. Specially for first usage, synqf may be different than hw
> register.
> 
> Or perhaps can keep continue to use synqf. Since synqf assigned to
> filter_info->syn_info after updated.
> 

Let me have a deeper think of this to reply you.


> >  			return -EINVAL;
> >  		synqf = (uint32_t)(((filter->queue <<
> IXGBE_SYN_FILTER_QUEUE_SHIFT) &
> >  			IXGBE_SYN_FILTER_QUEUE) |
> IXGBE_SYN_FILTER_ENABLE); @@ -5527,10
> > +5533,12 @@ ixgbe_syn_filter_set(struct rte_eth_dev *dev,
> >  		else
> >  			synqf &= ~IXGBE_SYN_FILTER_SYNQFP;
> >  	} else {
> > -		if (!(synqf & IXGBE_SYN_FILTER_ENABLE))
> > +		if (!(syn_info & IXGBE_SYN_FILTER_ENABLE))
> >  			return -ENOENT;
> >  		synqf &= ~(IXGBE_SYN_FILTER_QUEUE |
> IXGBE_SYN_FILTER_ENABLE);
> >  	}
> > +
> > +	filter_info->syn_info = synqf;
> >  	IXGBE_WRITE_REG(hw, IXGBE_SYNQF, synqf);
> >  	IXGBE_WRITE_FLUSH(hw);
> >  	return 0;
> <...>
> 

^ permalink raw reply

* Re: [PATCH 15/18] net/ixgbe: parse flow director filter
From: Zhao1, Wei @ 2016-12-22  9:19 UTC (permalink / raw)
  To: Yigit, Ferruh, dev@dpdk.org; +Cc: Lu, Wenzhuo
In-Reply-To: <fd2a2254-3243-b5eb-6011-4180eb121cdc@intel.com>

Hi, Yigit

> -----Original Message-----
> From: Yigit, Ferruh
> Sent: Wednesday, December 21, 2016 1:01 AM
> To: Zhao1, Wei <wei.zhao1@intel.com>; dev@dpdk.org
> Cc: Lu, Wenzhuo <wenzhuo.lu@intel.com>
> Subject: Re: [dpdk-dev] [PATCH 15/18] net/ixgbe: parse flow director filter
> 
> On 12/2/2016 10:43 AM, Wei Zhao wrote:
> > From: wei zhao1 <wei.zhao1@intel.com>
> >
> > check if the rule is a flow director rule, and get the flow director info.
> >
> > Signed-off-by: wei zhao1 <wei.zhao1@intel.com>
> > Signed-off-by: Wenzhuo Lu <wenzhuo.lu@intel.com>
> > ---
> 
> <...>
> 
> > +	PATTERN_SKIP_VOID(rule, struct ixgbe_fdir_rule,
> > +			  RTE_FLOW_ERROR_TYPE_ITEM_NUM);
> > +	if (item->type != RTE_FLOW_ITEM_TYPE_ETH &&
> > +	    item->type != RTE_FLOW_ITEM_TYPE_IPV4 &&
> > +	    item->type != RTE_FLOW_ITEM_TYPE_IPV6 &&
> > +	    item->type != RTE_FLOW_ITEM_TYPE_UDP &&
> > +	    item->type != RTE_FLOW_ITEM_TYPE_VXLAN &&
> > +	    item->type != RTE_FLOW_ITEM_TYPE_NVGRE) {
> 
> This gives build error [1], there are a few more same usage:
> 
> .../drivers/net/ixgbe/ixgbe_ethdev.c:9238:17: error: comparison of constant
> 241 with expression of type 'const enum rte_flow_item_type' is always true
> [-Werror,-Wtautological-constant-out-of-range-compare]
>             item->type != RTE_FLOW_ITEM_TYPE_NVGRE) {
> 
> 
> 

Ok, I will add two type definition RTE_FLOW_ITEM_TYPE_NVGRE and RTE_FLOW_ITEM_TYPE_E_TAG  into const enum rte_flow_item_type to eliminate this problem.
Thank you.

^ permalink raw reply

* Re: [PATCH 0/3] net/virtio: a temp to fix few multiple process issues
From: Xu, Qian Q @ 2016-12-22  8:29 UTC (permalink / raw)
  To: Yuanhan Liu, dev@dpdk.org, Yaron Illouz; +Cc: Yao, Lei A
In-Reply-To: <1482391123-8149-1-git-send-email-yuanhan.liu@linux.intel.com>

Yaron
Could you also share your test case step by step here? I want to learn from u on the cases. 

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yuanhan Liu
> Sent: Thursday, December 22, 2016 3:19 PM
> To: dev@dpdk.org
> Cc: Yaron Illouz <yaroni@radcom.com>; Yuanhan Liu
> <yuanhan.liu@linux.intel.com>
> Subject: [dpdk-dev] [PATCH 0/3] net/virtio: a temp to fix few multiple process
> issues
> 
> Here is a temp to fix few multiple process issues: only did some basic test, not
> positive sure (though it's likely) it will fix the issue reported by Yaron.
> 
> Yaron, please help testing. Thanks!
> 
> ---
> Yuanhan Liu (3):
>   net/virtio: fix crash for secondary process
>   net/virtio: fix multiple process support
>   net/virtio: fix wrong Rx/Tx method for secondary process
> 
>  drivers/net/virtio/virtio_ethdev.c      | 27 ++++++++++++++++++++++++++-
>  drivers/net/virtio/virtio_pci.c         |  4 ++--
>  drivers/net/virtio/virtio_pci.h         |  4 ++++
>  drivers/net/virtio/virtio_user_ethdev.c |  2 +-
>  4 files changed, 33 insertions(+), 4 deletions(-)
> 
> --
> 1.9.0

^ permalink raw reply

* Re: [PATCH v2] crypto/aesni_mb: enablement of avx512 support in IPsec_mb library
From: De Lara Guarch, Pablo @ 2016-12-22  8:26 UTC (permalink / raw)
  To: Doherty, Declan, dev@dpdk.org; +Cc: Doherty, Declan
In-Reply-To: <1482357900-8382-2-git-send-email-declan.doherty@intel.com>



> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Declan Doherty
> Sent: Wednesday, December 21, 2016 10:05 PM
> To: dev@dpdk.org
> Cc: Doherty, Declan
> Subject: [dpdk-dev] [PATCH v2] crypto/aesni_mb: enablement of avx512
> support in IPsec_mb library
> 
> Release v0.44 of Intel(R) Multi-Buffer Crypto for IPsec library adds
> support for AVX512 instructions. This patch enables the new AVX512
> accelerated functions from the aesni_mb_pmd crypto poll mode driver.
> 
> This patch set requires that the aesni_mb_pmd is linked against the
> version 0.44 or greater of the Multi-Buffer Crypto for IPsec library.
> 
> Signed-off-by: Declan Doherty <declan.doherty@intel.com>

Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>

^ permalink raw reply

* Re: [PATCH v5 00/29] Support VFD and DPDK PF + kernel VF on i40e
From: Chen, Jing D @ 2016-12-22  8:10 UTC (permalink / raw)
  To: Vincent JARDIN, Thomas Monjalon
  Cc: dev@dpdk.org, Yigit, Ferruh, Wu, Jingjing, Zhang, Helin
In-Reply-To: <932774d3-e0f7-7b44-1635-9015b8be6c0e@6wind.com>

Hi, Vincent,

> -----Original Message-----
> From: Vincent JARDIN [mailto:vincent.jardin@6wind.com]
> Sent: Tuesday, December 20, 2016 11:19 PM
> To: Chen, Jing D <jing.d.chen@intel.com>; Thomas Monjalon
> <thomas.monjalon@6wind.com>
> Cc: dev@dpdk.org; Yigit, Ferruh <ferruh.yigit@intel.com>; Wu, Jingjing
> <jingjing.wu@intel.com>; Zhang, Helin <helin.zhang@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v5 00/29] Support VFD and DPDK PF + kernel VF on
> i40e
> 
> Le 20/12/2016 à 05:48, Chen, Jing D a écrit :
> > That's a collaboration with another team. we'll follow-up that but not guarantee
> > it will happen.
> > May I ask if my reply make it clear? Still NAC for this patch?
> 
> Yes still nack, I am not confident with this PF approach since you are
> breaking Linux PF behavior. It does not provide guarantees with PF.
> Something is missing to guarantee the compatibilities.

Let me introduce the rationale of API between PF and VF.

There is a common head file visible to both PF and VF, which includes a set of command
and data structures, which is managed by a version number. Below is an example.

Major_verion=1
Minor_verion=1
enum i40e_virtchnl_ops {
	CMD_GET_VERSION,
	CMD_RESET_VF,
	......
}

struct i40e_virtchnl_version_info {
	u32 major;
	u32 minor;
};
......

So, both PF and VF strictly follow the spec. VF send request with expected command and 
data structures to PF host. PF do sanity check and configure, finally applied to HW. 

As developing the code, it's possible to have multiple version of API occurred with 'major_verion'
or 'minor_version' changed. So, at the initialization stage, VF and PF will use a command 
'CMD_GET_VERSION' to negotiate what language (what version of API) they should use
and setup conversation. 

So, from my perspective, there is no issue here.

In the meanwhile, we have some test models ongoing to validate combination of Linux and 
DPDK drivers for VF and PF. We'll fully support below 4 cases going forward.
1. DPDK PF + DPDK VF 
2. DPDK PF + Linux VF
3. Linux PF + DPDK VF 
4. Linux PF + Linux VF (it's not our scope)

After applied this patch, i've done below test without observing compatibility issue.
1. DPDK PF + DPDK VF (middle of 16.11 and 17.02 code base). PF to support API 1.0 while VF
    to support API 1.1/1.0	
2. DPDK PF + Linux VF 1.5.14. PF to support 1.0, while Linux to support 1.1/1.0

Linux PF + DPDK VF has been tested with 1.0 API long time ago. There is some test activities
ongoing.

Finally, please give strong reasons to support your NAC.

^ permalink raw reply

* Re: [PATCH v4 21/23] ethdev: Move filling of rte_eth_dev_info->pci_dev to dev_infos_get()
From: Shreyansh Jain @ 2016-12-22  8:11 UTC (permalink / raw)
  To: Stephen Hemminger, Jan Blunck; +Cc: dev, david.marchand
In-Reply-To: <20161221120953.0282b531@xeon-e3>

On Thursday 22 December 2016 01:39 AM, Stephen Hemminger wrote:
> On Wed, 21 Dec 2016 16:09:44 +0100
> Jan Blunck <jblunck@infradead.org> wrote:
>
>> Only the device itself can decide its PCI or not.
>>
>> Signed-off-by: Jan Blunck <jblunck@infradead.org>
>> Acked-by: Shreyansh Jain <shreyansh.jain@nxp.com>
>> ---
>
> I would still like to kill dev_pci from the dev_info API.
>

+1. It should be rte_dev reference instead.

(But, I am also OK about doing that immediately over this set.)

-
Shreyansh

^ permalink raw reply

* Re: Bug in virtqueue_dequeue_burst_rx()
From: Yuanhan Liu @ 2016-12-22  7:38 UTC (permalink / raw)
  To: Gopakumar Choorakkot Edakkunni; +Cc: huawei.xie, dev
In-Reply-To: <CABK1yFBkaePG37Qd2ebobgJtdeVMj1uNZi5szprFkfyZ5CFZuw@mail.gmail.com>

On Mon, Dec 19, 2016 at 09:59:33PM -0800, Gopakumar Choorakkot Edakkunni wrote:
> While I was testing virtio with ubuntu 14.04 kvm as host and dpdk16.07
> linux as guest, quite often I have seen that I get into a situation where
> virtio_recv_mergeable_pkts() gets into a forever loop, after sending
> traffic for a while. In the below API,  I see that it clearly leads to a
> while loop, I am not quite familiar with virtio or mergeable buffers, so
> thought of checking with dpdk alias on the intent here.
> 
> I checked the linux kernel virtio_net.c file which does the similar
> mergeable recieve, and the kernel code breaks out in case of error.
> Shouldnt dpdk be breaking out of here on error instead of continue ?

Yep, that looks buggy.

> 
> virtio_recv_mergeable_pkts()
> {
> <snip>
>     while (i < nb_used) {
>         <snip>
> *        num = virtqueue_dequeue_burst_rx(rxvq, rcv_pkts, len, 1);*
> *        if (num != 1)*
> *            continue;*

However, normally, virtqueue_dequeue_burst_rx() would be successful
here: the outer check (i < nb_used) somehow ascertains it.

Two options I can think of so far:

- will it go back working once you do "break" here?

- tell us how you managed to trigger this issue, so that I could
  also reproduce it that I can debug it.

	--yliu

^ permalink raw reply

* Re: [PATCH 0/3] net/virtio: a temp to fix few multiple process issues
From: Yuanhan Liu @ 2016-12-22  7:23 UTC (permalink / raw)
  To: dev; +Cc: Yaron Illouz
In-Reply-To: <1482391123-8149-1-git-send-email-yuanhan.liu@linux.intel.com>

On Thu, Dec 22, 2016 at 03:18:40PM +0800, Yuanhan Liu wrote:
> Here is a temp to fix few multiple process issues: only did some basic
            ^^^^

Sign.., I meant "attempt" :-/

	--yliu

^ permalink raw reply

* [PATCH 3/3] net/virtio: fix wrong Rx/Tx method for secondary process
From: Yuanhan Liu @ 2016-12-22  7:18 UTC (permalink / raw)
  To: dev; +Cc: Yaron Illouz, Yuanhan Liu, stable
In-Reply-To: <1482391123-8149-1-git-send-email-yuanhan.liu@linux.intel.com>

If the primary enables the vector Rx/Tx path, the current code would
let the secondary always choose the non vector Rx/Tx path. This results
to a Rx/Tx method mismatch between primary and secondary process. Werid
errors then may happen, something like:

    PMD: virtio_xmit_pkts() tx: virtqueue_enqueue error: -14

Fix it by choosing the correct Rx/Tx callbacks for the secondary process.
That is, use vector path if it's given.

Fixes: 8d8393fb1861 ("virtio: pick simple Rx/Tx")

Cc: stable@dpdk.org
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
---
 drivers/net/virtio/virtio_ethdev.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index a9f7ae4..04d93de 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1325,7 +1325,11 @@ static int virtio_dev_xstats_get_names(struct rte_eth_dev *dev,
 			return 1;
 
 		set_vtpci_ops(eth_dev, hw);
-		rx_func_get(eth_dev);
+		if (hw->use_simple_rxtx) {
+			eth_dev->tx_pkt_burst = virtio_xmit_pkts_simple;
+			eth_dev->rx_pkt_burst = virtio_recv_pkts_vec;
+		} else
+			rx_func_get(eth_dev);
 		return 0;
 	}
 
-- 
1.9.0

^ permalink raw reply related

* [PATCH 2/3] net/virtio: fix multiple process support
From: Yuanhan Liu @ 2016-12-22  7:18 UTC (permalink / raw)
  To: dev; +Cc: Yaron Illouz, Yuanhan Liu, stable
In-Reply-To: <1482391123-8149-1-git-send-email-yuanhan.liu@linux.intel.com>

The introduce of virtio 1.0 support brings yet another set of ops, badly,
it's not handled correctly, that it breaks the multiple process support.

The issue is the function pointer may vary from different processes, and
the current code just does one time set (for primary process only). That
said, the function pointer the secondary process saw is actually from the
primary process space. Accessing it could likely result to a crash.

The fix is somehow straightforward (at least, way more simpler than what
I have firstly thought): just reset the function pointers for the secondary
process on init stage.

Fixes: d5bbeefca826 ("virtio: introduce PCI implementation structure")

Cc: stable@dpdk.org
Reported-by: Yaron Illouz <yaroni@radcom.com>
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
---
 drivers/net/virtio/virtio_ethdev.c      | 12 ++++++++++++
 drivers/net/virtio/virtio_pci.c         |  4 ++--
 drivers/net/virtio/virtio_pci.h         |  4 ++++
 drivers/net/virtio/virtio_user_ethdev.c |  2 +-
 4 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index f2a803b..a9f7ae4 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1174,6 +1174,17 @@ static int virtio_dev_xstats_get_names(struct rte_eth_dev *dev,
 }
 
 static void
+set_vtpci_ops(struct rte_eth_dev *eth_dev, struct virtio_hw *hw)
+{
+	if (eth_dev->pci_dev == NULL)
+		hw->vtpci_ops = &virtio_user_ops;
+	else if (hw->modern)
+		hw->vtpci_ops = &modern_ops;
+	else
+		hw->vtpci_ops = &legacy_ops;
+}
+
+static void
 rx_func_get(struct rte_eth_dev *eth_dev)
 {
 	struct virtio_hw *hw = eth_dev->data->dev_private;
@@ -1313,6 +1324,7 @@ static int virtio_dev_xstats_get_names(struct rte_eth_dev *dev,
 		if (!hw)
 			return 1;
 
+		set_vtpci_ops(eth_dev, hw);
 		rx_func_get(eth_dev);
 		return 0;
 	}
diff --git a/drivers/net/virtio/virtio_pci.c b/drivers/net/virtio/virtio_pci.c
index 9b47165..e9e53a5 100644
--- a/drivers/net/virtio/virtio_pci.c
+++ b/drivers/net/virtio/virtio_pci.c
@@ -300,7 +300,7 @@
 	return 0;
 }
 
-static const struct virtio_pci_ops legacy_ops = {
+const struct virtio_pci_ops legacy_ops = {
 	.read_dev_cfg	= legacy_read_dev_config,
 	.write_dev_cfg	= legacy_write_dev_config,
 	.reset		= legacy_reset,
@@ -516,7 +516,7 @@
 	io_write16(1, vq->notify_addr);
 }
 
-static const struct virtio_pci_ops modern_ops = {
+const struct virtio_pci_ops modern_ops = {
 	.read_dev_cfg	= modern_read_dev_config,
 	.write_dev_cfg	= modern_write_dev_config,
 	.reset		= modern_reset,
diff --git a/drivers/net/virtio/virtio_pci.h b/drivers/net/virtio/virtio_pci.h
index de271bf..3e1e911 100644
--- a/drivers/net/virtio/virtio_pci.h
+++ b/drivers/net/virtio/virtio_pci.h
@@ -317,4 +317,8 @@ int vtpci_init(struct rte_pci_device *, struct virtio_hw *,
 
 uint16_t vtpci_irq_config(struct virtio_hw *, uint16_t);
 
+extern const struct virtio_pci_ops legacy_ops;
+extern const struct virtio_pci_ops modern_ops;
+extern const struct virtio_pci_ops virtio_user_ops;
+
 #endif /* _VIRTIO_PCI_H_ */
diff --git a/drivers/net/virtio/virtio_user_ethdev.c b/drivers/net/virtio/virtio_user_ethdev.c
index 406beea..26eca37 100644
--- a/drivers/net/virtio/virtio_user_ethdev.c
+++ b/drivers/net/virtio/virtio_user_ethdev.c
@@ -212,7 +212,7 @@
 			    strerror(errno));
 }
 
-static const struct virtio_pci_ops virtio_user_ops = {
+const struct virtio_pci_ops virtio_user_ops = {
 	.read_dev_cfg	= virtio_user_read_dev_config,
 	.write_dev_cfg	= virtio_user_write_dev_config,
 	.reset		= virtio_user_reset,
-- 
1.9.0

^ permalink raw reply related

* [PATCH 1/3] net/virtio: fix crash for secondary process
From: Yuanhan Liu @ 2016-12-22  7:18 UTC (permalink / raw)
  To: dev; +Cc: Yaron Illouz, Yuanhan Liu, stable
In-Reply-To: <1482391123-8149-1-git-send-email-yuanhan.liu@linux.intel.com>

If a virtio net device is managed by kernel driver, the primary
process would be aware of it and skip it.

For secondary process, however, it isn't aware of that. Instead,
the 'hw' would be NULL. If we keep going on with the rx_func_get(),
a crash would happen. For such case, we simply return 1 to let
EAL skip this device. Thus, the crashed could be fixed.

Fixes: c1f86306a026 ("virtio: add new driver")

Cc: stable@dpdk.org
Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
---
 drivers/net/virtio/virtio_ethdev.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index 079fd6c..f2a803b 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1304,6 +1304,15 @@ static int virtio_dev_xstats_get_names(struct rte_eth_dev *dev,
 	eth_dev->tx_pkt_burst = &virtio_xmit_pkts;
 
 	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
+		/*
+		 * if a virtio net device is managed by kernel driver,
+		 * the secondary process won't be aware of it. Instead,
+		 * it sees hw being set with NULL. For such case, here
+		 * we returns 1 to let EAL skip this device.
+		 */
+		if (!hw)
+			return 1;
+
 		rx_func_get(eth_dev);
 		return 0;
 	}
-- 
1.9.0

^ permalink raw reply related

* [PATCH 0/3] net/virtio: a temp to fix few multiple process issues
From: Yuanhan Liu @ 2016-12-22  7:18 UTC (permalink / raw)
  To: dev; +Cc: Yaron Illouz, Yuanhan Liu

Here is a temp to fix few multiple process issues: only did some basic
test, not positive sure (though it's likely) it will fix the issue
reported by Yaron.

Yaron, please help testing. Thanks!

---
Yuanhan Liu (3):
  net/virtio: fix crash for secondary process
  net/virtio: fix multiple process support
  net/virtio: fix wrong Rx/Tx method for secondary process

 drivers/net/virtio/virtio_ethdev.c      | 27 ++++++++++++++++++++++++++-
 drivers/net/virtio/virtio_pci.c         |  4 ++--
 drivers/net/virtio/virtio_pci.h         |  4 ++++
 drivers/net/virtio/virtio_user_ethdev.c |  2 +-
 4 files changed, 33 insertions(+), 4 deletions(-)

-- 
1.9.0

^ permalink raw reply

* Re: [PATCH v4 03/23] rte_device: make driver pointer const
From: Shreyansh Jain @ 2016-12-22  7:18 UTC (permalink / raw)
  To: Jan Blunck; +Cc: dev, Stephen Hemminger, david.marchand
In-Reply-To: <1482332986-7599-4-git-send-email-jblunck@infradead.org>

On Wednesday 21 December 2016 08:39 PM, Jan Blunck wrote:
> From: Stephen Hemminger <stephen@networkplumber.org>
>
> The info in rte_device about driver is immutable and
> shouldn't change.
> Acked-by: Jan Blunck <jblunck@infradead.org>
>
> Acked-by: Jan Blunck <jblunck@infradead.org>

Not being picky - but, multiple Acked-by can be removed when this is 
applied.

> ---
>  lib/librte_eal/common/include/rte_dev.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/lib/librte_eal/common/include/rte_dev.h b/lib/librte_eal/common/include/rte_dev.h
> index 8840380..e5471a2 100644
> --- a/lib/librte_eal/common/include/rte_dev.h
> +++ b/lib/librte_eal/common/include/rte_dev.h
> @@ -122,7 +122,7 @@ struct rte_driver;
>   */
>  struct rte_device {
>  	TAILQ_ENTRY(rte_device) next; /**< Next device */
> -	struct rte_driver *driver;    /**< Associated driver */
> +	const struct rte_driver *driver;/**< Associated driver */
>  	int numa_node;                /**< NUMA node connection */
>  	struct rte_devargs *devargs;  /**< Device user arguments */
>  };
>

Acked-by: Shreyansh Jain <shreyansh.jain@nxp.com>

^ permalink raw reply

* 答复: RE: [PATCH] net/fm10k/base: add a break statement
From: yao.chenghu @ 2016-12-22  6:01 UTC (permalink / raw)
  To: Chen, Jing D; +Cc: dev@dpdk.org
In-Reply-To: <4341B239C0EFF9468EE453F9E9F4604D3C5C6866@shsmsx102.ccr.corp.intel.com>

> 
> Thanks for contributing code. But there are 2 problems here.
> 
> 1. You are modifying base code under 'base' directory. It assumed 
> READ ONLY because
>     there is another Intel team are maintaining it.
> 2. Without your change, the code won't have any negative effect. 
> Yes, I appreciate your
>     change to make it stronger.
> 
> So, I'd to say 'NAC' for this patch.
> 

Ok, thanks for your reply.

^ permalink raw reply

* Re: [PATCH v3 02/12] eal/bus: introduce bus abstraction
From: Shreyansh Jain @ 2016-12-22  5:52 UTC (permalink / raw)
  To: Stephen Hemminger, Jan Blunck
  Cc: dev, David Marchand, Thomas Monjalon, Ferruh Yigit, jianbo.liu
In-Reply-To: <ed26e9ff-7e68-76db-8639-3a391211da42@nxp.com>

On Thursday 22 December 2016 10:42 AM, Shreyansh Jain wrote:
> On Thursday 22 December 2016 05:03 AM, Stephen Hemminger wrote:
>> On Wed, 21 Dec 2016 16:38:42 +0100
>> Jan Blunck <jblunck@infradead.org> wrote:
>>
>>> On Tue, Dec 20, 2016 at 6:11 PM, Stephen Hemminger
>>> <stephen@networkplumber.org> wrote:
>>>> On Tue, 20 Dec 2016 14:17:14 +0100
>>>> Jan Blunck <jblunck@infradead.org> wrote:
>>>>
>>>>> On Fri, Dec 16, 2016 at 2:10 PM, Shreyansh Jain
>>>>> <shreyansh.jain@nxp.com> wrote:
>>>>>> This patch introduces the rte_bus abstraction for devices and
>>>>>> drivers in
>>>>>> EAL framework. The model is:
>>>>>>  - One or more buses are connected to a CPU (or core)
>>>>>>  - One or more devices are conneted to a Bus
>>>>>>  - Drivers are running instances which manage one or more devices
>>>>>>  - Bus is responsible for identifying devices (and interrupt
>>>>>> propogation)
>>>>>>  - Driver is responsible for initializing the device
>>>>>>
>>>>>> This patch adds a 'rte_bus' class which rte_driver and rte_device
>>>>>> refer.
>>>>>> This way, each device (rte_xxx_device) would have reference to the
>>>>>> bus
>>>>>> it is based on. As well as, each driver (rte_xxx_driver) would
>>>>>> have link
>>>>>> to the bus and devices on it for servicing.
>>>>>>
>>>>>>                                   __ rte_bus_list
>>>>>>                                  /
>>>>>>                      +----------'---+
>>>>>>                      |rte_bus       |
>>>>>>                      | driver_list------> List of rte_bus specific
>>>>>>                      | device_list----    devices
>>>>>>                      |              | `-> List of rte_bus associated
>>>>>>                      |              |     drivers
>>>>>>                      +--|------|----+
>>>>>>               _________/        \_________
>>>>>>     +--------/----+                     +-\---------------+
>>>>>>     |rte_device   |                     |rte_driver       |
>>>>>>     | rte_bus     |                     | rte_bus         |
>>>>>>     | rte_driver  |                     | ...             |
>>>>>>     | ...         |                     +---------...-----+
>>>>>>     |             |                               |||
>>>>>>     +---||--------+                               |||
>>>>>>         ||                                        |||
>>>>>>         | \                                        \\\
>>>>>>         |  \_____________                           \\\
>>>>>>         |                \                          |||
>>>>>>  +------|---------+ +----|----------+               |||
>>>>>>  |rte_pci_device  | |rte_xxx_device |               |||
>>>>>>  | ....           | | ....          |               |||
>>>>>>  +----------------+ +---------------+              / | \
>>>>>>                                                   /  |  \
>>>>>>                             _____________________/  /    \
>>>>>>                            /                    ___/      \
>>>>>>             +-------------'--+    +------------'---+
>>>>>> +--'------------+
>>>>>>             |rte_pci_driver  |    |rte_vdev_driver |
>>>>>> |rte_xxx_driver |
>>>>>>             | ....           |    | ....           |    |
>>>>>> ....          |
>>>>>>             +----------------+    +----------------+
>>>>>> +---------------+
>>>>>>
>>>>>> This patch only enables the bus references on rte_driver and
>>>>>> rte_driver.
>>>>>> EAL wide global device and driver list continue to exist until an
>>>>>> instance
>>>>>> of bus is added in subsequent patches.
>>>>>>
>>>>>> This patch also introduces RTE_REGISTER_BUS macro on the lines of
>>>>>> RTE_PMD_REGISTER_XXX. Key difference is that the constructor
>>>>>> priority has
>>>>>> been explicitly set to 101 so as to execute bus registration
>>>>>> before PMD.
>>>>>>
>>>>>> Signed-off-by: Shreyansh Jain <shreyansh.jain@nxp.com>
>>>>>>
>>>>
>>>> Ok, but let's keep this as bus type not bus. It gets really hard and
>>>> complex
>>>> to enumerate all layers of PCI bus and bridges.
>>>
>>> As far as I understand it this isn't the intention to replicate the
>>> hierarchy of buses we have in the kernel. The PCI bus in this case
>>> becomes a list of PCI devices.
>>
>> One of the motivations seems to be "lets be able to handle lots of
>> devices",
>> but the current model with an array of ports is not going to scale
>> well for that.
>>
>> It is time to make rte_eth_devices into rb-tree and get rid of
>> MAX_PORTS config
>> option.
>>
>
> That is a nice idea. Probably once we get the EAL compatible for 'lots
> of devices', this would be next good change.

Just out of curiosity - I think only need here is to do away with 
'MAX_PORTS'. There is no need for a 'fast' lookup of ports as this part 
wouldn't be part of datapath. Am I wrong in assuming this?

>
> -
> Shreyansh
>
>

^ permalink raw reply

* Re: [PATCH] net/fm10k/base: add a break statement
From: Chen, Jing D @ 2016-12-22  5:22 UTC (permalink / raw)
  To: Chenghu Yao; +Cc: dev@dpdk.org
In-Reply-To: <1482289505-47591-1-git-send-email-yao.chenghu@zte.com.cn>

Hi, Chenghu,


> -----Original Message-----
> From: Chenghu Yao [mailto:yao.chenghu@zte.com.cn]
> Sent: Wednesday, December 21, 2016 11:05 AM
> To: Chen, Jing D <jing.d.chen@intel.com>
> Cc: dev@dpdk.org; Chenghu Yao <yao.chenghu@zte.com.cn>
> Subject: [PATCH] net/fm10k/base: add a break statement
> 
> In function fm10k_mbx_create_reply(), the last case branch
> has no break statement.
> 
> Signed-off-by: Chenghu Yao <yao.chenghu@zte.com.cn>
> ---
>  drivers/net/fm10k/base/fm10k_mbx.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/net/fm10k/base/fm10k_mbx.c
> b/drivers/net/fm10k/base/fm10k_mbx.c
> index 2e70434..45d6ddb 100644
> --- a/drivers/net/fm10k/base/fm10k_mbx.c
> +++ b/drivers/net/fm10k/base/fm10k_mbx.c
> @@ -1084,6 +1084,7 @@ STATIC s32 fm10k_mbx_create_reply(struct fm10k_hw
> *hw,
>  	case FM10K_STATE_CLOSED:
>  		/* generate new header based on data */
>  		fm10k_mbx_create_disconnect_hdr(mbx);
> +		break;
>  	default:
>  		break;
>  	}

Thanks for contributing code. But there are 2 problems here.

1. You are modifying base code under 'base' directory. It assumed READ ONLY because
    there is another Intel team are maintaining it.
2. Without your change, the code won't have any negative effect. Yes, I appreciate your
    change to make it stronger.

So, I'd to say 'NAC' for this patch.

^ permalink raw reply

* Re: [PATCH v3 02/12] eal/bus: introduce bus abstraction
From: Shreyansh Jain @ 2016-12-22  5:12 UTC (permalink / raw)
  To: Stephen Hemminger, Jan Blunck
  Cc: dev, David Marchand, Thomas Monjalon, Ferruh Yigit, jianbo.liu
In-Reply-To: <20161221153334.2b04f732@xeon-e3>

On Thursday 22 December 2016 05:03 AM, Stephen Hemminger wrote:
> On Wed, 21 Dec 2016 16:38:42 +0100
> Jan Blunck <jblunck@infradead.org> wrote:
>
>> On Tue, Dec 20, 2016 at 6:11 PM, Stephen Hemminger
>> <stephen@networkplumber.org> wrote:
>>> On Tue, 20 Dec 2016 14:17:14 +0100
>>> Jan Blunck <jblunck@infradead.org> wrote:
>>>
>>>> On Fri, Dec 16, 2016 at 2:10 PM, Shreyansh Jain <shreyansh.jain@nxp.com> wrote:
>>>>> This patch introduces the rte_bus abstraction for devices and drivers in
>>>>> EAL framework. The model is:
>>>>>  - One or more buses are connected to a CPU (or core)
>>>>>  - One or more devices are conneted to a Bus
>>>>>  - Drivers are running instances which manage one or more devices
>>>>>  - Bus is responsible for identifying devices (and interrupt propogation)
>>>>>  - Driver is responsible for initializing the device
>>>>>
>>>>> This patch adds a 'rte_bus' class which rte_driver and rte_device refer.
>>>>> This way, each device (rte_xxx_device) would have reference to the bus
>>>>> it is based on. As well as, each driver (rte_xxx_driver) would have link
>>>>> to the bus and devices on it for servicing.
>>>>>
>>>>>                                   __ rte_bus_list
>>>>>                                  /
>>>>>                      +----------'---+
>>>>>                      |rte_bus       |
>>>>>                      | driver_list------> List of rte_bus specific
>>>>>                      | device_list----    devices
>>>>>                      |              | `-> List of rte_bus associated
>>>>>                      |              |     drivers
>>>>>                      +--|------|----+
>>>>>               _________/        \_________
>>>>>     +--------/----+                     +-\---------------+
>>>>>     |rte_device   |                     |rte_driver       |
>>>>>     | rte_bus     |                     | rte_bus         |
>>>>>     | rte_driver  |                     | ...             |
>>>>>     | ...         |                     +---------...-----+
>>>>>     |             |                               |||
>>>>>     +---||--------+                               |||
>>>>>         ||                                        |||
>>>>>         | \                                        \\\
>>>>>         |  \_____________                           \\\
>>>>>         |                \                          |||
>>>>>  +------|---------+ +----|----------+               |||
>>>>>  |rte_pci_device  | |rte_xxx_device |               |||
>>>>>  | ....           | | ....          |               |||
>>>>>  +----------------+ +---------------+              / | \
>>>>>                                                   /  |  \
>>>>>                             _____________________/  /    \
>>>>>                            /                    ___/      \
>>>>>             +-------------'--+    +------------'---+    +--'------------+
>>>>>             |rte_pci_driver  |    |rte_vdev_driver |    |rte_xxx_driver |
>>>>>             | ....           |    | ....           |    | ....          |
>>>>>             +----------------+    +----------------+    +---------------+
>>>>>
>>>>> This patch only enables the bus references on rte_driver and rte_driver.
>>>>> EAL wide global device and driver list continue to exist until an instance
>>>>> of bus is added in subsequent patches.
>>>>>
>>>>> This patch also introduces RTE_REGISTER_BUS macro on the lines of
>>>>> RTE_PMD_REGISTER_XXX. Key difference is that the constructor priority has
>>>>> been explicitly set to 101 so as to execute bus registration before PMD.
>>>>>
>>>>> Signed-off-by: Shreyansh Jain <shreyansh.jain@nxp.com>
>>>>>
>>>
>>> Ok, but let's keep this as bus type not bus. It gets really hard and complex
>>> to enumerate all layers of PCI bus and bridges.
>>
>> As far as I understand it this isn't the intention to replicate the
>> hierarchy of buses we have in the kernel. The PCI bus in this case
>> becomes a list of PCI devices.
>
> One of the motivations seems to be "lets be able to handle lots of devices",
> but the current model with an array of ports is not going to scale well for that.
>
> It is time to make rte_eth_devices into rb-tree and get rid of MAX_PORTS config
> option.
>

That is a nice idea. Probably once we get the EAL compatible for 'lots 
of devices', this would be next good change.

-
Shreyansh

^ permalink raw reply

* Re: [PATCH v2 00/12] Add crypto PMD optimized for ARMv8
From: Jerin Jacob @ 2016-12-22  4:57 UTC (permalink / raw)
  To: Declan Doherty
  Cc: Zbigniew Bodek, Bruce Richardson, pablo.de.lara.guarch, dev
In-Reply-To: <898f972b-7d8a-384c-20bf-6ae3f7822136@intel.com>

On Wed, Dec 21, 2016 at 03:34:14PM +0000, Declan Doherty wrote:
> On 08/12/16 17:45, Jerin Jacob wrote:
> > On Thu, Dec 08, 2016 at 12:32:52PM +0100, Zbigniew Bodek wrote:
> > > On 08.12.2016 11:24, Bruce Richardson wrote:
> > > > On Tue, Dec 06, 2016 at 06:32:53PM -0800, zbigniew.bodek@caviumnetworks.com wrote:
> > > > > From: Zbigniew Bodek <zbigniew.bodek@caviumnetworks.com>
> > > > 
> > > 
> > > Hello Bruce,
> > > 
> > > I don't know to be honest. I didn't know the reasoning behind not including
> > > crypto code for Intel for example. I thought it was due to licensing and
> > > code control rather than export compliance.
> > > 
> > > Maybe someone from the DPDK community will know what are the constraints
> > > related to including crypto algorithms to DPDK.
> > 
> > One of the primary reason why we thought of going with this approach is
> > for out of the box "distribution" enablement. We thought, if the core crypto
> > algorithm sits in some git-hub code or public hosted tarball then the
> > PMD will never be added to standard distributions and which is a setback
> > for armv8 server ecosystem.
> > 
> > Having said that and as Zbigniew mentioned, We are open for revisiting
> > the crypto core algorithm and PMD split if there are community concerns
> > about export compliance. Let us know.
> > 
> > Jerin
> > 
> > > 
> > > Kind regards
> > > Zbigniew
> 
> Hey Jerin/Zbigniew,
> 
> 
> as Bruce said it's great to see you contributing to the crypto ecosystem in
> DPDK. I don't know if the export compliance with the core crypto code is an
> issue or not, that's definitely not my area of expertise, but I do have some
> concern which I think it relates somewhat to Thomas questions regarding
> implementing the core crypto algorithms in C rather than assembly.
> 
> I wonder is there the expertise within the DPDK community to review/maintain
> the core crypto code in terms of both the assembly code itself and also the
> details of the crypto algorithm's implementations themselves. I know I
> wouldn't feel I have the knowledge/expertise to be able to review the core
> crypto algorithm's implementations and the assembly code itself and sign-off
> on them.
> 
> I understand the advantage of having the code integrated directly into DPDK
> for packaging etc but this also puts the ownest on the DPDK community for
> the correctness of the underlying implementation of a particular algorithm.
> I think the approach of a separate library removes this responsibility from
> the community and places it on the distributor of the core crypto library.

OK. Taking Thomas and your feedback into account, We will move the core
crypto ARMv8 ASM code to separate library.

Jerin
> 
> Declan
> 
> 

^ permalink raw reply

* Re: [PATCH v6] vhost: allow for many vhost user ports
From: Yuanhan Liu @ 2016-12-22  3:16 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: dev, Patrik Andersson, Jan Wickbom
In-Reply-To: <20161221100636.71a27b22@xeon-e3>

On Wed, Dec 21, 2016 at 10:06:36AM -0800, Stephen Hemminger wrote:
> On Wed, 21 Dec 2016 17:45:13 +0800
> Yuanhan Liu <yuanhan.liu@linux.intel.com> wrote:
> 
> > From: Jan Wickbom <jan.wickbom@ericsson.com>
> > 
> > Currently select() is used to monitor file descriptors for vhostuser
> > ports. This limits the number of ports possible to create since the
> > fd number is used as index in the fd_set and we have seen fds > 1023.
> > This patch changes select() to poll(). This way we can keep an
> > packed (pollfd) array for the fds, e.g. as many fds as the size of
> > the array.
> > 
> > Also see:
> > http://dpdk.org/ml/archives/dev/2016-April/037024.html
> > 
> > Reported-by: Patrik Andersson <patrik.r.andersson@ericsson.com>
> > Signed-off-by: Jan Wickbom <jan.wickbom@ericsson.com>
> > Signed-off-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
> 
> Why not epoll()? It scales much better. The old System V poll
> is just as bad with 1023 fd's.

Indeed, there was a plan to use epoll() to fix this issue. It's been
delayed long enough that Jan came up with a fix with poll recently.

I don't have strong preference to epoll() over poll, for following
reasons:

- epoll() is Linux only, though I don't know other platforms (mainly
  *BSD) actually care vhost-user or not.

- epoll indeed scales much better, but I don't think the scale actually
  matters a lot here: vhost-user negotiation normally happens once on
  QEMU startup, and it's a short lived process. Both facts make it be
  non-performance critical.

- 1023 is also not a big problem so far: vhost lib hardcodes the max
  vhost devices we could support to be 1024 after all.

For above reasons and Jan already came up with a solution (that would
work for most platforms), I think I'm fine with this poll() so far.

	--yliu

^ permalink raw reply

* Re: [PATCH v2 00/30] update ixgbe shared code to version 16.11.21
From: Dai, Wei @ 2016-12-22  2:35 UTC (permalink / raw)
  To: Yigit, Ferruh, dev@dpdk.org; +Cc: Zhang, Helin, Ananyev, Konstantin
In-Reply-To: <6e5d4c65-f0ee-4b18-4935-fa9fce7c84d6@intel.com>

Many thanks to you for your taking time to review this patch set.

Thanks
-Wei

> -----Original Message-----
> From: Yigit, Ferruh
> Sent: Thursday, December 22, 2016 1:20 AM
> To: Dai, Wei <wei.dai@intel.com>; dev@dpdk.org
> Cc: Zhang, Helin <helin.zhang@intel.com>; Ananyev, Konstantin
> <konstantin.ananyev@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2 00/30] update ixgbe shared code to version
> 16.11.21
> 
> On 12/21/2016 9:47 AM, Wei Dai wrote:
> > update ixgbe shared code to version 16.11.21 .
> > v2 changes:
> >   modify subject line and message body of git log according to
> >   feedbacks from communtiy.
> >
> >   split the 24th patch into 2 separate ones.
> >   the 24th of v1 is net/ixgbe/base: add EEE support for DNL-controlled PHYs
> >   the 24th of v2 is net/ixgbe/base: add EEE support for some PHYs
> >   the 25th of v2 is net/ixgbe/base: remove unused enum type
> >
> >   correct RAEDME
> >
> > Wei Dai (30):
> >   net/ixgbe/base: fix PHY reset check for x550em-ext
> >   net/ixgbe/base: fix clearing SAN MAC address
> >   net/ixgbe/base: use fast MDIO for non-10G devices
> >   net/ixgbe/base: fix PHY identification for x550a
> >   net/ixgbe/base: clean up X557 link status check
> >   net/ixgbe/base: add driver version to firmware
> >   net/ixgbe/base: store link active LED
> >   net/ixgbe/base: cleanup X540 checksum calculation
> >   net/ixgbe/base: enable LASI only for X552 devices
> >   net/ixgbe/base: limit iXFI setup to X552 devices
> >   net/ixgbe/base: fix getting PHY type for some x550 devices
> >   net/ixgbe/base: fix SGMII link setup for M88 PHYs
> >   net/ixgbe/base: cleanup dead EEE code
> >   net/ixgbe/base: fix setting unsupported autoneg speeds
> >   net/ixgbe/base: support FW commands to control some PHYs
> >   net/ixgbe/base: use FW commands to control some PHYs
> >   net/ixgbe/base: support busy SGMII register reads
> >   net/ixgbe/base: include new speeds in VFLINK interpretation
> >   net/ixgbe/base: limit 5Gb support to X550 devices
> >   net/ixgbe/base: add physical layer options for FW PHY type
> >   net/ixgbe/base: remove unneeded MAC type check
> >   net/ixgbe/base: remove unused PHY ID
> >   net/ixgbe/base: update FW PHY flow control
> >   net/ixgbe/base: add EEE support for some PHYs
> >   net/ixgbe/base: remove unused enum type
> >   net/ixgbe/base: fix IXGBE LSWFW register
> >   net/ixgbe/base: remove unused EEE code
> >   net/ixgbe/base: add write flush required by Inphi PHY
> >   net/ixgbe/base: report physical layer for SGMII PHY type
> >   net/ixgbe/base: update shared code version to 2016.11.21
> <...>
> 
> Series applied to dpdk-next-net/master, thanks.
> 

^ permalink raw reply


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox