[PATCH] ixgbe: fix icc issue with mbuf initializer

[PATCH] ixgbe: fix icc issue with mbuf initializer

[Bruce Richardson]

When using Intel C++ compiler(icc) 14.0.1.106 or the older icc 13.x
version, the mbuf initializer variable was not getting configured
correctly, as the mb_def variable was not set correctly. This is due
to an issue with icc (DPD200249565 which already been fixed in
icc 14.0.2 and newer compiler release) where it incorrectly calculates
the field offsets with initializers when zero-sized fields
are used in a structure.
To work around this, the code in ixgbe_rxq_vec_setup does not setup the
fields using an initializer, but instead assigns the values individually
in code
NOTE: There is no performance impact to this change as the queue
setup functions are not data-plane APIs, but are only used at app
initialization.

Signed-off-by: Bruce Richardson <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
index e813e43..b57c588 100644
--- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
+++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
@@ -730,16 +730,15 @@ static struct ixgbe_txq_ops vec_txq_ops = {
 int
 ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq)
 {
-	struct rte_mbuf mb_def = {
-		.nb_segs = 1,
-		.data_off = RTE_PKTMBUF_HEADROOM,
-#ifdef RTE_MBUF_REFCNT
-		{ .refcnt = 1, }
-#endif
-	};
+	struct rte_mbuf mb_def = { .buf_addr = 0 }; /* zeroed mbuf */
 
+	mb_def.nb_segs = 1;
+	mb_def.data_off = RTE_PKTMBUF_HEADROOM;
 	mb_def.buf_len = rxq->mb_pool->elt_size - sizeof(struct rte_mbuf);
 	mb_def.port = rxq->port_id;
+#ifdef RTE_MBUF_REFCNT
+	mb_def.refcnt = 1;
+#endif
 	rxq->mbuf_initializer = *((uint64_t *)&mb_def.rearm_data);
 	return 0;
 }
-- 
1.9.3

Re: [PATCH] ixgbe: fix icc issue with mbuf initializer

[David Marchand]

Hello Bruce,

On Mon, Nov 3, 2014 at 12:11 PM, Bruce Richardson <
bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> index e813e43..b57c588 100644
> --- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> +++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> @@ -730,16 +730,15 @@ static struct ixgbe_txq_ops vec_txq_ops = {
>  int
>  ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq)
>  {
> -       struct rte_mbuf mb_def = {
> -               .nb_segs = 1,
> -               .data_off = RTE_PKTMBUF_HEADROOM,
> -#ifdef RTE_MBUF_REFCNT
> -               { .refcnt = 1, }
> -#endif
> -       };
> +       struct rte_mbuf mb_def = { .buf_addr = 0 }; /* zeroed mbuf */
>
> +       mb_def.nb_segs = 1;
> +       mb_def.data_off = RTE_PKTMBUF_HEADROOM;
>         mb_def.buf_len = rxq->mb_pool->elt_size - sizeof(struct rte_mbuf);
>         mb_def.port = rxq->port_id;
> +#ifdef RTE_MBUF_REFCNT
> +       mb_def.refcnt = 1;
> +#endif
>         rxq->mbuf_initializer = *((uint64_t *)&mb_def.rearm_data);
>         return 0;
>  }
>

I would expect we use rte_mbuf_refcnt_set / rte_mbuf_refcnt_read to access
this "refcnt" field.
This api handles both RTE_MBUF_REFCNT_ATOMIC and ! RTE_MBUF_REFCNT_ATOMIC
configs.
But I suppose this is fine at init time (since the union will initialize
properly the field).

By the way, why do we have this RTE_MBUF_REFCNT_ATOMIC option ?
>From my point of view, there is not much use of a refcnt that is not atomic
:-).


-- 
David Marchand

Re: [PATCH] ixgbe: fix icc issue with mbuf initializer

[Bruce Richardson]

On Mon, Nov 03, 2014 at 01:31:10PM +0100, David Marchand wrote:
> Hello Bruce,
> 
> On Mon, Nov 3, 2014 at 12:11 PM, Bruce Richardson <
> bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> 
> > diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> > b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> > index e813e43..b57c588 100644
> > --- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> > +++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
> > @@ -730,16 +730,15 @@ static struct ixgbe_txq_ops vec_txq_ops = {
> >  int
> >  ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq)
> >  {
> > -       struct rte_mbuf mb_def = {
> > -               .nb_segs = 1,
> > -               .data_off = RTE_PKTMBUF_HEADROOM,
> > -#ifdef RTE_MBUF_REFCNT
> > -               { .refcnt = 1, }
> > -#endif
> > -       };
> > +       struct rte_mbuf mb_def = { .buf_addr = 0 }; /* zeroed mbuf */
> >
> > +       mb_def.nb_segs = 1;
> > +       mb_def.data_off = RTE_PKTMBUF_HEADROOM;
> >         mb_def.buf_len = rxq->mb_pool->elt_size - sizeof(struct rte_mbuf);
> >         mb_def.port = rxq->port_id;
> > +#ifdef RTE_MBUF_REFCNT
> > +       mb_def.refcnt = 1;
> > +#endif
> >         rxq->mbuf_initializer = *((uint64_t *)&mb_def.rearm_data);
> >         return 0;
> >  }
> >
> 
> I would expect we use rte_mbuf_refcnt_set / rte_mbuf_refcnt_read to access
> this "refcnt" field.
> This api handles both RTE_MBUF_REFCNT_ATOMIC and ! RTE_MBUF_REFCNT_ATOMIC
> configs.
> But I suppose this is fine at init time (since the union will initialize
> properly the field).

It's a good point, I'll update patch to use the appropriate macro which will clean up the code a bit.

/Bruce

> 
> By the way, why do we have this RTE_MBUF_REFCNT_ATOMIC option ?
> From my point of view, there is not much use of a refcnt that is not atomic
> :-).
> 
> 
> -- 
> David Marchand

Re: [PATCH] ixgbe: fix icc issue with mbuf initializer

[Thomas Monjalon]

2014-11-03 12:47, Bruce Richardson:
> On Mon, Nov 03, 2014 at 01:31:10PM +0100, David Marchand wrote:
> > On Mon, Nov 3, 2014 at 12:11 PM, Bruce Richardson <
> > > +#ifdef RTE_MBUF_REFCNT
> > > +       mb_def.refcnt = 1;
> > > +#endif
> > 
> > I would expect we use rte_mbuf_refcnt_set / rte_mbuf_refcnt_read to access
> > this "refcnt" field.
> > This api handles both RTE_MBUF_REFCNT_ATOMIC and ! RTE_MBUF_REFCNT_ATOMIC
> > configs.
> > But I suppose this is fine at init time (since the union will initialize
> > properly the field).
> 
> It's a good point, I'll update patch to use the appropriate macro which will clean up the code a bit.

> > By the way, why do we have this RTE_MBUF_REFCNT_ATOMIC option ?
> > From my point of view, there is not much use of a refcnt that is not atomic
> > :-).

Bruce, I think it's a good question but you didn't answer.
Maybe we should remove this option to keep only atomic mode.

-- 
Thomas

Re: [PATCH] ixgbe: fix icc issue with mbuf initializer

[Bruce Richardson]

On Mon, Nov 03, 2014 at 01:59:16PM +0100, Thomas Monjalon wrote:
> 2014-11-03 12:47, Bruce Richardson:
> > On Mon, Nov 03, 2014 at 01:31:10PM +0100, David Marchand wrote:
> > > On Mon, Nov 3, 2014 at 12:11 PM, Bruce Richardson <
> > > > +#ifdef RTE_MBUF_REFCNT
> > > > +       mb_def.refcnt = 1;
> > > > +#endif
> > > 
> > > I would expect we use rte_mbuf_refcnt_set / rte_mbuf_refcnt_read to access
> > > this "refcnt" field.
> > > This api handles both RTE_MBUF_REFCNT_ATOMIC and ! RTE_MBUF_REFCNT_ATOMIC
> > > configs.
> > > But I suppose this is fine at init time (since the union will initialize
> > > properly the field).
> > 
> > It's a good point, I'll update patch to use the appropriate macro which will clean up the code a bit.
> 
> > > By the way, why do we have this RTE_MBUF_REFCNT_ATOMIC option ?
> > > From my point of view, there is not much use of a refcnt that is not atomic
> > > :-).
> 
> Bruce, I think it's a good question but you didn't answer.
> Maybe we should remove this option to keep only atomic mode.
> 

I didn't answer just because it wasn't directly relevant to the patch. It was not
meant as a snub. :-)

As for why the option is there, it's purely for performance, I suspect. The
cost of doing increments and decrements using atomic operations is far higher
than doing a read-modify-write on a single core. However, the downside is
obviously that you need to know what you are doing if you disable atomic refcnts
and, given that atomic is the default, I reckon we can probably get rid of the
option permanently - unless someone has a use case where they turn off the option,
and can't take the performance hit of the atomic instructions.

As a further asside, if we get the proposed changes made to the zero-copy vhost
implementation - discussed previously[1] - we should hopefully be able to get rid
of the refcnt option too, and leave it permanently enabled.

/Bruce

[1] Discussed in this thread: http://permalink.gmane.org/gmane.comp.networking.dpdk.devel/7098

[PATCH v2] ixgbe: fix icc issue with mbuf initializer

[Bruce Richardson]

When using Intel C++ compiler(icc) 14.0.1.106 or the older icc 13.x
version, the mbuf initializer variable was not getting configured
correctly, as the mb_def variable was not set correctly. This is due
to an issue with icc (DPD200249565 which already been fixed in
icc 14.0.2 and newer compiler release) where it incorrectly calculates
the field offsets with initializers when zero-sized fields
are used in a structure.
To work around this, the code in ixgbe_rxq_vec_setup does not setup the
fields using an initializer, but instead assigns the values individually
in code
NOTE: There is no performance impact to this change as the queue
setup functions are not data-plane APIs, but are only used at app
initialization.

Signed-off-by: Bruce Richardson <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

---
V2 change: use rte_mbuf_refcnt_set to update reference count

---
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
index e813e43..42c0f60 100644
--- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
+++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
@@ -730,16 +730,13 @@ static struct ixgbe_txq_ops vec_txq_ops = {
 int
 ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq)
 {
-	struct rte_mbuf mb_def = {
-		.nb_segs = 1,
-		.data_off = RTE_PKTMBUF_HEADROOM,
-#ifdef RTE_MBUF_REFCNT
-		{ .refcnt = 1, }
-#endif
-	};
+	struct rte_mbuf mb_def = { .buf_addr = 0 }; /* zeroed mbuf */
 
+	mb_def.nb_segs = 1;
+	mb_def.data_off = RTE_PKTMBUF_HEADROOM;
 	mb_def.buf_len = rxq->mb_pool->elt_size - sizeof(struct rte_mbuf);
 	mb_def.port = rxq->port_id;
+	rte_mbuf_refcnt_set(&mb_def, 1);
 	rxq->mbuf_initializer = *((uint64_t *)&mb_def.rearm_data);
 	return 0;
 }
-- 
1.9.3

Re: [PATCH v2] ixgbe: fix icc issue with mbuf initializer

[Thomas Monjalon]

> When using Intel C++ compiler(icc) 14.0.1.106 or the older icc 13.x
> version, the mbuf initializer variable was not getting configured
> correctly, as the mb_def variable was not set correctly. This is due
> to an issue with icc (DPD200249565 which already been fixed in
> icc 14.0.2 and newer compiler release) where it incorrectly calculates
> the field offsets with initializers when zero-sized fields
> are used in a structure.
> To work around this, the code in ixgbe_rxq_vec_setup does not setup the
> fields using an initializer, but instead assigns the values individually
> in code
> NOTE: There is no performance impact to this change as the queue
> setup functions are not data-plane APIs, but are only used at app
> initialization.
> 
> Signed-off-by: Bruce Richardson <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Acked-by: Thomas Monjalon <thomas.monjalon-pdR9zngts4EAvxtiuMwx3w@public.gmane.org>

Applied

Thanks
-- 
Thomas

Re: [PATCH v2] ixgbe: fix icc issue with mbuf initializer

[Cao, Min]

Tested-by: Min Cao <min.cao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Patch name: 		[dpdk-dev] [PATCH v2] ixgbe: fix icc issue with mbuf initializer
Test Flag: 			Tested-by
Tester name: 		min.cao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org
ICC version:            13.1.2
ICC package:            l_ccompxe_2013.4.183.tgz
Result summary:		total 6 cases, 6 passed, 0 failed

Test Case 1:		
Name:				l2fwd
Environment:		OS: Fedora20 3.11.10-301.fc20.x86_64
				CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
				NIC: Fortville eagle/spirit 
Test result(32bit):	PASSED
Test result(64bit):	PASSED

Test Case 2:		
Name:				l3fwd
Environment:		OS: Fedora20 3.11.10-301.fc20.x86_64
				CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
				NIC: Fortville eagle/spirit 
Test result(32bit):	PASSED
Test result(64bit):	PASSED
Test Case 3:		
Name:				pmd
Environment:		OS: Fedora20 3.11.10-301.fc20.x86_64
				CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
				NIC: Fortville eagle/spirit 
Test result(32bit):	PASSED
Test result(64bit):	PASSED

-----Original Message-----
From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Bruce Richardson
Sent: Tuesday, November 04, 2014 1:01 AM
To: dev-VfR2kkLFssw@public.gmane.org
Subject: [dpdk-dev] [PATCH v2] ixgbe: fix icc issue with mbuf initializer

When using Intel C++ compiler(icc) 14.0.1.106 or the older icc 13.x
version, the mbuf initializer variable was not getting configured
correctly, as the mb_def variable was not set correctly. This is due
to an issue with icc (DPD200249565 which already been fixed in
icc 14.0.2 and newer compiler release) where it incorrectly calculates
the field offsets with initializers when zero-sized fields
are used in a structure.
To work around this, the code in ixgbe_rxq_vec_setup does not setup the
fields using an initializer, but instead assigns the values individually
in code
NOTE: There is no performance impact to this change as the queue
setup functions are not data-plane APIs, but are only used at app
initialization.

Signed-off-by: Bruce Richardson <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

---
V2 change: use rte_mbuf_refcnt_set to update reference count

---
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
index e813e43..42c0f60 100644
--- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
+++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
@@ -730,16 +730,13 @@ static struct ixgbe_txq_ops vec_txq_ops = {
 int
 ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq)
 {
-	struct rte_mbuf mb_def = {
-		.nb_segs = 1,
-		.data_off = RTE_PKTMBUF_HEADROOM,
-#ifdef RTE_MBUF_REFCNT
-		{ .refcnt = 1, }
-#endif
-	};
+	struct rte_mbuf mb_def = { .buf_addr = 0 }; /* zeroed mbuf */
 
+	mb_def.nb_segs = 1;
+	mb_def.data_off = RTE_PKTMBUF_HEADROOM;
 	mb_def.buf_len = rxq->mb_pool->elt_size - sizeof(struct rte_mbuf);
 	mb_def.port = rxq->port_id;
+	rte_mbuf_refcnt_set(&mb_def, 1);
 	rxq->mbuf_initializer = *((uint64_t *)&mb_def.rearm_data);
 	return 0;
 }
-- 
1.9.3

Re: [PATCH] ixgbe: fix icc issue with mbuf initializer

[Cao, Min]

Tested-by: Min Cao <min.cao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Patch name: 		[dpdk-dev] [PATCH v2] ixgbe: fix icc issue with mbuf initializer
Test Flag: 			Tested-by
Tester name: 		min.cao-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org
ICC version:            13.1.2
ICC package:            l_ccompxe_2013.4.183.tgz
Result summary:		total 6 cases, 6 passed, 0 failed

Test Case 1:		
Name:				l2fwd
Environment:		OS: Fedora20 3.11.10-301.fc20.x86_64
				CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
				NIC: Fortville eagle/spirit 
Test result(32bit):	PASSED
Test result(64bit):	PASSED

Test Case 2:		
Name:				l3fwd
Environment:		OS: Fedora20 3.11.10-301.fc20.x86_64
				CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
				NIC: Fortville eagle/spirit 
Test result(32bit):	PASSED
Test result(64bit):	PASSED
Test Case 3:		
Name:				pmd
Environment:		OS: Fedora20 3.11.10-301.fc20.x86_64
				CPU: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
				NIC: Fortville eagle/spirit 
Test result(32bit):	PASSED
Test result(64bit):	PASSED

-----Original Message-----
From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org] On Behalf Of Bruce Richardson
Sent: Monday, November 03, 2014 7:11 PM
To: dev-VfR2kkLFssw@public.gmane.org
Subject: [dpdk-dev] [PATCH] ixgbe: fix icc issue with mbuf initializer

When using Intel C++ compiler(icc) 14.0.1.106 or the older icc 13.x
version, the mbuf initializer variable was not getting configured
correctly, as the mb_def variable was not set correctly. This is due
to an issue with icc (DPD200249565 which already been fixed in
icc 14.0.2 and newer compiler release) where it incorrectly calculates
the field offsets with initializers when zero-sized fields
are used in a structure.
To work around this, the code in ixgbe_rxq_vec_setup does not setup the
fields using an initializer, but instead assigns the values individually
in code
NOTE: There is no performance impact to this change as the queue
setup functions are not data-plane APIs, but are only used at app
initialization.

Signed-off-by: Bruce Richardson <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
index e813e43..b57c588 100644
--- a/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
+++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c
@@ -730,16 +730,15 @@ static struct ixgbe_txq_ops vec_txq_ops = {
 int
 ixgbe_rxq_vec_setup(struct igb_rx_queue *rxq)
 {
-	struct rte_mbuf mb_def = {
-		.nb_segs = 1,
-		.data_off = RTE_PKTMBUF_HEADROOM,
-#ifdef RTE_MBUF_REFCNT
-		{ .refcnt = 1, }
-#endif
-	};
+	struct rte_mbuf mb_def = { .buf_addr = 0 }; /* zeroed mbuf */
 
+	mb_def.nb_segs = 1;
+	mb_def.data_off = RTE_PKTMBUF_HEADROOM;
 	mb_def.buf_len = rxq->mb_pool->elt_size - sizeof(struct rte_mbuf);
 	mb_def.port = rxq->port_id;
+#ifdef RTE_MBUF_REFCNT
+	mb_def.refcnt = 1;
+#endif
 	rxq->mbuf_initializer = *((uint64_t *)&mb_def.rearm_data);
 	return 0;
 }
-- 
1.9.3