Getting physical packet counts with LRO enabled with ixgbe?

Getting physical packet counts with LRO enabled with ixgbe?

[Ben Greear]

I notice that with LRO enabled, the interface stats count the LRO'd pkts,
not the physical ones on the wire.

I also tried using ethtool -S, but it seems those counters are the same.

Is there any way to get the actual rx/tx packet count on the wire?

Also, for the rx/tx bytes, I assume that isn't counting the protocol headers
for the physical pkts that have been merged into a single LRO packet.  Is
there any way to get the wire stats for bytes as well?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Peter P Waskiewicz Jr]

On Wed, 2009-09-23 at 09:40 -0700, Ben Greear wrote:
> I notice that with LRO enabled, the interface stats count the LRO'd pkts,
> not the physical ones on the wire.
> 
> I also tried using ethtool -S, but it seems those counters are the same.
> 
> Is there any way to get the actual rx/tx packet count on the wire?
> 

Depending on which device you're using ixgbe with, there are slightly
different registers to get what you want.

The only suggestion I have for you though is to refer to the datasheets
for each device on our SourceForge site (e1000.sf.net).  Some of the
relevant counters to look at are PRC64, PRC127, etc, and GPRC/GPTC.  For
the per-queue stuff, you'll need to look at the TQSMR and RQSMR mapping
registers.  Let me know if you need assistance in using these registers.

> Also, for the rx/tx bytes, I assume that isn't counting the protocol headers
> for the physical pkts that have been merged into a single LRO packet.  Is
> there any way to get the wire stats for bytes as well?

The counters for per-byte are purely software-based, so if the packet is
LRO'd, you can probably do some somewhat trivial math with the MTU to
find the actual wire stats.  But we only compute what we pass to the
stack, so it'd be the LRO'd packet.

Cheers,
-PJ Waskiewicz

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Ben Greear]

On 09/23/2009 11:32 AM, Peter P Waskiewicz Jr wrote:
> On Wed, 2009-09-23 at 09:40 -0700, Ben Greear wrote:
>> I notice that with LRO enabled, the interface stats count the LRO'd pkts,
>> not the physical ones on the wire.
>>
>> I also tried using ethtool -S, but it seems those counters are the same.
>>
>> Is there any way to get the actual rx/tx packet count on the wire?
>>
>
> Depending on which device you're using ixgbe with, there are slightly
> different registers to get what you want.
>
> The only suggestion I have for you though is to refer to the datasheets
> for each device on our SourceForge site (e1000.sf.net).  Some of the
> relevant counters to look at are PRC64, PRC127, etc, and GPRC/GPTC.  For
> the per-queue stuff, you'll need to look at the TQSMR and RQSMR mapping
> registers.  Let me know if you need assistance in using these registers.

Thanks, I'll look at the data-sheet.

I don't care about per-queue stats at this time, just
over-all NIC stats.

>> Also, for the rx/tx bytes, I assume that isn't counting the protocol headers
>> for the physical pkts that have been merged into a single LRO packet.  Is
>> there any way to get the wire stats for bytes as well?
>
> The counters for per-byte are purely software-based, so if the packet is
> LRO'd, you can probably do some somewhat trivial math with the MTU to
> find the actual wire stats.  But we only compute what we pass to the
> stack, so it'd be the LRO'd packet.

That's a bummer.  I'm guessing you might get close to right on average with some
trivial math, but if someone is sending you pkts with size of 1000 and
your MTU is 1500, would there be any way to tell that the pkts were originally
1000 bytes instead of 1500?

Next time you guys re-compile your hardware, please consider adding byte counters :)

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Peter P Waskiewicz Jr]

On Wed, 2009-09-23 at 11:46 -0700, Ben Greear wrote:
> On 09/23/2009 11:32 AM, Peter P Waskiewicz Jr wrote:
> > On Wed, 2009-09-23 at 09:40 -0700, Ben Greear wrote:
> >> I notice that with LRO enabled, the interface stats count the LRO'd pkts,
> >> not the physical ones on the wire.
> >>
> >> I also tried using ethtool -S, but it seems those counters are the same.
> >>
> >> Is there any way to get the actual rx/tx packet count on the wire?
> >>
> >
> > Depending on which device you're using ixgbe with, there are slightly
> > different registers to get what you want.
> >
> > The only suggestion I have for you though is to refer to the datasheets
> > for each device on our SourceForge site (e1000.sf.net).  Some of the
> > relevant counters to look at are PRC64, PRC127, etc, and GPRC/GPTC.  For
> > the per-queue stuff, you'll need to look at the TQSMR and RQSMR mapping
> > registers.  Let me know if you need assistance in using these registers.
> 
> Thanks, I'll look at the data-sheet.
> 
> I don't care about per-queue stats at this time, just
> over-all NIC stats.
> 
> >> Also, for the rx/tx bytes, I assume that isn't counting the protocol headers
> >> for the physical pkts that have been merged into a single LRO packet.  Is
> >> there any way to get the wire stats for bytes as well?
> >
> > The counters for per-byte are purely software-based, so if the packet is
> > LRO'd, you can probably do some somewhat trivial math with the MTU to
> > find the actual wire stats.  But we only compute what we pass to the
> > stack, so it'd be the LRO'd packet.
> 
> That's a bummer.  I'm guessing you might get close to right on average with some
> trivial math, but if someone is sending you pkts with size of 1000 and
> your MTU is 1500, would there be any way to tell that the pkts were originally
> 1000 bytes instead of 1500?

Good point.

> Next time you guys re-compile your hardware, please consider adding byte counters :)

On 10G adapters, byte counters can skyrocket quickly, so we'd need to
read them often to avoid them wrapping.  But I will forward your request
to our HW design folks and see if they have other ideas to implement
these counters and make them efficient.

Cheers,
-PJ

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Ben Greear]

On 09/23/2009 11:50 AM, Peter P Waskiewicz Jr wrote:

>> That's a bummer.  I'm guessing you might get close to right on average with some
>> trivial math, but if someone is sending you pkts with size of 1000 and
>> your MTU is 1500, would there be any way to tell that the pkts were originally
>> 1000 bytes instead of 1500?
>
> Good point.
>
>> Next time you guys re-compile your hardware, please consider adding byte counters :)
>
> On 10G adapters, byte counters can skyrocket quickly, so we'd need to
> read them often to avoid them wrapping.  But I will forward your request
> to our HW design folks and see if they have other ideas to implement
> these counters and make them efficient.

It still takes a while to wrap 64-bit counters :)

But, you do have to read every 3 secs or so if you're using 32-bit counters.

This can be dealt with in user-space easily enough as long as polling
the NIC for counters is efficient.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Rick Jones]

>>Next time you guys re-compile your hardware, please consider adding byte counters :)
> 
> 
> On 10G adapters, byte counters can skyrocket quickly, so we'd need to
> read them often to avoid them wrapping. 

10G Ethernet is ~1.16 GB/s (GiB/s for purists I guess) for simplicity, call that 
2GB/s or 2^31 bytes per second.  If the counter is 64 bits, that would suggest 
wrap in 2^64/2^31 or 2^33 seconds right?  Or have I made some nasty math error? 
  I'm having quit a difficult time imagining that someone would have 32 bit 
counters in a 10G NIC.

rick jones

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Ben Greear]

I noticed that 'ethtool -d' has some counters that may be right,
including bytes.

(I'm looking at the 82599 chipset currently)

0x04074: gprc        (Good Packets Received Count)    0xF611E2C2
0x04078: bprc        (Broadcast Packets Rx Count)     0x00000000
0x0407C: mprc        (Multicast Packets Rx Count)     0x0000000C
0x04080: gptc        (Good Packets Transmitted Count) 0xC9962C10
0x04088: gorcl       (Good Octets Rx Count Low)       0x9A23F0B0
0x0408C: gorch       (Good Octets Rx Count High)      0x00000000
0x04090: gotcl       (Good Octets Tx Count Low)       0x5DFAACD4
0x04094: gotch       (Good Octets Tx Count High)      0x00000000


Any suggestions on a more efficient way to get these than dumping 'ethtool -d' and
searching all that data?  Maybe put it in 'ethtool -S' as well ?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Ben Greear]

I'm poking at the ixgbe_main code in 2.6.31.

It seems from the spec sheet that the 82599 supports the GORCH.


	/* 82598 hardware only has a 32 bit counter in the high register */
	if (hw->mac.type == ixgbe_mac_82599EB) {
		adapter->stats.gorc += IXGBE_READ_REG(hw, IXGBE_GORCL);
		IXGBE_READ_REG(hw, IXGBE_GORCH); /* to clear */


stats.gorc is 64-bit, so any reason not to grab the 4 high-bits out of GORCL
and add them to stats.gorc instead of just clearing them as this
code seems to do?

That gives us some precious extra seconds to read counters before they
wrap :)

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Herbert Xu]

Peter P Waskiewicz Jr <peter.p.waskiewicz.jr@intel.com> wrote:
>
>> That's a bummer.  I'm guessing you might get close to right on average with some
>> trivial math, but if someone is sending you pkts with size of 1000 and
>> your MTU is 1500, would there be any way to tell that the pkts were originally
>> 1000 bytes instead of 1500?
> 
> Good point.

Well if you did GRO instead of LRO then none of this would matter :)
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Ben Greear]

On 09/25/2009 02:45 PM, Herbert Xu wrote:
> Peter P Waskiewicz Jr<peter.p.waskiewicz.jr@intel.com>  wrote:
>>
>>> That's a bummer.  I'm guessing you might get close to right on average with some
>>> trivial math, but if someone is sending you pkts with size of 1000 and
>>> your MTU is 1500, would there be any way to tell that the pkts were originally
>>> 1000 bytes instead of 1500?
>>
>> Good point.
>
> Well if you did GRO instead of LRO then none of this would matter :)

I get aggregate 18Gbps with LRO v/s 12Gbps without, so it's very much a useful feature
for me.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Peter P Waskiewicz Jr]

On Fri, 2009-09-25 at 14:45 -0700, Herbert Xu wrote:
> Peter P Waskiewicz Jr <peter.p.waskiewicz.jr@intel.com> wrote:
> >
> >> That's a bummer.  I'm guessing you might get close to right on average with some
> >> trivial math, but if someone is sending you pkts with size of 1000 and
> >> your MTU is 1500, would there be any way to tell that the pkts were originally
> >> 1000 bytes instead of 1500?
> > 
> > Good point.
> 
> Well if you did GRO instead of LRO then none of this would matter :)

ixgbe uses GRO, but we have HW RSC/LRO running on 82599 adapters
underneath GRO.

-PJ

Re: Getting physical packet counts with LRO enabled with ixgbe?

[David Miller]

From: Ben Greear <greearb@candelatech.com>
Date: Fri, 25 Sep 2009 14:50:44 -0700

> I get aggregate 18Gbps with LRO v/s 12Gbps without, so it's very
> much a useful feature for me.

And it'll stay like that until you work with Herbert to fix
that performance difference.

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Herbert Xu]

On Fri, Sep 25, 2009 at 02:59:15PM -0700, David Miller wrote:
> From: Ben Greear <greearb@candelatech.com>
> Date: Fri, 25 Sep 2009 14:50:44 -0700
> 
> > I get aggregate 18Gbps with LRO v/s 12Gbps without, so it's very
> > much a useful feature for me.
> 
> And it'll stay like that until you work with Herbert to fix
> that performance difference.

Actually I believe he's talking about hardware LRO which is a
different beast.

AFAICS the ixgbe driver doesn't use software LRO anymore.

So if it is hardware LRO then what we need is for the hardware
to switch over :)

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Herbert Xu]

Ben Greear <greearb@candelatech.com> wrote:
>
>> Well if you did GRO instead of LRO then none of this would matter :)
> 
> I get aggregate 18Gbps with LRO v/s 12Gbps without, so it's very much a useful feature
> for me.

Sorry we are each talking about different things.  The performance
difference you observed is software GRO vs. hardware LRO.  What I
was talking about above is switching the Intel hardware to do GRO
instead of LRO.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Herbert Xu]

Peter P Waskiewicz Jr <peter.p.waskiewicz.jr@intel.com> wrote:
>
>> Well if you did GRO instead of LRO then none of this would matter :)
> 
> ixgbe uses GRO, but we have HW RSC/LRO running on 82599 adapters
> underneath GRO.

I know.  What I was suggesting is that the hardware should also
switch away from LRO to GRO, i.e., restrict what you merge so that
resegmentation can be performed.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Ben Greear]

On 09/25/2009 03:01 PM, Herbert Xu wrote:
> On Fri, Sep 25, 2009 at 02:59:15PM -0700, David Miller wrote:
>> From: Ben Greear<greearb@candelatech.com>
>> Date: Fri, 25 Sep 2009 14:50:44 -0700
>>
>>> I get aggregate 18Gbps with LRO v/s 12Gbps without, so it's very
>>> much a useful feature for me.
>>
>> And it'll stay like that until you work with Herbert to fix
>> that performance difference.
>
> Actually I believe he's talking about hardware LRO which is a
> different beast.
>
> AFAICS the ixgbe driver doesn't use software LRO anymore.
>
> So if it is hardware LRO then what we need is for the hardware
> to switch over :)

Either way, I will be happy to test & benchmark patches of this nature
if someone wants to post them.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Getting physical packet counts with LRO enabled with ixgbe?

[Peter P Waskiewicz Jr]

On Fri, 2009-09-25 at 15:12 -0700, Ben Greear wrote:
> On 09/25/2009 03:01 PM, Herbert Xu wrote:
> > On Fri, Sep 25, 2009 at 02:59:15PM -0700, David Miller wrote:
> >> From: Ben Greear<greearb@candelatech.com>
> >> Date: Fri, 25 Sep 2009 14:50:44 -0700
> >>
> >>> I get aggregate 18Gbps with LRO v/s 12Gbps without, so it's very
> >>> much a useful feature for me.
> >>
> >> And it'll stay like that until you work with Herbert to fix
> >> that performance difference.
> >
> > Actually I believe he's talking about hardware LRO which is a
> > different beast.
> >
> > AFAICS the ixgbe driver doesn't use software LRO anymore.
> >
> > So if it is hardware LRO then what we need is for the hardware
> > to switch over :)
> 
> Either way, I will be happy to test & benchmark patches of this nature
> if someone wants to post them.
> 

I can't post a patch that changes my hardware behavior.  :-)

So to be clear, ixgbe uses GRO, and does not use software LRO.  In 82599
adapters, we have a HW LRO engine.  So we run HW LRO and GRO together.
What Herbert is referring to is that we change our HW design of the HW
LRO engine to do more of what GRO is doing today.  That would require a
silicon spin, so that wouldn't happen soon unfortunately.

-PJ