How to count tx and rx bytes?

How to count tx and rx bytes?

[Ben Greear]

Is there an agreed upon standard for exactly what ethernet drivers
should be counting for rx-bytes and tx-bytes?  For example, should the
counters include the 4-byte FCS?  Should they include the ethernet header?

 From what I can tell, various drivers count the rx and tx bytes differently,
which cannot be correct....

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: How to count tx and rx bytes?

[David S. Miller]

On Mon, 15 Dec 2003 12:03:58 -0800
Ben Greear <greearb@candelatech.com> wrote:

> Is there an agreed upon standard for exactly what ethernet drivers
> should be counting for rx-bytes and tx-bytes?  For example, should the
> counters include the 4-byte FCS?  Should they include the ethernet header?

Good question.

It should be that all drivers use what skb->len ends up with at
rx/tx time.

However, it is often faster to just let the hardware keep track
of these statistics (tg3 is one example of a chip that can do
this).  And sometimes these mechanisms take the FCS or whatever
into account and this as you note makes the numbers different.

Re: How to count tx and rx bytes?

[Randy.Dunlap]

On Mon, 15 Dec 2003 14:17:29 -0800 "David S. Miller" <davem@redhat.com> wrote:

| On Mon, 15 Dec 2003 12:03:58 -0800
| Ben Greear <greearb@candelatech.com> wrote:
| 
| > Is there an agreed upon standard for exactly what ethernet drivers
| > should be counting for rx-bytes and tx-bytes?  For example, should the
| > counters include the 4-byte FCS?  Should they include the ethernet header?
| 
| Good question.
| 
| It should be that all drivers use what skb->len ends up with at
| rx/tx time.
| 
| However, it is often faster to just let the hardware keep track
| of these statistics (tg3 is one example of a chip that can do
| this).  And sometimes these mechanisms take the FCS or whatever
| into account and this as you note makes the numbers different.


RFC 1573:
RFC 2233:
RFC 1213:
all agree that on an "interface" the number of octets received (InOctets)
is:
 The total number of octets received on the interface, including
 framing characters.

so I read that as including MAC headers and FCS.  Without reading
these, I would have said include MAC headers but not FCS.

Similar for OutOctets.

--
~Randy
MOTD:  Always include version info.

Re: How to count tx and rx bytes?

[Donald Becker]

On Mon, 15 Dec 2003, Randy.Dunlap wrote:

> On Mon, 15 Dec 2003 14:17:29 -0800 "David S. Miller" <davem@redhat.com> wrote:
> 
> | On Mon, 15 Dec 2003 12:03:58 -0800
> | Ben Greear <greearb@candelatech.com> wrote:
> | 
> | > Is there an agreed upon standard for exactly what ethernet drivers
> | > should be counting for rx-bytes and tx-bytes?  For example, should the
> | > counters include the 4-byte FCS?

Yes.

> | >  Should they include the ethernet header?

Yes, but not the preamble.
  [[ Ethernet packets have an 8 byte preamble before the station
  addresses.  It doesn't make sense to count this, as
    all modern NICs strip it off, and
    a few leading bits may be dropped at each hop
  ]]

> | It should be that all drivers use what skb->len ends up with at
> | rx/tx time.

Not quite.  The skb->len value omits the 4 byte FCS/CRC.

> | However, it is often faster to just let the hardware keep track
> | of these statistics (tg3 is one example of a chip that can do
> | this).  And sometimes these mechanisms take the FCS or whatever
> | into account and this as you note makes the numbers different.

The driver should correct, although this is non-critical.
Using hardware counters where available used to make sense, but today
it's better to have the software driver maintain the statistics.  The
only thing the hardware counters are needed for is
otherwise-unobservable errors, such as missed packets and CRC errors.

> RFC 1573:
> RFC 2233:
> RFC 1213:
> all agree that on an "interface" the number of octets received (InOctets)
> is:
>  The total number of octets received on the interface, including
>  framing characters.

This is imprecise.  "Framing" might be misread to include the preamble,
which should be omitted from the count.

The original Linux errors counters were influenced by what the dp8390
chip reported.  Later, the definitions in appendix B of the dc21040
manual served as a more general guideline (although the software
counters were never a one-for-one copy of any specific hardware
semantics).


-- 
Donald Becker				becker@scyld.com
Scyld Computing Corporation		http://www.scyld.com
914 Bay Ridge Road, Suite 220		Scyld Beowulf cluster systems
Annapolis MD 21403			410-990-9993

Re: How to count tx and rx bytes?

[Randy.Dunlap]

On Mon, 15 Dec 2003 18:19:27 -0500 (EST) Donald Becker <becker@scyld.com> wrote:

| On Mon, 15 Dec 2003, Randy.Dunlap wrote:
| 
| > On Mon, 15 Dec 2003 14:17:29 -0800 "David S. Miller" <davem@redhat.com> wrote:
| > 
| > | On Mon, 15 Dec 2003 12:03:58 -0800
| > | Ben Greear <greearb@candelatech.com> wrote:
| > | 
| > | > Is there an agreed upon standard for exactly what ethernet drivers
| > | > should be counting for rx-bytes and tx-bytes?  For example, should the
| > | > counters include the 4-byte FCS?
| 
| Yes.
| 
| > | >  Should they include the ethernet header?
| 
| Yes, but not the preamble.
|   [[ Ethernet packets have an 8 byte preamble before the station
|   addresses.  It doesn't make sense to count this, as
|     all modern NICs strip it off, and
|     a few leading bits may be dropped at each hop
|   ]]
| 
| > | It should be that all drivers use what skb->len ends up with at
| > | rx/tx time.
| 
| Not quite.  The skb->len value omits the 4 byte FCS/CRC.
| 
| > | However, it is often faster to just let the hardware keep track
| > | of these statistics (tg3 is one example of a chip that can do
| > | this).  And sometimes these mechanisms take the FCS or whatever
| > | into account and this as you note makes the numbers different.
| 
| The driver should correct, although this is non-critical.
| Using hardware counters where available used to make sense, but today
| it's better to have the software driver maintain the statistics.  The
| only thing the hardware counters are needed for is
| otherwise-unobservable errors, such as missed packets and CRC errors.
| 
| > RFC 1573:
| > RFC 2233:
| > RFC 1213:
| > all agree that on an "interface" the number of octets received (InOctets)
| > is:
| >  The total number of octets received on the interface, including
| >  framing characters.
| 
| This is imprecise.  "Framing" might be misread to include the preamble,
| which should be omitted from the count.

Yes, sorry.

| The original Linux errors counters were influenced by what the dp8390
| chip reported.  Later, the definitions in appendix B of the dc21040
| manual served as a more general guideline (although the software
| counters were never a one-for-one copy of any specific hardware
| semantics).

I agree with Don's comments, based on my experiences in former lives.
I don't know about the DC21040, but I worked at/on both National and
Intel LAN drivers.

--
~Randy
MOTD:  Always include version info.

Re: How to count tx and rx bytes?

[Ben Greear]

David S. Miller wrote:
> On Mon, 15 Dec 2003 12:03:58 -0800
> Ben Greear <greearb@candelatech.com> wrote:
> 
> 
>>Is there an agreed upon standard for exactly what ethernet drivers
>>should be counting for rx-bytes and tx-bytes?  For example, should the
>>counters include the 4-byte FCS?  Should they include the ethernet header?
> 
> 
> Good question.
> 
> It should be that all drivers use what skb->len ends up with at
> rx/tx time.

A more noticeable problem is what I found in e100:  It was adding
skb->len after doing eth_type_trans, which yanks off the ethernet
header...  I moved it before the eth_type_trans and I am getting
much better numbers.  It does appear that the e1000 is counting the
FCS in transmitted bytes, however, and unless we tell the e100 to
receive the FCS, it will not count those on receive.

> However, it is often faster to just let the hardware keep track
> of these statistics (tg3 is one example of a chip that can do
> this).  And sometimes these mechanisms take the FCS or whatever
> into account and this as you note makes the numbers different.

Maybe the stats-polling code in the driver could do the necessary subtraction
to remove the FCS from the results (ie, subtract (packets-since-last-sample * 4) bytes.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: How to count tx and rx bytes?

[David S. Miller]

On Mon, 15 Dec 2003 14:46:59 -0800
Ben Greear <greearb@candelatech.com> wrote:

> Maybe the stats-polling code in the driver could do the necessary subtraction
> to remove the FCS from the results (ie, subtract (packets-since-last-sample * 4) bytes.

The whole reason to use the chip internally computed stats is to avoid
having to do "stats->foo++" at all, your suggestion basically eliminates
this purpose.

Re: How to count tx and rx bytes?

[Ben Greear]

David S. Miller wrote:
> On Mon, 15 Dec 2003 14:46:59 -0800
> Ben Greear <greearb@candelatech.com> wrote:
> 
> 
>>Maybe the stats-polling code in the driver could do the necessary subtraction
>>to remove the FCS from the results (ie, subtract (packets-since-last-sample * 4) bytes.
> 
> 
> The whole reason to use the chip internally computed stats is to avoid
> having to do "stats->foo++" at all, your suggestion basically eliminates
> this purpose.

Well, I was assuming that the stats polling is a fixed cost O(1), where-as the per-packet
calculation is O(n).  I am quite sure this assumption is true for e1000,
but I have not looked at tg3.  So, for e1000, the cost of subtracting out the FCS
would be basically free.

All that said, from Randy's email, it appears we should be including the FCS
anyway...

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: How to count tx and rx bytes?

[David S. Miller]

On Mon, 15 Dec 2003 15:17:29 -0800
Ben Greear <greearb@candelatech.com> wrote:

> Well, I was assuming that the stats polling is a fixed cost O(1),
> where-as the per-packet calculation is O(n).  I am quite sure this
> assumption is true for e1000, but I have not looked at tg3.  So, for
> e1000, the cost of subtracting out the FCS would be basically free.

I see, I misunderstood your idea.  That would work, and yes all of
the effort would be expended at netdev->get_stats() time.

> All that said, from Randy's email, it appears we should be including
> the FCS anyway...

Yes, indeed.  So the drivers that do not have hardware statistics
doing this, and are using skb->len, need to add on the FCS length
(which is 4 bytes right?) when accumulating stats->* values.

Sounds like a nice janitor job :)

Re: How to count tx and rx bytes?

[Rask Ingemann Lambertsen]

On Mon, Dec 15, 2003 at 12:03:58PM -0800, Ben Greear wrote:
> 
> Is there an agreed upon standard for exactly what ethernet drivers
> should be counting for rx-bytes and tx-bytes?  For example, should the
> counters include the 4-byte FCS?  Should they include the ethernet header?

Should they include the frame padding in tx-bytes? Some do, some don't.
IMHO they should, because you do actually transmit those bytes on the wire.

-- 
Regards,
Rask Ingemann Lambertsen

Re: How to count tx and rx bytes?

[Randy.Dunlap]

On Tue, 16 Dec 2003 14:38:16 +0100 Rask Ingemann Lambertsen <rask@sygehus.dk> wrote:

| On Mon, Dec 15, 2003 at 12:03:58PM -0800, Ben Greear wrote:
| > 
| > Is there an agreed upon standard for exactly what ethernet drivers
| > should be counting for rx-bytes and tx-bytes?  For example, should the
| > counters include the 4-byte FCS?  Should they include the ethernet header?
| 
| Should they include the frame padding in tx-bytes? Some do, some don't.
| IMHO they should, because you do actually transmit those bytes on the wire.

If you are referring to padding that is put there by software,
then Yes.

--
~Randy

Re: How to count tx and rx bytes?

[Ben Greear]

Randy.Dunlap wrote:
> On Tue, 16 Dec 2003 14:38:16 +0100 Rask Ingemann Lambertsen <rask@sygehus.dk> wrote:
> 
> | On Mon, Dec 15, 2003 at 12:03:58PM -0800, Ben Greear wrote:
> | > 
> | > Is there an agreed upon standard for exactly what ethernet drivers
> | > should be counting for rx-bytes and tx-bytes?  For example, should the
> | > counters include the 4-byte FCS?  Should they include the ethernet header?
> | 
> | Should they include the frame padding in tx-bytes? Some do, some don't.
> | IMHO they should, because you do actually transmit those bytes on the wire.
> 
> If you are referring to padding that is put there by software,
> then Yes.

The padding is often done in the NIC itself, and since the bytes are put
on the wire, they should be counted...

> 
> --
> ~Randy
> 


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com