[e1000]: flow control on by default - good idea really?

[e1000]: flow control on by default - good idea really?

[jamal]

CCing anybody who may have stakes on this. Ignore the email if this
doesnt interest you.
Ok, folks - i had deferred this discussion but it bit me in the ass. 
I just spend an hour debugging it (and in the process blew up a gbic i
borrowed, so my day aint going well since i actually have to pay for
this and cant really do the testing i was planning to;-<).
 
I have a device connected to a e1000 that was erroneously advertising
both tx/rx flow control but wasnt properly reacting to it. 
The default setup on the e1000 has rx flow control turned on.
I was sending at wire rate gige from the device - which is about
1.48Mpps. The e1000 was in turn sending me flow control packets
as per default/expected behavior. Unfortunately, it was sending
a very large amount of packets. At one point i was seeing upto
1Mpps and on average, the flow control packets were consuming
60-70% of the bandwidth. Even when i fixed this behavior to act
properly, allowing flow control on consumed up to 15% of the bandwidth. 
Clearly, this is a bad thing. Yes, the device in the first instance was
at fault. But i have argued in the past that NAPI does just fine without
flow control being turned on, so even chewing 5% of bandwidth on flow
control is a bad thing..

As a compromise, can we declare flow control as an advanced feature
and turn it off by default? People who feel it is valuable and know
what they are doing can turn it off.

If you want more details just shoot.

cheers,
jamal 

PS:- BTW, even turning off flow control on e1000 didnt give as good
performance as in the old days on this machine - but i dont want to go
into that discussion.

Re: [e1000]: flow control on by default - good idea really?

[jamal]

On Tue, 2006-04-07 at 13:11 -0400, jamal wrote:
> CCing anybody who may have stakes on this. Ignore the email if this
> doesnt interest you.
> Ok, folks - i had deferred this discussion but it bit me in the ass. 
> I just spend an hour debugging it (and in the process blew up a gbic i
> borrowed, so my day aint going well since i actually have to pay for
> this and cant really do the testing i was planning to;-<).
>  
> I have a device connected to a e1000 that was erroneously advertising
> both tx/rx flow control but wasnt properly reacting to it. 
> The default setup on the e1000 has rx flow control turned on.
> I was sending at wire rate gige from the device - which is about
> 1.48Mpps. The e1000 was in turn sending me flow control packets
> as per default/expected behavior. Unfortunately, it was sending
> a very large amount of packets. At one point i was seeing upto
> 1Mpps and on average, the flow control packets were consuming
> 60-70% of the bandwidth. Even when i fixed this behavior to act
> properly, allowing flow control on consumed up to 15% of the bandwidth. 
> Clearly, this is a bad thing. Yes, the device in the first instance was
> at fault. But i have argued in the past that NAPI does just fine without
> flow control being turned on, so even chewing 5% of bandwidth on flow
> control is a bad thing..
> 
> As a compromise, can we declare flow control as an advanced feature
> and turn it off by default? People who feel it is valuable and know
> what they are doing can turn it off.

I meant turn it on.

BTW, As an addendum this default behavior changed around 2.6.16 it
seems.

cheers,
jamal

Re: [e1000]: flow control on by default - good idea really?

[Auke Kok]

jamal wrote:
> On Tue, 2006-04-07 at 13:11 -0400, jamal wrote:
>> I have a device connected to a e1000 that was erroneously advertising
>> both tx/rx flow control but wasnt properly reacting to it. 
>> The default setup on the e1000 has rx flow control turned on.
>> I was sending at wire rate gige from the device - which is about
>> 1.48Mpps. The e1000 was in turn sending me flow control packets
>> as per default/expected behavior. Unfortunately, it was sending
>> a very large amount of packets. At one point i was seeing upto
>> 1Mpps and on average, the flow control packets were consuming
>> 60-70% of the bandwidth. Even when i fixed this behavior to act
>> properly, allowing flow control on consumed up to 15% of the bandwidth. 
>> Clearly, this is a bad thing. Yes, the device in the first instance was
>> at fault. But i have argued in the past that NAPI does just fine without
>> flow control being turned on, so even chewing 5% of bandwidth on flow
>> control is a bad thing..
>>
>> As a compromise, can we declare flow control as an advanced feature
>> and turn it off by default? People who feel it is valuable and know
>> what they are doing can turn it off.
> 
> I meant turn it on.
> 
> BTW, As an addendum this default behavior changed around 2.6.16 it
> seems.

Flow Control is using the EEPROM provided value, the module driver itself does 
not choose a default:

e1000_param.c:

/* User Specified Flow Control Override
  *
  * Valid Range: 0-3
  *  - 0 - No Flow Control
  *  - 1 - Rx only, respond to PAUSE frames but do not generate them
  *  - 2 - Tx only, generate PAUSE frames but ignore them on receive
  *  - 3 - Full Flow Control Support
  *
  * Default Value: Read flow control settings from the EEPROM
  */

Turning flow control off usually (i.e. almost always) causes (significantly) 
_degraded_ performance. We should really leave it the way it is (as per eeprom 
setting), and this is best for most if not all people. The card itself has 
this value programmed, which makes it possible for the user to turn on/off 
flowcontrol per card consistently, which makes much more sense to me. Also 
considering e1000 hardware varies significantly.

Moreover, most support calls here where people complain about degraded 
performance are about users who turned flow control _off_. :)


Cheers,

Auke

Re: [e1000]: flow control on by default - good idea really?

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2455 bytes --]



On Wed, 5 Jul 2006, Auke Kok wrote:

> jamal wrote:
>> On Tue, 2006-04-07 at 13:11 -0400, jamal wrote:
>>> I have a device connected to a e1000 that was erroneously advertising
>>> both tx/rx flow control but wasnt properly reacting to it. The default 
>>> setup on the e1000 has rx flow control turned on.
>>> I was sending at wire rate gige from the device - which is about
>>> 1.48Mpps. The e1000 was in turn sending me flow control packets
>>> as per default/expected behavior. Unfortunately, it was sending
>>> a very large amount of packets. At one point i was seeing upto
>>> 1Mpps and on average, the flow control packets were consuming
>>> 60-70% of the bandwidth. Even when i fixed this behavior to act
>>> properly, allowing flow control on consumed up to 15% of the bandwidth. 
>>> Clearly, this is a bad thing. Yes, the device in the first instance was
>>> at fault. But i have argued in the past that NAPI does just fine without
>>> flow control being turned on, so even chewing 5% of bandwidth on flow
>>> control is a bad thing..
>>> 
>>> As a compromise, can we declare flow control as an advanced feature
>>> and turn it off by default? People who feel it is valuable and know
>>> what they are doing can turn it off.
>> 
>> I meant turn it on.
>> 
>> BTW, As an addendum this default behavior changed around 2.6.16 it
>> seems.
>
> Flow Control is using the EEPROM provided value, the module driver itself 
> does not choose a default:
>
> e1000_param.c:
>
> /* User Specified Flow Control Override
> *
> * Valid Range: 0-3
> *  - 0 - No Flow Control
> *  - 1 - Rx only, respond to PAUSE frames but do not generate them
> *  - 2 - Tx only, generate PAUSE frames but ignore them on receive
> *  - 3 - Full Flow Control Support
> *
> * Default Value: Read flow control settings from the EEPROM
> */
>
> Turning flow control off usually (i.e. almost always) causes (significantly) 
> _degraded_ performance. We should really leave it the way it is (as per 
> eeprom setting), and this is best for most if not all people. The card itself 
> has this value programmed, which makes it possible for the user to turn 
> on/off flowcontrol per card consistently, which makes much more sense to me. 
> Also considering e1000 hardware varies significantly.

I was never able to find such tool for Linux or at least DOS. Where should 
I look for it?

Best regards,

 				Krzysztof Olędzki

Re: [e1000]: flow control on by default - good idea really?

[David Miller]

From: jamal <hadi@cyberus.ca>
Date: Tue, 04 Jul 2006 15:20:39 -0400

> BTW, As an addendum this default behavior changed around 2.6.16 it
> seems.

Flow control has been on by default in the tg3 driver since the
beginning, maybe e1000 only recently started to behave that way
but it's the right thing to do IMHO.

Re: [e1000]: flow control on by default - good idea really?

[Auke Kok]

David Miller wrote:
> From: jamal <hadi@cyberus.ca>
> Date: Tue, 04 Jul 2006 15:20:39 -0400
> 
>> BTW, As an addendum this default behavior changed around 2.6.16 it
>> seems.
> 
> Flow control has been on by default in the tg3 driver since the
> beginning, maybe e1000 only recently started to behave that way
> but it's the right thing to do IMHO.

As said earlier, e1000 always honors the EEPROM setting for this, which has 
been _on_ by default for all cards (AFAIK, that is).

Cheers,

Auke

Re: [e1000]: flow control on by default - good idea really?

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1928 bytes --]



On Wed, 5 Jul 2006, Auke Kok wrote:

> David Miller wrote:
>> From: jamal <hadi@cyberus.ca>
>> Date: Tue, 04 Jul 2006 15:20:39 -0400
>> 
>>> BTW, As an addendum this default behavior changed around 2.6.16 it
>>> seems.
>> 
>> Flow control has been on by default in the tg3 driver since the
>> beginning, maybe e1000 only recently started to behave that way
>> but it's the right thing to do IMHO.
>
> As said earlier, e1000 always honors the EEPROM setting for this, which has 
> been _on_ by default for all cards (AFAIK, that is).

I'm not sure:

root@r1:~# mii-tool -v eth0
eth0: negotiated 100baseTx-FD, link ok
   product info: vendor 00:aa:00, model 56 rev 0
   basic mode:   autonegotiation enabled
   basic status: autonegotiation complete, link ok
   capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
   advertising:  100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
   link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control

root@r1:~# ethtool -d eth0|grep flow
       Receive flow control:              disabled
       Transmit flow control:             disabled

root@r1:~# uname -r
2.6.14.3

root@r2:~# mii-tool -v eth0
eth0: negotiated 100baseTx-FD flow-control, link ok
   product info: vendor 00:aa:00, model 56 rev 0
   basic mode:   autonegotiation enabled
   basic status: autonegotiation complete, link ok
   capabilities: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD
   advertising:  100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control
   link partner: 100baseTx-FD 100baseTx-HD 10baseT-FD 10baseT-HD flow-control

root@r2:~# ethtool -d eth0|grep flow
       Receive flow control:              enabled
       Transmit flow control:             enabled

root@r2:~# uname -r
2.6.16.19

This is exactly the same hardware, only kernel was recently upgraded on 
the r2.

Best regards,


 				Krzysztof Olędzki

Re: [e1000]: flow control on by default - good idea really?

[Auke Kok]

Krzysztof Oledzki wrote:
> 
> 
> On Wed, 5 Jul 2006, Auke Kok wrote:
> 
>> David Miller wrote:
>>> From: jamal <hadi@cyberus.ca>
>>> Date: Tue, 04 Jul 2006 15:20:39 -0400
>>>
>>>> BTW, As an addendum this default behavior changed around 2.6.16 it
>>>> seems.
>>>
>>> Flow control has been on by default in the tg3 driver since the
>>> beginning, maybe e1000 only recently started to behave that way
>>> but it's the right thing to do IMHO.
>>
>> As said earlier, e1000 always honors the EEPROM setting for this, 
>> which has been _on_ by default for all cards (AFAIK, that is).
> 
> I'm not sure:
> 
> root@r1:~# ethtool -d eth0|grep flow
>       Receive flow control:              disabled
>       Transmit flow control:             disabled
> root@r2:~# ethtool -d eth0|grep flow
>       Receive flow control:              enabled
>       Transmit flow control:             enabled


Same physical NIC or different ones? In the latter case your NICs may have 
different EEPROM settings. You can dump the EEPROM and compare, do something 
like `ethtool -e eth0 raw on length 64| hexdump` for both interfaces and send 
it to me. Only the first 64 bytes should matter.

Cheers,

Auke

Re: [e1000]: flow control on by default - good idea really?

[jamal]

On Wed, 2006-05-07 at 22:45 +0200, Krzysztof Oledzki wrote:
> 
> On Wed, 5 Jul 2006, Auke Kok wrote:
> 
> > David Miller wrote:
> >> From: jamal <hadi@cyberus.ca>
> >> Date: Tue, 04 Jul 2006 15:20:39 -0400
> >> 
> >>> BTW, As an addendum this default behavior changed around 2.6.16 it
> >>> seems.
> >> 
> >> Flow control has been on by default in the tg3 driver since the
> >> beginning, 

Are you sure about this Dave?;-> because I do have a tg3 on my laptop.

hadi@jzny2:~/Desktop/maemo$ sudo ethtool -a eth0
Pause parameters for eth0:
Autonegotiate:  on
RX:             off
TX:             off

hadi@jzny2:~/Desktop/maemo$ uname -a
Linux jzny2 2.6.16 #6 Fri Jun 9 15:29:40 EDT 2006 i686 GNU/Linux
hadi@jzny2:~/Desktop/maemo$

Maybe it is read from the eeprom and mine has it off?

> maybe e1000 only recently started to behave that way
> >> but it's the right thing to do IMHO.
> >

I will continue testing when i get proper hardware and post. And if only
Robert and I are whining about this, we are adults and we can turn it
off. I never tried it with large packets, so it may be helpful there
for forwarding.

Again, note that: It is consuming > 10% (13-15% range) of my bandwidth.
Granted that is at high speeds with small packets so may not be
reflective of 96% of the world. But that would be > 50kpps of my
forwarding capacity being chewed unreasonably. So Auke, did you say
"performance" was what people mostly bitched about? ;->

> > As said earlier, e1000 always honors the EEPROM setting for this, which has 
> > been _on_ by default for all cards (AFAIK, that is).
> 

It has _never ever_ worked on e1000 for as long as i have used e1000. If
it was intended to work, it must have been fixed in 2.6.16. So it is new
behavior.

> I'm not sure:
> 
> root@r1:~# mii-tool -v eth0

Try ethtool -a eth0 to read and -A to set.

cheers,
jamal

Re: [e1000]: flow control on by default - good idea really?

[Auke Kok]

jamal wrote:
> On Wed, 2006-05-07 at 22:45 +0200, Krzysztof Oledzki wrote:
>> On Wed, 5 Jul 2006, Auke Kok wrote:
>>
>>> David Miller wrote:
>>>> Flow control has been on by default in the tg3 driver since the
>>>> beginning,
> 
> Are you sure about this Dave?;-> because I do have a tg3 on my laptop.
> 
> hadi@jzny2:~/Desktop/maemo$ sudo ethtool -a eth0
> Pause parameters for eth0:
> Autonegotiate:  on
> RX:             off
> TX:             off

mine says it's on :)

> Maybe it is read from the eeprom and mine has it off?
> 
> Again, note that: It is consuming > 10% (13-15% range) of my bandwidth.
> Granted that is at high speeds with small packets so may not be
> reflective of 96% of the world. But that would be > 50kpps of my
> forwarding capacity being chewed unreasonably. So Auke, did you say
> "performance" was what people mostly bitched about? ;->

yes, but that's linked with hardware that doesn't handle flowcontrol events 
properly, if you were doing large message TCP transfers over that you'd 
probably see even worse performance I bet (retransmits being dropped etc).

Jesse is working on performance stuff, he'll gladly look into it :)

>>> As said earlier, e1000 always honors the EEPROM setting for this, which has 
>>> been _on_ by default for all cards (AFAIK, that is).
> 
> It has _never ever_ worked on e1000 for as long as i have used e1000. If
> it was intended to work, it must have been fixed in 2.6.16. So it is new
> behavior.

Turns out that of the e1000 cards I can find around here that are plugged in 
actually are 50-50 distributed on/off, so I was wrong about it being on by 
default everywhere.

Looking back through the code I see no changes affecting flow control setup as 
early as 2.6.12 ... There are some minor (new) HW changes but nothing that 
should have boken fc.

Cheers,

Auke

Re: [e1000]: flow control on by default - good idea really?

[jamal]

On Thu, 2006-06-07 at 11:25 -0700, Auke Kok wrote:
> jamal wrote:

> > hadi@jzny2:~/Desktop/maemo$ sudo ethtool -a eth0
> > Pause parameters for eth0:
> > Autonegotiate:  on
> > RX:             off
> > TX:             off
> 
> mine says it's on :)


Dell D610:
hadi@jzny2:~$ sudo lspci | grep -i bcm
0000:02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5751
Gigabit Ethernet PCI Express (rev 01)

Whats yours?
So ok, from what you say below i take it that even with the e1000s
you didnt find consistency on different machines.
Can you double check at least the kernels are >= 2.6.16?
I can tell you this hardware that caused me problems never had flow
control working in the earlier kernels.

> > Maybe it is read from the eeprom and mine has it off?
> > 
> > Again, note that: It is consuming > 10% (13-15% range) of my bandwidth.
> > Granted that is at high speeds with small packets so may not be
> > reflective of 96% of the world. But that would be > 50kpps of my
> > forwarding capacity being chewed unreasonably. So Auke, did you say
> > "performance" was what people mostly bitched about? ;->
> 
> yes, but that's linked with hardware that doesn't handle flowcontrol events 
> properly, 

As i mentioned earlier: This is actually hardware that works with flow
control ;->
When it doesnt work - as in the case of the one i found advertising but
not respecting flow control the bandwidth being consumed was > 60% ;->
To give you perspective, on average that is > 500Mbps

> if you were doing large message TCP transfers over that you'd 
> probably see even worse performance I bet (retransmits being dropped etc).
> 

I havent tested that, but it does seem unlikely.

> Jesse is working on performance stuff, he'll gladly look into it :)
> 

Jesse, if you want to reproduce this talk to me and i will give you a
description of how to do it. Make sure you can record flow control
packets both in send and receive.


> >>> As said earlier, e1000 always honors the EEPROM setting for this, which has 
> >>> been _on_ by default for all cards (AFAIK, that is).
> > 
> > It has _never ever_ worked on e1000 for as long as i have used e1000. If
> > it was intended to work, it must have been fixed in 2.6.16. So it is new
> > behavior.
> 
> Turns out that of the e1000 cards I can find around here that are plugged in 
> actually are 50-50 distributed on/off, so I was wrong about it being on by 
> default everywhere.
> 
> Looking back through the code I see no changes affecting flow control setup as 
> early as 2.6.12 ... There are some minor (new) HW changes but nothing that 
> should have boken fc.
> 

It could be something very basic. Since you have access to a variety of
hardware (and variety of kernels i take it), you can go by elimination
and find it.

cheers,
jamal

Re: [e1000]: flow control on by default - good idea really?

[David Miller]

From: jamal <hadi@cyberus.ca>
Date: Thu, 06 Jul 2006 09:03:38 -0400

> On Wed, 2006-05-07 at 22:45 +0200, Krzysztof Oledzki wrote:
> > 
> > On Wed, 5 Jul 2006, Auke Kok wrote:
> > 
> > > David Miller wrote:
> > >> From: jamal <hadi@cyberus.ca>
> > >> Date: Tue, 04 Jul 2006 15:20:39 -0400
> > >> 
> > >>> BTW, As an addendum this default behavior changed around 2.6.16 it
> > >>> seems.
> > >> 
> > >> Flow control has been on by default in the tg3 driver since the
> > >> beginning, 
> 
> Are you sure about this Dave?;-> because I do have a tg3 on my laptop.
> 
> hadi@jzny2:~/Desktop/maemo$ sudo ethtool -a eth0
> Pause parameters for eth0:
> Autonegotiate:  on
> RX:             off
> TX:             off

It's autonegotiated, check you kernel message logs when the link
came up, you'll see this:

tg3: eth0: Flow control is on for TX and on for RX.

Take care.

Re: [e1000]: flow control on by default - good idea really?

[jamal]

On Thu, 2006-06-07 at 23:59 -0700, David Miller wrote:

> 
> It's autonegotiated, check you kernel message logs when the link
> came up, you'll see this:
> 
> tg3: eth0: Flow control is on for TX and on for RX.
> 

yikes - yes, this would be it.

I  could be wrong and i will double check:
I think when the e1000 says via ethtool "rx is on" - it means that it 
is _advertising_ flow control as opposed to detecting partner has flow
control capability.
Auke, can you also check this as well?

cheers,
jamal 

Bug in e1000 + semantics of flow control WAS(Re: [e1000]: flow control on by default - good idea really?

[jamal]

I went back to this today.  I am typing this from a scribbled sticky
note in a big hurry -  but i still believe I took the correct notes.

It does seem there is no distinction between what ethernet advertises
for flow control capability vs what it ends up negotiating with its
partner i.e there is some ambiguity. I havent checked tg3, this on e1000
only.

On Fri, 2006-07-07 at 08:28 -0400, jamal wrote:
> On Thu, 2006-06-07 at 23:59 -0700, David Miller wrote:
> 
> > 
> > It's autonegotiated, check you kernel message logs when the link
> > came up, you'll see this:
> > 
> > tg3: eth0: Flow control is on for TX and on for RX.
> > 
> 
> yikes - yes, this would be it.
> 
> I  could be wrong and i will double check:
> I think when the e1000 says via ethtool "rx is on" - it means that it 
> is _advertising_ flow control as opposed to detecting partner has flow
> control capability.
> Auke, can you also check this as well?

Semantic #1

For example, configure:
ethtool -A eth0 rx off
ethtool -A eth0 tx on

debopolis:~# ethtool -a eth0
Pause parameters for eth0:
Autonegotiate:  on
RX:             off
TX:             on

The other side was set to do symmetric TX flow control only.

Now enforce autonegotiation:
ethtool -r eth0

ethtool -a eth0
Pause parameters for eth0:
Autonegotiate:  on
RX:             off
TX:             off

Ok, this is what i expected if this thing (output of ethtool) was
supposed to store state as opposed to configuration.
But if it is state that is stored, then what about that the values
before autonegotian - surely that state is invalid, no?

It would be nice (for debug/usability reasons) to be able to see what i
configured vs what i end up negotiating with the link partner. I think
this may be an ethtool issue, but it could also be a driver issue.

I send 1 Mpps to eth0 and see no flow control packets back. good. So it
does store state

#2: The other semantic

debopolis:~# ethtool -A eth0 rx on
debopolis:~# ethtool -A eth0 tx on

debopolis:~# ethtool -a eth0
Pause parameters for eth0:
Autonegotiate:  on
RX:             on
TX:             on

Other side was set to do symmetric TX flow control only.

Now enforce autonegotiation:
debopolis:~# ethtool -r eth0

lets see what we came up with:
debopolis:~# ethtool -a eth0
Pause parameters for eth0:
Autonegotiate:  on
RX:             on
TX:             on

Now that is contradictory to #1 semantic - I would have expected this TX
flow control on the e1000 to be off. 
Unless it is meant to store configuration info and not what you have
negotiated.
 
Trying sending traffic to the e1000 at about 1Mpps.
I observe that the e1000 is sending out about 800Kpps of flow control
packets back ;->

So which semantics are correct? I would claim #2 flow control behavior
to be a bug. I just dont have time to chase a fix - hopefully whoever
reads this from the e1000 crowd can fix it.

More importantly can we have two variables storing the two pieces on
information separately instead of the ambiguity of just one?


cheers,
jamal

Re: Bug in e1000 + semantics of flow control WAS(Re: [e1000]: flow control on by default - good idea really?

[jamal]

To the good folks at Intel:


Do you need more info on how to reproduce the issue below?
Note, there are 2 issues - one being a larger ethtool semantic issue and
the other being what i perceive to be a bug in the e1000.

cheers,
jamal

On Thu, 2006-20-07 at 16:15 -0400, jamal wrote:
> I went back to this today.  I am typing this from a scribbled sticky
> note in a big hurry -  but i still believe I took the correct notes.
> 
> It does seem there is no distinction between what ethernet advertises
> for flow control capability vs what it ends up negotiating with its
> partner i.e there is some ambiguity. I havent checked tg3, this on e1000
> only.
> 
> On Fri, 2006-07-07 at 08:28 -0400, jamal wrote:
> > On Thu, 2006-06-07 at 23:59 -0700, David Miller wrote:
> > 
> > > 
> > > It's autonegotiated, check you kernel message logs when the link
> > > came up, you'll see this:
> > > 
> > > tg3: eth0: Flow control is on for TX and on for RX.
> > > 
> > 
> > yikes - yes, this would be it.
> > 
> > I  could be wrong and i will double check:
> > I think when the e1000 says via ethtool "rx is on" - it means that it 
> > is _advertising_ flow control as opposed to detecting partner has flow
> > control capability.
> > Auke, can you also check this as well?
> 
> Semantic #1
> 
> For example, configure:
> ethtool -A eth0 rx off
> ethtool -A eth0 tx on
> 
> debopolis:~# ethtool -a eth0
> Pause parameters for eth0:
> Autonegotiate:  on
> RX:             off
> TX:             on
> 
> The other side was set to do symmetric TX flow control only.
> 
> Now enforce autonegotiation:
> ethtool -r eth0
> 
> ethtool -a eth0
> Pause parameters for eth0:
> Autonegotiate:  on
> RX:             off
> TX:             off
> 
> Ok, this is what i expected if this thing (output of ethtool) was
> supposed to store state as opposed to configuration.
> But if it is state that is stored, then what about that the values
> before autonegotian - surely that state is invalid, no?
> 
> It would be nice (for debug/usability reasons) to be able to see what i
> configured vs what i end up negotiating with the link partner. I think
> this may be an ethtool issue, but it could also be a driver issue.
> 
> I send 1 Mpps to eth0 and see no flow control packets back. good. So it
> does store state
> 
> #2: The other semantic
> 
> debopolis:~# ethtool -A eth0 rx on
> debopolis:~# ethtool -A eth0 tx on
> 
> debopolis:~# ethtool -a eth0
> Pause parameters for eth0:
> Autonegotiate:  on
> RX:             on
> TX:             on
> 
> Other side was set to do symmetric TX flow control only.
> 
> Now enforce autonegotiation:
> debopolis:~# ethtool -r eth0
> 
> lets see what we came up with:
> debopolis:~# ethtool -a eth0
> Pause parameters for eth0:
> Autonegotiate:  on
> RX:             on
> TX:             on
> 
> Now that is contradictory to #1 semantic - I would have expected this TX
> flow control on the e1000 to be off. 
> Unless it is meant to store configuration info and not what you have
> negotiated.
>  
> Trying sending traffic to the e1000 at about 1Mpps.
> I observe that the e1000 is sending out about 800Kpps of flow control
> packets back ;->
> 
> So which semantics are correct? I would claim #2 flow control behavior
> to be a bug. I just dont have time to chase a fix - hopefully whoever
> reads this from the e1000 crowd can fix it.
> 
> More importantly can we have two variables storing the two pieces on
> information separately instead of the ambiguity of just one?
> 
> 
> cheers,
> jamal

Re: [e1000]: flow control on by default - good idea really?

[Auke Kok]

jamal wrote:
> On Thu, 2006-06-07 at 23:59 -0700, David Miller wrote:
> 
>> It's autonegotiated, check you kernel message logs when the link
>> came up, you'll see this:
>>
>> tg3: eth0: Flow control is on for TX and on for RX.
>>
> 
> yikes - yes, this would be it.
> 
> I  could be wrong and i will double check:
> I think when the e1000 says via ethtool "rx is on" - it means that it 
> is _advertising_ flow control as opposed to detecting partner has flow
> control capability.
> Auke, can you also check this as well?

Just found this in my todo box - a bit late :(

yes, that appears to be the correct interpretation: we never read back the detected FC 
state from the hardware.

Auke

Re: [e1000]: flow control on by default - good idea really?

[jamal]

On Mon, 2006-16-10 at 11:55 -0700, Auke Kok wrote:
> jamal wrote:

> > I think when the e1000 says via ethtool "rx is on" - it means that it 
> > is _advertising_ flow control as opposed to detecting partner has flow
> > control capability.
> > Auke, can you also check this as well?
> 
> Just found this in my todo box - a bit late :(
> 
> yes, that appears to be the correct interpretation: we never read back the 
> detected FC  state from the hardware.
> 

It sounds to me that ethttool needs to have this semantic fix.
IOW, ethttool doesnt differentiate the two items:
a) advertised parameters.
b) link partner negotiated parameters.

and instead #a becomes #b after negotiation.

methinks this needs fixing. Dave? Jeff?

cheers,
jamal

Re: [e1000]: flow control on by default - good idea really?

[Auke Kok]

jamal wrote:
> On Mon, 2006-16-10 at 11:55 -0700, Auke Kok wrote:
>> jamal wrote:
> 
>>> I think when the e1000 says via ethtool "rx is on" - it means that it 
>>> is _advertising_ flow control as opposed to detecting partner has flow
>>> control capability.
>>> Auke, can you also check this as well?
>> Just found this in my todo box - a bit late :(
>>
>> yes, that appears to be the correct interpretation: we never read back the 
>> detected FC  state from the hardware.
> 
> It sounds to me that ethttool needs to have this semantic fix.
> IOW, ethttool doesnt differentiate the two items:
> a) advertised parameters.
> b) link partner negotiated parameters.
> 
> and instead #a becomes #b after negotiation.

that's odd and confusing, we should keep them separate.

> methinks this needs fixing. Dave? Jeff?

Alternatively, we can report in the driver at link up time what FC settings were 
succesfull using printk. The output of `eththool eth0` would be the best place to put 
this output as well since that displays "status" and not "settings" which are displayed 
with '-a|-A' options of ethtool. Lets keep `-a|-A` the same and work on showing 
negotiated FC parameters in `ethtool ethX` instead.

Auke

Re: [e1000]: flow control on by default - good idea really?

[Stephen Hemminger]

On Tue, 17 Oct 2006 09:05:31 -0400
jamal <hadi@cyberus.ca> wrote:

> On Mon, 2006-16-10 at 11:55 -0700, Auke Kok wrote:
> > jamal wrote:
> 
> > > I think when the e1000 says via ethtool "rx is on" - it means that it 
> > > is _advertising_ flow control as opposed to detecting partner has flow
> > > control capability.
> > > Auke, can you also check this as well?
> > 
> > Just found this in my todo box - a bit late :(
> > 
> > yes, that appears to be the correct interpretation: we never read back the 
> > detected FC  state from the hardware.
> > 
> 
> It sounds to me that ethttool needs to have this semantic fix.
> IOW, ethttool doesnt differentiate the two items:
> a) advertised parameters.
> b) link partner negotiated parameters.
> 
> and instead #a becomes #b after negotiation.
> 
> methinks this needs fixing. Dave? Jeff?
> 
> cheers,
> jamal

If #a becomes #b after negotiation, that is a bug.
Otherwise, if imagine that connection was moved from gigabit with flow control
to 10mbit with no flow control, then back to gigabit with flow control.
You would want the advertised parameters to stay the same and not change.

I think driver need to keep track of advertising and status bits
in separate fields.

Another pet bug, is that drivers need to accept and remember ethtool
settings when offline before device is brought up. The existing if_mii 
interface doesn't do that, and so drivers that use it are broken.

-- 
Stephen Hemminger <shemminger@osdl.org>

Re: [e1000]: flow control on by default - good idea really?

[Auke Kok]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

jamal wrote:
> On Mon, 2006-16-10 at 11:55 -0700, Auke Kok wrote:
>> jamal wrote:
> 
>>> I think when the e1000 says via ethtool "rx is on" - it means that it 
>>> is _advertising_ flow control as opposed to detecting partner has flow
>>> control capability.
>>> Auke, can you also check this as well?
 >>
>> Just found this in my todo box - a bit late :(
>>
>> yes, that appears to be the correct interpretation: we never read back the 
>> detected FC  state from the hardware.

For now, we should really report the FC status in e1000 at link up time. Jamal: this 
should help you out for now, I'll send something like this upstream later on.

Cheers,

Auke




[-- Attachment #2: e1000_display_fc_mode_on_link_up.patch --]
[-- Type: text/x-patch, Size: 1422 bytes --]


e1000: Display Flow Control setting used after link negotiation

No part of e1000 was reporting which Fc settings were effectively
negotiated with the link partner so that it would be impossible
to know if FC was actually used at all.

Signed-off-by: Auke Kok <auke-jan.h.kok@intel.com>

diff --git a/drivers/net/e1000/e1000_main.c b/drivers/net/e1000/e1000_main.c
index ce0d35f..ff7f396 100644
--- a/drivers/net/e1000/e1000_main.c
+++ b/drivers/net/e1000/e1000_main.c
@@ -2426,15 +2426,22 @@ e1000_watchdog(unsigned long data)
 
 	if (link) {
 		if (!netif_carrier_ok(netdev)) {
+			uint32_t ctrl;
 			boolean_t txb2b = 1;
 			e1000_get_speed_and_duplex(&adapter->hw,
 			                           &adapter->link_speed,
 			                           &adapter->link_duplex);
 
-			DPRINTK(LINK, INFO, "NIC Link is Up %d Mbps %s\n",
+			ctrl = E1000_READ_REG(&adapter->hw, CTRL);
+			DPRINTK(LINK, INFO, "NIC Link is Up %d Mbps %s, "
+			       "Flow Control: %s\n",
 			       adapter->link_speed,
 			       adapter->link_duplex == FULL_DUPLEX ?
-			       "Full Duplex" : "Half Duplex");
+			       "Full Duplex" : "Half Duplex",
+			       ((ctrl & E1000_CTRL_TFCE) && (ctrl &
+			       E1000_CTRL_RFCE)) ? "RX/TX" : ((ctrl &
+			       E1000_CTRL_RFCE) ? "RX" : ((ctrl &
+			       E1000_CTRL_TFCE) ? "TX" : "None" )));
 
 			/* tweak tx_queue_len according to speed/duplex
 			 * and adjust the timeout factor */

Re: [e1000]: flow control on by default - good idea really?

[jamal]

On Tue, 2006-17-10 at 14:02 -0700, Auke Kok wrote:

> For now, we should really report the FC status in e1000 at link up time. Jamal: this 
> should help you out for now, I'll send something like this upstream later on.
> 

Thanks - this puts you at par with the tg3 at least.


On Tue, 2006-17-10 at 14:46 -0700, David Miller wrote: 
> From: jamal <hadi@cyberus.ca>
> Date: Tue, 17 Oct 2006 09:05:31 -0400
> 
> > It sounds to me that ethttool needs to have this semantic fix.
> > IOW, ethttool doesnt differentiate the two items:
> > a) advertised parameters.
> > b) link partner negotiated parameters.
> > 
> > and instead #a becomes #b after negotiation.
> > 
> >
> > methinks this needs fixing. Dave? Jeff?
> 
> The way I understand it the ethernet autonegotiation mechanisms don't
> really give you a way to seperate these two things.
> 
> Either you negotiate the link and flow control settings, or nothing.

True - but I was thinking more of the state stored in the driver
either by ethtool or some other part of the driver.

If i remember correctly, Donald Beckers old mii tool was able to 
display
"here's what you have configured the driver for link and flow control and 
these are what i advertise to link peers"
and
"here's what current negotiated link and flow control parameters with 
link peer"

That distinction doesnt exist with ethtool. Or i am missing something.

cheers,
jamal

Re: [e1000]: flow control on by default - good idea really?

[Auke Kok]

jamal wrote:
> On Tue, 2006-17-10 at 14:02 -0700, Auke Kok wrote:
> 
>> For now, we should really report the FC status in e1000 at link up time. Jamal: this 
>> should help you out for now, I'll send something like this upstream later on.
>>
> 
> Thanks - this puts you at par with the tg3 at least.
> 
> 
> On Tue, 2006-17-10 at 14:46 -0700, David Miller wrote: 
>> From: jamal <hadi@cyberus.ca>
>> Date: Tue, 17 Oct 2006 09:05:31 -0400
>>
>>> It sounds to me that ethttool needs to have this semantic fix.
>>> IOW, ethttool doesnt differentiate the two items:
>>> a) advertised parameters.
>>> b) link partner negotiated parameters.
>>>
>>> and instead #a becomes #b after negotiation.
>>>
>>>
>>> methinks this needs fixing. Dave? Jeff?
>> The way I understand it the ethernet autonegotiation mechanisms don't
>> really give you a way to seperate these two things.
>>
>> Either you negotiate the link and flow control settings, or nothing.
> 
> True - but I was thinking more of the state stored in the driver
> either by ethtool or some other part of the driver.
> 
> If i remember correctly, Donald Beckers old mii tool was able to 
> display
> "here's what you have configured the driver for link and flow control and 
> these are what i advertise to link peers"
> and
> "here's what current negotiated link and flow control parameters with 
> link peer"
> 
> That distinction doesnt exist with ethtool. Or i am missing something.

nope, there's not even an ethtool cmd to query for that data AFAICS

Auke

Re: [e1000]: flow control on by default - good idea really?

[David Miller]

From: jamal <hadi@cyberus.ca>
Date: Tue, 17 Oct 2006 09:05:31 -0400

> It sounds to me that ethttool needs to have this semantic fix.
> IOW, ethttool doesnt differentiate the two items:
> a) advertised parameters.
> b) link partner negotiated parameters.
> 
> and instead #a becomes #b after negotiation.
> 
> methinks this needs fixing. Dave? Jeff?

The way I understand it the ethernet autonegotiation mechanisms don't
really give you a way to seperate these two things.

Either you negotiate the link and flow control settings, or nothing.

[e1000]: flow control on by default - good idea really?

[Robert Olsson]

jamal writes:

 > The default setup on the e1000 has rx flow control turned on.
 > I was sending at wire rate gige from the device - which is about
 > 1.48Mpps. The e1000 was in turn sending me flow control packets
 > as per default/expected behavior. Unfortunately, it was sending
 > a very large amount of packets. At one point i was seeing upto
 > 1Mpps and on average, the flow control packets were consuming
 > 60-70% of the bandwidth. Even when i fixed this behavior to act
 > properly, allowing flow control on consumed up to 15% of the bandwidth. 
 > Clearly, this is a bad thing. Yes, the device in the first instance was
 > at fault. But i have argued in the past that NAPI does just fine without
 > flow control being turned on, so even chewing 5% of bandwidth on flow
 > control is a bad thing..

 Interesting numbers. But one can argue that if there were no control packets 
 there would be data packets sent and dropped instead. But I'll agree with you
 as the flow control seems hard to monitor and tune and takes debugging
 down to link and chiplevels. Much easier and robust just to read the 
 dropped packet counter on the box that couldn't handle the load rather than 
 following secret flow control packets somewhere else and try to figure out 
 whats going on. 
  
 > As a compromise, can we declare flow control as an advanced feature
 > and turn it off by default? People who feel it is valuable and know
 > what they are doing can turn it off.

 At least for NAPI-scheme drivers.

 > If you want more details just shoot.

 Cheers.
						--ro

Re: [e1000]: flow control on by default - good idea really?

[David Miller]

From: jamal <hadi@cyberus.ca>
Date: Tue, 04 Jul 2006 13:11:56 -0400

> Clearly, this is a bad thing. Yes, the device in the first instance was
> at fault. But i have argued in the past that NAPI does just fine without
> flow control being turned on, so even chewing 5% of bandwidth on flow
> control is a bad thing..

I think it should be on by default.

If the machine is really busy, proper flow control keeps the
sender from overflowing the RX ring of the receiver.

Yes, with NAPI this happens less, but it does still happen.  And
I know that this is probably one of the least tested code paths
in every single driver, RX ring emptying out, and in fact the Tigon2
and the earliest Tigon3 chips would lock up on you if the RX ring
was emptied completely requiring a full chip reset when the watchdog
timer fired.