[e1000]: flow control on by default - good idea really?

[jamal <hadi@cyberus.ca>]

CCing anybody who may have stakes on this. Ignore the email if this
doesnt interest you.
Ok, folks - i had deferred this discussion but it bit me in the ass. 
I just spend an hour debugging it (and in the process blew up a gbic i
borrowed, so my day aint going well since i actually have to pay for
this and cant really do the testing i was planning to;-<).
 
I have a device connected to a e1000 that was erroneously advertising
both tx/rx flow control but wasnt properly reacting to it. 
The default setup on the e1000 has rx flow control turned on.
I was sending at wire rate gige from the device - which is about
1.48Mpps. The e1000 was in turn sending me flow control packets
as per default/expected behavior. Unfortunately, it was sending
a very large amount of packets. At one point i was seeing upto
1Mpps and on average, the flow control packets were consuming
60-70% of the bandwidth. Even when i fixed this behavior to act
properly, allowing flow control on consumed up to 15% of the bandwidth. 
Clearly, this is a bad thing. Yes, the device in the first instance was
at fault. But i have argued in the past that NAPI does just fine without
flow control being turned on, so even chewing 5% of bandwidth on flow
control is a bad thing..

As a compromise, can we declare flow control as an advanced feature
and turn it off by default? People who feel it is valuable and know
what they are doing can turn it off.

If you want more details just shoot.

cheers,
jamal 

PS:- BTW, even turning off flow control on e1000 didnt give as good
performance as in the old days on this machine - but i dont want to go
into that discussion.