e1000e: reset of tx_queue_len

e1000e: reset of tx_queue_len

[Franco Fichtner]

Hi all,

while working on a new server, I noticed that tx_queue_len is reset
to the default of 1000 while ADDRCONF(NETDEV_CHANGE).
This happens with 2.6.30.8, but I could not see any obvious
differences to the current state of e1000e, while trying to find the
problem in the code.

The interface has been configured including a customized qlen
via /etc/network/interfaces. The ports are not connected at startup
so obviously the Kernel reports

ADDRCONF(NETDEV_UP): eth2: link is not ready

while ifconfig properly displays the customized qlen.

When plugging the server tx_queue_len is reset to 1000.

Any help is appreciated. Thanks.


Franco

Re: e1000e: reset of tx_queue_len

[Brandeburg, Jesse]

On Tue, 2 Mar 2010, Franco Fichtner wrote:
> while working on a new server, I noticed that tx_queue_len is reset
> to the default of 1000 while ADDRCONF(NETDEV_CHANGE).
> This happens with 2.6.30.8, but I could not see any obvious
> differences to the current state of e1000e, while trying to find the
> problem in the code.

The driver tries to set the queuelen back to 1000 if the link changes 
speed, because the driver tries to shorten the txqueuelen to 100 to reduce 
latency when the link changes to 100/10 Mb/s

> The interface has been configured including a customized qlen
> via /etc/network/interfaces. The ports are not connected at startup
> so obviously the Kernel reports
> 
> ADDRCONF(NETDEV_UP): eth2: link is not ready

do your scripts try to configure the queuelen when link is not up?  Why 
are the ports not connected at startup?

> 
> while ifconfig properly displays the customized qlen.
> 
> When plugging the server tx_queue_len is reset to 1000.
> 
> Any help is appreciated. Thanks.

you can safely comment out the code in netdev.c on lines 3532 through 
3549, as well as any other lines of code in the driver that modify or use 
the variable netdev->tx_queue_len

http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.30.y.git;a=blob;f=drivers/net/e1000e/netdev.c;h=42055a580f7ccd6cc2ac59ea5d7d864157b57107;hb=HEAD#l3532

Its possible we should just remove this code from the in-kernel 
driver, as I don't think many other drivers mess with the tx_queue_len 
either.  We did introduce the code to fix some users complaints about the 
amount of work that can get queued up in the hardware with tx_queue_len = 
1000 when at 10/100 speeds.

Hope this helps!

Re: e1000e: reset of tx_queue_len

[Franco Fichtner]

Brandeburg, Jesse wrote:
> On Tue, 2 Mar 2010, Franco Fichtner wrote:
>   
>> while working on a new server, I noticed that tx_queue_len is reset
>> to the default of 1000 while ADDRCONF(NETDEV_CHANGE).
>> This happens with 2.6.30.8, but I could not see any obvious
>> differences to the current state of e1000e, while trying to find the
>> problem in the code.
>>     
>
> The driver tries to set the queuelen back to 1000 if the link changes 
> speed, because the driver tries to shorten the txqueuelen to 100 to reduce 
> latency when the link changes to 100/10 Mb/s
>   

Thanks, that explains it. Since we don't connect ports at startup, the 
plugging causes
a speed "change".

>> The interface has been configured including a customized qlen
>> via /etc/network/interfaces. The ports are not connected at startup
>> so obviously the Kernel reports
>>
>> ADDRCONF(NETDEV_UP): eth2: link is not ready
>>     
>
> do your scripts try to configure the queuelen when link is not up?  Why 
> are the ports not connected at startup?
>   

I have a modified up command that sets appropriate values of mtu and 
qlen via ip. The device is
brought up at startup automatically with the correct values. At least 
that's what ifconfig shows
even though the port is not yet connected.

There has been a lot of testing back and forth with single hosts and 
much larger quantities of traffic.
This got really messy when testing single host first (connected via 100 
Mb/s, so qlen to 100) and then
switching to 1000 Mb/s. qlen would still be 100, causing lots of packet 
drops. This took a few
days to track down, unfortunately.

> Hope this helps

Greatly appreciated, thanks again. :)


Franco

Re: e1000e: reset of tx_queue_len

[Franco Fichtner]

Hi Jesse,

> Its possible we should just remove this code from the in-kernel 
> driver, as I don't think many other drivers mess with the tx_queue_len 
> either.  We did introduce the code to fix some users complaints about the 
> amount of work that can get queued up in the hardware with tx_queue_len = 
> 1000 when at 10/100 speeds.
>   

I've tested the changes you suggested and now tx_queue_len
is no longer reset. I have also prepared a patch for net-next
for you to sign or ack (or nack) because I'd like to see this go,
also.


Thanks again,
Franco