Re: Re: limited network bandwidth with 3.2.x kernels

[Eric Dumazet <eric.dumazet@gmail.com>]

Le mardi 21 février 2012 à 06:21 +0200, alekcejk@googlemail.com a 
> I tested last released 3.2.7 kernel (2.6.42.7-1.fc15.i686).
> Speed problems in 3.2.7 the same as in 3.2.6.
> 

I am not sure what you expected from 3.2.7, given we still are
discussing of issues.

> I also have Fedora Rawhide installed in virtual machine (e1000 driver used there)
> running on host with 3.1.10 kernel.
> And I noticed that after one of 3.3 rc's (or even rc's of 3.2) kernel updates download speed of
> updates from dl.fedoraproject.org in VM became extremely slow - about 160 kilobytes/sec
> instead of 1,2 megabytes/sec as it was before.
> But recently (somewhere between 3.3 rc3 and rc4) speed became again
> as it should 1,2 MB/s.
> 
> Don't know is this problem related with speed in 3.2.x kernels
> and it is hard to debug this now.
> 

We have a pretty clear idea of what happens, we have to decide if we 
must change kernel defaults now or not.

On prior kernels, memory accounting was wrong, and a socket could
consume far more memory than the limit given.

See the 3rd column in :

net.ipv4.tcp_wmem = 4096        16384   897664
net.ipv4.tcp_rmem = 4096        87380   897664

We now have a more correct 'truesize' accounting, and this means a
socket cannot lie anymore : Instead of consuming 1.200.000 bytes for receive
buffers, it really cannot consume more than 897664 bytes.
(unless yet another driver try to lie)

On your 32bit kernel, this means the tcp_rmem[2] (897664) really 
can be too small for your case with a 50 ms RTT, since memory/rtt is
limiting the bandwidth.

On a 64bit kernel, we hit the memory/rtt limit less often because
tcp_rmem[2] default value is 4127616, unless you have very small
available memory.

Maybe we should default tcp_rmem[2] to 4Mbytes, even on 32bit kernel.

Could you try :

sysctl net.ipv4.tcp_rmem="4096 87380 4127616"
or
echo "4096 87380 4127616" >/proc/sys/net/ipv4/tcp_rmem