A question about MTUs and TCP stack

A question about MTUs and TCP stack

[Pawan Singh]

Hi

I am posting this question to "netdev" mailing list because I could no longer find "linux-net" mailing list.

I find that the Linux TCP stack consumes huge amount of CPU if the MTU of an interface is set to 2400 and it is receiving 1000 byte Ethernet packets. On the other hand, if the MTU is set to 1500, the CPU consumption is reduced drastically. Increased CPU usage causes network throughput to drop considerably (from 800-900 Mbps to 200 Mbps). My kernel version is fedora core 6 and we are using 1 Gig NICs (Intel 82546GB and Broadcom NetXtreme BCM5721):

Linux he7700-tg 2.6.22.14-72.fc6 #1 SMP Wed Nov 21 14:10:25 EST 2007 x86_64 x86_64 x86_64 GNU/Linux

I do not know the TCP buffer management internals and how they are affected by MTU. Is there some FAQ/information online or do I have to open up the source code and try to identify the source of the problem. I guess I can also try newer versions of the kernel and see if the issue has been resolved.

-Pawan

Re: A question about MTUs and TCP stack

[Rick Jones]

On 08/16/2011 11:33 AM, Pawan Singh wrote:
> Hi
>
> I am posting this question to "netdev" mailing list because I could
> no longer find "linux-net" mailing list.
>
> I find that the Linux TCP stack consumes huge amount of CPU if the
> MTU of an interface is set to 2400 and it is receiving 1000 byte
> Ethernet packets. On the other hand, if the MTU is set to 1500, the
> CPU consumption is reduced drastically. Increased CPU usage causes
> network throughput to drop considerably (from 800-900 Mbps to 200
> Mbps). My kernel version is fedora core 6 and we are using 1 Gig NICs
> (Intel 82546GB and Broadcom NetXtreme BCM5721):
>
> Linux he7700-tg 2.6.22.14-72.fc6 #1 SMP Wed Nov 21 14:10:25 EST 2007
> x86_64 x86_64 x86_64 GNU/Linux

Based on my experience, you are best off using much more contemporary 
kernels when bringing things to the attention of the netdev list.  The 
list is much more concerned with current bits than those from years past.

Issues with bits from years past are probably best addressed towards the 
distro(s) using them, if any.

Looking into getting a oprofile profile of the behaviour, particularly 
if it still exists in a contemporary kernel would probably be a good idea.

Also good ideas would be checking the netstat statistics before and 
after a test run, and looking at packet traces.  You want to look for 
things like stats on collapsing buffers and packets being dropped.

rick jones

>
> I do not know the TCP buffer management internals and how they are
> affected by MTU. Is there some FAQ/information online or do I have to
> open up the source code and try to identify the source of the
> problem. I guess I can also try newer versions of the kernel and see
> if the issue has been resolved.
>
> -Pawan

Re: A question about MTUs and TCP stack

[Eric Dumazet]

Le mardi 16 août 2011 à 11:33 -0700, Pawan Singh a écrit :
> Hi
> 
> I am posting this question to "netdev" mailing list because I could no
> longer find "linux-net" mailing list.
> 
> I find that the Linux TCP stack consumes huge amount of CPU if the MTU
> of an interface is set to 2400 and it is receiving 1000 byte Ethernet
> packets. On the other hand, if the MTU is set to 1500, the CPU
> consumption is reduced drastically. Increased CPU usage causes network
> throughput to drop considerably (from 800-900 Mbps to 200 Mbps). My
> kernel version is fedora core 6 and we are using 1 Gig NICs (Intel
> 82546GB and Broadcom NetXtreme BCM5721):
> 
> Linux he7700-tg 2.6.22.14-72.fc6 #1 SMP Wed Nov 21 14:10:25 EST 2007
> x86_64 x86_64 x86_64 GNU/Linux
> 
> I do not know the TCP buffer management internals and how they are
> affected by MTU. Is there some FAQ/information online or do I have to
> open up the source code and try to identify the source of the problem.
> I guess I can also try newer versions of the kernel and see if the
> issue has been resolved.
> 

I suspect you have a CPU problem on the receiver side, not on transmit
one ?

MTU has a direct impact on skb 'truesize'. As the linux tcp stack takes
care of not using too much kernel ram for the sole use of one socket,
you might hit a per socket limit faster with big MTU (and not filled RX
buffers)

When this limit is hit, tcp stack tries to 'collapse' consecutive skbs
to remove the unused parts of skbs. This is a very expensive process.

With a recent kernel, you could use 'perf top' tool and easily spot the
hot paths in kernel.