Re: big picture UDP/IP performance question re 2.6.18 -> 2.6.32

[Eric Dumazet <eric.dumazet@gmail.com>]

Le samedi 01 octobre 2011 à 01:30 -0400, starlight@binnacle.cx a écrit :
> Hello,
> 
> I'm hoping someone can provide a brief big-picture
> perspective on the dramatic UDP/IP multicast
> receive performance reduction from 2.6.18 to
> 2.6.32 that I just benchmarked.
> 
> Have helped out in the past, mainly by identifying
> a bug in hugepage handling and providing a solid
> testcase that helped in quickly identifying and
> correcting the problem.
> 
> Have a very-high-volume UDP multicast receiver
> application.  Just finished benchmarking latest RH
> variant of 2.6.18 against latest RH 2.6.32 and
> vanilla 2.6.32.27 on the same 12 core Opteron
> 6174 processor system, one CPU.
> 
> Application reads on 250 sockets with large socket
> buffer maximums.  Zero data loss.  Four Intel
> 'e1000e' 82571 gigabit NICs, or two Intel 'igb'
> 82571 gigabit NICs or two Intel 82599 10 gigabit
> NICs.  Results similar on all.
> 
> With 2.6.18, system CPU is reported in
> /proc/<pid>/stat as 25% of total.  With 2.6.32,
> system consumption is 45% with the same exact data
> playback test.  Jiffy count for user CPU is same
> for both kernels, but .32 system CPU is double
> .18 system CPU.
> 
> Overall maximum performance capacity is reduced in
> proportion to the increased system overhead.
> 
> ------
> 
> My question is why is the performance significantly
> worse in the more recent kernels?  Apparently
> network performance is worse for TCP by about the
> same amount--double the system overhead for the
> same amount of work.
> 
> http://www.phoronix.com/scan.php?page=article&item=linux_2612_2637&num=6
> 
> Is there any chance that network performance will
> improve in future kernels?  Or is the situation
> a permanent trade-off for security, reliability
> or scalability reasons?
> 

CC netdev

Since you have 2.6.32, you could use perf tool and provide us a
performance report.

In my experience, I have the exact opposite : performance greatly
improved in recent kernels. Unless you compile your kernel to include
new features that might reduce performance (namespaces, cgroup, ...)

It can vary a lot depending on many parameters, like cpu affinities,
device parameters (coalescing, interrupt mitigation...).

You cant expect switching from 2.6.18 to 2.6.32 and have exactly same
system behavior.

If your app is performance sensitive, you'll have to make some analysis
to find out what needs to be tuned.

One known problem of old kernels and UDP is that they was no memory
accouting, so an application could easily consume all kernel memory and
crash the machine.

So in 2.6.25, Hideo Aoki added memory limiting to UDP, slowing down a
lot of UDP operations because of added socket locking, both on transmit
and receive path.

If your application is multithreaded and use a single socket, you can
hit lock contention since 2.6.25.

Step by step, we tried to remove part of the scalability problems
introduced in 2.6.25

In 2.6.35, we speedup receive path a bit (avoiding backlog processing)

In 2.6.39, transmit path became lockless again, thanks to Herbert Xu.

I advise you to try a recent kernel if you need UDP performance, 2.6.32
is quite old

Multicast is quite a stress for process scheduler, so we experimented a
way to group all wakeups at the end of softirq handler.

Work is in progress in this area : Peter Zijlstra named this "delayed
wakeup". A further idea would be to be able to delegate the wakeups to
another cpu, since I suspect you have one CPU busy in softirq
processing, and other cpus are ile.