Re: SO_REUSEPORT - can it be done in kernel?

Eric Dumazet
2011-03-01T11:13:11Z

Le mardi 01 mars 2011 à 06:07 -0500, Thomas Graf a écrit :
> On Tue, Mar 01, 2011 at 11:33:22AM +0100, Eric Dumazet wrote:
> > > I retested with net-2.6 w/o Herbert's patch:
> > >
> > > named -n 1: 36.9kqps
> > > named: 16.2kqps
> >
> > Thats better ;)
> >
> > You could do "cat /proc/net/udp" to check if drops occur on port 53
> > socket (last column)
> >
> > But maybe your queryperf is limited to few queries in flight (default is
> > 20 per queryperf instance)
>
> I tried -q 10, 20, 30, 50, 100. Starting with 20 I see drops, at q=50
> queryperf reports 99% drops.
>
> I also tested again on the Intel machine that did ~650kqps using SO_REUSEPORT.
>
> net-2.6: 106.3kqps, 101.2kqps
> net-2.6 lockless udp: 251.7kqps, 250.4kqps
>
> I see drops in both test cases occur so I believe the rate supplied by the
> clients is sufficient.
>
> The difference is obvious when looking at top and mpstat:
>
> UDP lockless (250kqps):
>
> Cpu0 : 46.4%us, 28.8%sy, 0.0%ni, 24.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu1 : 2.0%us, 1.3%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 93.6%si, 0.0%st
> Cpu2 : 45.9%us, 28.2%sy, 0.0%ni, 25.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu3 : 50.0%us, 21.6%sy, 0.0%ni, 28.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu4 : 45.4%us, 27.8%sy, 0.0%ni, 26.5%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
> Cpu5 : 50.7%us, 23.2%sy, 0.0%ni, 26.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu6 : 45.2%us, 28.9%sy, 0.0%ni, 25.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu7 : 50.5%us, 22.0%sy, 0.0%ni, 27.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu8 : 45.3%us, 29.3%sy, 0.0%ni, 25.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu9 : 50.8%us, 20.8%sy, 0.0%ni, 28.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu10 : 46.1%us, 27.8%sy, 0.0%ni, 26.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu11 : 27.2%us, 11.3%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 58.1%si, 0.0%st

Its a bit strange two cpus spend time in softirq, unless you have two
queryperf sources, and a multiqueue NIC, or maybe you use two NICS ?

Mind use "perf top -C 1" and "perf top -C 11" to check what these cpus
do ?