Re: SO_REUSEPORT - can it be done in kernel?

Eric Dumazet
2011-02-28T16:23:05Z

Le lundi 28 février 2011 à 09:13 -0500, Thomas Graf a écrit :
> On Mon, Feb 28, 2011 at 07:36:59PM +0800, Herbert Xu wrote:
> > But please do test them heavily, especially if you have an AMD
> > NUMA machine as that's where scalability problems really show
> > up. Intel tends to be a lot more forgiving. My last AMD machine
> > blew up years ago :)
>
> This is just a preliminary test result and not 100% reliable
> because half through the testing the machine reported memory
> issues and disabled a DIMM before booting the tested kernels.
>
> Nevertheless, bind 9.7.3:
>
> 2.6.38-rc5+: 62kqps
> 2.6.38-rc5+ w/ Herbert's patch: 442kqps
>
> This is on a 2 NUMA Intel Xeon X5560 @ 2.80GHz with 16 cores
>
> Again, this number is not 100% reliably but clearly shows that
> the concept of the patch is working very well.
>
> Will test Herbert's patch on the machine that did 650kqps with
> SO_REUSEPORT and also on some AMD machines.
> --

I suspect your queryperf input file hits many zones ?

With a single zone, my machine is able to give 250kps : most of the time
is consumed in bind code, dealing with rwlocks and false sharing
things...

(bind-9.7.2-P3)
Using two remote machines to perform queries, on bnx2x adapter, RSS
enabled : two cpus receive UDP frames for the same socket, so we also
hit false sharing in kernel receive path.

---------------------------------------------------------------------------------------------------------------------------------
PerfTop: 558863 irqs/sec kernel:40.8% exact: 0.0% [1000Hz cpu-clock-msecs], (all, 16 CPUs)
---------------------------------------------------------------------------------------------------------------------------------

samples pcnt function DSO
_______ _____ _____________________________ ______________________________________

137175.00 12.4% acpi_idle_enter_bm [kernel.kallsyms]
63784.00 5.8% _raw_spin_unlock_irqrestore [kernel.kallsyms]
54140.00 4.9% isc_rwlock_lock /opt/src/bind-9.7.2-P3/bin/named/named
32682.00 2.9% isc_rwlock_unlock /opt/src/bind-9.7.2-P3/bin/named/named
21823.00 2.0% dns_rbt_findnode /opt/src/bind-9.7.2-P3/bin/named/named
20306.00 1.8% __ticket_spin_lock [kernel.kallsyms]
16881.00 1.5% finish_task_switch [kernel.kallsyms]
15335.00 1.4% zone_find /opt/src/bind-9.7.2-P3/bin/named/named
14082.00 1.3% decrement_reference /opt/src/bind-9.7.2-P3/bin/named/named
14064.00 1.3% __pthread_mutex_lock_internal /lib/tls/libpthread-2.3.4.so
13519.00 1.2% isc_stats_increment /opt/src/bind-9.7.2-P3/bin/named/named
13027.00 1.2% __GI_memcpy /lib/tls/libc-2.3.4.so
12516.00 1.1% dns_name_concatenate /opt/src/bind-9.7.2-P3/bin/named/named
12499.00 1.1% currentversion /opt/src/bind-9.7.2-P3/bin/named/named
11412.00 1.0% dns_name_fullcompare /opt/src/bind-9.7.2-P3/bin/named/named
10814.00 1.0% new_reference.clone.6 /opt/src/bind-9.7.2-P3/bin/named/named
10580.00 1.0% attach /opt/src/bind-9.7.2-P3/bin/named/named
9805.00 0.9% zone_zonecut_callback /opt/src/bind-9.7.2-P3/bin/named/named