Re: SO_REUSEPORT - can it be done in kernel?

[Eric Dumazet <eric.dumazet@gmail.com>]

Le lundi 28 février 2011 à 09:13 -0500, Thomas Graf a écrit :
> On Mon, Feb 28, 2011 at 07:36:59PM +0800, Herbert Xu wrote:
> > But please do test them heavily, especially if you have an AMD
> > NUMA machine as that's where scalability problems really show
> > up.  Intel tends to be a lot more forgiving.  My last AMD machine
> > blew up years ago :)
> 
> This is just a preliminary test result and not 100% reliable
> because half through the testing the machine reported memory
> issues and disabled a DIMM before booting the tested kernels.
> 
> Nevertheless, bind 9.7.3:
> 
> 2.6.38-rc5+: 62kqps
> 2.6.38-rc5+ w/ Herbert's patch: 442kqps
> 
> This is on a 2 NUMA Intel Xeon X5560 @ 2.80GHz with 16 cores
> 
> Again, this number is not 100% reliably but clearly shows that
> the concept of the patch is working very well.
> 
> Will test Herbert's patch on the machine that did 650kqps with
> SO_REUSEPORT and also on some AMD machines.
> --

I suspect your queryperf input file hits many zones ?

With a single zone, my machine is able to give 250kps : most of the time
is consumed in bind code, dealing with rwlocks and false sharing
things...

(bind-9.7.2-P3)
Using two remote machines to perform queries, on bnx2x adapter, RSS
enabled : two cpus receive UDP frames for the same socket, so we also
hit false sharing in kernel receive path.


---------------------------------------------------------------------------------------------------------------------------------
   PerfTop:  558863 irqs/sec  kernel:40.8%  exact:  0.0% [1000Hz cpu-clock-msecs],  (all, 16 CPUs)
---------------------------------------------------------------------------------------------------------------------------------

             samples  pcnt function                      DSO
             _______ _____ _____________________________ ______________________________________

           137175.00 12.4% acpi_idle_enter_bm            [kernel.kallsyms]                     
            63784.00  5.8% _raw_spin_unlock_irqrestore   [kernel.kallsyms]                     
            54140.00  4.9% isc_rwlock_lock               /opt/src/bind-9.7.2-P3/bin/named/named
            32682.00  2.9% isc_rwlock_unlock             /opt/src/bind-9.7.2-P3/bin/named/named
            21823.00  2.0% dns_rbt_findnode              /opt/src/bind-9.7.2-P3/bin/named/named
            20306.00  1.8% __ticket_spin_lock            [kernel.kallsyms]                     
            16881.00  1.5% finish_task_switch            [kernel.kallsyms]                     
            15335.00  1.4% zone_find                     /opt/src/bind-9.7.2-P3/bin/named/named
            14082.00  1.3% decrement_reference           /opt/src/bind-9.7.2-P3/bin/named/named
            14064.00  1.3% __pthread_mutex_lock_internal /lib/tls/libpthread-2.3.4.so          
            13519.00  1.2% isc_stats_increment           /opt/src/bind-9.7.2-P3/bin/named/named
            13027.00  1.2% __GI_memcpy                   /lib/tls/libc-2.3.4.so                
            12516.00  1.1% dns_name_concatenate          /opt/src/bind-9.7.2-P3/bin/named/named
            12499.00  1.1% currentversion                /opt/src/bind-9.7.2-P3/bin/named/named
            11412.00  1.0% dns_name_fullcompare          /opt/src/bind-9.7.2-P3/bin/named/named
            10814.00  1.0% new_reference.clone.6         /opt/src/bind-9.7.2-P3/bin/named/named
            10580.00  1.0% attach                        /opt/src/bind-9.7.2-P3/bin/named/named
             9805.00  0.9% zone_zonecut_callback         /opt/src/bind-9.7.2-P3/bin/named/named