Re: [PATCH net-next-2.6] inetpeer: lower false sharing effect

[Eric Dumazet <eric.dumazet@gmail.com>]

Le jeudi 09 juin 2011 à 17:03 -0700, Tim Chen a écrit :
> On Thu, 2011-06-09 at 08:26 +0200, Eric Dumazet wrote:
> > Profiles show false sharing in addr_compare() because refcnt/dtime
> > changes dirty the first inet_peer cache line, where are lying the keys
> > used at lookup time. If many cpus are calling inet_getpeer() and
> > inet_putpeer(), or need frag ids, addr_compare() is in 2nd position in
> > "perf top".
> > 
> 
> I've applied both inetpeer patches.  I also no longer have inet_getpeer
> and inet_putpeer and addr_compare in my profile.  Instead, neighbor
> lookup is now dominant.  See profile below.
> 
> When I retest with original 3.0-rc2 kernel, inet_putpeer no longer shows
> up, wonder if dst->peer was not set for some reason. 
> 
> Tim
> 
> -     27.06%     memcached  [kernel.kallsyms]             [k] atomic_add_unless.clone.34
>    - atomic_add_unless.clone.34
>       - 99.97% neigh_lookup
>            __neigh_lookup_errno.clone.17
>            arp_bind_neighbour
>            rt_intern_hash
>            __ip_route_output_key
>            ip_route_output_flow
>            udp_sendmsg
>            inet_sendmsg
>            __sock_sendmsg
>            sock_sendmsg
>            __sys_sendmsg
>            sys_sendmsg
>            system_call_fastpath
>            __sendmsg
> -     13.33%     memcached  [kernel.kallsyms]             [k] atomic_dec_and_test
>    - atomic_dec_and_test
>       - 99.89% dst_destroy
>          - dst_release
>             - 98.12% skb_dst_drop.clone.55
>                  dev_hard_start_xmit
>                + sch_direct_xmit
>             + 1.88% skb_release_head_state
> -      3.26%     memcached  [kernel.kallsyms]             [k] do_raw_spin_lock
>    - do_raw_spin_lock
>       - 92.24% _raw_spin_lock
>          + 41.39% sch_direct_xmit
> 
> 

Thanks Tim

I have some questions for further optimizations.

1) How many different destinations are used in your stress load ?
2) Could you provide a distribution of the size of packet lengthes ?
   Or maybe the average length would be OK