simple mod_timer patch

simple mod_timer patch

[Steve Modica]

I pulled this back from the 2.6 kernel to reduce some serious contention on the 
timerlist_lock when I had 8 gigabit cards runnings.


diff -u -r1.23 -r1.24
--- linux/linux/kernel/timer.c       2003/08/11 20:16:19     1.23
+++ linux/linux/kernel/timer.c       2003/10/01 21:09:20     1.24
@@ -207,6 +207,14 @@
         int ret;
         unsigned long flags;

+        /*
+        * This is a common optimization triggered by the
+        * networking code - if the timer is re-modified
+        * to be the same thing then just return:
+        */
+        if (timer->expires == expires && timer_pending(timer))
+               return 1;
+
         spin_lock_irqsave(&timerlist_lock, flags);
         timer->expires = expires;
         ret = detach_timer(timer);



-- 
Steve Modica
work: 651-683-3224
MTS-Technical Lead
"Give a man a fish, and he will eat for a day, hit him with a fish and
he leaves you alone" - me

Re: simple mod_timer patch

[Steve Modica]

Forgot to mention.. please leave me on the CC list. Thanks!
Steve



Steve Modica wrote:
> I pulled this back from the 2.6 kernel to reduce some serious contention 
> on the timerlist_lock when I had 8 gigabit cards runnings.
> 
> 
> diff -u -r1.23 -r1.24
> --- linux/linux/kernel/timer.c       2003/08/11 20:16:19     1.23
> +++ linux/linux/kernel/timer.c       2003/10/01 21:09:20     1.24
> @@ -207,6 +207,14 @@
>         int ret;
>         unsigned long flags;
> 
> +        /*
> +        * This is a common optimization triggered by the
> +        * networking code - if the timer is re-modified
> +        * to be the same thing then just return:
> +        */
> +        if (timer->expires == expires && timer_pending(timer))
> +               return 1;
> +
>         spin_lock_irqsave(&timerlist_lock, flags);
>         timer->expires = expires;
>         ret = detach_timer(timer);
> 
> 
> 


-- 
Steve Modica
work: 651-683-3224
MTS-Technical Lead
"Give a man a fish, and he will eat for a day, hit him with a fish and
he leaves you alone" - me

Re: simple mod_timer patch

[Andrew Morton]

Steve Modica <modica@sgi.com> wrote:
>
> I pulled this back from the 2.6 kernel to reduce some serious contention on the 
> timerlist_lock when I had 8 gigabit cards runnings.
> 

By how much did it reduce contention?

Re: simple mod_timer patch

[Steve Modica]

Andrew Morton wrote:
> Steve Modica <modica@sgi.com> wrote:
> 
>>I pulled this back from the 2.6 kernel to reduce some serious contention on the 
>>timerlist_lock when I had 8 gigabit cards runnings.
>>
> 
> By how much did it reduce contention?


I got some profile results on the recv side for a 64p system running 8 recv 
threads using 1500 byte frames and the 1.5 tg3 driver (with NAPI)

The 4 cpus I was using were pegged.. Here's the sorted profile useage:

23581672 total                                      3.7362
22193477 cpu_idle                                 43346.6348
282048 tg3_poll                                 275.4375
218033 mod_timer                                272.5412
158467 __copy_user                               66.9202
  71956 sn_gettimeoffset                         249.8472
  60643 tcp_v4_rcv                                11.8443
  59058 tcp_rcv_established                        9.5625
  42402 kmalloc                                   31.5491
  41140 alloc_skb                                 33.8322
  38985 __kfree_skb                               46.8570
  36601 skb_release_data                          81.6987
  31168 kfree                                     57.2941
  24531 __wake_up                                 45.0938
  22551 kmem_cache_alloc                          19.0465

Here's an 8 cpu run (1 cpu per interface):


35664353 cpu_idle                                 69656.9395
 > > 550978 mod_timer                                688.7225
 > > 141781 sn_gettimeoffset                         492.2951
 > > 428917 tg3_poll                                 418.8643
 > > 310004 __copy_user                              130.9139
 > >   50981 skb_release_data                         113.7969
 > >   30982 do_gettimeofday                           74.4760
 > >   52091 __kfree_skb                               62.6094
 > >   25806 kfree                                     47.4375
 > >   36212 tcp_v4_do_rcv                             37.7208
 > >   44090 alloc_skb                                 36.2582
 > >   11238 tg3_recycle_rx                            35.1187


Here's a run with 8 cards receiving, 4 interrupt cpus with the mod_timer fix and 
a change so the stack isn't time stamping every packet (it's sorted based on the 
weighted load, but mod_timer had just dropped way down into the noise for these 
runs.)

[root@ascender2 modica]# head profile.recv.1500
40057535 cpu_idle                                 78237.3730
228061 tg3_poll                                 222.7158
  36780 skb_release_data                          82.0982
  13100 pciio_dmatrans_addr                       81.8750
  39542 __wake_up                                 72.6875
   4494 ia64_page_valid                           70.2188
  35029 kfree                                     64.3915
140595 __copy_user                               59.3729
  16307 __release_sock                            46.3267
  18897 kmem_cache_free                           45.4255



-- 
Steve Modica
work: 651-683-3224
MTS-Technical Lead
"Give a man a fish, and he will eat for a day, hit him with a fish and
he leaves you alone" - me