Re: 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache

cold cold
2010-01-28T09:14:29Z

On Wed, Jan 27, 2010 at 11:18 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Le mercredi 27 janvier 2010 à 21:53 +0200, cold cold a écrit :
>
>>
>> btw atm i have 1k ip on that router but plan to put 10k so route cache
>> will grow 10 time fast.
>> i try to keep low  rhash_entries but garbage collector  is non stop
>> running and eat CPU
>> atm i have around 7k  entries per second for second for first minute.
>>
>> i think it will be better if i flush cache on secret interval instead
>> of giving work to gc.
>> i test with 512MB cache and CPU is 0% and flushing entire hash dont take a lot,
>> I'm not sure is there some side effects from flushing.
>
> Flush is immediate, it only marks entries as invalid and they are
> cleaned up later.
>
> 512 MB cache is quite small for your needs, each entry uses 384 bytes
> (assuming you use a 64bit kernel)
>
> In my experiments, I found using gc (and no flushing) was the most
> reliable way to have an equilibrium.
>
> WHen setting gc_interval to 1,  the garbage collector is fired every
> second and handles 1/300 of entries, from a work queue (thus doesnt stop
> packets to be handled by irqs), in a smooth way.
>
> You can post "perf top" results to check where cpu is consumed.
>
>
>

RX Kpps : 57 TX Kpps : 53 RX Kbits : 331184 TX Kbits : 306213
RX Kpps : 59 TX Kpps : 54 RX Kbits : 345517 TX Kbits : 304323
RX Kpps : 56 TX Kpps : 52 RX Kbits : 331418 TX Kbits : 296032
RX Kpps : 60 TX Kpps : 54 RX Kbits : 362007 TX Kbits : 297371
RX Kpps : 59 TX Kpps : 52 RX Kbits : 360455 TX Kbits : 280603

ON one cpu, gc_interval to 1, gc_elasticity 2

Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 72.0%id, 0.0%wa, 8.3%hi, 19.7%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

------------------------------------------------------------------------------
PerfTop: 17064 irqs/sec kernel:98.0% [100000 cycles], (all, 4 CPUs)
------------------------------------------------------------------------------

samples pcnt kernel function
_______ _____ _______________

40388.00 - 27.8% : acpi_idle_do_entry
24651.00 - 17.0% : read_hpet
4271.00 - 2.9% : _spin_lock
3388.00 - 2.3% : pskb_expand_head
3288.00 - 2.3% : igb_poll [igb]
3246.00 - 2.2% : irq_entries_start
2868.00 - 2.0% : dev_gro_receive
2665.00 - 1.8% : igb_xmit_frame_adv [igb]
2513.00 - 1.7% : ip_route_input
2144.00 - 1.5% : igb_clean_tx_irq [igb]
1842.00 - 1.3% : __slab_free
1544.00 - 1.1% : dev_queue_xmit
1423.00 - 1.0% : igb_msix_rx [igb]
1353.00 - 0.9% : __alloc_skb
1285.00 - 0.9% : eth_type_trans