* 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache @ 2010-01-26 16:58 cold cold 2010-01-27 15:26 ` Eric Dumazet 0 siblings, 1 reply; 12+ messages in thread From: cold cold @ 2010-01-26 16:58 UTC (permalink / raw) To: netdev HI, i have expiriance some CPU usage spikes up to 10% on each four cpus after the first kernel route cache flush. After mashine start first 20 min CPU is 0%si 300Mbits/s full duplex and arount 100k pps forwarded traffic, without any firewall, just plain routing. route -n |wc -l 34 ip route show cache | wc -l 2140842 cat /proc/sys/net/ipv4/route/secret_interval 600 cat /proc/sys/net/ipv4/route/max_size 33554432 after kernel flush i got 10% on all CPUs for 5-6 mins. It's not from rebuilding route cashe becouse after fresh boot or network restrat there is no CPU usage until kernel flush route cache. I try to play with rhash_entries= 300000 to 2000000 same result. Network Card: Ethernet controller: Intel Corporation 82576 Gigabit Network Connection CPU: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz RAM: 4G Kernel: vanilla 2.6.32.4 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache 2010-01-26 16:58 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache cold cold @ 2010-01-27 15:26 ` Eric Dumazet 2010-01-27 19:53 ` cold cold 0 siblings, 1 reply; 12+ messages in thread From: Eric Dumazet @ 2010-01-27 15:26 UTC (permalink / raw) To: cold cold; +Cc: netdev Le mardi 26 janvier 2010 à 18:58 +0200, cold cold a écrit : > HI, > > > i have expiriance some CPU usage spikes up to 10% on each four cpus > after the first kernel route cache flush. > After mashine start first 20 min CPU is 0%si 300Mbits/s full duplex > and arount 100k pps forwarded traffic, without any firewall, just > plain routing. > > route -n |wc -l > 34 > > ip route show cache | wc -l > 2140842 > > cat /proc/sys/net/ipv4/route/secret_interval > 600 > cat /proc/sys/net/ipv4/route/max_size > 33554432 > > after kernel flush i got 10% on all CPUs for 5-6 mins. It's not from > rebuilding route cashe becouse after > fresh boot or network restrat there is no CPU usage until kernel flush > route cache. > I try to play with rhash_entries= 300000 to 2000000 same result. If you have one million dst entries to flush, it takes some time. You could try to not increase the rhash_entries (or keep it low, say 131072) but tune /proc/sys/net/ipv4/route settings. Try to reduce gc_elasticity from 8 to 2 Try to reduce gc_interval from 60 to 1 Important thing to consider is to irq affinities (so that one cpu handles network interrupts, to minimize cache ping poings ) ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache 2010-01-27 15:26 ` Eric Dumazet @ 2010-01-27 19:53 ` cold cold 2010-01-27 21:18 ` Eric Dumazet 0 siblings, 1 reply; 12+ messages in thread From: cold cold @ 2010-01-27 19:53 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev On Wed, Jan 27, 2010 at 5:26 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > Le mardi 26 janvier 2010 à 18:58 +0200, cold cold a écrit : >> HI, >> >> >> i have expiriance some CPU usage spikes up to 10% on each four cpus >> after the first kernel route cache flush. >> After mashine start first 20 min CPU is 0%si 300Mbits/s full duplex >> and arount 100k pps forwarded traffic, without any firewall, just >> plain routing. >> >> route -n |wc -l >> 34 >> >> ip route show cache | wc -l >> 2140842 >> >> cat /proc/sys/net/ipv4/route/secret_interval >> 600 >> cat /proc/sys/net/ipv4/route/max_size >> 33554432 >> >> after kernel flush i got 10% on all CPUs for 5-6 mins. It's not from >> rebuilding route cashe becouse after >> fresh boot or network restrat there is no CPU usage until kernel flush >> route cache. >> I try to play with rhash_entries= 300000 to 2000000 same result. > > If you have one million dst entries to flush, it takes some time. > > You could try to not increase the rhash_entries > (or keep it low, say 131072) > but tune /proc/sys/net/ipv4/route settings. > > Try to reduce gc_elasticity from 8 to 2 > Try to reduce gc_interval from 60 to 1 > > Important thing to consider is to irq affinities (so that one cpu > handles network interrupts, to minimize cache ping poings ) > btw atm i have 1k ip on that router but plan to put 10k so route cache will grow 10 time fast. i try to keep low rhash_entries but garbage collector is non stop running and eat CPU atm i have around 7k entries per second for second for first minute. i think it will be better if i flush cache on secret interval instead of giving work to gc. i test with 512MB cache and CPU is 0% and flushing entire hash dont take a lot, I'm not sure is there some side effects from flushing. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache 2010-01-27 19:53 ` cold cold @ 2010-01-27 21:18 ` Eric Dumazet 2010-01-28 9:14 ` cold cold 0 siblings, 1 reply; 12+ messages in thread From: Eric Dumazet @ 2010-01-27 21:18 UTC (permalink / raw) To: cold cold; +Cc: netdev Le mercredi 27 janvier 2010 à 21:53 +0200, cold cold a écrit : > > btw atm i have 1k ip on that router but plan to put 10k so route cache > will grow 10 time fast. > i try to keep low rhash_entries but garbage collector is non stop > running and eat CPU > atm i have around 7k entries per second for second for first minute. > > i think it will be better if i flush cache on secret interval instead > of giving work to gc. > i test with 512MB cache and CPU is 0% and flushing entire hash dont take a lot, > I'm not sure is there some side effects from flushing. Flush is immediate, it only marks entries as invalid and they are cleaned up later. 512 MB cache is quite small for your needs, each entry uses 384 bytes (assuming you use a 64bit kernel) In my experiments, I found using gc (and no flushing) was the most reliable way to have an equilibrium. WHen setting gc_interval to 1, the garbage collector is fired every second and handles 1/300 of entries, from a work queue (thus doesnt stop packets to be handled by irqs), in a smooth way. You can post "perf top" results to check where cpu is consumed. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache 2010-01-27 21:18 ` Eric Dumazet @ 2010-01-28 9:14 ` cold cold 2010-01-28 16:09 ` Eric Dumazet 0 siblings, 1 reply; 12+ messages in thread From: cold cold @ 2010-01-28 9:14 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev On Wed, Jan 27, 2010 at 11:18 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > Le mercredi 27 janvier 2010 à 21:53 +0200, cold cold a écrit : > >> >> btw atm i have 1k ip on that router but plan to put 10k so route cache >> will grow 10 time fast. >> i try to keep low rhash_entries but garbage collector is non stop >> running and eat CPU >> atm i have around 7k entries per second for second for first minute. >> >> i think it will be better if i flush cache on secret interval instead >> of giving work to gc. >> i test with 512MB cache and CPU is 0% and flushing entire hash dont take a lot, >> I'm not sure is there some side effects from flushing. > > Flush is immediate, it only marks entries as invalid and they are > cleaned up later. > > 512 MB cache is quite small for your needs, each entry uses 384 bytes > (assuming you use a 64bit kernel) > > In my experiments, I found using gc (and no flushing) was the most > reliable way to have an equilibrium. > > WHen setting gc_interval to 1, the garbage collector is fired every > second and handles 1/300 of entries, from a work queue (thus doesnt stop > packets to be handled by irqs), in a smooth way. > > You can post "perf top" results to check where cpu is consumed. > > > RX Kpps : 57 TX Kpps : 53 RX Kbits : 331184 TX Kbits : 306213 RX Kpps : 59 TX Kpps : 54 RX Kbits : 345517 TX Kbits : 304323 RX Kpps : 56 TX Kpps : 52 RX Kbits : 331418 TX Kbits : 296032 RX Kpps : 60 TX Kpps : 54 RX Kbits : 362007 TX Kbits : 297371 RX Kpps : 59 TX Kpps : 52 RX Kbits : 360455 TX Kbits : 280603 ON one cpu, gc_interval to 1, gc_elasticity 2 Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 72.0%id, 0.0%wa, 8.3%hi, 19.7%si, 0.0%st Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st ------------------------------------------------------------------------------ PerfTop: 17064 irqs/sec kernel:98.0% [100000 cycles], (all, 4 CPUs) ------------------------------------------------------------------------------ samples pcnt kernel function _______ _____ _______________ 40388.00 - 27.8% : acpi_idle_do_entry 24651.00 - 17.0% : read_hpet 4271.00 - 2.9% : _spin_lock 3388.00 - 2.3% : pskb_expand_head 3288.00 - 2.3% : igb_poll [igb] 3246.00 - 2.2% : irq_entries_start 2868.00 - 2.0% : dev_gro_receive 2665.00 - 1.8% : igb_xmit_frame_adv [igb] 2513.00 - 1.7% : ip_route_input 2144.00 - 1.5% : igb_clean_tx_irq [igb] 1842.00 - 1.3% : __slab_free 1544.00 - 1.1% : dev_queue_xmit 1423.00 - 1.0% : igb_msix_rx [igb] 1353.00 - 0.9% : __alloc_skb 1285.00 - 0.9% : eth_type_trans ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache 2010-01-28 9:14 ` cold cold @ 2010-01-28 16:09 ` Eric Dumazet 2010-01-28 16:26 ` cold cold 0 siblings, 1 reply; 12+ messages in thread From: Eric Dumazet @ 2010-01-28 16:09 UTC (permalink / raw) To: cold cold; +Cc: netdev Le jeudi 28 janvier 2010 à 11:14 +0200, cold cold a écrit : > > RX Kpps : 57 TX Kpps : 53 RX Kbits : 331184 TX Kbits : 306213 > RX Kpps : 59 TX Kpps : 54 RX Kbits : 345517 TX Kbits : 304323 > RX Kpps : 56 TX Kpps : 52 RX Kbits : 331418 TX Kbits : 296032 > RX Kpps : 60 TX Kpps : 54 RX Kbits : 362007 TX Kbits : 297371 > RX Kpps : 59 TX Kpps : 52 RX Kbits : 360455 TX Kbits : 280603 > > > ON one cpu, gc_interval to 1, gc_elasticity 2 > > Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 72.0%id, 0.0%wa, 8.3%hi, 19.7%si, 0.0%st > Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st > Cpu2 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st > Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st > > ------------------------------------------------------------------------------ > PerfTop: 17064 irqs/sec kernel:98.0% [100000 cycles], (all, 4 CPUs) > ------------------------------------------------------------------------------ > > samples pcnt kernel function > _______ _____ _______________ > > 40388.00 - 27.8% : acpi_idle_do_entry > 24651.00 - 17.0% : read_hpet > 4271.00 - 2.9% : _spin_lock > 3388.00 - 2.3% : pskb_expand_head > 3288.00 - 2.3% : igb_poll [igb] > 3246.00 - 2.2% : irq_entries_start > 2868.00 - 2.0% : dev_gro_receive > 2665.00 - 1.8% : igb_xmit_frame_adv [igb] > 2513.00 - 1.7% : ip_route_input > 2144.00 - 1.5% : igb_clean_tx_irq [igb] > 1842.00 - 1.3% : __slab_free > 1544.00 - 1.1% : dev_queue_xmit > 1423.00 - 1.0% : igb_msix_rx [igb] > 1353.00 - 0.9% : __alloc_skb > 1285.00 - 0.9% : eth_type_trans > -- All this seems pretty normal profile (regarding networking functions), your machine should scale without problem. Of course, the two first functions (acpi_idle_do_entry() & read_hpet()) look suspicious but I have no idea why. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache 2010-01-28 16:09 ` Eric Dumazet @ 2010-01-28 16:26 ` cold cold 2010-01-28 17:06 ` Eric Dumazet 0 siblings, 1 reply; 12+ messages in thread From: cold cold @ 2010-01-28 16:26 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev On Thu, Jan 28, 2010 at 6:09 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > Le jeudi 28 janvier 2010 à 11:14 +0200, cold cold a écrit : > >> >> RX Kpps : 57 TX Kpps : 53 RX Kbits : 331184 TX Kbits : 306213 >> RX Kpps : 59 TX Kpps : 54 RX Kbits : 345517 TX Kbits : 304323 >> RX Kpps : 56 TX Kpps : 52 RX Kbits : 331418 TX Kbits : 296032 >> RX Kpps : 60 TX Kpps : 54 RX Kbits : 362007 TX Kbits : 297371 >> RX Kpps : 59 TX Kpps : 52 RX Kbits : 360455 TX Kbits : 280603 >> >> >> ON one cpu, gc_interval to 1, gc_elasticity 2 >> >> Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 72.0%id, 0.0%wa, 8.3%hi, 19.7%si, 0.0%st >> Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st >> Cpu2 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st >> Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st >> >> ------------------------------------------------------------------------------ >> PerfTop: 17064 irqs/sec kernel:98.0% [100000 cycles], (all, 4 CPUs) >> ------------------------------------------------------------------------------ >> >> samples pcnt kernel function >> _______ _____ _______________ >> >> 40388.00 - 27.8% : acpi_idle_do_entry >> 24651.00 - 17.0% : read_hpet >> 4271.00 - 2.9% : _spin_lock >> 3388.00 - 2.3% : pskb_expand_head >> 3288.00 - 2.3% : igb_poll [igb] >> 3246.00 - 2.2% : irq_entries_start >> 2868.00 - 2.0% : dev_gro_receive >> 2665.00 - 1.8% : igb_xmit_frame_adv [igb] >> 2513.00 - 1.7% : ip_route_input >> 2144.00 - 1.5% : igb_clean_tx_irq [igb] >> 1842.00 - 1.3% : __slab_free >> 1544.00 - 1.1% : dev_queue_xmit >> 1423.00 - 1.0% : igb_msix_rx [igb] >> 1353.00 - 0.9% : __alloc_skb >> 1285.00 - 0.9% : eth_type_trans >> -- > > All this seems pretty normal profile (regarding networking functions), > your machine should scale without problem. > > Of course, the two first functions (acpi_idle_do_entry() & read_hpet()) > look suspicious but I have no idea why. > i make with flushing without gc on 2 cpu 2 time more traffic and CPU usage about 5 times less top - 11:22:04 up 6:45, 5 users, load average: 0.00, 0.10, 0.25 Tasks: 84 total, 1 running, 83 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 0.3%sy, 0.0%ni, 94.0%id, 0.0%wa, 1.7%hi, 4.0%si, 0.0%st Cpu1 : 0.3%us, 0.0%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.7%hi, 1.3%si, 0.0%st Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st RX Kpps : 90 TX Kpps : 75 RX Kbits : 582791 TX Kbits : 413873 RX Kpps : 87 TX Kpps : 74 RX Kbits : 546327 TX Kbits : 415852 RX Kpps : 87 TX Kpps : 74 RX Kbits : 544820 TX Kbits : 418339 RX Kpps : 88 TX Kpps : 73 RX Kbits : 569143 TX Kbits : 406438 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache 2010-01-28 16:26 ` cold cold @ 2010-01-28 17:06 ` Eric Dumazet 2010-01-28 18:49 ` cold cold 0 siblings, 1 reply; 12+ messages in thread From: Eric Dumazet @ 2010-01-28 17:06 UTC (permalink / raw) To: cold cold; +Cc: netdev Le jeudi 28 janvier 2010 à 18:26 +0200, cold cold a écrit : > On Thu, Jan 28, 2010 at 6:09 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > > Le jeudi 28 janvier 2010 à 11:14 +0200, cold cold a écrit : > > > >> > >> RX Kpps : 57 TX Kpps : 53 RX Kbits : 331184 TX Kbits : 306213 > >> RX Kpps : 59 TX Kpps : 54 RX Kbits : 345517 TX Kbits : 304323 > >> RX Kpps : 56 TX Kpps : 52 RX Kbits : 331418 TX Kbits : 296032 > >> RX Kpps : 60 TX Kpps : 54 RX Kbits : 362007 TX Kbits : 297371 > >> RX Kpps : 59 TX Kpps : 52 RX Kbits : 360455 TX Kbits : 280603 > >> > >> > >> ON one cpu, gc_interval to 1, gc_elasticity 2 > >> > >> Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 72.0%id, 0.0%wa, 8.3%hi, 19.7%si, 0.0%st > >> Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st > >> Cpu2 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st > >> Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st > >> > >> ------------------------------------------------------------------------------ > >> PerfTop: 17064 irqs/sec kernel:98.0% [100000 cycles], (all, 4 CPUs) > >> ------------------------------------------------------------------------------ > >> > >> samples pcnt kernel function > >> _______ _____ _______________ > >> > >> 40388.00 - 27.8% : acpi_idle_do_entry > >> 24651.00 - 17.0% : read_hpet > >> 4271.00 - 2.9% : _spin_lock > >> 3388.00 - 2.3% : pskb_expand_head > >> 3288.00 - 2.3% : igb_poll [igb] > >> 3246.00 - 2.2% : irq_entries_start > >> 2868.00 - 2.0% : dev_gro_receive > >> 2665.00 - 1.8% : igb_xmit_frame_adv [igb] > >> 2513.00 - 1.7% : ip_route_input > >> 2144.00 - 1.5% : igb_clean_tx_irq [igb] > >> 1842.00 - 1.3% : __slab_free > >> 1544.00 - 1.1% : dev_queue_xmit > >> 1423.00 - 1.0% : igb_msix_rx [igb] > >> 1353.00 - 0.9% : __alloc_skb > >> 1285.00 - 0.9% : eth_type_trans > >> -- > > > > All this seems pretty normal profile (regarding networking functions), > > your machine should scale without problem. > > > > Of course, the two first functions (acpi_idle_do_entry() & read_hpet()) > > look suspicious but I have no idea why. > > > > i make with flushing without gc on 2 cpu 2 time more traffic and CPU > usage about 5 times less > > top - 11:22:04 up 6:45, 5 users, load average: 0.00, 0.10, 0.25 > Tasks: 84 total, 1 running, 83 sleeping, 0 stopped, 0 zombie > Cpu0 : 0.0%us, 0.3%sy, 0.0%ni, 94.0%id, 0.0%wa, 1.7%hi, 4.0%si, 0.0%st > Cpu1 : 0.3%us, 0.0%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.7%hi, 1.3%si, 0.0%st > Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st > Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st > pure luck I guess, since you use two cpus here instead of one, but nothing guarantee this. > > RX Kpps : 90 TX Kpps : 75 RX Kbits : 582791 TX Kbits : 413873 > RX Kpps : 87 TX Kpps : 74 RX Kbits : 546327 TX Kbits : 415852 > RX Kpps : 87 TX Kpps : 74 RX Kbits : 544820 TX Kbits : 418339 > RX Kpps : 88 TX Kpps : 73 RX Kbits : 569143 TX Kbits : 406438 > -- Not sure I understand your goals. Previous numbers were with less trafic ? What do you want ? your cpus being idle, or your router being able to forward packets without drops ? Good, but if you drop packets when real garbage collection is done, you loose. Make your choice :) ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache 2010-01-28 17:06 ` Eric Dumazet @ 2010-01-28 18:49 ` cold cold 2010-01-28 21:16 ` Eric Dumazet 0 siblings, 1 reply; 12+ messages in thread From: cold cold @ 2010-01-28 18:49 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev On Thu, Jan 28, 2010 at 7:06 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > Le jeudi 28 janvier 2010 à 18:26 +0200, cold cold a écrit : >> On Thu, Jan 28, 2010 at 6:09 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: >> > Le jeudi 28 janvier 2010 à 11:14 +0200, cold cold a écrit : >> > >> >> >> >> RX Kpps : 57 TX Kpps : 53 RX Kbits : 331184 TX Kbits : 306213 >> >> RX Kpps : 59 TX Kpps : 54 RX Kbits : 345517 TX Kbits : 304323 >> >> RX Kpps : 56 TX Kpps : 52 RX Kbits : 331418 TX Kbits : 296032 >> >> RX Kpps : 60 TX Kpps : 54 RX Kbits : 362007 TX Kbits : 297371 >> >> RX Kpps : 59 TX Kpps : 52 RX Kbits : 360455 TX Kbits : 280603 >> >> >> >> >> >> ON one cpu, gc_interval to 1, gc_elasticity 2 >> >> >> >> Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 72.0%id, 0.0%wa, 8.3%hi, 19.7%si, 0.0%st >> >> Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st >> >> Cpu2 : 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st >> >> Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st >> >> >> >> ------------------------------------------------------------------------------ >> >> PerfTop: 17064 irqs/sec kernel:98.0% [100000 cycles], (all, 4 CPUs) >> >> ------------------------------------------------------------------------------ >> >> >> >> samples pcnt kernel function >> >> _______ _____ _______________ >> >> >> >> 40388.00 - 27.8% : acpi_idle_do_entry >> >> 24651.00 - 17.0% : read_hpet >> >> 4271.00 - 2.9% : _spin_lock >> >> 3388.00 - 2.3% : pskb_expand_head >> >> 3288.00 - 2.3% : igb_poll [igb] >> >> 3246.00 - 2.2% : irq_entries_start >> >> 2868.00 - 2.0% : dev_gro_receive >> >> 2665.00 - 1.8% : igb_xmit_frame_adv [igb] >> >> 2513.00 - 1.7% : ip_route_input >> >> 2144.00 - 1.5% : igb_clean_tx_irq [igb] >> >> 1842.00 - 1.3% : __slab_free >> >> 1544.00 - 1.1% : dev_queue_xmit >> >> 1423.00 - 1.0% : igb_msix_rx [igb] >> >> 1353.00 - 0.9% : __alloc_skb >> >> 1285.00 - 0.9% : eth_type_trans >> >> -- >> > >> > All this seems pretty normal profile (regarding networking functions), >> > your machine should scale without problem. >> > >> > Of course, the two first functions (acpi_idle_do_entry() & read_hpet()) >> > look suspicious but I have no idea why. >> > >> >> i make with flushing without gc on 2 cpu 2 time more traffic and CPU >> usage about 5 times less >> >> top - 11:22:04 up 6:45, 5 users, load average: 0.00, 0.10, 0.25 >> Tasks: 84 total, 1 running, 83 sleeping, 0 stopped, 0 zombie >> Cpu0 : 0.0%us, 0.3%sy, 0.0%ni, 94.0%id, 0.0%wa, 1.7%hi, 4.0%si, 0.0%st >> Cpu1 : 0.3%us, 0.0%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.7%hi, 1.3%si, 0.0%st >> Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st >> Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st >> > > pure luck I guess, since you use two cpus here instead of one, but > nothing guarantee this. > >> >> RX Kpps : 90 TX Kpps : 75 RX Kbits : 582791 TX Kbits : 413873 >> RX Kpps : 87 TX Kpps : 74 RX Kbits : 546327 TX Kbits : 415852 >> RX Kpps : 87 TX Kpps : 74 RX Kbits : 544820 TX Kbits : 418339 >> RX Kpps : 88 TX Kpps : 73 RX Kbits : 569143 TX Kbits : 406438 >> -- > > Not sure I understand your goals. Previous numbers were with less > trafic ? > > What do you want ? your cpus being idle, or your router being able to > forward packets without drops ? > > Good, but if you drop packets when real garbage collection is done, you > loose. Make your choice :) > > what you mean drop packets ? i test 2 different things and shearing results with you first test with high CPU is with garbage collection function second results represent CPU usage with totally disabled garbage collection ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache 2010-01-28 18:49 ` cold cold @ 2010-01-28 21:16 ` Eric Dumazet 2010-01-29 7:38 ` cold cold 0 siblings, 1 reply; 12+ messages in thread From: Eric Dumazet @ 2010-01-28 21:16 UTC (permalink / raw) To: cold cold; +Cc: netdev Le jeudi 28 janvier 2010 à 20:49 +0200, cold cold a écrit : > what you mean drop packets ? > > i test 2 different things and shearing results with you > first test with high CPU is with garbage collection function > second results represent CPU usage with totally disabled garbage collection ell, you didnt describe your benchmark method. 1) your results were on different rx/tx workload, and describing your workload is very important to be able to compare results. Then it should be exactly same workload. For example, when tx/tx load is high enough, less cpu overhead is spent on irq processing, since each IRQ delivers more packets per round. 2) you didnt sent "perf top" results for the second/last one. But the first "perf top" results showed less than 1% of cpu time was used by cache cleanup. I guess you dont want to focus on this, since its already very good. Usually, when we want to bench a router, we study how it deals with DDOS workload. Feeding lot of packets to the device and study what percentage of them are actually transmitted. Goal being 100% of legit packets of course. Route cache settings matter in DDOS situations, and the flush operation can have a big impact on dropped frames because of cpu/ram congestion. Because of 600 seconds oscillations, its pretty hard to study exact cpu use of a router, unless taking samples on long periods. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache 2010-01-28 21:16 ` Eric Dumazet @ 2010-01-29 7:38 ` cold cold 2010-01-29 9:06 ` Eric Dumazet 0 siblings, 1 reply; 12+ messages in thread From: cold cold @ 2010-01-29 7:38 UTC (permalink / raw) To: netdev On Thu, Jan 28, 2010 at 11:16 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > Le jeudi 28 janvier 2010 à 20:49 +0200, cold cold a écrit : >> what you mean drop packets ? >> >> i test 2 different things and shearing results with you >> first test with high CPU is with garbage collection function >> second results represent CPU usage with totally disabled garbage collection > > ell, you didnt describe your benchmark method. > > 1) your results were on different rx/tx workload, and describing your > workload is very important to be able to compare results. Then it should > be exactly same workload. > > For example, when tx/tx load is high enough, less cpu overhead is spent > on irq processing, since each IRQ delivers more packets per round. > > 2) you didnt sent "perf top" results for the second/last one. > > But the first "perf top" results showed less than 1% of cpu time was > used by cache cleanup. I guess you dont want to focus on this, since > its already very good. > > > > Usually, when we want to bench a router, we study how it deals with DDOS > workload. Feeding lot of packets to the device and study what percentage > of them are actually transmitted. Goal being 100% of legit packets of > course. > > Route cache settings matter in DDOS situations, and the flush operation > can have a big impact on dropped frames because of cpu/ram congestion. > > Because of 600 seconds oscillations, its pretty hard to study exact cpu > use of a router, unless taking samples on long periods. > > > > I'm totally agree with you there must be some scheduler to release route cache over the time. So far i see garbage collector do this, but it cost a lot of CPU probably its is a bug or design problem don't know for this i make this 2 test to compare CPU usage with and without GC. / secret_interval 10 min, 1300000 route entries in cash ofter 10 min, 7k new route on empty cache 2k on 1300000 / all route cache parameters default. I try also with gc_elasticity from 8 to 2 and gc_interval from 60 to 1 but don't have too much difference. What I'm trying to say is that flush cash is almost instant ( less then second on 1 CPU) so releasing of cash is not so heavy job ( you are right can have a big impact on dropped frames because of cpu/ram congestion ) but my point is why GC need 5-6min 10% no 4 CPU to do same job ? ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache 2010-01-29 7:38 ` cold cold @ 2010-01-29 9:06 ` Eric Dumazet 0 siblings, 0 replies; 12+ messages in thread From: Eric Dumazet @ 2010-01-29 9:06 UTC (permalink / raw) To: cold cold; +Cc: netdev Le vendredi 29 janvier 2010 à 09:38 +0200, cold cold a écrit : > I'm totally agree with you there must be some scheduler to release > route cache over the time. > So far i see garbage collector do this, but it cost a lot of CPU > probably its is a bug or design problem don't know > for this i make this 2 test to compare CPU usage with and without GC. > > > / secret_interval 10 min, 1300000 route entries in cash ofter 10 min, > 7k new route on empty cache 2k on 1300000 / > all route cache parameters default. I try also with gc_elasticity from > 8 to 2 and gc_interval from 60 to 1 but don't have > too much difference. > > What I'm trying to say is that flush cash is almost instant ( less > then second on 1 CPU) so releasing of cash is not so heavy job > ( you are right can have a big impact on dropped frames because of > cpu/ram congestion ) but my point is why GC need 5-6min > 10% no 4 CPU to do same job ? > -- Once again, 'flushing cache' is immediate, it only increments a global variable (aka a generation number) Then, later, when ip routing hits an entry with an old generation number, this entry is discarded. This slows down processing, and your router might drop packets during 5 to 60 seconds, while stale entries are eliminated. This delays the real cost of 'flush cache' in a smooth way, depending on trafic you have. Releasing 1.300.000 dst entries is expensive, no matter how you trigger the release, because it has to go through RCU queueing, spinlocks, kernel memory allocator logic, and touch a lot of memory. In your previous "perf top" results, we saw most of kernel cpu cycles were consumed outside of network stack, you might investigate why. Using HPET time keeping is probably not very good for your machine... ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2010-01-29 9:06 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-01-26 16:58 0% cpu usasge after fresh boot or net restart but 10% CPU if kernel flush route cache cold cold 2010-01-27 15:26 ` Eric Dumazet 2010-01-27 19:53 ` cold cold 2010-01-27 21:18 ` Eric Dumazet 2010-01-28 9:14 ` cold cold 2010-01-28 16:09 ` Eric Dumazet 2010-01-28 16:26 ` cold cold 2010-01-28 17:06 ` Eric Dumazet 2010-01-28 18:49 ` cold cold 2010-01-28 21:16 ` Eric Dumazet 2010-01-29 7:38 ` cold cold 2010-01-29 9:06 ` Eric Dumazet
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).