* Re: Questions about your dual Opteron packetfiltering tests [not found] <20040716015152.GA29337@soohrt.org> @ 2004-07-16 13:18 ` Harald Welte 2004-09-06 20:56 ` Karsten Desler 0 siblings, 1 reply; 8+ messages in thread From: Harald Welte @ 2004-07-16 13:18 UTC (permalink / raw) To: Karsten Desler; +Cc: Netfilter Development Mailinglist [-- Attachment #1: Type: text/plain, Size: 1907 bytes --] On Fri, Jul 16, 2004 at 03:51:52AM +0200, Karsten Desler wrote: > I've got a couple of questions about the "Doing lots of benchmarks / > tuning / profiling lately" post on your weblog and I'd really appreciate > any pointers to more precise results and developments, or ballpark > figures to help me decide on hardware for an upcoming project. unfortunately I still didn't have the time to put the information into some presentable/publishable format. (Even though I'm now allowed to publish the numbers). I'll try to do this soon. > What impact do any of the following statements of you have on the > overall performance of the router (rough estimates)? > - i386 kernels give you higher PPS than x86_64 (because sk_buff is > smaller) about 5% performance increase > - e1000 are way faster than tg3 boards (could be hardware or driver > issue) it's actually a hardware issue, I've just been talking about this to David Miller yesterday. Apparently the tg3 receive path needs to do two read operations in the receive path, as opposed to one read; this increases latency. I would say about 15% to 20%. > - Intel PRO/1000MT Quad e1000 boards suck (apparently problems with the > onboard PCI-X bridge) About 25% to 30% less pps with 64byte packets on tg3 versus e1000. > And do the dual Intel Pro/1000 PCIX card suffer the same fate? no, not at all. they don't have a PCI-X bridge on the board but attach the two chips directly [electrically] to the pci slot of the mainboard. > Thanks in advance, > Karsten You might be interested into the various numbers published by Robert Olsson on robur.slu.se. -- - Harald Welte <laforge@gnumonks.org> http://www.gnumonks.org/ ============================================================================ Programming is like sex: One mistake and you have to support it your lifetime [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Questions about your dual Opteron packetfiltering tests 2004-07-16 13:18 ` Questions about your dual Opteron packetfiltering tests Harald Welte @ 2004-09-06 20:56 ` Karsten Desler 2004-09-07 8:41 ` Harald Welte 0 siblings, 1 reply; 8+ messages in thread From: Karsten Desler @ 2004-09-06 20:56 UTC (permalink / raw) To: Harald Welte, Netfilter Development Mailinglist Hi, again referring to your Weblog about the Sun V20z boxes for high-speed packet filtering: After spending a few days googling and trying to recreate results that are at least somewhere around your numbers, I'm out of ideas. quoting from http://gnumonks.org/~laforge/weblog/2004/04/21 * ip_tables performance sucks, even if the ruleset is empty ?!? [...] * You can route up to 1mpps at 64bytes packet size * ip_conntrack and iptable_filter at suck at least 300kpps, giving 700kpps as a result Just two quick questions: a) How? :), or b) is that the expected 'ip_tables performance sucks'-performance? I'm using two Opteron 244 on a Tyan S2882 mainboard with 2gb of RAM and a vanilla 64bit 2.6.9-rc1-bk11 kernel. I'm pushing 50mbit/s with 60kpps, about 100 iptables rules and both CPUs are about 65% idle. - interrupt 201 (e1000 eth0) is bound to cpu0, and 209 (e1000 eth1) is bound to cpu1. - e1000 is compiled with NAPI. - tso is activated for both cards - I've increased ip_conntrack_htable_size to 65536. - My traffic is largely udp traffic (around 90%) with a distribution of: 20% 0 - 75 bytes, 60% 76 - 150 bytes, 10% 151 - 225 bytes and 10% 226 - 1500 bytes Thanks in advance, Karsten eth0 is: 0000:01:01.0 Ethernet controller: Intel Corp. 82545EM Gigabit Ethernet Controller (Fiber) (rev 01) Subsystem: Intel Corp. PRO/1000 MF Server Adapter Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 201 Memory at fc7e0000 (64-bit, non-prefetchable) [size=128K] I/O ports at 9c00 [size=64] Capabilities: [dc] Power Management version 2 Capabilities: [e4] PCI-X non-bridge device. Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable- eth1 is: 0000:01:03.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03) Subsystem: Intel Corp. PRO/1000 MT Dual Port Network Connection Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 209 Memory at fc720000 (64-bit, non-prefetchable) [size=128K] Memory at fc6c0000 (64-bit, non-prefetchable) [size=256K] I/O ports at 9400 [size=64] Expansion ROM at fc680000 [disabled] [size=256K] Capabilities: [dc] Power Management version 2 Capabilities: [e4] /proc/interrupts: CPU0 CPU1 0: 67093304 0 IO-APIC-edge timer 8: 4 0 IO-APIC-edge rtc 9: 0 0 IO-APIC-level acpi 169: 117226 0 IO-APIC-level libata 201: 213918484 0 IO-APIC-level eth0 209: 11 211891491 IO-APIC-level eth1 NMI: 10377 11910 LOC: 67085557 67085955 ERR: 0 MIS: 0 /etc/sysctl.conf: net/ipv4/icmp_ignore_bogus_error_responses=1 net/ipv4/conf/all/accept_redirects=0 net/ipv4/conf/all/rp_filter=1 net/ipv4/route/gc_elasticity=4 net/ipv4/neigh/default/gc_thresh1=1024 net/ipv4/neigh/default/gc_thresh2=2048 net/ipv4/neigh/default/gc_thresh3=4096 net/core/wmem_max=262144 net/core/rmem_max=262144 vm/min_free_kbytes=16000 net/ipv4/ip_forward=1 wc -l /proc/net/ip_conntrack 54243 /proc/net/ip_conntrack rtstat -i 10 size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot mc GC: tot ignored goal_miss ovrf HASH: in_search out_search 36723 84998 1435 0 0 1 0 0 172 2 0 1438 1436 0 0 328787 232 41192 84884 1147 0 0 0 0 0 125 2 0 1149 1147 0 0 375680 261 44635 85263 1186 0 0 1 0 0 80 2 0 1189 1187 0 0 406300 63 47397 86269 1032 0 0 0 0 0 72 3 0 1035 1033 0 0 433299 80 42786 86713 1287 0 0 0 0 0 53 1 0 1288 1286 0 0 428865 81 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Questions about your dual Opteron packetfiltering tests 2004-09-06 20:56 ` Karsten Desler @ 2004-09-07 8:41 ` Harald Welte 2004-09-10 14:06 ` Karsten Desler 2004-09-10 21:29 ` David S. Miller 0 siblings, 2 replies; 8+ messages in thread From: Harald Welte @ 2004-09-07 8:41 UTC (permalink / raw) To: Karsten Desler; +Cc: Netfilter Development Mailinglist, Harald Welte [-- Attachment #1: Type: text/plain, Size: 2943 bytes --] On Mon, Sep 06, 2004 at 10:56:53PM +0200, Karsten Desler wrote: > Hi, > > again referring to your Weblog about the Sun V20z boxes for high-speed > packet filtering: > After spending a few days googling and trying to recreate results that > are at least somewhere around your numbers, I'm out of ideas. > Just two quick questions: > a) How? :), or > b) is that the expected 'ip_tables performance sucks'-performance? no, you're definitely getting worse results. However, I never tested with packet sizes > 64bytes, since I was only interested in the worst case performance. > I'm using two Opteron 244 on a Tyan S2882 mainboard with 2gb of RAM > and a vanilla 64bit 2.6.9-rc1-bk11 kernel. - 32bit kernels are faster for packet formwarding (5-10%) - how fast are your pci busses? (I had PCI-X 133) - how fast is your memory (I had DDR400) _VERY_ important! > - e1000 is compiled with NAPI. > - tso is activated for both cards tso doesn't make any difference for packet filtering > - I've increased ip_conntrack_htable_size to 65536. maybe still too little, > - My traffic is largely udp traffic (around 90%) with a distribution of: > 20% 0 - 75 bytes, > 60% 76 - 150 bytes, > 10% 151 - 225 bytes and > 10% 226 - 1500 bytes > > Thanks in advance, > Karsten > > eth0 is: > 0000:01:01.0 Ethernet controller: Intel Corp. 82545EM Gigabit Ethernet Controller (Fiber) (rev 01) > eth1 is: > 0000:01:03.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03) This seems like both e1000 seem to be attached to the same PCI bus, which is probably also not good for highest performance > /proc/interrupts: > CPU0 CPU1 > 0: 67093304 0 IO-APIC-edge timer > 8: 4 0 IO-APIC-edge rtc > 9: 0 0 IO-APIC-level acpi > 169: 117226 0 IO-APIC-level libata > 201: 213918484 0 IO-APIC-level eth0 > 209: 11 211891491 IO-APIC-level eth1 are you sure you have NAPI enabled? You shouldn't get that much interrupts if using NAPI and going into saturation > net/ipv4/conf/all/rp_filter=1 never ever enable rp_filter, that makes a huge difference. rp_filter is not even recommended as default, and probably Debian is the only distribution doing that mistake (read netdev archives on this). > wc -l /proc/net/ip_conntrack > 54243 /proc/net/ip_conntrack Ok. I was testing single-flow UDP performance, not 50k different flows... -- - Harald Welte <laforge@netfilter.org> http://www.netfilter.org/ ============================================================================ "Fragmentation is like classful addressing -- an interesting early architectural error that shows how much experimentation was going on while IP was being designed." -- Paul Vixie [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Questions about your dual Opteron packetfiltering tests 2004-09-07 8:41 ` Harald Welte @ 2004-09-10 14:06 ` Karsten Desler 2004-09-12 1:23 ` David S. Miller [not found] ` <20040911212703.GB19871@obroa-skai.de.gnumonks.org> 2004-09-10 21:29 ` David S. Miller 1 sibling, 2 replies; 8+ messages in thread From: Karsten Desler @ 2004-09-10 14:06 UTC (permalink / raw) To: Harald Welte, Harald Welte, Netfilter Development Mailinglist * Harald Welte wrote: > On Mon, Sep 06, 2004 at 10:56:53PM +0200, Karsten Desler wrote: > > I'm using two Opteron 244 on a Tyan S2882 mainboard with 2gb of RAM > > and a vanilla 64bit 2.6.9-rc1-bk11 kernel. > > - how fast are your pci busses? (I had PCI-X 133) PCI-X 133 here too. > - how fast is your memory (I had DDR400) _VERY_ important! The memory itself is DDR400, but opterons 244 only support DDR333. Opterons >= 246 (which support DDR400) are virtually impossible to get on short notice on the german market. > > - I've increased ip_conntrack_htable_size to 65536. > > maybe still too little, Ok, I'm going to test with increased values tomorrow. Is there a point in lowering the factor in the ip_conntrack_max calculation to reduce the length of the linked list per bucket? > > eth0 is: > > 0000:01:01.0 Ethernet controller: Intel Corp. 82545EM Gigabit Ethernet Controller (Fiber) (rev 01) > > eth1 is: > > 0000:01:03.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03) > > This seems like both e1000 seem to be attached to the same PCI bus, > which is probably also not good for highest performance True, they are in the same riser card and the fibre card only supports 33MHz/64bit PCI while the dual-copper adapter supports PCIX-133. > > /proc/interrupts: > > CPU0 CPU1 > > 0: 67093304 0 IO-APIC-edge timer > > 8: 4 0 IO-APIC-edge rtc > > 9: 0 0 IO-APIC-level acpi > > 169: 117226 0 IO-APIC-level libata > > 201: 213918484 0 IO-APIC-level eth0 > > 209: 11 211891491 IO-APIC-level eth1 > > are you sure you have NAPI enabled? You shouldn't get that much > interrupts if using NAPI and going into saturation Pretty sure, yes. I'm getting around 7000 interrupts per second, which fits perfectly in the 3000 interrupts/s/card and 1000 timer interrupts/s picture. > > net/ipv4/conf/all/rp_filter=1 > > never ever enable rp_filter, that makes a huge difference. rp_filter is > not even recommended as default, and probably Debian is the only > distribution doing that mistake (read netdev archives on this). Ok, I've disabled rp_filter and added rp_filter-like iptables rules, doesn't make much (any?) difference though. before: r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 1649708 111172 179052 0 0 0 10 6997 60 0 21 79 0 0 0 0 1649708 111172 179052 0 0 0 2 7032 61 0 21 79 0 0 0 0 1649700 111172 179052 0 0 0 0 7019 149 1 21 78 0 0 0 0 1649708 111172 179052 0 0 0 122 7052 72 0 21 79 0 after: r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 1649836 111176 179048 0 0 0 10 7181 64 0 21 79 0 0 0 0 1649620 111176 179048 0 0 0 16 7135 171 1 21 78 0 0 0 0 1649620 111180 179044 0 0 0 122 7050 71 0 21 79 0 0 0 0 1649644 111180 179044 0 0 0 36 6981 102 0 21 79 0 > > wc -l /proc/net/ip_conntrack > > 54243 /proc/net/ip_conntrack > > Ok. I was testing single-flow UDP performance, not 50k different > flows... And could that be the cause for such vastly different results? If desired, I guess I could give oprofile a try. Thanks, Karsten ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Questions about your dual Opteron packetfiltering tests 2004-09-10 14:06 ` Karsten Desler @ 2004-09-12 1:23 ` David S. Miller 2004-09-12 16:13 ` Karsten Desler [not found] ` <20040911212703.GB19871@obroa-skai.de.gnumonks.org> 1 sibling, 1 reply; 8+ messages in thread From: David S. Miller @ 2004-09-12 1:23 UTC (permalink / raw) To: Karsten Desler; +Cc: laforge, netfilter-devel, laforge On Fri, 10 Sep 2004 16:06:17 +0200 Karsten Desler <kdesler@soohrt.org> wrote: > > never ever enable rp_filter, that makes a huge difference. rp_filter is > > not even recommended as default, and probably Debian is the only > > distribution doing that mistake (read netdev archives on this). > > Ok, I've disabled rp_filter and added rp_filter-like iptables > rules, doesn't make much (any?) difference though. Please only disable rp_filter, then test. You're making it difficult to determine the source of the bad performance if you add a new set of overhead. So please don't add the new rp_filter-like iptables rules, and test like that. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Questions about your dual Opteron packetfiltering tests 2004-09-12 1:23 ` David S. Miller @ 2004-09-12 16:13 ` Karsten Desler 0 siblings, 0 replies; 8+ messages in thread From: Karsten Desler @ 2004-09-12 16:13 UTC (permalink / raw) To: David S. Miller; +Cc: laforge, netfilter-devel, laforge * David S. Miller wrote: > On Fri, 10 Sep 2004 16:06:17 +0200 > Karsten Desler <kdesler@soohrt.org> wrote: > > Ok, I've disabled rp_filter and added rp_filter-like iptables > > rules, doesn't make much (any?) difference though. > > Please only disable rp_filter, then test. > > You're making it difficult to determine the source of the > bad performance if you add a new set of overhead. So please > don't add the new rp_filter-like iptables rules, and test > like that. Ok, I've done some measurements/profiles today. Traffic information: 40mbit/s, 50kpps, 53444 flows. iptables: on refers to around 180 iptables rules iptables: off refers to a iptables/conntrack still being in the kernel, but no currently active iptables rules. kernel booted with profile=1; readprofile commandline: readprofile -r; sleep 60; readprofile -m /boot/System.map iptables: on, rp_filter: off: r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 5876 570184 9548 1341064 0 0 0 92 7235 395 1 43 56 0 0 0 5876 570256 9568 1341180 0 0 0 152 7215 349 0 44 56 0 0 0 5876 569512 9576 1341376 0 0 0 74 7270 317 0 44 56 0 0 0 5876 569576 9584 1341640 0 0 0 48 7213 316 0 44 56 0 183 pfifo_fast_dequeue 1.9062 194 ip_refrag 1.5156 198 ip_forward_finish 1.5469 253 hash_conntrack 1.5813 254 udp_packet 3.1750 256 kmem_cache_alloc 4.0000 274 e1000_irq_enable 5.7083 274 vgacon_scroll 0.5189 276 e1000_intr 1.0781 289 kmem_cache_free 3.0104 336 dma_map_single 1.4000 339 __kmalloc 2.6484 359 ip_rcv_finish 0.6599 384 ip_ct_find_proto 4.8000 389 memcpy 2.2102 392 ip_confirm 6.1250 393 ip_rcv 0.3512 407 nf_hook_slow 1.4132 442 netif_receive_skb 1.0625 443 dev_queue_xmit 0.7691 512 ip_finish_output2 1.0000 527 ip_conntrack_in 0.2967 560 ip_forward 0.8140 602 e1000_alloc_rx_buffers 1.7917 686 ip_conntrack_find_get 8.5750 718 nf_iterate 3.7396 735 csum_partial 0.5889 771 alloc_skb 2.6771 778 skb_release_data 4.4205 818 eth_type_trans 3.4083 833 local_bh_enable 6.5078 857 kfree 3.5708 1130 qdisc_restart 2.6157 1177 __mod_timer 3.5030 1213 e1000_xmit_frame 0.5616 1218 del_timer 9.5156 1310 memset 6.9681 1466 ip_output 10.1806 1561 ip_ct_refresh_acct 4.4347 1825 __ip_conntrack_find 9.5052 2660 e1000_clean 1.3516 3292 __kfree_skb 12.8594 6064 ip_route_input 2.1173 10901 ipt_do_table 12.1663 68252 default_idle 1421.9167 119731 total 0.0562 ------ iptables: on, rp_filter: on: r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 5876 562976 10076 1348048 0 0 0 57 7060 318 0 42 58 0 0 0 5876 562656 10084 1348244 0 0 0 50 7095 317 0 42 58 0 0 0 5876 562720 10092 1348440 0 0 0 54 7098 321 0 42 57 0 0 0 5876 562592 10100 1348568 0 0 0 48 7127 317 0 43 57 0 139 ip_conntrack_defrag 2.1719 140 udp_error 0.2823 144 pfifo_fast_dequeue 1.5000 186 ip_refrag 1.4531 191 __read_lock_failed 9.5500 193 ip_forward_finish 1.5078 204 udp_packet 2.5500 208 hash_conntrack 1.3000 231 kmem_cache_alloc 3.6094 271 e1000_irq_enable 5.6458 279 kmem_cache_free 2.9062 282 memcpy 1.6023 290 e1000_intr 1.1328 295 dma_map_single 1.2292 299 ip_rcv_finish 0.5496 301 __kmalloc 2.3516 334 dev_queue_xmit 0.5799 344 ip_ct_find_proto 4.3000 367 ip_confirm 5.7344 378 nf_hook_slow 1.3125 403 ip_rcv 0.3601 403 netif_receive_skb 0.9688 498 ip_finish_output2 0.9727 502 e1000_alloc_rx_buffers 1.4940 506 ip_conntrack_in 0.2849 523 ip_forward 0.7602 633 ip_conntrack_find_get 7.9125 650 nf_iterate 3.3854 757 csum_partial 0.6066 758 skb_release_data 4.3068 764 alloc_skb 2.6528 790 kfree 3.2917 831 eth_type_trans 3.4625 842 local_bh_enable 6.5781 1066 qdisc_restart 2.4676 1141 e1000_xmit_frame 0.5282 1148 __mod_timer 3.4167 1162 del_timer 9.0781 1236 memset 6.5745 1364 ip_output 9.4722 1510 ip_ct_refresh_acct 4.2898 1690 __ip_conntrack_find 8.8021 2557 e1000_clean 1.2993 2805 ipt_do_table 3.1306 3163 __kfree_skb 12.3555 5304 ip_route_input 1.8520 80113 default_idle 1669.0208 119832 total 0.0563 iptables: off, rp_filter: off: r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 5876 558860 10408 1351932 0 0 0 10 7271 11 0 34 66 0 0 0 5876 558404 10408 1351932 0 0 0 2 7183 37 1 34 65 0 0 0 5876 558716 10444 1351964 0 0 1 139 7265 97 0 35 65 0 0 0 5876 558724 10444 1351964 0 0 0 0 7252 7 0 34 66 0 139 ip_conntrack_defrag 2.1719 140 udp_error 0.2823 144 pfifo_fast_dequeue 1.5000 186 ip_refrag 1.4531 191 __read_lock_failed 9.5500 193 ip_forward_finish 1.5078 204 udp_packet 2.5500 208 hash_conntrack 1.3000 231 kmem_cache_alloc 3.6094 271 e1000_irq_enable 5.6458 279 kmem_cache_free 2.9062 282 memcpy 1.6023 290 e1000_intr 1.1328 295 dma_map_single 1.2292 299 ip_rcv_finish 0.5496 301 __kmalloc 2.3516 334 dev_queue_xmit 0.5799 344 ip_ct_find_proto 4.3000 367 ip_confirm 5.7344 378 nf_hook_slow 1.3125 403 ip_rcv 0.3601 403 netif_receive_skb 0.9688 498 ip_finish_output2 0.9727 502 e1000_alloc_rx_buffers 1.4940 506 ip_conntrack_in 0.2849 523 ip_forward 0.7602 633 ip_conntrack_find_get 7.9125 650 nf_iterate 3.3854 757 csum_partial 0.6066 758 skb_release_data 4.3068 764 alloc_skb 2.6528 790 kfree 3.2917 831 eth_type_trans 3.4625 842 local_bh_enable 6.5781 1066 qdisc_restart 2.4676 1141 e1000_xmit_frame 0.5282 1148 __mod_timer 3.4167 1162 del_timer 9.0781 1236 memset 6.5745 1364 ip_output 9.4722 1510 ip_ct_refresh_acct 4.2898 1690 __ip_conntrack_find 8.8021 2557 e1000_clean 1.2993 2805 ipt_do_table 3.1306 3163 __kfree_skb 12.3555 5304 ip_route_input 1.8520 80113 default_idle 1669.0208 119832 total 0.0563 iptables: off, rp_filter: off: r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 5872 558852 10540 1352008 0 0 0 10 7380 10 0 33 67 0 0 0 5872 558596 10540 1352008 0 0 0 2 7429 9 0 34 66 0 0 0 5872 558596 10540 1352008 0 0 0 0 7457 7 0 34 66 0 0 0 5872 558956 10556 1351992 0 0 0 97 7476 113 1 33 66 0 112 __write_lock_failed 3.5000 154 __read_lock_failed 7.7000 161 ip_refrag 1.2578 168 pfifo_fast_dequeue 1.7500 171 udp_error 0.3448 188 ip_forward_finish 1.4688 200 kmem_cache_alloc 3.1250 214 udp_packet 2.6750 223 hash_conntrack 1.3938 237 kmem_cache_free 2.4688 239 e1000_intr 0.9336 256 e1000_irq_enable 5.3333 266 memcpy 1.5114 292 ip_rcv_finish 0.5368 296 dma_map_single 1.2333 310 __kmalloc 2.4219 338 ip_ct_find_proto 4.2250 342 dev_queue_xmit 0.5938 370 ip_confirm 5.7812 400 ip_rcv 0.3575 420 nf_hook_slow 1.4583 424 netif_receive_skb 1.0192 466 ip_finish_output2 0.9102 514 ip_forward 0.7471 526 ip_conntrack_in 0.2962 583 e1000_alloc_rx_buffers 1.7351 598 ip_conntrack_find_get 7.4750 658 nf_iterate 3.4271 710 csum_partial 0.5689 740 skb_release_data 4.2045 757 alloc_skb 2.6285 762 eth_type_trans 3.1750 772 kfree 3.2167 819 local_bh_enable 6.3984 1080 qdisc_restart 2.5000 1132 __mod_timer 3.3690 1174 memset 6.2447 1195 e1000_xmit_frame 0.5532 1224 del_timer 9.5625 1353 ip_output 9.3958 1593 ip_ct_refresh_acct 4.5256 1692 __ip_conntrack_find 8.8125 2599 e1000_clean 1.3206 2872 ipt_do_table 3.2054 3100 __kfree_skb 12.1094 5676 ip_route_input 1.9818 79665 default_idle 1659.6875 119723 total 0.0562 - Karsten ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <20040911212703.GB19871@obroa-skai.de.gnumonks.org>]
* Re: Questions about your dual Opteron packetfiltering tests [not found] ` <20040911212703.GB19871@obroa-skai.de.gnumonks.org> @ 2004-10-13 11:16 ` Karsten Desler 0 siblings, 0 replies; 8+ messages in thread From: Karsten Desler @ 2004-10-13 11:16 UTC (permalink / raw) To: Harald Welte, Netfilter Development Mailinglist * Harald Welte wrote: > Any possibility you could test it with single-flow udp performance in > order to make sure that we both actually get similar performance, and > the slowdown really only results from the 50k concurrent flows, and not > from some other configuration/setup/... issue. I've tested the single flow performance now. |test1|-----|eth1 - router - eth0|-----|test2| test1 was sending 1000byte udp packets at a rate of 70mbit/s from port 10000 to test2:10000. In a second test, i've reversed the roles of test1 and test2, which made no difference. The router is still a 64bit 2.6.9-rc2, I haven't had time to test 2.6.9-rc4 yet. Watching top during the flood, almost all additional systime was accounted on the receiving cpu (eth0 bound to cpu0 and eth1 is bound to cpu1) vmstat on the router (in is zero, because it's a 32bit vmstat and a couple of numbers in /proc/interrupts are bigger than 2^32): procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 0 1715628 110740 155888 0 0 0 0 0 7 0 8 92 0 0 0 0 1715628 110740 155888 0 0 0 0 0 5 0 7 93 0 0 0 0 1715652 110740 155888 0 0 0 392 0 35 0 8 93 0 0 0 0 1715652 110740 155888 0 0 0 0 0 3 0 15 85 0 <- flood is starting 0 0 0 1715652 110740 155888 0 0 0 0 0 7 0 18 82 0 0 0 0 1715652 110740 155888 0 0 0 0 0 3 0 18 82 0 0 0 0 1715652 110740 155888 0 0 0 0 0 9 0 19 81 0 0 0 0 1715652 110740 155888 0 0 0 0 0 5 0 19 81 0 0 0 0 1715652 110740 155888 0 0 0 0 0 9 0 19 81 0 0 0 0 1715652 110740 155888 0 0 0 0 0 3 0 19 82 0 0 0 0 1715716 110740 155888 0 0 0 0 0 7 0 18 82 0 0 0 0 1715716 110740 155888 0 0 0 0 0 5 0 19 81 0 0 0 0 1715716 110740 155888 0 0 0 0 0 9 0 19 81 0 0 0 0 1715716 110740 155888 0 0 0 0 0 3 0 19 81 0 0 0 0 1715716 110740 155888 0 0 0 0 0 7 0 12 89 0 0 0 0 1715716 110740 155888 0 0 0 0 0 3 0 8 93 0 <- stop 0 0 0 1715716 110740 155888 0 0 0 0 0 9 0 7 93 0 0 0 0 1715716 110740 155888 0 0 0 0 0 5 0 7 93 0 0 0 0 1715716 110740 155888 0 0 0 0 0 11 0 6 94 0 rtstat: size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot mc GC: tot ignored goal_miss ovrf HASH: in_search out_search 46630 30092 344 0 0 0 0 0 12 1 0 345 343 0 0 152963 19 46840 30021 403 0 0 0 0 0 70 1 0 405 403 0 0 152929 39 47138 30495 424 0 0 0 0 0 101 2 0 426 424 0 0 153787 128 47432 30739 516 0 0 0 0 0 10 2 0 518 516 0 0 153909 24 47723 34964 594 0 0 0 0 0 5 1 0 596 594 0 0 156555 32 48044 90336 676 0 0 0 0 0 44 1 0 677 675 0 0 155515 28 <- start 48322 89834 567 0 0 0 0 0 6 1 0 568 566 0 0 166415 22 48536 89941 499 0 0 0 0 0 72 2 0 501 499 0 0 215003 39 48804 89973 562 0 0 0 0 0 51 2 0 564 562 0 0 216194 28 49021 89315 545 0 0 0 0 0 22 2 0 547 545 0 0 215410 52 41444 91284 691 0 0 0 0 0 5 1 0 692 690 0 0 203104 24 41795 90726 554 0 0 0 0 0 6 1 0 555 553 0 0 197768 32 42060 63387 373 0 0 0 0 0 5 0 0 373 371 0 0 172197 27 <- stop 42394 31652 470 0 0 0 0 0 4 1 0 471 469 0 0 142762 23 41466 30315 438 0 0 0 0 0 7 2 0 440 438 0 0 139246 36 41915 30735 605 0 0 0 0 0 5 1 0 606 604 0 0 140605 29 diffprofile from before and during the flood: 205 168.0% e1000_clean 198 291.2% _spin_lock 133 391.2% _write_trylock 127 77.9% ip_append_data 115 132.2% __kfree_skb 93 197.9% memset 82 256.2% skb_release_data 71 263.0% alloc_skb 66 169.2% _spin_lock_irq 62 144.2% e1000_irq_enable 50 172.4% _write_unlock 49 140.0% _write_lock 44 89.8% tcp_setsockopt 43 116.2% kfree 39 121.9% eth_type_trans 37 336.4% local_bh_enable 36 105.9% e1000_intr 35 500.0% kmem_cache_free 32 290.9% __kmalloc 30 176.5% dev_queue_xmit 27 270.0% kmem_cache_alloc 24 171.4% _spin_unlock_irqrestore 20 153.8% _read_lock 20 71.4% tcp_sendmsg 19 118.8% netif_receive_skb 17 141.7% e1000_xmit_frame 15 68.2% e1000_alloc_rx_buffers 15 166.7% tcp_sendpage 14 350.0% eth_header_parse 14 140.0% tcp_listen_start 13 216.7% __do_softirq 13 433.3% ip_options_compile 12 200.0% dma_unmap_single 11 122.2% tcp_disconnect -348 -1.8% total -2258 -12.1% default_idle - Karsten ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Questions about your dual Opteron packetfiltering tests 2004-09-07 8:41 ` Harald Welte 2004-09-10 14:06 ` Karsten Desler @ 2004-09-10 21:29 ` David S. Miller 1 sibling, 0 replies; 8+ messages in thread From: David S. Miller @ 2004-09-10 21:29 UTC (permalink / raw) To: Harald Welte; +Cc: kdesler, netfilter-devel, laforge On Tue, 7 Sep 2004 10:41:51 +0200 Harald Welte <laforge@netfilter.org> wrote: > > net/ipv4/conf/all/rp_filter=1 > > never ever enable rp_filter, that makes a huge difference. rp_filter is > not even recommended as default, and probably Debian is the only > distribution doing that mistake (read netdev archives on this). Absolutely correct. This setting causes routing lookups to be 2 to 3 times more expensive. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2004-10-13 11:16 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20040716015152.GA29337@soohrt.org>
2004-07-16 13:18 ` Questions about your dual Opteron packetfiltering tests Harald Welte
2004-09-06 20:56 ` Karsten Desler
2004-09-07 8:41 ` Harald Welte
2004-09-10 14:06 ` Karsten Desler
2004-09-12 1:23 ` David S. Miller
2004-09-12 16:13 ` Karsten Desler
[not found] ` <20040911212703.GB19871@obroa-skai.de.gnumonks.org>
2004-10-13 11:16 ` Karsten Desler
2004-09-10 21:29 ` David S. Miller
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.