* Re: Questions about your dual Opteron packetfiltering tests
[not found] <20040716015152.GA29337@soohrt.org>
@ 2004-07-16 13:18 ` Harald Welte
2004-09-06 20:56 ` Karsten Desler
0 siblings, 1 reply; 8+ messages in thread
From: Harald Welte @ 2004-07-16 13:18 UTC (permalink / raw)
To: Karsten Desler; +Cc: Netfilter Development Mailinglist
[-- Attachment #1: Type: text/plain, Size: 1907 bytes --]
On Fri, Jul 16, 2004 at 03:51:52AM +0200, Karsten Desler wrote:
> I've got a couple of questions about the "Doing lots of benchmarks /
> tuning / profiling lately" post on your weblog and I'd really appreciate
> any pointers to more precise results and developments, or ballpark
> figures to help me decide on hardware for an upcoming project.
unfortunately I still didn't have the time to put the information into
some presentable/publishable format. (Even though I'm now allowed to
publish the numbers). I'll try to do this soon.
> What impact do any of the following statements of you have on the
> overall performance of the router (rough estimates)?
> - i386 kernels give you higher PPS than x86_64 (because sk_buff is
> smaller)
about 5% performance increase
> - e1000 are way faster than tg3 boards (could be hardware or driver
> issue)
it's actually a hardware issue, I've just been talking about this to
David Miller yesterday. Apparently the tg3 receive path needs to do two
read operations in the receive path, as opposed to one read; this
increases latency.
I would say about 15% to 20%.
> - Intel PRO/1000MT Quad e1000 boards suck (apparently problems with the
> onboard PCI-X bridge)
About 25% to 30% less pps with 64byte packets on tg3 versus e1000.
> And do the dual Intel Pro/1000 PCIX card suffer the same fate?
no, not at all. they don't have a PCI-X bridge on the board but attach
the two chips directly [electrically] to the pci slot of the mainboard.
> Thanks in advance,
> Karsten
You might be interested into the various numbers published by Robert Olsson
on robur.slu.se.
--
- Harald Welte <laforge@gnumonks.org> http://www.gnumonks.org/
============================================================================
Programming is like sex: One mistake and you have to support it your lifetime
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Questions about your dual Opteron packetfiltering tests
2004-07-16 13:18 ` Questions about your dual Opteron packetfiltering tests Harald Welte
@ 2004-09-06 20:56 ` Karsten Desler
2004-09-07 8:41 ` Harald Welte
0 siblings, 1 reply; 8+ messages in thread
From: Karsten Desler @ 2004-09-06 20:56 UTC (permalink / raw)
To: Harald Welte, Netfilter Development Mailinglist
Hi,
again referring to your Weblog about the Sun V20z boxes for high-speed
packet filtering:
After spending a few days googling and trying to recreate results that
are at least somewhere around your numbers, I'm out of ideas.
quoting from http://gnumonks.org/~laforge/weblog/2004/04/21
* ip_tables performance sucks, even if the ruleset is empty ?!?
[...]
* You can route up to 1mpps at 64bytes packet size
* ip_conntrack and iptable_filter at suck at least 300kpps,
giving 700kpps as a result
Just two quick questions:
a) How? :), or
b) is that the expected 'ip_tables performance sucks'-performance?
I'm using two Opteron 244 on a Tyan S2882 mainboard with 2gb of RAM
and a vanilla 64bit 2.6.9-rc1-bk11 kernel.
I'm pushing 50mbit/s with 60kpps, about 100 iptables rules and both
CPUs are about 65% idle.
- interrupt 201 (e1000 eth0) is bound to cpu0, and 209 (e1000 eth1) is
bound to cpu1.
- e1000 is compiled with NAPI.
- tso is activated for both cards
- I've increased ip_conntrack_htable_size to 65536.
- My traffic is largely udp traffic (around 90%) with a distribution of:
20% 0 - 75 bytes,
60% 76 - 150 bytes,
10% 151 - 225 bytes and
10% 226 - 1500 bytes
Thanks in advance,
Karsten
eth0 is:
0000:01:01.0 Ethernet controller: Intel Corp. 82545EM Gigabit Ethernet Controller (Fiber) (rev 01)
Subsystem: Intel Corp. PRO/1000 MF Server Adapter
Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 201
Memory at fc7e0000 (64-bit, non-prefetchable) [size=128K]
I/O ports at 9c00 [size=64]
Capabilities: [dc] Power Management version 2
Capabilities: [e4] PCI-X non-bridge device.
Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
eth1 is:
0000:01:03.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03)
Subsystem: Intel Corp. PRO/1000 MT Dual Port Network Connection
Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 209
Memory at fc720000 (64-bit, non-prefetchable) [size=128K]
Memory at fc6c0000 (64-bit, non-prefetchable) [size=256K]
I/O ports at 9400 [size=64]
Expansion ROM at fc680000 [disabled] [size=256K]
Capabilities: [dc] Power Management version 2
Capabilities: [e4]
/proc/interrupts:
CPU0 CPU1
0: 67093304 0 IO-APIC-edge timer
8: 4 0 IO-APIC-edge rtc
9: 0 0 IO-APIC-level acpi
169: 117226 0 IO-APIC-level libata
201: 213918484 0 IO-APIC-level eth0
209: 11 211891491 IO-APIC-level eth1
NMI: 10377 11910
LOC: 67085557 67085955
ERR: 0
MIS: 0
/etc/sysctl.conf:
net/ipv4/icmp_ignore_bogus_error_responses=1
net/ipv4/conf/all/accept_redirects=0
net/ipv4/conf/all/rp_filter=1
net/ipv4/route/gc_elasticity=4
net/ipv4/neigh/default/gc_thresh1=1024
net/ipv4/neigh/default/gc_thresh2=2048
net/ipv4/neigh/default/gc_thresh3=4096
net/core/wmem_max=262144
net/core/rmem_max=262144
vm/min_free_kbytes=16000
net/ipv4/ip_forward=1
wc -l /proc/net/ip_conntrack
54243 /proc/net/ip_conntrack
rtstat -i 10
size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot mc GC: tot ignored goal_miss ovrf HASH: in_search out_search
36723 84998 1435 0 0 1 0 0 172 2 0 1438 1436 0 0 328787 232
41192 84884 1147 0 0 0 0 0 125 2 0 1149 1147 0 0 375680 261
44635 85263 1186 0 0 1 0 0 80 2 0 1189 1187 0 0 406300 63
47397 86269 1032 0 0 0 0 0 72 3 0 1035 1033 0 0 433299 80
42786 86713 1287 0 0 0 0 0 53 1 0 1288 1286 0 0 428865 81
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Questions about your dual Opteron packetfiltering tests
2004-09-06 20:56 ` Karsten Desler
@ 2004-09-07 8:41 ` Harald Welte
2004-09-10 14:06 ` Karsten Desler
2004-09-10 21:29 ` David S. Miller
0 siblings, 2 replies; 8+ messages in thread
From: Harald Welte @ 2004-09-07 8:41 UTC (permalink / raw)
To: Karsten Desler; +Cc: Netfilter Development Mailinglist, Harald Welte
[-- Attachment #1: Type: text/plain, Size: 2943 bytes --]
On Mon, Sep 06, 2004 at 10:56:53PM +0200, Karsten Desler wrote:
> Hi,
>
> again referring to your Weblog about the Sun V20z boxes for high-speed
> packet filtering:
> After spending a few days googling and trying to recreate results that
> are at least somewhere around your numbers, I'm out of ideas.
> Just two quick questions:
> a) How? :), or
> b) is that the expected 'ip_tables performance sucks'-performance?
no, you're definitely getting worse results. However, I never tested
with packet sizes > 64bytes, since I was only interested in the worst
case performance.
> I'm using two Opteron 244 on a Tyan S2882 mainboard with 2gb of RAM
> and a vanilla 64bit 2.6.9-rc1-bk11 kernel.
- 32bit kernels are faster for packet formwarding (5-10%)
- how fast are your pci busses? (I had PCI-X 133)
- how fast is your memory (I had DDR400) _VERY_ important!
> - e1000 is compiled with NAPI.
> - tso is activated for both cards
tso doesn't make any difference for packet filtering
> - I've increased ip_conntrack_htable_size to 65536.
maybe still too little,
> - My traffic is largely udp traffic (around 90%) with a distribution of:
> 20% 0 - 75 bytes,
> 60% 76 - 150 bytes,
> 10% 151 - 225 bytes and
> 10% 226 - 1500 bytes
>
> Thanks in advance,
> Karsten
>
> eth0 is:
> 0000:01:01.0 Ethernet controller: Intel Corp. 82545EM Gigabit Ethernet Controller (Fiber) (rev 01)
> eth1 is:
> 0000:01:03.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03)
This seems like both e1000 seem to be attached to the same PCI bus,
which is probably also not good for highest performance
> /proc/interrupts:
> CPU0 CPU1
> 0: 67093304 0 IO-APIC-edge timer
> 8: 4 0 IO-APIC-edge rtc
> 9: 0 0 IO-APIC-level acpi
> 169: 117226 0 IO-APIC-level libata
> 201: 213918484 0 IO-APIC-level eth0
> 209: 11 211891491 IO-APIC-level eth1
are you sure you have NAPI enabled? You shouldn't get that much
interrupts if using NAPI and going into saturation
> net/ipv4/conf/all/rp_filter=1
never ever enable rp_filter, that makes a huge difference. rp_filter is
not even recommended as default, and probably Debian is the only
distribution doing that mistake (read netdev archives on this).
> wc -l /proc/net/ip_conntrack
> 54243 /proc/net/ip_conntrack
Ok. I was testing single-flow UDP performance, not 50k different
flows...
--
- Harald Welte <laforge@netfilter.org> http://www.netfilter.org/
============================================================================
"Fragmentation is like classful addressing -- an interesting early
architectural error that shows how much experimentation was going
on while IP was being designed." -- Paul Vixie
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Questions about your dual Opteron packetfiltering tests
2004-09-07 8:41 ` Harald Welte
@ 2004-09-10 14:06 ` Karsten Desler
2004-09-12 1:23 ` David S. Miller
[not found] ` <20040911212703.GB19871@obroa-skai.de.gnumonks.org>
2004-09-10 21:29 ` David S. Miller
1 sibling, 2 replies; 8+ messages in thread
From: Karsten Desler @ 2004-09-10 14:06 UTC (permalink / raw)
To: Harald Welte, Harald Welte, Netfilter Development Mailinglist
* Harald Welte wrote:
> On Mon, Sep 06, 2004 at 10:56:53PM +0200, Karsten Desler wrote:
> > I'm using two Opteron 244 on a Tyan S2882 mainboard with 2gb of RAM
> > and a vanilla 64bit 2.6.9-rc1-bk11 kernel.
>
> - how fast are your pci busses? (I had PCI-X 133)
PCI-X 133 here too.
> - how fast is your memory (I had DDR400) _VERY_ important!
The memory itself is DDR400, but opterons 244 only support DDR333.
Opterons >= 246 (which support DDR400) are virtually impossible to get
on short notice on the german market.
> > - I've increased ip_conntrack_htable_size to 65536.
>
> maybe still too little,
Ok, I'm going to test with increased values tomorrow.
Is there a point in lowering the factor in the ip_conntrack_max
calculation to reduce the length of the linked list per bucket?
> > eth0 is:
> > 0000:01:01.0 Ethernet controller: Intel Corp. 82545EM Gigabit Ethernet Controller (Fiber) (rev 01)
> > eth1 is:
> > 0000:01:03.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller (rev 03)
>
> This seems like both e1000 seem to be attached to the same PCI bus,
> which is probably also not good for highest performance
True, they are in the same riser card and the fibre card only supports
33MHz/64bit PCI while the dual-copper adapter supports PCIX-133.
> > /proc/interrupts:
> > CPU0 CPU1
> > 0: 67093304 0 IO-APIC-edge timer
> > 8: 4 0 IO-APIC-edge rtc
> > 9: 0 0 IO-APIC-level acpi
> > 169: 117226 0 IO-APIC-level libata
> > 201: 213918484 0 IO-APIC-level eth0
> > 209: 11 211891491 IO-APIC-level eth1
>
> are you sure you have NAPI enabled? You shouldn't get that much
> interrupts if using NAPI and going into saturation
Pretty sure, yes. I'm getting around 7000 interrupts per second, which
fits perfectly in the 3000 interrupts/s/card and 1000 timer interrupts/s
picture.
> > net/ipv4/conf/all/rp_filter=1
>
> never ever enable rp_filter, that makes a huge difference. rp_filter is
> not even recommended as default, and probably Debian is the only
> distribution doing that mistake (read netdev archives on this).
Ok, I've disabled rp_filter and added rp_filter-like iptables
rules, doesn't make much (any?) difference though.
before:
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 1649708 111172 179052 0 0 0 10 6997 60 0 21 79 0
0 0 0 1649708 111172 179052 0 0 0 2 7032 61 0 21 79 0
0 0 0 1649700 111172 179052 0 0 0 0 7019 149 1 21 78 0
0 0 0 1649708 111172 179052 0 0 0 122 7052 72 0 21 79 0
after:
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 1649836 111176 179048 0 0 0 10 7181 64 0 21 79 0
0 0 0 1649620 111176 179048 0 0 0 16 7135 171 1 21 78 0
0 0 0 1649620 111180 179044 0 0 0 122 7050 71 0 21 79 0
0 0 0 1649644 111180 179044 0 0 0 36 6981 102 0 21 79 0
> > wc -l /proc/net/ip_conntrack
> > 54243 /proc/net/ip_conntrack
>
> Ok. I was testing single-flow UDP performance, not 50k different
> flows...
And could that be the cause for such vastly different results?
If desired, I guess I could give oprofile a try.
Thanks,
Karsten
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Questions about your dual Opteron packetfiltering tests
2004-09-07 8:41 ` Harald Welte
2004-09-10 14:06 ` Karsten Desler
@ 2004-09-10 21:29 ` David S. Miller
1 sibling, 0 replies; 8+ messages in thread
From: David S. Miller @ 2004-09-10 21:29 UTC (permalink / raw)
To: Harald Welte; +Cc: kdesler, netfilter-devel, laforge
On Tue, 7 Sep 2004 10:41:51 +0200
Harald Welte <laforge@netfilter.org> wrote:
> > net/ipv4/conf/all/rp_filter=1
>
> never ever enable rp_filter, that makes a huge difference. rp_filter is
> not even recommended as default, and probably Debian is the only
> distribution doing that mistake (read netdev archives on this).
Absolutely correct. This setting causes routing lookups to be
2 to 3 times more expensive.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Questions about your dual Opteron packetfiltering tests
2004-09-10 14:06 ` Karsten Desler
@ 2004-09-12 1:23 ` David S. Miller
2004-09-12 16:13 ` Karsten Desler
[not found] ` <20040911212703.GB19871@obroa-skai.de.gnumonks.org>
1 sibling, 1 reply; 8+ messages in thread
From: David S. Miller @ 2004-09-12 1:23 UTC (permalink / raw)
To: Karsten Desler; +Cc: laforge, netfilter-devel, laforge
On Fri, 10 Sep 2004 16:06:17 +0200
Karsten Desler <kdesler@soohrt.org> wrote:
> > never ever enable rp_filter, that makes a huge difference. rp_filter is
> > not even recommended as default, and probably Debian is the only
> > distribution doing that mistake (read netdev archives on this).
>
> Ok, I've disabled rp_filter and added rp_filter-like iptables
> rules, doesn't make much (any?) difference though.
Please only disable rp_filter, then test.
You're making it difficult to determine the source of the
bad performance if you add a new set of overhead. So please
don't add the new rp_filter-like iptables rules, and test
like that.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Questions about your dual Opteron packetfiltering tests
2004-09-12 1:23 ` David S. Miller
@ 2004-09-12 16:13 ` Karsten Desler
0 siblings, 0 replies; 8+ messages in thread
From: Karsten Desler @ 2004-09-12 16:13 UTC (permalink / raw)
To: David S. Miller; +Cc: laforge, netfilter-devel, laforge
* David S. Miller wrote:
> On Fri, 10 Sep 2004 16:06:17 +0200
> Karsten Desler <kdesler@soohrt.org> wrote:
> > Ok, I've disabled rp_filter and added rp_filter-like iptables
> > rules, doesn't make much (any?) difference though.
>
> Please only disable rp_filter, then test.
>
> You're making it difficult to determine the source of the
> bad performance if you add a new set of overhead. So please
> don't add the new rp_filter-like iptables rules, and test
> like that.
Ok, I've done some measurements/profiles today.
Traffic information: 40mbit/s, 50kpps, 53444 flows.
iptables: on refers to around 180 iptables rules
iptables: off refers to a iptables/conntrack still being in the kernel,
but no currently active iptables rules.
kernel booted with profile=1; readprofile commandline:
readprofile -r; sleep 60; readprofile -m /boot/System.map
iptables: on, rp_filter: off:
r b swpd free buff cache si so bi bo in cs us sy id wa
0 1 5876 570184 9548 1341064 0 0 0 92 7235 395 1 43 56 0
0 0 5876 570256 9568 1341180 0 0 0 152 7215 349 0 44 56 0
0 0 5876 569512 9576 1341376 0 0 0 74 7270 317 0 44 56 0
0 0 5876 569576 9584 1341640 0 0 0 48 7213 316 0 44 56 0
183 pfifo_fast_dequeue 1.9062
194 ip_refrag 1.5156
198 ip_forward_finish 1.5469
253 hash_conntrack 1.5813
254 udp_packet 3.1750
256 kmem_cache_alloc 4.0000
274 e1000_irq_enable 5.7083
274 vgacon_scroll 0.5189
276 e1000_intr 1.0781
289 kmem_cache_free 3.0104
336 dma_map_single 1.4000
339 __kmalloc 2.6484
359 ip_rcv_finish 0.6599
384 ip_ct_find_proto 4.8000
389 memcpy 2.2102
392 ip_confirm 6.1250
393 ip_rcv 0.3512
407 nf_hook_slow 1.4132
442 netif_receive_skb 1.0625
443 dev_queue_xmit 0.7691
512 ip_finish_output2 1.0000
527 ip_conntrack_in 0.2967
560 ip_forward 0.8140
602 e1000_alloc_rx_buffers 1.7917
686 ip_conntrack_find_get 8.5750
718 nf_iterate 3.7396
735 csum_partial 0.5889
771 alloc_skb 2.6771
778 skb_release_data 4.4205
818 eth_type_trans 3.4083
833 local_bh_enable 6.5078
857 kfree 3.5708
1130 qdisc_restart 2.6157
1177 __mod_timer 3.5030
1213 e1000_xmit_frame 0.5616
1218 del_timer 9.5156
1310 memset 6.9681
1466 ip_output 10.1806
1561 ip_ct_refresh_acct 4.4347
1825 __ip_conntrack_find 9.5052
2660 e1000_clean 1.3516
3292 __kfree_skb 12.8594
6064 ip_route_input 2.1173
10901 ipt_do_table 12.1663
68252 default_idle 1421.9167
119731 total 0.0562
------
iptables: on, rp_filter: on:
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 5876 562976 10076 1348048 0 0 0 57 7060 318 0 42 58 0
0 0 5876 562656 10084 1348244 0 0 0 50 7095 317 0 42 58 0
0 0 5876 562720 10092 1348440 0 0 0 54 7098 321 0 42 57 0
0 0 5876 562592 10100 1348568 0 0 0 48 7127 317 0 43 57 0
139 ip_conntrack_defrag 2.1719
140 udp_error 0.2823
144 pfifo_fast_dequeue 1.5000
186 ip_refrag 1.4531
191 __read_lock_failed 9.5500
193 ip_forward_finish 1.5078
204 udp_packet 2.5500
208 hash_conntrack 1.3000
231 kmem_cache_alloc 3.6094
271 e1000_irq_enable 5.6458
279 kmem_cache_free 2.9062
282 memcpy 1.6023
290 e1000_intr 1.1328
295 dma_map_single 1.2292
299 ip_rcv_finish 0.5496
301 __kmalloc 2.3516
334 dev_queue_xmit 0.5799
344 ip_ct_find_proto 4.3000
367 ip_confirm 5.7344
378 nf_hook_slow 1.3125
403 ip_rcv 0.3601
403 netif_receive_skb 0.9688
498 ip_finish_output2 0.9727
502 e1000_alloc_rx_buffers 1.4940
506 ip_conntrack_in 0.2849
523 ip_forward 0.7602
633 ip_conntrack_find_get 7.9125
650 nf_iterate 3.3854
757 csum_partial 0.6066
758 skb_release_data 4.3068
764 alloc_skb 2.6528
790 kfree 3.2917
831 eth_type_trans 3.4625
842 local_bh_enable 6.5781
1066 qdisc_restart 2.4676
1141 e1000_xmit_frame 0.5282
1148 __mod_timer 3.4167
1162 del_timer 9.0781
1236 memset 6.5745
1364 ip_output 9.4722
1510 ip_ct_refresh_acct 4.2898
1690 __ip_conntrack_find 8.8021
2557 e1000_clean 1.2993
2805 ipt_do_table 3.1306
3163 __kfree_skb 12.3555
5304 ip_route_input 1.8520
80113 default_idle 1669.0208
119832 total 0.0563
iptables: off, rp_filter: off:
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 5876 558860 10408 1351932 0 0 0 10 7271 11 0 34 66 0
0 0 5876 558404 10408 1351932 0 0 0 2 7183 37 1 34 65 0
0 0 5876 558716 10444 1351964 0 0 1 139 7265 97 0 35 65 0
0 0 5876 558724 10444 1351964 0 0 0 0 7252 7 0 34 66 0
139 ip_conntrack_defrag 2.1719
140 udp_error 0.2823
144 pfifo_fast_dequeue 1.5000
186 ip_refrag 1.4531
191 __read_lock_failed 9.5500
193 ip_forward_finish 1.5078
204 udp_packet 2.5500
208 hash_conntrack 1.3000
231 kmem_cache_alloc 3.6094
271 e1000_irq_enable 5.6458
279 kmem_cache_free 2.9062
282 memcpy 1.6023
290 e1000_intr 1.1328
295 dma_map_single 1.2292
299 ip_rcv_finish 0.5496
301 __kmalloc 2.3516
334 dev_queue_xmit 0.5799
344 ip_ct_find_proto 4.3000
367 ip_confirm 5.7344
378 nf_hook_slow 1.3125
403 ip_rcv 0.3601
403 netif_receive_skb 0.9688
498 ip_finish_output2 0.9727
502 e1000_alloc_rx_buffers 1.4940
506 ip_conntrack_in 0.2849
523 ip_forward 0.7602
633 ip_conntrack_find_get 7.9125
650 nf_iterate 3.3854
757 csum_partial 0.6066
758 skb_release_data 4.3068
764 alloc_skb 2.6528
790 kfree 3.2917
831 eth_type_trans 3.4625
842 local_bh_enable 6.5781
1066 qdisc_restart 2.4676
1141 e1000_xmit_frame 0.5282
1148 __mod_timer 3.4167
1162 del_timer 9.0781
1236 memset 6.5745
1364 ip_output 9.4722
1510 ip_ct_refresh_acct 4.2898
1690 __ip_conntrack_find 8.8021
2557 e1000_clean 1.2993
2805 ipt_do_table 3.1306
3163 __kfree_skb 12.3555
5304 ip_route_input 1.8520
80113 default_idle 1669.0208
119832 total 0.0563
iptables: off, rp_filter: off:
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 5872 558852 10540 1352008 0 0 0 10 7380 10 0 33 67 0
0 0 5872 558596 10540 1352008 0 0 0 2 7429 9 0 34 66 0
0 0 5872 558596 10540 1352008 0 0 0 0 7457 7 0 34 66 0
0 0 5872 558956 10556 1351992 0 0 0 97 7476 113 1 33 66 0
112 __write_lock_failed 3.5000
154 __read_lock_failed 7.7000
161 ip_refrag 1.2578
168 pfifo_fast_dequeue 1.7500
171 udp_error 0.3448
188 ip_forward_finish 1.4688
200 kmem_cache_alloc 3.1250
214 udp_packet 2.6750
223 hash_conntrack 1.3938
237 kmem_cache_free 2.4688
239 e1000_intr 0.9336
256 e1000_irq_enable 5.3333
266 memcpy 1.5114
292 ip_rcv_finish 0.5368
296 dma_map_single 1.2333
310 __kmalloc 2.4219
338 ip_ct_find_proto 4.2250
342 dev_queue_xmit 0.5938
370 ip_confirm 5.7812
400 ip_rcv 0.3575
420 nf_hook_slow 1.4583
424 netif_receive_skb 1.0192
466 ip_finish_output2 0.9102
514 ip_forward 0.7471
526 ip_conntrack_in 0.2962
583 e1000_alloc_rx_buffers 1.7351
598 ip_conntrack_find_get 7.4750
658 nf_iterate 3.4271
710 csum_partial 0.5689
740 skb_release_data 4.2045
757 alloc_skb 2.6285
762 eth_type_trans 3.1750
772 kfree 3.2167
819 local_bh_enable 6.3984
1080 qdisc_restart 2.5000
1132 __mod_timer 3.3690
1174 memset 6.2447
1195 e1000_xmit_frame 0.5532
1224 del_timer 9.5625
1353 ip_output 9.3958
1593 ip_ct_refresh_acct 4.5256
1692 __ip_conntrack_find 8.8125
2599 e1000_clean 1.3206
2872 ipt_do_table 3.2054
3100 __kfree_skb 12.1094
5676 ip_route_input 1.9818
79665 default_idle 1659.6875
119723 total 0.0562
- Karsten
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Questions about your dual Opteron packetfiltering tests
[not found] ` <20040911212703.GB19871@obroa-skai.de.gnumonks.org>
@ 2004-10-13 11:16 ` Karsten Desler
0 siblings, 0 replies; 8+ messages in thread
From: Karsten Desler @ 2004-10-13 11:16 UTC (permalink / raw)
To: Harald Welte, Netfilter Development Mailinglist
* Harald Welte wrote:
> Any possibility you could test it with single-flow udp performance in
> order to make sure that we both actually get similar performance, and
> the slowdown really only results from the 50k concurrent flows, and not
> from some other configuration/setup/... issue.
I've tested the single flow performance now.
|test1|-----|eth1 - router - eth0|-----|test2|
test1 was sending 1000byte udp packets at a rate of 70mbit/s from port
10000 to test2:10000.
In a second test, i've reversed the roles of test1 and test2, which made
no difference.
The router is still a 64bit 2.6.9-rc2, I haven't had time to test 2.6.9-rc4
yet.
Watching top during the flood, almost all additional systime was accounted
on the receiving cpu (eth0 bound to cpu0 and eth1 is bound to cpu1)
vmstat on the router (in is zero, because it's a 32bit vmstat and a couple
of numbers in /proc/interrupts are bigger than 2^32):
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 1715628 110740 155888 0 0 0 0 0 7 0 8 92 0
0 0 0 1715628 110740 155888 0 0 0 0 0 5 0 7 93 0
0 0 0 1715652 110740 155888 0 0 0 392 0 35 0 8 93 0
0 0 0 1715652 110740 155888 0 0 0 0 0 3 0 15 85 0 <- flood is starting
0 0 0 1715652 110740 155888 0 0 0 0 0 7 0 18 82 0
0 0 0 1715652 110740 155888 0 0 0 0 0 3 0 18 82 0
0 0 0 1715652 110740 155888 0 0 0 0 0 9 0 19 81 0
0 0 0 1715652 110740 155888 0 0 0 0 0 5 0 19 81 0
0 0 0 1715652 110740 155888 0 0 0 0 0 9 0 19 81 0
0 0 0 1715652 110740 155888 0 0 0 0 0 3 0 19 82 0
0 0 0 1715716 110740 155888 0 0 0 0 0 7 0 18 82 0
0 0 0 1715716 110740 155888 0 0 0 0 0 5 0 19 81 0
0 0 0 1715716 110740 155888 0 0 0 0 0 9 0 19 81 0
0 0 0 1715716 110740 155888 0 0 0 0 0 3 0 19 81 0
0 0 0 1715716 110740 155888 0 0 0 0 0 7 0 12 89 0
0 0 0 1715716 110740 155888 0 0 0 0 0 3 0 8 93 0 <- stop
0 0 0 1715716 110740 155888 0 0 0 0 0 9 0 7 93 0
0 0 0 1715716 110740 155888 0 0 0 0 0 5 0 7 93 0
0 0 0 1715716 110740 155888 0 0 0 0 0 11 0 6 94 0
rtstat:
size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot mc GC: tot ignored goal_miss ovrf HASH: in_search out_search
46630 30092 344 0 0 0 0 0 12 1 0 345 343 0 0 152963 19
46840 30021 403 0 0 0 0 0 70 1 0 405 403 0 0 152929 39
47138 30495 424 0 0 0 0 0 101 2 0 426 424 0 0 153787 128
47432 30739 516 0 0 0 0 0 10 2 0 518 516 0 0 153909 24
47723 34964 594 0 0 0 0 0 5 1 0 596 594 0 0 156555 32
48044 90336 676 0 0 0 0 0 44 1 0 677 675 0 0 155515 28 <- start
48322 89834 567 0 0 0 0 0 6 1 0 568 566 0 0 166415 22
48536 89941 499 0 0 0 0 0 72 2 0 501 499 0 0 215003 39
48804 89973 562 0 0 0 0 0 51 2 0 564 562 0 0 216194 28
49021 89315 545 0 0 0 0 0 22 2 0 547 545 0 0 215410 52
41444 91284 691 0 0 0 0 0 5 1 0 692 690 0 0 203104 24
41795 90726 554 0 0 0 0 0 6 1 0 555 553 0 0 197768 32
42060 63387 373 0 0 0 0 0 5 0 0 373 371 0 0 172197 27 <- stop
42394 31652 470 0 0 0 0 0 4 1 0 471 469 0 0 142762 23
41466 30315 438 0 0 0 0 0 7 2 0 440 438 0 0 139246 36
41915 30735 605 0 0 0 0 0 5 1 0 606 604 0 0 140605 29
diffprofile from before and during the flood:
205 168.0% e1000_clean
198 291.2% _spin_lock
133 391.2% _write_trylock
127 77.9% ip_append_data
115 132.2% __kfree_skb
93 197.9% memset
82 256.2% skb_release_data
71 263.0% alloc_skb
66 169.2% _spin_lock_irq
62 144.2% e1000_irq_enable
50 172.4% _write_unlock
49 140.0% _write_lock
44 89.8% tcp_setsockopt
43 116.2% kfree
39 121.9% eth_type_trans
37 336.4% local_bh_enable
36 105.9% e1000_intr
35 500.0% kmem_cache_free
32 290.9% __kmalloc
30 176.5% dev_queue_xmit
27 270.0% kmem_cache_alloc
24 171.4% _spin_unlock_irqrestore
20 153.8% _read_lock
20 71.4% tcp_sendmsg
19 118.8% netif_receive_skb
17 141.7% e1000_xmit_frame
15 68.2% e1000_alloc_rx_buffers
15 166.7% tcp_sendpage
14 350.0% eth_header_parse
14 140.0% tcp_listen_start
13 216.7% __do_softirq
13 433.3% ip_options_compile
12 200.0% dma_unmap_single
11 122.2% tcp_disconnect
-348 -1.8% total
-2258 -12.1% default_idle
- Karsten
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2004-10-13 11:16 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20040716015152.GA29337@soohrt.org>
2004-07-16 13:18 ` Questions about your dual Opteron packetfiltering tests Harald Welte
2004-09-06 20:56 ` Karsten Desler
2004-09-07 8:41 ` Harald Welte
2004-09-10 14:06 ` Karsten Desler
2004-09-12 1:23 ` David S. Miller
2004-09-12 16:13 ` Karsten Desler
[not found] ` <20040911212703.GB19871@obroa-skai.de.gnumonks.org>
2004-10-13 11:16 ` Karsten Desler
2004-09-10 21:29 ` David S. Miller
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.