From: Jorrit Kronjee <j.kronjee@infopact.nl>
To: Eric Dumazet <eric.dumazet@gmail.com>, netfilter-devel@vger.kernel.org
Subject: Re: debugging kernel during packet drops
Date: Wed, 24 Mar 2010 16:20:37 +0100 [thread overview]
Message-ID: <4BAA2DC5.7000409@infopact.nl> (raw)
In-Reply-To: <1269364893.2983.296.camel@edumazet-laptop>
On 3/23/2010 6:21 PM, Eric Dumazet wrote:
>
> Could you post more information about your machine ?
>
> cat /proc/interrupts
>
> If running a recent kernel, a "perf top" would be useful
>
> Maybe RPS will help your setup (included in net-next-2.?6 tree)
>
Eric,
To make things easier, I just installed the latest net-next tree.
Traffic flows from eth3 (e1000 7.3.21-k5-NAPI) to eth4 (e1000e
1.0.2-k2). After a reboot and 5 minutes of flooding it with 300 kpps,
perftop shows this:
------------------------------------------------------------------------------------------------------------------------------------------------
PerfTop: 918 irqs/sec kernel:99.6% [1000Hz cycles], (all, 4 CPUs)
------------------------------------------------------------------------------------------------------------------------------------------------
samples pcnt function DSO
_______ _____ ________________________
_______________________________________________________________________
1588.00 11.4% __slab_free
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
1571.00 11.3% dsthash_find
/lib/modules/2.6.34-rc1-net-next/kernel/net/netfilter/xt_hashlimit.ko
1117.00 8.0% _raw_spin_lock
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
899.00 6.4% e1000_clean_tx_irq
/lib/modules/2.6.34-rc1-net-next/kernel/drivers/net/e1000e/e1000e.ko
702.00 5.0% skb_release_head_state
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
650.00 4.7% kfree
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
540.00 3.9% __slab_alloc
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
514.00 3.7% memcpy
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
481.00 3.4% e1000_xmit_frame
/lib/modules/2.6.34-rc1-net-next/kernel/drivers/net/e1000e/e1000e.ko
335.00 2.4% nf_iterate
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
285.00 2.0% e1000_clean
/lib/modules/2.6.34-rc1-net-next/kernel/drivers/net/e1000e/e1000e.ko
264.00 1.9% kmem_cache_free
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
258.00 1.8% nf_hook_slow
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
257.00 1.8% e1000_intr_msi
/lib/modules/2.6.34-rc1-net-next/kernel/drivers/net/e1000e/e1000e.ko
207.00 1.5% ipt_do_table
/lib/modules/2.6.34-rc1-net-next/kernel/net/ipv4/netfilter/ip_tables.ko
202.00 1.4% dev_queue_xmit
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
180.00 1.3% memset
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
173.00 1.2% __alloc_skb
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
165.00 1.2% br_nf_pre_routing
/lib/modules/2.6.34-rc1-net-next/kernel/net/bridge/bridge.ko
159.00 1.1% __kmalloc_track_caller
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
159.00 1.1% br_nf_pre_routing_finish
/lib/modules/2.6.34-rc1-net-next/kernel/net/bridge/bridge.ko
158.00 1.1% e1000_clean_rx_irq
/lib/modules/2.6.34-rc1-net-next/kernel/drivers/net/e1000/e1000.ko
147.00 1.1% kmem_cache_alloc
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
140.00 1.0% kmem_cache_alloc_notrace
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
130.00 0.9% _raw_spin_lock_bh
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
122.00 0.9% hashlimit_mt_init
/lib/modules/2.6.34-rc1-net-next/kernel/net/netfilter/xt_hashlimit.ko
101.00 0.7% br_fdb_update
/lib/modules/2.6.34-rc1-net-next/kernel/net/bridge/bridge.ko
101.00 0.7% br_handle_frame
/lib/modules/2.6.34-rc1-net-next/kernel/net/bridge/bridge.ko
100.00 0.7% __netif_receive_skb
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
90.00 0.6% __netdev_alloc_skb
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
88.00 0.6% irq_entries_start
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
85.00 0.6% br_handle_frame_finish
/lib/modules/2.6.34-rc1-net-next/kernel/net/bridge/bridge.ko
83.00 0.6% br_nf_post_routing
/lib/modules/2.6.34-rc1-net-next/kernel/net/bridge/bridge.ko
76.00 0.5% __br_fdb_get
/lib/modules/2.6.34-rc1-net-next/kernel/net/bridge/bridge.ko
64.00 0.5% eth_type_trans
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
59.00 0.4% br_nf_forward_ip
/lib/modules/2.6.34-rc1-net-next/kernel/net/bridge/bridge.ko
55.00 0.4% add_partial
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
52.00 0.4% __kfree_skb
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
50.00 0.4% e1000_put_txbuf
/lib/modules/2.6.34-rc1-net-next/kernel/drivers/net/e1000e/e1000e.ko
50.00 0.4% local_bh_disable
/lib/modules/2.6.34-rc1-net-next/build/vmlinux
50.00 0.4% e1000_alloc_rx_buffers
/lib/modules/2.6.34-rc1-net-next/kernel/drivers/net/e1000/e1000.ko
49.00 0.4% htable_selective_cleanup
/lib/modules/2.6.34-rc1-net-next/kernel/net/netfilter/xt_hashlimit.ko
And /proc/interrupts shows this:
CPU0 CPU1 CPU2 CPU3
0: 46 0 1 0 IO-APIC-edge timer
1: 0 1 0 1 IO-APIC-edge i8042
6: 0 1 1 0 IO-APIC-edge floppy
8: 0 0 1 0 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 1 1 1 1 IO-APIC-edge i8042
14: 25 20 20 21 IO-APIC-edge
ata_piix
15: 0 0 0 0 IO-APIC-edge
ata_piix
16: 622 640 655 667 IO-APIC-fasteoi arcmsr
17: 0 0 0 0 IO-APIC-fasteoi
ehci_hcd:usb1
18: 31149 31209 31023 30680 IO-APIC-fasteoi
uhci_hcd:usb3, uhci_hcd:usb7, eth3
19: 0 0 0 0 IO-APIC-fasteoi
uhci_hcd:usb6
21: 0 0 0 0 IO-APIC-fasteoi
ata_piix, uhci_hcd:usb4
23: 1 1 0 0 IO-APIC-fasteoi
ehci_hcd:usb2, uhci_hcd:usb5
27: 541048 540974 541145 541475 PCI-MSI-edge eth4
NMI: 80763 83546 37524 37703 Non-maskable interrupts
LOC: 26176 24807 10336 13595 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 80763 83546 37524 37703 Performance
monitoring interrupts
PND: 79733 82513 36495 36674 Performance pending work
RES: 34 196 110 93 Rescheduling interrupts
CAL: 801 566 54 53 Function call interrupts
TLB: 145 152 89 72 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
I hope this helps! Is there anything special I need to do to use RPS?
Regards,
Jorrit Kronjee
next prev parent reply other threads:[~2010-03-24 15:20 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-03-22 10:41 debugging kernel during packet drops Jorrit Kronjee
2010-03-22 17:16 ` Patrick McHardy
2010-03-22 17:53 ` Jan Engelhardt
2010-03-22 18:02 ` Patrick McHardy
2010-03-23 15:14 ` Jorrit Kronjee
2010-03-23 15:39 ` Patrick McHardy
2010-03-23 17:21 ` Eric Dumazet
2010-03-23 20:07 ` Eric Dumazet
2010-03-24 15:20 ` Jorrit Kronjee [this message]
2010-03-24 16:21 ` Eric Dumazet
2010-03-24 16:28 ` Jan Engelhardt
2010-03-24 17:04 ` Eric Dumazet
2010-03-24 17:25 ` Jan Engelhardt
2010-03-25 9:32 ` Eric Dumazet
2010-03-25 10:35 ` Patrick McHardy
2010-03-25 11:02 ` Eric Dumazet
2010-03-31 12:23 ` [PATCH nf-next-2.6] xt_hashlimit: RCU conversion Eric Dumazet
2010-04-01 11:03 ` Patrick McHardy
2010-04-01 12:10 ` Eric Dumazet
2010-04-01 12:36 ` Patrick McHardy
2010-03-25 12:42 ` debugging kernel during packet drops Jan Engelhardt
2010-03-30 12:06 ` Jan Engelhardt
2010-03-30 14:12 ` Patrick McHardy
2010-03-26 10:41 ` Jorrit Kronjee
2010-03-26 11:21 ` Eric Dumazet
2010-03-26 14:17 ` Eric Dumazet
2010-03-26 15:54 ` Jorrit Kronjee
2010-03-23 17:04 ` James King
2010-03-23 17:23 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4BAA2DC5.7000409@infopact.nl \
--to=j.kronjee@infopact.nl \
--cc=eric.dumazet@gmail.com \
--cc=netfilter-devel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).