From: "Denys Fedoryshchenko" <denys@visp.net.lb>
To: netdev@vger.kernel.org
Subject: NAPI, rx_no_buffer_count, e1000, r8169 and other actors
Date: Sun, 15 Jun 2008 23:24:43 +0300 [thread overview]
Message-ID: <20080615200013.M67401@visp.net.lb> (raw)
Hi
Since i am using PC routers for my network, and i reach significant numbers
(for me significant) i start noticing minor problems. So all this talk about
networking performance in my case.
For example.
Sun server, AMD based (two CPU - AMD Opteron(tm) Processor 248).
e1000 connected over PCI-X ([ 4.919249] e1000: 0000:01:01.0: e1000_probe:
(PCI-X:100MHz:64-bit) 00:14:4f:20:89:f4)
All traffic processed over eth0, 5 VLAN, 1 second average around 110-200Mbps
of traffic. Host running also conntrack (max 1000000 entries, when packetloss
happen - around 256k entries). Around 1300 routes (FIB_TRIE) running. What is
worrying me, that ok, i win time by increasing rx descriptors from 256 to
4096, but how much time i win? if it "cracks" on 100 Mbps RX, it means by
interpolating descriptors increase from 256 to 4096 (4 times), i cannot
process more than 400Mbps RX?
The CPU is not so busy after all... maybe there is a way to change some
parameter to force NAPI poll interface more often?
I tried nice, changing realtime priority to FIFO, changing kernel to
preemptible... no luck, except increasing descriptors.
Router-Dora ~ # mpstat -P ALL 1
Linux 2.6.26-rc6-git2-build-0029 (Router-Dora) 06/15/08
22:51:02 CPU %user %nice %sys %iowait %irq %soft %steal
%idle intr/s
22:51:03 all 1.00 0.00 0.00 0.00 2.50 29.00 0.00
67.50 12927.00
22:51:03 0 2.00 0.00 0.00 0.00 4.00 59.00 0.00
35.00 11935.00
22:51:03 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
100.00 993.00
22:51:03 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
PID PPID USER STAT VSZ %MEM %CPU COMMAND
1544 1 root S 5824 0.2 0.0 /usr/sbin/snmpd -c /config/snmpd.conf
1530 1 squid S 2880 0.1 0.0 /usr/sbin/ripd -d
1524 1 squid S 2740 0.1 0.0 /usr/sbin/zebra -d
1 0 root S 2384 0.1 0.0 /bin/sh /init
1576 1115 root S 2384 0.1 0.0 /sbin/getty 38400 tty1
1577 1115 root S 2384 0.1 0.0 /sbin/getty 38400 tty2
1581 1115 root S 2384 0.1 0.0 /sbin/getty 38400 tty3
I have another host running, Core 2 Duo, e1000e+3 x e100, also conntrack, same
kernel configuration and similar amount of traffic, higher load (ifb + plenty
of shapers running) - almost no errors on default settings.
Linux 2.6.26-rc6-git2-build-0029 (Kup) 06/16/08
07:00:27 CPU %user %nice %sys %iowait %irq %soft %steal
%idle intr/s
07:00:28 all 0.00 0.00 0.50 0.00 4.00 31.50 0.00
64.00 32835.00
07:00:29 all 0.00 0.00 0.50 0.00 2.50 29.00 0.00
68.00 33164.36
Third host r8169 (PCI! This is important, seems i am running out of PCI
capacity), 400Mbit/s rx+tx summary load, e1000e interface also - around
200Mbps load. What is worrying me - interrupts rate, it seems generated by
realtek card... is there any way to drop it down?
Also some packetloss, around 0.0005% (i prefer to have clear zero :-)) ). No
nat, no shapers, same kernel configuration.
17:36:51 CPU %user %nice %sys %iowait %irq %soft %steal
%idle intr/s
17:36:52 all 0.50 0.00 0.50 0.00 1.49 32.34 0.00
65.17 88993.07
17:36:53 all 0.00 0.00 0.50 0.00 0.50 32.00 0.00
67.00 88655.00
17:36:54 all 0.00 0.00 0.50 0.00 1.49 31.84 0.00
66.17 89484.00
MegaRouter-KARAM ~ # cat /proc/interrupts ;sleep 10;cat /proc/interrupts
CPU0 CPU1
0: 806263699 0 IO-APIC-edge timer
1: 2 0 IO-APIC-edge i8042
9: 0 0 IO-APIC-fasteoi acpi
12: 5 0 IO-APIC-edge i8042
16: 0 0 IO-APIC-fasteoi uhci_hcd:usb3
18: 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb7
19: 0 0 IO-APIC-fasteoi uhci_hcd:usb6
21: 1191830952 0 IO-APIC-fasteoi uhci_hcd:usb4, eth0
23: 1245 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb5
217: 3 1584682152 PCI-MSI-edge eth1
NMI: 806263639 806263443 Non-maskable interrupts
LOC: 0 806263442 Local timer interrupts
RES: 99130 71199 Rescheduling interrupts
CAL: 62651 3871 function call interrupts
TLB: 239 187 TLB shootdowns
TRM: 0 0 Thermal event interrupts
SPU: 0 0 Spurious interrupts
ERR: 0
MIS: 0
CPU0 CPU1
0: 806273702 0 IO-APIC-edge timer
1: 2 0 IO-APIC-edge i8042
9: 0 0 IO-APIC-fasteoi acpi
12: 5 0 IO-APIC-edge i8042
16: 0 0 IO-APIC-fasteoi uhci_hcd:usb3
18: 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb7
19: 0 0 IO-APIC-fasteoi uhci_hcd:usb6
21: 1192549139 0 IO-APIC-fasteoi uhci_hcd:usb4, eth0
23: 1245 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb5
217: 3 1584840861 PCI-MSI-edge eth1
NMI: 806273642 806273446 Non-maskable interrupts
LOC: 0 806273445 Local timer interrupts
RES: 99130 71199 Rescheduling interrupts
CAL: 62653 3871 function call interrupts
TLB: 239 187 TLB shootdowns
TRM: 0 0 Thermal event interrupts
SPU: 0 0 Spurious interrupts
ERR: 0
MIS: 0
--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.
next reply other threads:[~2008-06-15 20:25 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-06-15 20:24 Denys Fedoryshchenko [this message]
2008-06-15 20:57 ` NAPI, rx_no_buffer_count, e1000, r8169 and other actors Francois Romieu
2008-06-15 21:32 ` Denys Fedoryshchenko
2008-06-15 21:32 ` Denys Fedoryshchenko
2008-06-15 23:46 ` Ben Hutchings
2008-06-16 2:59 ` Stephen Hemminger
2008-06-16 4:05 ` Denys Fedoryshchenko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20080615200013.M67401@visp.net.lb \
--to=denys@visp.net.lb \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).