NAPI, rx_no_buffer_count, e1000, r8169 and other actors

["Denys Fedoryshchenko" <denys@visp.net.lb>]

Hi

Since i am using PC routers for my network, and i reach significant numbers
(for me significant) i start noticing minor problems. So all this talk about
networking performance in my case.

For example.
Sun server, AMD based (two CPU -  AMD Opteron(tm) Processor 248).
e1000 connected over PCI-X ([    4.919249] e1000: 0000:01:01.0: e1000_probe:
(PCI-X:100MHz:64-bit) 00:14:4f:20:89:f4)

All traffic processed over eth0, 5 VLAN, 1 second average around 110-200Mbps
of traffic. Host running also conntrack (max 1000000 entries, when packetloss
happen - around 256k entries). Around 1300 routes (FIB_TRIE) running. What is
worrying me, that ok, i win time by increasing rx descriptors from 256 to
4096, but how much time i win? if it "cracks" on 100 Mbps RX, it means by
interpolating descriptors increase from 256 to 4096 (4 times), i cannot
process more than 400Mbps RX?
The CPU is not so busy after all... maybe there is a way to change some
parameter to force NAPI poll interface more often?
I tried nice, changing realtime priority to FIFO, changing kernel to
preemptible... no luck, except increasing descriptors.

Router-Dora ~ # mpstat -P ALL 1
Linux 2.6.26-rc6-git2-build-0029 (Router-Dora)  06/15/08

22:51:02     CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal  
%idle    intr/s
22:51:03     all    1.00    0.00    0.00    0.00    2.50   29.00    0.00  
67.50  12927.00
22:51:03       0    2.00    0.00    0.00    0.00    4.00   59.00    0.00  
35.00  11935.00
22:51:03       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00 
100.00    993.00
22:51:03       2    0.00    0.00    0.00    0.00    0.00    0.00    0.00   
0.00      0.00

  PID  PPID USER     STAT   VSZ %MEM %CPU COMMAND
 1544     1 root     S     5824  0.2  0.0 /usr/sbin/snmpd -c /config/snmpd.conf
 1530     1 squid    S     2880  0.1  0.0 /usr/sbin/ripd -d
 1524     1 squid    S     2740  0.1  0.0 /usr/sbin/zebra -d
    1     0 root     S     2384  0.1  0.0 /bin/sh /init
 1576  1115 root     S     2384  0.1  0.0 /sbin/getty 38400 tty1
 1577  1115 root     S     2384  0.1  0.0 /sbin/getty 38400 tty2
 1581  1115 root     S     2384  0.1  0.0 /sbin/getty 38400 tty3



I have another host running, Core 2 Duo, e1000e+3 x e100, also conntrack, same
kernel configuration and similar amount of traffic, higher load (ifb + plenty
of shapers running) - almost no errors on default settings.
Linux 2.6.26-rc6-git2-build-0029 (Kup)  06/16/08

07:00:27     CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal  
%idle    intr/s
07:00:28     all    0.00    0.00    0.50    0.00    4.00   31.50    0.00  
64.00  32835.00
07:00:29     all    0.00    0.00    0.50    0.00    2.50   29.00    0.00  
68.00  33164.36

Third host r8169 (PCI! This is important, seems i am running out of PCI
capacity), 400Mbit/s rx+tx summary load, e1000e interface also - around
200Mbps load. What is worrying me - interrupts rate, it seems generated by
realtek card... is there any way to drop it down? 
Also some packetloss, around 0.0005% (i prefer to have clear zero :-)) ). No
nat, no shapers, same kernel configuration.

17:36:51     CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal  
%idle    intr/s
17:36:52     all    0.50    0.00    0.50    0.00    1.49   32.34    0.00  
65.17  88993.07
17:36:53     all    0.00    0.00    0.50    0.00    0.50   32.00    0.00  
67.00  88655.00
17:36:54     all    0.00    0.00    0.50    0.00    1.49   31.84    0.00  
66.17  89484.00




MegaRouter-KARAM ~ # cat /proc/interrupts ;sleep 10;cat /proc/interrupts
           CPU0       CPU1
  0:  806263699          0   IO-APIC-edge      timer
  1:          2          0   IO-APIC-edge      i8042
  9:          0          0   IO-APIC-fasteoi   acpi
 12:          5          0   IO-APIC-edge      i8042
 16:          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
 18:          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb7
 19:          0          0   IO-APIC-fasteoi   uhci_hcd:usb6
 21: 1191830952          0   IO-APIC-fasteoi   uhci_hcd:usb4, eth0
 23:       1245          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb5
217:          3 1584682152   PCI-MSI-edge      eth1
NMI:  806263639  806263443   Non-maskable interrupts
LOC:          0  806263442   Local timer interrupts
RES:      99130      71199   Rescheduling interrupts
CAL:      62651       3871   function call interrupts
TLB:        239        187   TLB shootdowns
TRM:          0          0   Thermal event interrupts
SPU:          0          0   Spurious interrupts
ERR:          0
MIS:          0
           CPU0       CPU1
  0:  806273702          0   IO-APIC-edge      timer
  1:          2          0   IO-APIC-edge      i8042
  9:          0          0   IO-APIC-fasteoi   acpi
 12:          5          0   IO-APIC-edge      i8042
 16:          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
 18:          0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb7
 19:          0          0   IO-APIC-fasteoi   uhci_hcd:usb6
 21: 1192549139          0   IO-APIC-fasteoi   uhci_hcd:usb4, eth0
 23:       1245          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb5
217:          3 1584840861   PCI-MSI-edge      eth1
NMI:  806273642  806273446   Non-maskable interrupts
LOC:          0  806273445   Local timer interrupts
RES:      99130      71199   Rescheduling interrupts
CAL:      62653       3871   function call interrupts
TLB:        239        187   TLB shootdowns
TRM:          0          0   Thermal event interrupts
SPU:          0          0   Spurious interrupts
ERR:          0
MIS:          0


--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.