From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Denys Fedoryshchenko" Subject: NAPI, rx_no_buffer_count, e1000, r8169 and other actors Date: Sun, 15 Jun 2008 23:24:43 +0300 Message-ID: <20080615200013.M67401@visp.net.lb> Mime-Version: 1.0 Content-Type: text/plain; charset=koi8-r To: netdev@vger.kernel.org Return-path: Received: from usermail.globalproof.net ([194.146.153.18]:50573 "EHLO usermail.globalproof.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751969AbYFOUZC (ORCPT ); Sun, 15 Jun 2008 16:25:02 -0400 Received: from visp.net.lb (localhost [127.0.0.1]) by usermail.globalproof.net (Postfix) with ESMTP id 791DDBC395 for ; Sun, 15 Jun 2008 23:24:43 +0300 (EEST) Sender: netdev-owner@vger.kernel.org List-ID: Hi Since i am using PC routers for my network, and i reach significant numbers (for me significant) i start noticing minor problems. So all this talk about networking performance in my case. For example. Sun server, AMD based (two CPU - AMD Opteron(tm) Processor 248). e1000 connected over PCI-X ([ 4.919249] e1000: 0000:01:01.0: e1000_probe: (PCI-X:100MHz:64-bit) 00:14:4f:20:89:f4) All traffic processed over eth0, 5 VLAN, 1 second average around 110-200Mbps of traffic. Host running also conntrack (max 1000000 entries, when packetloss happen - around 256k entries). Around 1300 routes (FIB_TRIE) running. What is worrying me, that ok, i win time by increasing rx descriptors from 256 to 4096, but how much time i win? if it "cracks" on 100 Mbps RX, it means by interpolating descriptors increase from 256 to 4096 (4 times), i cannot process more than 400Mbps RX? The CPU is not so busy after all... maybe there is a way to change some parameter to force NAPI poll interface more often? I tried nice, changing realtime priority to FIFO, changing kernel to preemptible... no luck, except increasing descriptors. Router-Dora ~ # mpstat -P ALL 1 Linux 2.6.26-rc6-git2-build-0029 (Router-Dora) 06/15/08 22:51:02 CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 22:51:03 all 1.00 0.00 0.00 0.00 2.50 29.00 0.00 67.50 12927.00 22:51:03 0 2.00 0.00 0.00 0.00 4.00 59.00 0.00 35.00 11935.00 22:51:03 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 993.00 22:51:03 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 PID PPID USER STAT VSZ %MEM %CPU COMMAND 1544 1 root S 5824 0.2 0.0 /usr/sbin/snmpd -c /config/snmpd.conf 1530 1 squid S 2880 0.1 0.0 /usr/sbin/ripd -d 1524 1 squid S 2740 0.1 0.0 /usr/sbin/zebra -d 1 0 root S 2384 0.1 0.0 /bin/sh /init 1576 1115 root S 2384 0.1 0.0 /sbin/getty 38400 tty1 1577 1115 root S 2384 0.1 0.0 /sbin/getty 38400 tty2 1581 1115 root S 2384 0.1 0.0 /sbin/getty 38400 tty3 I have another host running, Core 2 Duo, e1000e+3 x e100, also conntrack, same kernel configuration and similar amount of traffic, higher load (ifb + plenty of shapers running) - almost no errors on default settings. Linux 2.6.26-rc6-git2-build-0029 (Kup) 06/16/08 07:00:27 CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 07:00:28 all 0.00 0.00 0.50 0.00 4.00 31.50 0.00 64.00 32835.00 07:00:29 all 0.00 0.00 0.50 0.00 2.50 29.00 0.00 68.00 33164.36 Third host r8169 (PCI! This is important, seems i am running out of PCI capacity), 400Mbit/s rx+tx summary load, e1000e interface also - around 200Mbps load. What is worrying me - interrupts rate, it seems generated by realtek card... is there any way to drop it down? Also some packetloss, around 0.0005% (i prefer to have clear zero :-)) ). No nat, no shapers, same kernel configuration. 17:36:51 CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s 17:36:52 all 0.50 0.00 0.50 0.00 1.49 32.34 0.00 65.17 88993.07 17:36:53 all 0.00 0.00 0.50 0.00 0.50 32.00 0.00 67.00 88655.00 17:36:54 all 0.00 0.00 0.50 0.00 1.49 31.84 0.00 66.17 89484.00 MegaRouter-KARAM ~ # cat /proc/interrupts ;sleep 10;cat /proc/interrupts CPU0 CPU1 0: 806263699 0 IO-APIC-edge timer 1: 2 0 IO-APIC-edge i8042 9: 0 0 IO-APIC-fasteoi acpi 12: 5 0 IO-APIC-edge i8042 16: 0 0 IO-APIC-fasteoi uhci_hcd:usb3 18: 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb7 19: 0 0 IO-APIC-fasteoi uhci_hcd:usb6 21: 1191830952 0 IO-APIC-fasteoi uhci_hcd:usb4, eth0 23: 1245 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb5 217: 3 1584682152 PCI-MSI-edge eth1 NMI: 806263639 806263443 Non-maskable interrupts LOC: 0 806263442 Local timer interrupts RES: 99130 71199 Rescheduling interrupts CAL: 62651 3871 function call interrupts TLB: 239 187 TLB shootdowns TRM: 0 0 Thermal event interrupts SPU: 0 0 Spurious interrupts ERR: 0 MIS: 0 CPU0 CPU1 0: 806273702 0 IO-APIC-edge timer 1: 2 0 IO-APIC-edge i8042 9: 0 0 IO-APIC-fasteoi acpi 12: 5 0 IO-APIC-edge i8042 16: 0 0 IO-APIC-fasteoi uhci_hcd:usb3 18: 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb7 19: 0 0 IO-APIC-fasteoi uhci_hcd:usb6 21: 1192549139 0 IO-APIC-fasteoi uhci_hcd:usb4, eth0 23: 1245 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb5 217: 3 1584840861 PCI-MSI-edge eth1 NMI: 806273642 806273446 Non-maskable interrupts LOC: 0 806273445 Local timer interrupts RES: 99130 71199 Rescheduling interrupts CAL: 62653 3871 function call interrupts TLB: 239 187 TLB shootdowns TRM: 0 0 Thermal event interrupts SPU: 0 0 Spurious interrupts ERR: 0 MIS: 0 -- Denys Fedoryshchenko Technical Manager Virtual ISP S.A.L.