From: "Kok, Auke" <auke-jan.h.kok@intel.com>
To: Willy Tarreau <w@1wt.eu>
Cc: Anton Titov <a.titov@host.bg>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
linux-net@vger.kernel.org,
Jesse Brandeburg <jesse.brandeburg@intel.com>
Subject: Re: Bad network performance over 2Gbps
Date: Tue, 15 Apr 2008 13:40:29 -0700 [thread overview]
Message-ID: <480512BD.3060407@intel.com> (raw)
In-Reply-To: <20080415201441.GE8474@1wt.eu>
Willy Tarreau wrote:
> On Tue, Apr 15, 2008 at 09:06:44PM +0300, Anton Titov wrote:
>> I use Linux for serving a huge amount of static web on few servers. When
>> network traffic goes above 2Gbit/sec ksoftirqd/5 (not every time 5, but
>> every time just one) starts using exactly 100% CPU time and packet
>> packet loss starts preventing traffic from going up. When the network
>> traffic is lower than 1.9Gbit ksoftirqds use 0% CPU according to top.
>>
>> Uplink is 6 gigabit Intel cards bonded together using 802.3ad algorithm
>> with xmit_hash_policy set to layer3+4. On the other side is Cisco 2960
>> switch. Machine is with two quad core Intel Xeons @2.33GHz.
>>
>> Here goes a screen snapshot of "top" command. The described behavior
>> have nothing to do with 13% io-wait. It happens even if it is 0%
>> io-wait.
>> http://www.titov.net/misc/top-snap.png
>>
>> kernel configuration:
>> http://www.titov.net/misc/config.gz
>>
>> /proc/interrupts, lspci, dmesg (nothing intresting there), ifconfig,
>> uname -a:
>> http://www.titov.net/misc/misc.txt.gz
>>
>> Is it a Linux bug or some hardware limitation?
>
> possibly some missing parameters when loading your e1000 drivers.
> e1000 NICs support interrupt rate limitation, which proves very
> efficient in cases such as yours. I'm used to limit them to about
> 5k ints/s. Do a "modinfo e1000" to get the parameter name, I don't
> have it quite right in mind.
>
> Also, I've CCed linux-net.
# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6
CPU7
0: 342 261 258 278 271 253 264
283 IO-APIC-edge timer
1: 0 0 1 0 1 0 0
0 IO-APIC-edge i8042
6: 0 1 0 1 0 0 1
0 IO-APIC-edge floppy
9: 0 0 0 0 0 0 0
0 IO-APIC-fasteoi acpi
12: 1 1 0 0 0 1 1
0 IO-APIC-edge i8042
17: 180 190 178 183 182 186 186
188 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb4
18: 843504 842514 843653 842033 842416 842742 841903
842960 IO-APIC-fasteoi 3w-9xxx, uhci_hcd:usb3
19: 0 0 0 0 0 0 0
0 IO-APIC-fasteoi uhci_hcd:usb2
498: 534642903 534635899 534726883 534732377 534701710 534708588 534730550
534742730 PCI-MSI-edge eth5
499: 531832274 531846609 531917849 531942676 531855140 531850692 531885565
531863468 PCI-MSI-edge eth4
500: 487251627 487279206 487248030 487220044 487239637 487231454 487281672
487227202 PCI-MSI-edge eth3
501: 486083953 486062203 486109925 486075793 486036977 486035152 486097551
486117164 PCI-MSI-edge eth2
502: 528889380 528863624 528760188 528798619 528891886 528890760 528807939
528822746 PCI-MSI-edge eth1
503: 529043135 529056706 528980250 528975209 529018995 529027386 528941583
528970472 PCI-MSI-edge eth0
NMI: 0 0 0 0 0 0 0
0 Non-maskable interrupts
LOC: 62893699 62809502 62744208 62746035 62708815 62709055 62739182
62620363 Local timer interrupts
RES: 15454866 15827970 16235695 15386970 15761053 16097167 16190851
16159843 Rescheduling interrupts
CAL: 85 98 85 84 98 93 94
91 function call interrupts
TLB: 3565361 3561798 3570271 3566272 3556996 3555866 3578257
3564557 TLB shootdowns
TRM: 0 0 0 0 0 0 0
0 Thermal event interrupts
THR: 0 0 0 0 0 0 0
0 Threshold APIC interrupts
SPU: 0 0 0 0 0 0 0
0 Spurious interrupts
Yikes! all wrong!
the network irq's are being ping-ponged around all the cores! bad!
1) turn the in-kernel IRQBALANCE option off !
2) use either the userspace `irqbalance` daemon or
3) set smp_affinity manually
Auke
>
> Regards,
> Willy
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
next prev parent reply other threads:[~2008-04-15 20:41 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-04-15 18:06 Bad network performance over 2Gbps Anton Titov
2008-04-15 20:14 ` Willy Tarreau
2008-04-15 20:40 ` Kok, Auke [this message]
2008-04-15 22:36 ` Anton Titov
2008-04-16 4:27 ` Willy Tarreau
2008-04-15 20:15 ` H. Willstrand
2008-04-15 20:34 ` Kok, Auke
2008-04-15 20:59 ` Chris Snook
2008-04-15 21:05 ` Kok, Auke
2008-04-17 10:02 ` Anton Titov
2008-04-17 17:37 ` [PATCH] " Kok, Auke
2008-04-20 12:08 ` Denys Fedoryshchenko
2008-04-21 13:19 ` Pavel Machek
2008-04-21 16:38 ` Kok, Auke
2008-04-21 15:28 ` Ingo Molnar
2008-04-21 16:58 ` Kok, Auke
2008-04-21 18:35 ` Andi Kleen
2008-04-22 5:07 ` Bill Fink
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=480512BD.3060407@intel.com \
--to=auke-jan.h.kok@intel.com \
--cc=a.titov@host.bg \
--cc=jesse.brandeburg@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-net@vger.kernel.org \
--cc=w@1wt.eu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.