* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
[not found] <AANLkTin7GBSTcfZgr_9sNZ8CPMkW7Vstni+fs2v1-ink@mail.gmail.com>
@ 2011-02-13 7:17 ` Eric Dumazet
2011-02-13 13:56 ` Francois Romieu
0 siblings, 1 reply; 16+ messages in thread
From: Eric Dumazet @ 2011-02-13 7:17 UTC (permalink / raw)
To: Seblu; +Cc: lkml, netdev, Francois Romieu, Ivan Vecera
Le dimanche 13 février 2011 à 02:35 +0100, Seblu a écrit :
> Hi,
>
CC netdev
> Some days ago, one of my computer poweroff without any warning during
> a long rsync. Every time i run this long rsync, my computer power off
> after a random time.
> Firstly i suspected a heat stroke. But it resisted all my heat tests
> (cpuburn,ffmpeg,etc).
> Secondly i suposed a power issue. But after some tests, it does not
> seem to be kind of problem.
> Thirdly, i trying to load disk by a lot of read. But system is stable.
>
> Which is not practical, is that there is no message or trace in logs
> file. And why power off? BIOS is configured to restart after a
> powerloss.
>
> So, maybe a network issue? From one another 1Gbit/s wired linux
> computer i tryed an udp iperf at full speed (got ~950mbps) and after
> some time, host reboot. o0
> I tryed again, and host reboot again. I tryed with "ping -s 65000 -f"
> and my host reboot again. I've tryed this with a 2.6.32 (debian
> squeeze) and a 2.6.37 (debian experimental) kernel, in both case
> host reboot.
>
> This doesn't explain why my host power off when rsync, but it seems to
> have a big issue with kernel driver r8169.
>
> After starting my flood ping or udp iperf, dmesg show a lot of line:
> [ 254.896055] r8169 0000:04:00.0: eth0: link up
> [ 254.919976] r8169 0000:04:00.0: eth0: link up
> [ 254.943916] r8169 0000:04:00.0: eth0: link up
> [ 254.983784] r8169 0000:04:00.0: eth0: link up
> [ 255.007710] r8169 0000:04:00.0: eth0: link up
> [ 255.031657] r8169 0000:04:00.0: eth0: link up
> [ 255.103444] r8169 0000:04:00.0: eth0: link up
>
> Reboot is curious because it doesnt look like a kernel panic and print
> there is no kernel trace.
>
> My OS is a debian squeeze amd64. My hardware is a intel core i3 +
> gigabyte H55N-UBS3 with 4G DDR3.
>
> Do you need more trace / test? Do you think power off and reboot is linked?
>
r8169 driver is known to trigger a reset in case of RX overflow (but a
NIC reset should not power off the machine)
Some attempts were done to avoid a reset on some chipsets.
You could try latest linux-2.6 tree. This includes commits
f60ac8e7ab7cbb413a0131d5665b053f9f386526 (r8169: prevent RxFIFO induced
loops in the irq handler.)
1519e57fe81c14bb8fa4855579f19264d1ef63b4 (r8169: RxFIFO overflow
oddities with 8168 chipsets.)
b5ba6d12bdac21bc0620a5089e0f24e362645efd (r8169: use RxFIFO overflow
workaround for 8168c chipset.)
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
2011-02-13 7:17 ` Mass udp flow reboot linux with RealTek RTL-8169 Gigabit Eric Dumazet
@ 2011-02-13 13:56 ` Francois Romieu
2011-02-13 17:27 ` Seblu
0 siblings, 1 reply; 16+ messages in thread
From: Francois Romieu @ 2011-02-13 13:56 UTC (permalink / raw)
To: Seblu; +Cc: Eric Dumazet, lkml, netdev, Ivan Vecera
Eric Dumazet <eric.dumazet@gmail.com> :
> Le dimanche 13 f??vrier 2011 ?? 02:35 +0100, Seblu a ??crit :
[...]
> Some attempts were done to avoid a reset on some chipsets.
>
> You could try latest linux-2.6 tree. This includes commits
>
> f60ac8e7ab7cbb413a0131d5665b053f9f386526 (r8169: prevent RxFIFO induced
> loops in the irq handler.)
>
> 1519e57fe81c14bb8fa4855579f19264d1ef63b4 (r8169: RxFIFO overflow
> oddities with 8168 chipsets.)
>
> b5ba6d12bdac21bc0620a5089e0f24e362645efd (r8169: use RxFIFO overflow
> workaround for 8168c chipset.)
If it still does not perform better, please include the XID line of
the r8169 driver from dmesg.
--
Ueimor
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
2011-02-13 13:56 ` Francois Romieu
@ 2011-02-13 17:27 ` Seblu
2011-02-13 18:02 ` Seblu
0 siblings, 1 reply; 16+ messages in thread
From: Seblu @ 2011-02-13 17:27 UTC (permalink / raw)
To: Francois Romieu; +Cc: Eric Dumazet, lkml, netdev, Ivan Vecera
[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]
On Sun, Feb 13, 2011 at 2:56 PM, Francois Romieu <romieu@fr.zoreil.com> wrote:
> Eric Dumazet <eric.dumazet@gmail.com> :
> If it still does not perform better, please include the XID line of
> the r8169 driver from dmesg.
>
I've tryed on archlinux i686 distro with 2.6.37 and host reboot like
under debian.
After this, i tryed wiith a 2.6.38-rc4 (amd64), which include your
recent patchs.
# dmesg|grep -i xid
r8169 0000:04:00.0: eth0: RTL8168b/8111b at 0xffffc90005260000,
1c:6f:65:56:d9:17, XID 0c200000 IRQ 47
NIC seems to be reset frequently but host stop rebooting. \o//
r8169 0000:04:00.0: eth0: link up
net_ratelimit: 317 callbacks suppressed
r8169 0000:04:00.0: eth0: link up
...
net_ratelimit: 312 callbacks suppressed
r8169 0000:04:00.0: eth0: link up
i noticed a strange behaviour:
I run an iperf from computer A (gigabit plugged).
I run an ping -f from computer A.
A lot of pintk like previous one is printed on console (dmesg -n 8)
if i run an iperf from a computer B (fast eth plugged)
pings is not answered (computer A) and printk message disapear (on target)
if i run this 2nd iperf from computer A, disable iperf from computer
B, printk and ping answers are back.
i attached dmesg output.
Regards,
--
Sébastien Luttringer
www.seblu.net
[-- Attachment #2: dmesg.2.6.38-rc4.xz --]
[-- Type: application/x-xz, Size: 10388 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
2011-02-13 17:27 ` Seblu
@ 2011-02-13 18:02 ` Seblu
2011-02-13 20:34 ` Francois Romieu
0 siblings, 1 reply; 16+ messages in thread
From: Seblu @ 2011-02-13 18:02 UTC (permalink / raw)
To: Francois Romieu; +Cc: Eric Dumazet, lkml, netdev, Ivan Vecera
On Sun, Feb 13, 2011 at 6:27 PM, Seblu <seblu@seblu.net> wrote:
> On Sun, Feb 13, 2011 at 2:56 PM, Francois Romieu <romieu@fr.zoreil.com> wrote:
>> Eric Dumazet <eric.dumazet@gmail.com> :
>> If it still does not perform better, please include the XID line of
>> the r8169 driver from dmesg.
>>
> I've tryed on archlinux i686 distro with 2.6.37 and host reboot like
> under debian.
>
> After this, i tryed wiith a 2.6.38-rc4 (amd64), which include your
> recent patchs.
>
> # dmesg|grep -i xid
> r8169 0000:04:00.0: eth0: RTL8168b/8111b at 0xffffc90005260000,
> 1c:6f:65:56:d9:17, XID 0c200000 IRQ 47
>
> NIC seems to be reset frequently but host stop rebooting. \o//
ok after about 1 hour of iperf, host reboot.
--
Sébastien Luttringer
www.seblu.net
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
2011-02-13 18:02 ` Seblu
@ 2011-02-13 20:34 ` Francois Romieu
2011-02-18 2:54 ` Seblu
0 siblings, 1 reply; 16+ messages in thread
From: Francois Romieu @ 2011-02-13 20:34 UTC (permalink / raw)
To: Seblu; +Cc: Eric Dumazet, lkml, netdev, Ivan Vecera
Seblu <seblu@seblu.net> :
[...]
> > NIC seems to be reset frequently but host stop rebooting. \o//
> ok after about 1 hour of iperf, host reboot.
Can you apply the patch below on top of 2.6.38-rc4 ?
diff --git a/drivers/net/r8169.c b/drivers/net/r8169.c
index 59ccf0c..5d64b80 100644
--- a/drivers/net/r8169.c
+++ b/drivers/net/r8169.c
@@ -4645,6 +4645,7 @@ static irqreturn_t rtl8169_interrupt(int irq, void *dev_instance)
switch (tp->mac_version) {
/* Work around for rx fifo overflow */
case RTL_GIGA_MAC_VER_11:
+ case RTL_GIGA_MAC_VER_20:
case RTL_GIGA_MAC_VER_22:
case RTL_GIGA_MAC_VER_26:
netif_stop_queue(dev);
@@ -4653,7 +4654,6 @@ static irqreturn_t rtl8169_interrupt(int irq, void *dev_instance)
/* Testers needed. */
case RTL_GIGA_MAC_VER_17:
case RTL_GIGA_MAC_VER_19:
- case RTL_GIGA_MAC_VER_20:
case RTL_GIGA_MAC_VER_21:
case RTL_GIGA_MAC_VER_23:
case RTL_GIGA_MAC_VER_24:
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
2011-02-13 20:34 ` Francois Romieu
@ 2011-02-18 2:54 ` Seblu
2011-02-18 9:30 ` Francois Romieu
0 siblings, 1 reply; 16+ messages in thread
From: Seblu @ 2011-02-18 2:54 UTC (permalink / raw)
To: Francois Romieu; +Cc: Eric Dumazet, lkml, netdev, Ivan Vecera
[-- Attachment #1: Type: text/plain, Size: 571 bytes --]
On Sun, Feb 13, 2011 at 9:34 PM, Francois Romieu <romieu@fr.zoreil.com> wrote:
> Seblu <seblu@seblu.net> :
> [...]
>> > NIC seems to be reset frequently but host stop rebooting. \o//
>> ok after about 1 hour of iperf, host reboot.
>
> Can you apply the patch below on top of 2.6.38-rc4 ?
>
I've applyed your patch on 2.6.38-rc5. Host have rebooted 2mn after udp start.
After this reboot, host is still on after 2 hour under a 1Gbit/s udp flow.
I attached a dmesg output before reboot. Do you need anything else?
--
Sébastien Luttringer
www.seblu.net
[-- Attachment #2: dmesg.2.6.38-rc5-seblu.xz --]
[-- Type: application/x-xz, Size: 12392 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
2011-02-18 2:54 ` Seblu
@ 2011-02-18 9:30 ` Francois Romieu
2011-03-06 0:29 ` Seblu
0 siblings, 1 reply; 16+ messages in thread
From: Francois Romieu @ 2011-02-18 9:30 UTC (permalink / raw)
To: Seblu; +Cc: Eric Dumazet, lkml, netdev, Ivan Vecera
[-- Attachment #1: Type: text/plain, Size: 813 bytes --]
Seblu <seblu@seblu.net> :
[...]
> I've applyed your patch on 2.6.38-rc5. Host have rebooted 2mn after udp start.
> After this reboot, host is still on after 2 hour under a 1Gbit/s udp flow.
Thanks for testing.
> I attached a dmesg output before reboot. Do you need anything else?
Mostly :
1. .config
2. the size of the udp packets and the mtu
As an option :
3. a few seconds of 'vmstat 1' from the host under test
4. an 'ethtool -s eth0' from the host under test
5. /proc/interrupts from the host under test
6. lspci -tv
Can you apply the two attached patches on top of the previous ones and
give it a try ? The debug should not be too verbose if things are stationary
enough.
Do you have a serial cable and a second computer at hand by chance (don't
go for netconsole with the r8169 driver) ?
--
Ueimor
[-- Attachment #2: 0001-r8169-TX-stop-queue-race-fix.patch --]
[-- Type: text/plain, Size: 1129 bytes --]
diff --git a/drivers/net/r8169.c b/drivers/net/r8169.c
index 59ccf0c..712231f 100644
--- a/drivers/net/r8169.c
+++ b/drivers/net/r8169.c
@@ -4361,13 +4361,13 @@ static netdev_tx_t rtl8169_start_xmit(struct sk_buff *skb,
tp->cur_tx += frags + 1;
- wmb();
-
RTL_W8(TxPoll, NPQ); /* set polling bit */
+ mmiowb();
+
if (TX_BUFFS_AVAIL(tp) < MAX_SKB_FRAGS) {
netif_stop_queue(dev);
- smp_rmb();
+ smp_mb();
if (TX_BUFFS_AVAIL(tp) >= MAX_SKB_FRAGS)
netif_wake_queue(dev);
}
@@ -4468,10 +4468,14 @@ static void rtl8169_tx_interrupt(struct net_device *dev,
if (tp->dirty_tx != dirty_tx) {
tp->dirty_tx = dirty_tx;
- smp_wmb();
- if (netif_queue_stopped(dev) &&
- (TX_BUFFS_AVAIL(tp) >= MAX_SKB_FRAGS)) {
- netif_wake_queue(dev);
+ smp_mb();
+ if (unlikely(netif_queue_stopped(dev) &&
+ (TX_BUFFS_AVAIL(tp) >= (NUM_TX_DESC / 4)))) {
+ netif_tx_lock(dev);
+ if (netif_queue_stopped(dev) &&
+ (TX_BUFFS_AVAIL(tp) >= (NUM_TX_DESC / 4)))
+ netif_wake_queue(dev);
+ netif_tx_unlock(dev);
}
/*
* 8168 hack: TxPoll requests are lost when the Tx packets are
--
1.7.3.4
[-- Attachment #3: 0002-r8169-debuginator.patch --]
[-- Type: text/plain, Size: 856 bytes --]
diff --git a/drivers/net/r8169.c b/drivers/net/r8169.c
index 712231f..5eaccbb 100644
--- a/drivers/net/r8169.c
+++ b/drivers/net/r8169.c
@@ -4622,6 +4622,11 @@ static int rtl8169_rx_interrupt(struct net_device *dev,
return count;
}
+static struct {
+ u16 status[4];
+ u16 idx;
+} x = { { 0 }, 0 };
+
static irqreturn_t rtl8169_interrupt(int irq, void *dev_instance)
{
struct net_device *dev = dev_instance;
@@ -4637,6 +4642,12 @@ static irqreturn_t rtl8169_interrupt(int irq, void *dev_instance)
while (status && status != 0xffff) {
handled = 1;
+ x.status[x.idx++ % 4] = status;
+ if (net_ratelimit()) {
+ printk(KERN_INFO "%04x %04x %04x %04x\n",
+ x.status[0], x.status[1],
+ x.status[2], x.status[3]);
+ }
/* Handle all of the error cases first. These will reset
* the chip, so just exit the loop.
*/
--
1.7.3.4
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
2011-02-18 9:30 ` Francois Romieu
@ 2011-03-06 0:29 ` Seblu
2011-03-10 12:08 ` Francois Romieu
0 siblings, 1 reply; 16+ messages in thread
From: Seblu @ 2011-03-06 0:29 UTC (permalink / raw)
To: Francois Romieu; +Cc: Eric Dumazet, lkml, netdev, Ivan Vecera
[-- Attachment #1: Type: text/plain, Size: 13538 bytes --]
On Fri, Feb 18, 2011 at 10:30 AM, Francois Romieu <romieu@fr.zoreil.com> wrote:
> Seblu <seblu@seblu.net> :
> [...]
>> I've applyed your patch on 2.6.38-rc5. Host have rebooted 2mn after udp start.
>> After this reboot, host is still on after 2 hour under a 1Gbit/s udp flow.
>
> Thanks for testing.
Thanks for debugging! Sorry for the late reply.
I catched the following trace during my previous torture session.
Maybe it can help.
[ 1448.189961] ------------[ cut here ]------------
[ 1448.189968] WARNING: at net/sched/sch_generic.c:256
dev_watchdog+0x104/0x1ad()
[ 1448.189970] Hardware name: H55N-USB3
[ 1448.189971] NETDEV WATCHDOG: eth0 (r8169): transmit queue 0 timed out
[ 1448.189972] Modules linked in: ext2 it87 hwmon_vid ds1621 i915
drm_kms_helper drm i2c_algo_bit video dm_crypt raid456
async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy
async_tx md_mod option usb_wwan r8169 snd_pcm snd_timer snd soundcore
usbserial uhci_hcd xhci_hcd ehci_hcd usbcore tpm_tis evdev i2c_i801
tpm processor snd_page_alloc mii sata_mv pcspkr serio_raw i2c_core
tpm_bios button thermal_sys nls_base ext4 mbcache jbd2 crc16 dm_mod
btrfs zlib_deflate crc32c libcrc32c sd_mod crc_t10dif ahci libahci
libata scsi_mod
[ 1448.190006] Pid: 0, comm: kworker/0:1 Not tainted 2.6.38-rc5-seblu #2
[ 1448.190007] Call Trace:
[ 1448.190009] <IRQ> [<ffffffff8103a198>] ? warn_slowpath_common+0x78/0x8c
[ 1448.190016] [<ffffffff8103a24b>] ? warn_slowpath_fmt+0x45/0x4a
[ 1448.190018] [<ffffffff8126a718>] ? netif_tx_lock+0x43/0x74
[ 1448.190019] [<ffffffff8126a84d>] ? dev_watchdog+0x104/0x1ad
[ 1448.190023] [<ffffffff8104e80f>] ? __queue_work+0x30c/0x334
[ 1448.190026] [<ffffffff81045ff3>] ? run_timer_softirq+0x1b8/0x27b
[ 1448.190028] [<ffffffff8126a749>] ? dev_watchdog+0x0/0x1ad
[ 1448.190031] [<ffffffff8105a726>] ? ktime_get+0x5f/0xb8
[ 1448.190034] [<ffffffff8103fa09>] ? __do_softirq+0xbd/0x194
[ 1448.190037] [<ffffffff810037cc>] ? call_softirq+0x1c/0x28
[ 1448.190040] [<ffffffff81004bb5>] ? do_softirq+0x31/0x63
[ 1448.190042] [<ffffffff8103f8b1>] ? irq_exit+0x36/0x7b
[ 1448.190045] [<ffffffff8101843c>] ? smp_apic_timer_interrupt+0x87/0x94
[ 1448.190047] [<ffffffff81003293>] ? apic_timer_interrupt+0x13/0x20
[ 1448.190048] <EOI> [<ffffffff81056250>] ? hrtimer_get_next_event+0x8a/0xa2
[ 1448.190056] [<ffffffffa0128d11>] ? acpi_idle_enter_c1+0x7b/0x95 [processor]
[ 1448.190059] [<ffffffffa0128cf1>] ? acpi_idle_enter_c1+0x5b/0x95 [processor]
[ 1448.190063] [<ffffffff8123c097>] ? cpuidle_idle_call+0x119/0x1c6
[ 1448.190065] [<ffffffff81001c9c>] ? cpu_idle+0xa5/0xdb
[ 1448.190068] [<ffffffff812ffd4a>] ? start_secondary+0x1ad/0x1b3
[ 1448.190070] ---[ end trace dd5a8baf361d8b47 ]---
>> I attached a dmesg output before reboot. Do you need anything else?
ok. next informations is about test with patch
r8169-TX-stop-queue-race-fix and r8169-debuginator on 2.6.38-rc7.
> Mostly :
> 1. .config
see attachment
> 2. the size of the udp packets and the mtu
eth0 mtu is 1500
iperf output:
Client connecting to bitume, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 122 KByte (default)
and icmp echo have size of 65500.
> As an option :
> 3. a few seconds of 'vmstat 1' from the host under test
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 3720532 11292 43292 0 0 15 2 2356 4160 0 1 98 1
0 0 0 3720524 11300 43316 0 0 0 20 21459 41637 0 2 97 2
1 0 0 3720420 11300 43316 0 0 0 0 20565 39521 1 1 98 0
0 0 0 3720588 11300 43316 0 0 0 0 21451 41569 0 1 99 0
0 0 0 3720492 11300 43316 0 0 0 0 21492 41441 0 1 98 0
0 0 0 3720564 11300 43316 0 0 0 0 21475 41435 0 2 98 0
1 0 0 3720500 11308 43320 0 0 0 16 21485 41675 1 1 97 1
1 0 0 3720556 11308 43320 0 0 0 0 17940 33773 0 1 99 0
0 0 0 3720588 11308 43320 0 0 0 0 9472 16384 0 1 100 0
0 0 0 3720660 11308 43320 0 0 0 0 8769 15217 1 1 99 0
0 0 0 3720484 11308 43320 0 0 0 0 10137 17766 0 1 99 0
2 0 0 3720524 11316 43324 0 0 0 16 9760 16907 0 1 98 2
0 0 0 3720492 11316 43324 0 0 0 0 9956 17138 0 1 99 0
0 0 0 3720492 11316 43324 0 0 0 0 9927 17220 0 0 99 0
1 0 0 3720524 11316 43324 0 0 0 0 10543 18502
0 0 100 0
1 0 0 3720596 11316 43324 0 0 0 0 9600 16409 0 1 99 0
0 0 0 3720564 11324 43324 0 0 0 60 9953 17339 0 1 98 1
0 0 0 3720460 11324 43324 0 0 0 0 9503 16583 0 0 99 0
0 0 0 3720684 11324 43324 0 0 0 0 9106 15637 0 1 99 0
0 0 0 3720460 11324 43324 0 0 0 0 9800 17019 0 1 99 0
0 0 0 3720492 11324 43324 0 0 0 0 9522 16559 0 0 99 0
0 0 0 3720428 11332 43328 0 0 0 28 9789 17106 0 1 97 2
0 0 0 3720460 11332 43328 0 0 0 0 10157 17717 0 1 99 0
0 0 0 3720564 11332 43328 0 0 0 0 9882 16940 0 0 100 0
0 0 0 3720492 11332 43328 0 0 0 0 9360 15991 0 0 100 0
0 0 0 3720492 11332 43328 0 0 0 0 10096 17436 0 0 99 0
0 0 0 3720556 11340 43328 0 0 0 28 9224 16028 0 1 97 2
0 0 0 3720500 11340 43328 0 0 0 0 9154 15942 0 1 99 0
1 0 0 3720524 11340 43328 0 0 0 0 10309 17781 0 1 99 0
> 4. an 'ethtool -s eth0' from the host under test
# ethtool -S eth0
NIC statistics:
tx_packets: 2598554
rx_packets: 35146015
tx_errors: 0
rx_errors: 0
rx_missed: 27571
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 35145983
broadcast: 24
multicast: 8
tx_aborted: 0
tx_underrun: 0
> 5. /proc/interrupts from the host under test
CPU0 CPU1 CPU2 CPU3
0: 40 1 0 2 IO-APIC-edge timer
1: 1 0 0 1 IO-APIC-edge i8042
8: 0 0 1 0 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 135 135 130 124 IO-APIC-fasteoi
sata_mv, uhci_hcd:usb4
18: 0 0 0 0 IO-APIC-fasteoi
ehci_hcd:usb1, uhci_hcd:usb6, uhci_hcd:usb9
19: 0 0 0 0 IO-APIC-fasteoi
uhci_hcd:usb8
21: 0 0 0 0 IO-APIC-fasteoi
uhci_hcd:usb5
23: 0 0 0 0 IO-APIC-fasteoi
ehci_hcd:usb3, uhci_hcd:usb7
41: 1196 1164 1206 1197 PCI-MSI-edge ahci
42: 0 0 0 1 PCI-MSI-edge xhci_hcd
43: 0 0 0 0 PCI-MSI-edge xhci_hcd
44: 0 0 0 0 PCI-MSI-edge xhci_hcd
45: 0 0 0 0 PCI-MSI-edge xhci_hcd
46: 0 0 0 0 PCI-MSI-edge xhci_hcd
47: 2517359 2517695 2517327 2517689 PCI-MSI-edge eth0
48: 0 0 0 0 PCI-MSI-edge i915
NMI: 64 64 63 61 Non-maskable interrupts
LOC: 189057 173286 183692 172647 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 64 64 63 61 Performance
monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 75 82 102 65 Rescheduling interrupts
CAL: 93 129 107 125 Function call interrupts
TLB: 105 111 238 322 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 5 5 5 5 Machine check polls
ERR: 3
MIS: 0
> 6. lspci -tv
-[0000:00]-+-00.0 Intel Corporation Core Processor DRAM Controller
+-01.0-[01]----00.0 Adaptec Serial ATA II RAID 1430SA
+-02.0 Intel Corporation Core Processor Integrated
Graphics Controller
+-16.0 Intel Corporation 5 Series/3400 Series Chipset HECI
Controller
+-1a.0 Intel Corporation 5 Series/3400 Series Chipset USB
Universal Host Controller
+-1a.1 Intel Corporation 5 Series/3400 Series Chipset USB
Universal Host Controller
+-1a.2 Intel Corporation 5 Series/3400 Series Chipset USB
Universal Host Controller
+-1a.7 Intel Corporation 5 Series/3400 Series Chipset USB2
Enhanced Host Controller
+-1c.0-[02]--
+-1c.3-[03]----00.0 NEC Corporation uPD720200 USB 3.0 Host
Controller
+-1c.5-[04]----00.0 Realtek Semiconductor Co., Ltd.
RTL8111/8168B PCI Express Gigabit Ethernet controller
+-1d.0 Intel Corporation 5 Series/3400 Series Chipset USB
Universal Host Controller
+-1d.1 Intel Corporation 5 Series/3400 Series Chipset USB
Universal Host Controller
+-1d.2 Intel Corporation 5 Series/3400 Series Chipset USB
Universal Host Controller
+-1d.7 Intel Corporation 5 Series/3400 Series Chipset USB2
Enhanced Host Controller
+-1e.0-[05]--
+-1f.0 Intel Corporation 5 Series Chipset LPC Interface Controller
+-1f.2 Intel Corporation 5 Series/3400 Series Chipset 6
port SATA AHCI Controller
\-1f.3 Intel Corporation 5 Series/3400 Series Chipset
SMBus Controller
>
> Can you apply the two attached patches on top of the previous ones and
> give it a try ? The debug should not be too verbose if things are stationary
> enough.
2.6.38-rc7 with your 2 previous patch change the game. No reboot. No
strange message in dmesg.
But somes sended packet are lost from some host. Example:
thunder ~ 1 # ping bitume
PING bitume.spn (192.168.242.4) 56(84) bytes of data.
64 bytes from bitume.spn (192.168.242.4): icmp_req=9 ttl=64 time=1005 ms
64 bytes from bitume.spn (192.168.242.4): icmp_req=10 ttl=64 time=0.538 ms
64 bytes from bitume.spn (192.168.242.4): icmp_req=12 ttl=64 time=0.817 ms
64 bytes from bitume.spn (192.168.242.4): icmp_req=15 ttl=64 time=0.639 ms
64 bytes from bitume.spn (192.168.242.4): icmp_req=20 ttl=64 time=5.32 ms
64 bytes from bitume.spn (192.168.242.4): icmp_req=22 ttl=64 time=0.866 ms
64 bytes from bitume.spn (192.168.242.4): icmp_req=26 ttl=64 time=0.607 ms
bitume ~ 0 # tcpdump -lni eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
00:23:45.713828 IP 192.168.242.5 > 192.168.242.4: ICMP echo request,
id 19486, seq 9, length 64
00:23:46.718479 IP 192.168.242.4 > 192.168.242.5: ICMP echo reply, id
19486, seq 9, length 64
00:23:46.718482 IP 192.168.242.4 > 192.168.242.5: ICMP ip reassembly
time exceeded, length 556
00:23:46.720291 IP 192.168.242.5 > 192.168.242.4: ICMP echo request,
id 19486, seq 10, length 64
00:23:46.720297 IP 192.168.242.4 > 192.168.242.5: ICMP echo reply, id
19486, seq 10, length 64
00:23:48.141013 IP 192.168.242.4 > 192.168.242.5: ICMP ip reassembly
time exceeded, length 556
00:23:48.722329 IP 192.168.242.5 > 192.168.242.4: ICMP echo request,
id 19486, seq 12, length 64
00:23:48.722335 IP 192.168.242.4 > 192.168.242.5: ICMP echo reply, id
19486, seq 12, length 64
00:23:51.741686 IP 192.168.242.5 > 192.168.242.4: ICMP echo request,
id 19486, seq 15, length 64
00:23:51.741692 IP 192.168.242.4 > 192.168.242.5: ICMP echo reply, id
19486, seq 15, length 64
00:23:56.778367 IP 192.168.242.5 > 192.168.242.4: ICMP echo request,
id 19486, seq 20, length 64
00:23:56.778373 IP 192.168.242.4 > 192.168.242.5: ICMP echo reply, id
19486, seq 20, length 64
00:23:57.988239 IP 192.168.242.2 > 192.168.242.4: ICMP ip reassembly
time exceeded, length 556
00:23:57.988243 IP 192.168.242.2 > 192.168.242.4: ICMP ip reassembly
time exceeded, length 556
00:23:57.988327 IP 192.168.242.2 > 192.168.242.4: ICMP ip reassembly
time exceeded, length 556
00:23:58.775425 IP 192.168.242.5 > 192.168.242.4: ICMP echo request,
id 19486, seq 22, length 64
00:23:58.775431 IP 192.168.242.4 > 192.168.242.5: ICMP echo reply, id
19486, seq 22, length 64
^C
17 packets captured
82 packets received by filter
0 packets dropped by kernel
This is maybe normal under stress, card discard packet after all.
> Do you have a serial cable and a second computer at hand by chance (don't
> go for netconsole with the r8169 driver) ?
I've a serial cable and a second computer, but my first computer
doesn't have a com port. Is it then possible?
Do you need more test?
--
Sébastien Luttringer
www.seblu.net
[-- Attachment #2: dmesg.2.6.38-rc7-seblu-patched.xz --]
[-- Type: application/x-xz, Size: 13516 bytes --]
[-- Attachment #3: config-2.6.38-rc7-seblu-patched.xz --]
[-- Type: application/x-xz, Size: 26984 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
2011-03-06 0:29 ` Seblu
@ 2011-03-10 12:08 ` Francois Romieu
0 siblings, 0 replies; 16+ messages in thread
From: Francois Romieu @ 2011-03-10 12:08 UTC (permalink / raw)
To: Seblu; +Cc: Eric Dumazet, lkml, netdev, Ivan Vecera
Seblu <seblu@seblu.net> :
[...]
> I catched the following trace during my previous torture session.
> Maybe it can help.
It's the usual r8169 TX timeout watchdog.
[...]
> > Can you apply the two attached patches on top of the previous ones and
> > give it a try ? The debug should not be too verbose if things are stationary
> > enough.
> 2.6.38-rc7 with your 2 previous patch change the game. No reboot. No
> strange message in dmesg.
?
"strange message" as :
[ ] netdev watchdog messages
[ ] 0001 0001 0001 0001 (or similar) message
[ ] net_ratelimit message
> But some sent packets are lost from some host. Example:
[...]
> This is maybe normal under stress, card discard packet after all.
It seems so. 0.08% packet loss. 10 ~ 20kpps (right ?). Sample at 0.1 Hz (ping).
[...]
> I've a serial cable and a second computer, but my first computer
> doesn't have a com port. Is it then possible?
Hardly. Forget it for now.
> Do you need more test?
1. 2.6.38-rc7 without the patches
2. 2.6.38-rc7 with the r8169.c driver of 2.6.38-rc5, without the patches
3. current setup + pktgen. Lower the packet size as long as it increases
the sender's pps.
I do not understand why the bug would be gone if it was in the r8169
proper.
--
Ueimor
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
@ 2011-02-21 11:56 Hans Nieser
0 siblings, 0 replies; 16+ messages in thread
From: Hans Nieser @ 2011-02-21 11:56 UTC (permalink / raw)
To: Francois Romieu; +Cc: netdev, linux-kernel
Francois Romieu wrote:
> Seblu <seblu@seblu.net> :
> [...]
> > I've applyed your patch on 2.6.38-rc5. Host have rebooted 2mn after udp start.
> > After this reboot, host is still on after 2 hour under a 1Gbit/s udp flow.
>
> Thanks for testing.
>
> > I attached a dmesg output before reboot. Do you need anything else?
>
> Mostly :
> 1. .config
> 2. the size of the udp packets and the mtu
>
> As an option :
> 3. a few seconds of 'vmstat 1' from the host under test
> 4. an 'ethtool -s eth0' from the host under test
> 5. /proc/interrupts from the host under test
> 6. lspci -tv
>
> Can you apply the two attached patches on top of the previous ones and
> give it a try ? The debug should not be too verbose if things are stationary
> enough.
>
<...>
Hi there, I just wanted to chime in on the discussion as I've been having similar
problems with similar hardware; I have a Gigabyte P55-USB3 motherboard
with an on-board Realtek NIC:
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:03:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
r8169 0000:03:00.0: setting latency timer to 64
r8169 0000:03:00.0: (unregistered net_device): unknown MAC, using family default
r8169 0000:03:00.0: irq 46 for MSI/MSI-X
r8169 0000:03:00.0: eth0: RTL8168b/8111b at 0xffffc9000001a000, 1c:6f:65:28:2f:2a, XID 0c100000 IRQ 46
A few days ago I noticed my machine had locked up while I was copying
some backup archives over the local gbit LAN over sftp. I then found out
that any kind of high-speed transfer to my machine would cause it to
lock up rather quickly (within seconds), wether that was via sftp, samba
or simply http (wget) from a webserver on my LAN. Slow(ish) transfers of
at most 120mbps don't seem to cause any issues, as I've been able to
download packages via my internet connection for updating my Gentoo
system for months without trouble.
I also found that on dmesg I would get hundreds of "r8169 0000:03:00.0:
eth0: link up" in the few seconds before my machine locks up (or
sometimes it just reboots - but never shutdowns unlike Sébastien).
I have managed to reproduce the hangs/reboots with the following
kernels:
2.6.38-rc5 (also including all three patches you posted in this thread)
2.6.37
2.6.36
With 2.6.36 it seems to take a bit longer to reproduce the hang/reboot
than it does with 2.6.37 and 2.6.38-rc5, and at some point I even got a
backtrace before it locked up (I suppose some stuff has scrolled off the
screen though, not sure how useful this is):
[<ffffffff814a3f8f>] page_fault+0x1f/0x30
[<ffffffff812c529a>] ? ahci_interrupt+0xea/0x700
[<ffffffff813b6901>] ? skb_checksum+0x51/0x2f0
[<ffffffff8108006a>] handle_IRQ_event+0x3a/0xd0
[<ffffffff8108211e>] handle_edge_irq+0xbe/0x170
[<ffffffff810052cd>] handle_irq+0x1d/0x30
[<ffffffff810047e7>] do_IRQ+0x67/0xf0
[<ffffffff814a3d53>] ret_from_intr+0x0/0xa
[<ffffffff8120110b>] ? memcpy+0xb/0xb0
[<ffffffff8120ce7e>] ? swiotlb_bounce+0x1e/0x40
[<ffffffff8120cedb>] ? swiotlb_tbl_sync_single+0x3b/0x70
[<ffffffff8120cf6b>] ? swiotlb_sync_single+0x5b/0x80
[<ffffffff8120d08c>] ? swiotlb_sync_single_for_cpu+0xc/0x10
[<ffffffff812c85da>] ? rtl8169_rx_interrupt+0x25a/0x550
[<ffffffff81046c9d>] ? update_process_times+0x5d/0x70
[<ffffffff812cb828>] ? rtl8169_poll+0x38/0x260
[<ffffffff813c0f0e>] ? net_rx_action+0x8e/0x1a0
[<ffffffff812caab1>] ? rtl8169_interrupt+0x101/0x350
[<ffffffff810404a6>] ? __do_softirq+0xa6/0x130
[<ffffffff8100320c>] ? call_softirq+0x1c/0x30
[<ffffffff8100527d>] ? do_softirq+0x4d/0x80
[<ffffffff8103fdad>] ? irq_exit+0x4d/0x50
[<ffffffff810047f0>] ? do_IRQ+0x70/0xf0
[<ffffffff814a3d53>] ? ret_from_intr+0x0/0xa
<EOI>
(I had to manually type this over so there may be typos in there)
On all the kernel versions on which I was able to reproduce the problem
my transer speed was also much slower than expected; somewhere around
10-20MiB/s (it seems to start out at 20MiB/s, then go down a bit to
<10MiB/s before the machine finally locks up, or sometimes the reverse
of this).
I was not able to reproduce the problem on 2.6.35.9, and managed to get
consistent transfer speeds of around 107MiB/s (using wget) with that
kernel. While I haven't spent too much time trying to reproduce it (just
a couple dozen of transfers of a 1GB file), at the very least it is much
harder to reproduce than on the newer kernels. There were also much less
'link up' messages on dmesg with this kernel, just one every few seconds
instead of dozens per second.
I'm not sure if it's worth the effort to try and git bisect between
2.6.35 and 2.6.36, but let me know if you think it is and I'll give it a
shot.
One other thing I observed (not sure if it's relevant, but just in case)
was that for all the kernels that I was able to reproduce the problem
with, the MSI irq was 46, while with 2.6.35.9 the MSI irq was 50.
I'll spend some more time this evening or tomorrow doing some more
testing and getting the other things you requested from Sébastien if you
think that useful to know in my case as well
Here is at least the output of lspci -tv:
lspci -tv:
-[0000:00]-+-00.0 Intel Corporation Core Processor DMI
+-03.0-[01]--+-00.0 ATI Technologies Inc Cypress [Radeon HD 5800 Series]
| \-00.1 ATI Technologies Inc Cypress HDMI Audio [Radeon HD 5800 Series]
+-08.0 Intel Corporation Core Processor System Management Registers
+-08.1 Intel Corporation Core Processor Semaphore and Scratchpad Registers
+-08.2 Intel Corporation Core Processor System Control and Status Registers
+-08.3 Intel Corporation Core Processor Miscellaneous Registers
+-10.0 Intel Corporation Core Processor QPI Link
+-10.1 Intel Corporation Core Processor QPI Routing and Protocol Registers
+-1a.0 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1a.1 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1a.2 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1a.7 Intel Corporation 5 Series/3400 Series Chipset USB2 Enhanced Host Controller
+-1b.0 Intel Corporation 5 Series/3400 Series Chipset High Definition Audio
+-1c.0-[02]--+-00.0 JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller
| \-00.1 JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller
+-1c.1-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
+-1c.2-[04]----00.0 NEC Corporation Device 0194
+-1d.0 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1d.1 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1d.2 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1d.3 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1d.7 Intel Corporation 5 Series/3400 Series Chipset USB2 Enhanced Host Controller
+-1e.0-[05]----04.0 Texas Instruments TSB12LV23 IEEE-1394 Controller
+-1f.0 Intel Corporation 5 Series Chipset LPC Interface Controller
+-1f.2 Intel Corporation 5 Series/3400 Series Chipset 6 port SATA AHCI Controller
\-1f.3 Intel Corporation 5 Series/3400 Series Chipset SMBus Controller
and lspci -vvxxx for my device (the motherboard reported is incorrect, it's definitely a GA-P55-USB3):
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 46
Region 0: I/O ports at de00 [size=256]
Region 2: Memory at fbeff000 (64-bit, prefetchable) [size=4K]
Region 4: Memory at fbef8000 (64-bit, prefetchable) [size=16K]
[virtual] Expansion ROM at fbe00000 [disabled] [size=128K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee0f00c Data: 4189
Capabilities: [70] Express (v2) Endpoint, MSI 01
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00000800
Capabilities: [d0] Vital Product Data
Unknown small resource type 00, will not decode more.
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [160 v1] Device Serial Number 12-34-56-78-12-34-56-78
Kernel driver in use: r8169
00: ec 10 68 81 07 04 10 00 06 00 00 02 10 00 00 00
10: 01 de 00 00 00 00 00 00 0c f0 ef fb 00 00 00 00
20: 0c 80 ef fb 00 00 00 00 00 00 00 00 58 14 00 e0
30: 00 00 00 00 40 00 00 00 00 00 00 00 0b 01 00 00
40: 01 50 c3 ff 08 00 00 00 00 00 00 00 00 00 00 00
50: 05 70 81 00 0c f0 e0 fe 00 00 00 00 89 41 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 b0 02 02 c1 8c 28 00 10 50 11 00 11 3c 07 00
80: 40 00 11 10 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 11 d0 03 00 04 00 00 00 04 08 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 03 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
@ 2011-02-22 17:33 Hans Nieser
2011-02-23 9:55 ` Francois Romieu
0 siblings, 1 reply; 16+ messages in thread
From: Hans Nieser @ 2011-02-22 17:33 UTC (permalink / raw)
To: Francois Romieu; +Cc: netdev, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 4140 bytes --]
Francois Romieu wrote:
> Seblu <seblu@seblu.net> :
> [...]
> > I've applyed your patch on 2.6.38-rc5. Host have rebooted 2mn after udp start.
> > After this reboot, host is still on after 2 hour under a 1Gbit/s udp flow.
>
> Thanks for testing.
>
> > I attached a dmesg output before reboot. Do you need anything else?
>
> Mostly :
> 1. .config
> 2. the size of the udp packets and the mtu
>
> As an option :
> 3. a few seconds of 'vmstat 1' from the host under test
> 4. an 'ethtool -s eth0' from the host under test
> 5. /proc/interrupts from the host under test
> 6. lspci -tv
>
> Can you apply the two attached patches on top of the previous ones and
> give it a try ? The debug should not be too verbose if things are stationary
> enough.
With your patches applied to 2.6.38-rc6, I have gathered some of the
info you requested from Seblu as well, I hope it's helpful:
1: see attachment
2: I'm not sure how to check the size of the packets, but I'm just
fetching a (large) file over http/tcp, so I guess they are mostly of the
size of my MTU which is 1500 looking at ifconfig output
For the other vmstat/ethtool/interrupts output, I started the following
commands remotely via ssh a second or two before starting the download,
and the machine locked up a few seconds later:
# vmstat 1
# while true; do echo .; ethtool -S eth0; sleep 0.5; done
(I assume you actually meant the -S option to ethtool btw, -s seemed to
be a noop)
# while true; do echo .; cat /proc/interrupts; sleep 0.5; done
I've attached the outputs of the above commands
I also included dmesg output generated during the transfer (the actual
transfer starts at the 256~ printk timestamp I believe), the dmesg is
probably missing a second or so at the end before machine locked up
6:
-[0000:00]-+-00.0 Intel Corporation Core Processor DMI
+-03.0-[01]--+-00.0 ATI Technologies Inc Cypress [Radeon HD 5800 Series]
| \-00.1 ATI Technologies Inc Cypress HDMI Audio [Radeon HD 5800 Series]
+-08.0 Intel Corporation Core Processor System Management Registers
+-08.1 Intel Corporation Core Processor Semaphore and Scratchpad Registers
+-08.2 Intel Corporation Core Processor System Control and Status Registers
+-08.3 Intel Corporation Core Processor Miscellaneous Registers
+-10.0 Intel Corporation Core Processor QPI Link
+-10.1 Intel Corporation Core Processor QPI Routing and Protocol Registers
+-1a.0 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1a.1 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1a.2 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1a.7 Intel Corporation 5 Series/3400 Series Chipset USB2 Enhanced Host Controller
+-1b.0 Intel Corporation 5 Series/3400 Series Chipset High Definition Audio
+-1c.0-[02]--+-00.0 JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller
| \-00.1 JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller
+-1c.1-[03]----00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller
+-1c.2-[04]----00.0 NEC Corporation Device 0194
+-1d.0 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1d.1 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1d.2 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1d.3 Intel Corporation 5 Series/3400 Series Chipset USB Universal Host Controller
+-1d.7 Intel Corporation 5 Series/3400 Series Chipset USB2 Enhanced Host Controller
+-1e.0-[05]----04.0 Texas Instruments TSB12LV23 IEEE-1394 Controller
+-1f.0 Intel Corporation 5 Series Chipset LPC Interface Controller
+-1f.2 Intel Corporation 5 Series/3400 Series Chipset 6 port SATA AHCI Controller
\-1f.3 Intel Corporation 5 Series/3400 Series Chipset SMBus Controller
[-- Attachment #2: 2.6.38-rc6-git-config.gz --]
[-- Type: application/x-gzip, Size: 13087 bytes --]
[-- Attachment #3: debug-dmesg --]
[-- Type: text/plain, Size: 12788 bytes --]
[ 7.362642] Freeing unused kernel memory: 432k freed
[ 9.800495] udev[1095]: starting version 164
[ 12.317466] EXT3-fs (sda6): using internal journal
[ 13.072807] EXT4-fs (sdb2): warning: maximal mount count reached, running e2fsck is recommended
[ 13.073229] EXT4-fs (sdb2): mounted filesystem with ordered data mode. Opts: (null)
[ 13.109537] EXT4-fs (sdb3): warning: maximal mount count reached, running e2fsck is recommended
[ 13.125235] EXT4-fs (sdb3): mounted filesystem with ordered data mode. Opts: (null)
[ 14.710943] Adding 4192928k swap on /dev/sda5. Priority:-1 extents:1 across:4192928k
[ 15.764690] r8169 0000:03:00.0: eth0: link down
[ 15.764695] 0020 0000 0000 0000
[ 15.764702] r8169 0000:03:00.0: eth0: link down
[ 15.765151] ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 19.020958] 0020 0020 0000 0000
[ 19.020970] r8169 0000:03:00.0: eth0: link up
[ 19.021554] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 19.023429] 0020 0020 0084 0000
[ 19.130371] 0020 0020 0084 0084
[ 19.753752] 0084 0020 0084 0084
[ 20.131392] 0084 0084 0084 0084
[ 20.131480] 0084 0084 0001 0084
[ 20.131508] 0084 0084 0001 0084
[ 20.145422] 0001 0084 0001 0084
[ 20.947682] net_ratelimit: 14 callbacks suppressed
[ 20.947686] 0001 0084 0001 0084
[ 20.961562] 0001 0084 0001 0084
[ 21.566002] 0001 0084 0001 0084
[ 21.946693] 0001 0084 0084 0084
[ 21.956978] 0001 0084 0084 0001
[ 22.945712] 0084 0084 0084 0001
[ 22.954632] 0084 0001 0084 0001
[ 22.957817] 0084 0001 0001 0001
[ 23.944726] 0084 0001 0001 0084
[ 23.955658] 0001 0001 0001 0084
[ 26.397322] net_ratelimit: 4 callbacks suppressed
[ 26.397324] 0001 0084 0084 0001
[ 26.941849] 0001 0084 0084 0001
[ 26.952315] 0001 0084 0084 0001
[ 26.952516] CE: hpet4 increased min_delta_ns to 7500 nsec
[ 26.952521] CE: hpet4 increased min_delta_ns to 11250 nsec
[ 26.952544] CE: hpet3 increased min_delta_ns to 7500 nsec
[ 26.952549] CE: hpet3 increased min_delta_ns to 11250 nsec
[ 27.940861] 0084 0084 0084 0001
[ 27.940878] 0084 0084 0084 0001
[ 27.951797] 0084 0084 0001 0001
[ 27.952032] 0084 0084 0001 0001
[ 28.939867] 0004 0084 0001 0001
[ 28.939870] 0004 0080 0001 0001
[ 28.948839] 0004 0080 0001 0001
[ 32.936012] net_ratelimit: 6 callbacks suppressed
[ 32.936014] 0001 0084 0081 0001
[ 32.945916] 0001 0084 0001 0001
[ 34.934079] 0001 0084 0001 0084
[ 34.942371] 0001 0084 0001 0084
[ 37.810633] 0001 0001 0001 0084
[ 37.812208] 0001 0001 0084 0084
[ 37.812268] 0001 0001 0084 0001
[ 37.812298] 0084 0001 0084 0001
[ 37.812385] 0084 0001 0084 0001
[ 37.815847] 0084 0001 0084 0001
[ 37.951150] net_ratelimit: 17 callbacks suppressed
[ 37.951152] 0084 0001 0084 0001
[ 38.100181] 0084 0084 0084 0001
[ 38.120969] 0084 0084 0001 0001
[ 38.120998] 0084 0084 0001 0084
[ 38.127134] 0084 0084 0001 0084
[ 38.127817] 0084 0001 0001 0084
[ 38.167842] 0084 0001 0084 0084
[ 38.296883] 0084 0001 0084 0084
[ 38.305463] 0001 0001 0084 0084
[ 38.305746] 0001 0084 0084 0084
[ 47.924331] net_ratelimit: 86 callbacks suppressed
[ 47.924336] 0001 0001 0084 0001
[ 47.924449] 0001 0001 0084 0001
[ 47.927575] 0001 0001 0084 0001
[ 47.927721] 0001 0001 0084 0001
[ 48.116315] 0001 0001 0084 0001
[ 48.118212] 0001 0084 0084 0001
[ 48.118349] 0001 0084 0001 0001
[ 48.120608] 0001 0084 0001 0084
[ 48.120668] 0084 0084 0001 0084
[ 48.120765] 0084 0001 0001 0084
[ 61.863340] net_ratelimit: 52 callbacks suppressed
[ 61.863344] 0001 0001 0001 0084
[ 61.863373] 0001 0001 0001 0084
[ 61.863500] 0001 0001 0001 0084
[ 61.866560] 0001 0084 0001 0084
[ 61.866630] 0001 0084 0001 0084
[ 61.866890] 0001 0084 0001 0001
[ 61.867048] 0084 0084 0001 0001
[ 61.867533] 0084 0084 0001 0001
[ 61.867582] 0084 0084 0001 0001
[ 61.906994] 0084 0084 0001 0084
[ 67.379512] net_ratelimit: 58 callbacks suppressed
[ 67.379517] 0084 0001 0001 0084
[ 67.742456] 0084 0001 0001 0001
[ 67.745380] 0084 0001 0001 0001
[ 67.786305] 0084 0001 0001 0001
[ 68.117021] 0084 0001 0001 0001
[ 68.120059] 0084 0001 0001 0084
[ 68.120169] 0001 0001 0001 0084
[ 68.309914] 0001 0001 0001 0084
[ 68.311914] 0001 0001 0084 0084
[ 68.311932] 0001 0001 0084 0001
[ 73.466822] net_ratelimit: 35 callbacks suppressed
[ 73.466827] 0001 0001 0001 0001
[ 73.466861] 0084 0001 0001 0001
[ 73.466982] 0084 0001 0001 0001
[ 73.470317] 0084 0001 0084 0001
[ 73.470379] 0084 0001 0084 0001
[ 73.470663] 0001 0001 0084 0001
[ 73.470827] 0001 0084 0084 0001
[ 73.471316] 0001 0084 0084 0001
[ 73.471337] 0001 0084 0084 0001
[ 73.510796] 0084 0084 0084 0001
[ 78.515173] net_ratelimit: 97 callbacks suppressed
[ 78.515178] 0084 0001 0001 0001
[ 78.515269] 0084 0001 0001 0084
[ 78.515411] 0001 0001 0001 0084
[ 78.642063] 0001 0001 0001 0084
[ 78.643693] 0001 0001 0084 0084
[ 78.643863] 0001 0001 0084 0001
[ 78.666617] 0084 0001 0084 0001
[ 78.666784] 0084 0001 0084 0001
[ 78.667760] 0084 0001 0084 0001
[ 78.668114] 0084 0001 0084 0001
[ 83.997640] net_ratelimit: 40 callbacks suppressed
[ 83.997644] 0001 0001 0084 0001
[ 84.460476] 0001 0001 0084 0001
[ 84.462195] 0001 0001 0084 0001
[ 84.462433] 0001 0001 0084 0001
[ 84.696262] 0001 0001 0084 0001
[ 84.698921] 0001 0084 0084 0001
[ 84.699251] 0001 0084 0001 0001
[ 84.725252] 0001 0084 0001 0001
[ 84.727891] 0084 0084 0001 0001
[ 84.754215] 0084 0001 0001 0001
[ 89.468722] net_ratelimit: 125 callbacks suppressed
[ 89.468727] 0084 0001 0084 0001
[ 89.471309] 0084 0001 0084 0001
[ 89.499664] 0084 0001 0084 0001
[ 89.502279] 0084 0001 0084 0001
[ 89.527636] 0084 0001 0084 0001
[ 89.530254] 0084 0001 0084 0001
[ 89.555624] 0084 0001 0084 0001
[ 89.557291] 0084 0001 0084 0001
[ 89.585584] 0084 0001 0084 0001
[ 89.588197] 0084 0001 0084 0001
[ 95.393176] net_ratelimit: 30 callbacks suppressed
[ 95.393181] 0001 0001 0084 0001
[ 95.395565] 0084 0001 0084 0001
[ 95.395901] 0084 0001 0084 0001
[ 95.875207] 0084 0001 0084 0001
[ 95.888142] 0084 0001 0084 0001
[ 96.874263] 0084 0001 0084 0001
[ 96.882576] 0084 0001 0084 0001
[ 97.873300] 0084 0001 0084 0001
[ 97.883204] 0084 0001 0084 0001
[ 97.885801] 0001 0001 0084 0001
[ 101.655799] 0001 0001 0084 0001
[ 104.495510] 0001 0001 0001 0001
[ 104.497779] 0001 0001 0001 0084
[ 104.498115] 0001 0001 0001 0084
[ 107.863176] 0001 0001 0001 0084
[ 110.408705] 0001 0001 0001 0084
[ 110.411139] 0001 0001 0001 0084
[ 110.411278] 0001 0001 0001 0084
[ 112.476458] 0001 0001 0001 0084
[ 121.637630] 0001 0001 0001 0084
[ 127.431241] 0001 0001 0001 0001
[ 129.074757] 0001 0001 0001 0001
[ 129.077067] 0001 0084 0001 0001
[ 129.117642] 0001 0084 0001 0001
[ 129.194754] 0001 0084 0001 0001
[ 129.198026] 0084 0084 0001 0001
[ 129.198126] 0084 0001 0001 0001
[ 129.266678] 0084 0001 0001 0001
[ 129.269954] 0084 0001 0001 0084
[ 132.439479] net_ratelimit: 418 callbacks suppressed
[ 132.439484] 0001 0084 0001 0084
[ 132.439563] 0001 0084 0001 0084
[ 132.441741] 0084 0084 0001 0084
[ 132.444111] 0084 0004 0001 0084
[ 132.444119] 0084 0004 0004 0084
[ 132.444130] 0084 0004 0004 0084
[ 132.444281] 0004 0004 0004 0084
[ 132.444292] 0004 0004 0004 0084
[ 132.444302] 0004 0004 0084 0084
[ 132.444440] 0004 0004 0084 0001
[ 137.530889] net_ratelimit: 118 callbacks suppressed
[ 137.530894] 0084 0001 0001 0001
[ 143.936457] 0084 0001 0001 0001
[ 149.702560] 0001 0001 0001 0001
[ 151.610378] 0001 0001 0001 0001
[ 151.610410] 0001 0001 0084 0001
[ 161.601436] 0001 0001 0084 0001
[ 162.810531] 0084 0001 0084 0001
[ 162.818457] 0084 0001 0084 0001
[ 163.809576] 0084 0001 0084 0001
[ 163.819093] 0084 0001 0084 0001
[ 165.807641] 0084 0001 0084 0001
[ 165.819746] 0084 0001 0084 0001
[ 165.826577] 0084 0001 0001 0001
[ 165.828636] 0084 0001 0001 0084
[ 165.828747] 0001 0001 0001 0084
[ 169.528058] net_ratelimit: 37 callbacks suppressed
[ 169.528062] 0001 0001 0084 0084
[ 169.528399] 0001 0001 0084 0001
[ 169.949627] 0001 0001 0084 0001
[ 169.953609] 0001 0084 0084 0001
[ 169.953737] 0001 0084 0001 0001
[ 169.989627] 0001 0084 0001 0001
[ 169.992489] 0084 0084 0001 0001
[ 170.033484] 0084 0001 0001 0001
[ 170.621022] 0084 0001 0001 0001
[ 170.621114] 0084 0001 0001 0084
[ 175.207674] net_ratelimit: 28 callbacks suppressed
[ 175.207678] 0001 0001 0084 0001
[ 175.209525] 0001 0084 0084 0001
[ 175.209648] 0001 0084 0001 0001
[ 175.304530] 0001 0084 0001 0001
[ 175.307443] 0084 0084 0001 0001
[ 175.307573] 0084 0001 0001 0001
[ 175.488392] 0084 0001 0001 0001
[ 175.490060] 0084 0001 0001 0084
[ 175.490135] 0001 0001 0001 0084
[ 175.491155] 0001 0084 0001 0084
[ 181.583322] net_ratelimit: 43 callbacks suppressed
[ 181.583326] 0001 0001 0004 0080
[ 183.413557] 0001 0001 0001 0080
[ 183.414571] 0001 0001 0001 0084
[ 183.454717] 0001 0001 0001 0084
[ 183.470421] 0001 0001 0001 0084
[ 183.471511] 0001 0001 0084 0084
[ 183.471842] 0001 0001 0084 0001
[ 183.525445] 0001 0001 0084 0001
[ 183.526486] 0001 0084 0084 0001
[ 183.526789] 0001 0084 0001 0001
[ 191.574104] net_ratelimit: 25 callbacks suppressed
[ 191.574109] 0001 0084 0001 0001
[ 193.780628] 0001 0001 0001 0001
[ 193.780660] 0001 0001 0084 0001
[ 193.905505] 0001 0001 0084 0001
[ 194.357981] 0001 0001 0084 0001
[ 194.358019] 0001 0084 0084 0001
[ 194.358114] 0001 0084 0001 0001
[ 194.361314] 0001 0084 0001 0084
[ 194.361368] 0001 0084 0001 0084
[ 194.361519] 0001 0001 0001 0084
[ 198.806025] net_ratelimit: 44 callbacks suppressed
[ 198.806030] 0084 0001 0001 0001
[ 198.806121] 0084 0001 0001 0084
[ 198.806215] 0001 0001 0001 0084
[ 199.085823] 0001 0001 0001 0084
[ 199.085916] 0001 0001 0084 0084
[ 199.086042] 0001 0001 0084 0001
[ 199.150748] 0001 0001 0084 0001
[ 199.150839] 0001 0084 0084 0001
[ 199.150936] 0001 0084 0001 0001
[ 199.397526] 0001 0084 0001 0001
[ 203.823428] net_ratelimit: 46 callbacks suppressed
[ 203.823432] 0084 0001 0001 0001
[ 203.823806] 0084 0001 0001 0084
[ 203.824129] 0001 0001 0001 0084
[ 203.875381] 0001 0001 0001 0084
[ 203.878773] 0001 0001 0001 0084
[ 203.879824] 0001 0001 0001 0084
[ 203.880166] 0001 0001 0001 0084
[ 203.935719] 0001 0001 0001 0084
[ 203.936733] 0001 0001 0084 0084
[ 203.937087] 0001 0001 0084 0001
[ 209.641493] net_ratelimit: 63 callbacks suppressed
[ 209.641498] 0001 0084 0001 0001
[ 211.555931] 0001 0084 0001 0001
[ 211.555967] 0001 0084 0001 0001
[ 220.007824] 0001 0084 0001 0001
[ 220.008175] 0001 0084 0001 0001
[ 221.546824] 0001 0084 0001 0001
[ 222.532390] 0001 0001 0001 0001
[ 222.534784] 0001 0001 0084 0001
[ 222.534919] 0001 0001 0084 0001
[ 222.629283] 0001 0001 0084 0001
[ 222.630632] 0001 0084 0084 0001
[ 222.630737] 0001 0084 0001 0001
[ 222.708212] 0001 0084 0001 0001
[ 225.609443] net_ratelimit: 90 callbacks suppressed
[ 225.609447] 0001 0084 0001 0084
[ 225.609723] 0001 0084 0001 0084
[ 225.610782] 0084 0084 0001 0084
[ 225.651247] 0084 0001 0001 0084
[ 225.846189] 0084 0001 0001 0084
[ 225.848584] 0084 0001 0001 0084
[ 225.848876] 0001 0001 0001 0084
[ 225.875187] 0001 0001 0001 0084
[ 225.877592] 0001 0001 0084 0084
[ 225.905157] 0001 0001 0084 0001
[ 231.743925] net_ratelimit: 112 callbacks suppressed
[ 231.743930] 0084 0001 0084 0001
[ 231.756830] 0084 0001 0084 0001
[ 232.354338] 0084 0001 0001 0001
[ 234.741003] 0084 0001 0001 0084
[ 234.753403] 0001 0001 0001 0084
[ 239.743146] 0001 0084 0001 0084
[ 239.743220] 0001 0084 0001 0084
[ 241.528701] 0001 0084 0001 0001
[ 247.531915] 0001 0084 0001 0001
[ 251.519734] 0001 0001 0001 0001
[ 253.844485] 0001 0001 0001 0001
[ 253.962372] 0001 0001 0001 0001
[ 253.964335] 0084 0001 0001 0001
[ 253.964446] 0084 0001 0001 0001
[ 253.965448] 0084 0001 0084 0001
[ 253.965735] 0084 0001 0084 0001
[ 254.492895] 0084 0001 0084 0001
[ 254.492913] 0084 0001 0084 0001
[ 256.649743] net_ratelimit: 46 callbacks suppressed
[ 256.649748] 0084 0004 0080 0001
[ 256.649818] 0084 0001 0080 0001
[ 256.650744] 0084 0001 0084 0001
[ 256.651019] 0084 0001 0084 0001
[ 257.001333] 0084 0001 0084 0001
[ 257.001431] 0084 0001 0084 0001
[ 257.002568] 0084 0001 0084 0001
[ 257.002839] 0084 0001 0084 0001
[ 257.151255] 0084 0001 0084 0001
[ 257.151377] 0084 0001 0084 0001
[ 261.645823] net_ratelimit: 17905 callbacks suppressed
[ 261.645828] 0001 0001 0084 0001
[ 261.645862] 0084 0001 0084 0001
[ 261.646133] 0084 0001 0084 0001
[ 261.646171] 0084 0001 0084 0001
[ 261.646336] 0084 0001 0084 0001
[ 261.646423] 0001 0001 0084 0001
[ 261.646460] 0001 0084 0084 0001
[ 261.646627] 0001 0084 0001 0001
[ 261.646799] 0001 0084 0001 0001
[ 261.646818] 0084 0084 0001 0001
[-- Attachment #4: interrupts --]
[-- Type: text/plain, Size: 51473 bytes --]
# while true; do echo .; cat /proc/interrupts; sleep 0.5; done
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 253857 0 0 0 HPET_MSI-edge hpet2
41: 0 253814 0 0 HPET_MSI-edge hpet3
42: 0 0 253746 0 HPET_MSI-edge hpet4
43: 0 0 0 253646 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5847 0 0 0 PCI-MSI-edge ahci
47: 0 1822 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 115 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 254388 0 0 0 HPET_MSI-edge hpet2
41: 0 254343 0 0 HPET_MSI-edge hpet3
42: 0 0 254275 0 HPET_MSI-edge hpet4
43: 0 0 0 254175 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5850 0 0 0 PCI-MSI-edge ahci
47: 0 1828 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 115 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 254891 0 0 0 HPET_MSI-edge hpet2
41: 0 254845 0 0 HPET_MSI-edge hpet3
42: 0 0 254777 0 HPET_MSI-edge hpet4
43: 0 0 0 254677 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5862 0 0 0 PCI-MSI-edge ahci
47: 0 1835 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 115 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 255394 0 0 0 HPET_MSI-edge hpet2
41: 0 255348 0 0 HPET_MSI-edge hpet3
42: 0 0 255280 0 HPET_MSI-edge hpet4
43: 0 0 0 255181 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5864 0 0 0 PCI-MSI-edge ahci
47: 0 1847 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 115 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 255896 0 0 0 HPET_MSI-edge hpet2
41: 0 255850 0 0 HPET_MSI-edge hpet3
42: 0 0 255784 0 HPET_MSI-edge hpet4
43: 0 0 0 255683 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5876 0 0 0 PCI-MSI-edge ahci
47: 0 1857 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 115 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 256399 0 0 0 HPET_MSI-edge hpet2
41: 0 256352 0 0 HPET_MSI-edge hpet3
42: 0 0 256287 0 HPET_MSI-edge hpet4
43: 0 0 0 256185 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5876 0 0 0 PCI-MSI-edge ahci
47: 0 1868 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 115 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 256901 0 0 0 HPET_MSI-edge hpet2
41: 0 256854 0 0 HPET_MSI-edge hpet3
42: 0 0 256791 0 HPET_MSI-edge hpet4
43: 0 0 0 256687 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5876 0 0 0 PCI-MSI-edge ahci
47: 0 1877 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 115 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 257404 0 0 0 HPET_MSI-edge hpet2
41: 0 257356 0 0 HPET_MSI-edge hpet3
42: 0 0 257294 0 HPET_MSI-edge hpet4
43: 0 0 0 257190 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5890 0 0 0 PCI-MSI-edge ahci
47: 0 2067 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 115 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 257906 0 0 0 HPET_MSI-edge hpet2
41: 0 257859 0 0 HPET_MSI-edge hpet3
42: 0 0 257797 0 HPET_MSI-edge hpet4
43: 0 0 0 257692 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5902 0 0 0 PCI-MSI-edge ahci
47: 0 4019 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 115 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 258409 0 0 0 HPET_MSI-edge hpet2
41: 0 258361 0 0 HPET_MSI-edge hpet3
42: 0 0 258300 0 HPET_MSI-edge hpet4
43: 0 0 0 258195 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5902 0 0 0 PCI-MSI-edge ahci
47: 0 6806 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 115 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 258911 0 0 0 HPET_MSI-edge hpet2
41: 0 258863 0 0 HPET_MSI-edge hpet3
42: 0 0 258803 0 HPET_MSI-edge hpet4
43: 0 0 0 258698 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5902 0 0 0 PCI-MSI-edge ahci
47: 0 9302 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 115 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 259413 0 0 0 HPET_MSI-edge hpet2
41: 0 259365 0 0 HPET_MSI-edge hpet3
42: 0 0 259307 0 HPET_MSI-edge hpet4
43: 0 0 0 259201 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5902 0 0 0 PCI-MSI-edge ahci
47: 0 12494 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 115 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 259916 0 0 0 HPET_MSI-edge hpet2
41: 0 259868 0 0 HPET_MSI-edge hpet3
42: 0 0 259811 0 HPET_MSI-edge hpet4
43: 0 0 0 259704 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5919 0 0 0 PCI-MSI-edge ahci
47: 0 13625 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 115 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 260419 0 0 0 HPET_MSI-edge hpet2
41: 0 260370 0 0 HPET_MSI-edge hpet3
42: 0 0 260314 0 HPET_MSI-edge hpet4
43: 0 0 0 260207 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5919 0 0 0 PCI-MSI-edge ahci
47: 0 13748 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 116 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 260921 0 0 0 HPET_MSI-edge hpet2
41: 0 260872 0 0 HPET_MSI-edge hpet3
42: 0 0 260816 0 HPET_MSI-edge hpet4
43: 0 0 0 260711 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5919 0 0 0 PCI-MSI-edge ahci
47: 0 15062 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 116 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 261424 0 0 0 HPET_MSI-edge hpet2
41: 0 261375 0 0 HPET_MSI-edge hpet3
42: 0 0 261320 0 HPET_MSI-edge hpet4
43: 0 0 0 261214 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5919 0 0 0 PCI-MSI-edge ahci
47: 0 17204 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 116 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
.
CPU0 CPU1 CPU2 CPU3
0: 116 0 0 1 IO-APIC-edge timer
1: 0 0 0 2 IO-APIC-edge i8042
4: 0 0 0 2 IO-APIC-edge
8: 0 0 0 81 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
16: 0 0 0 0 IO-APIC-fasteoi ahci, uhci_hcd:usb3, uhci_hcd:usb9
17: 0 0 0 0 IO-APIC-fasteoi pata_jmicron
18: 0 0 0 269 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb5, uhci_hcd:usb8
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb7
21: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
23: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6
40: 261926 0 0 0 HPET_MSI-edge hpet2
41: 0 261877 0 0 HPET_MSI-edge hpet3
42: 0 0 261824 0 HPET_MSI-edge hpet4
43: 0 0 0 261716 HPET_MSI-edge hpet5
45: 0 0 0 611 PCI-MSI-edge radeon
46: 5931 0 0 0 PCI-MSI-edge ahci
47: 0 20063 0 0 PCI-MSI-edge eth0
48: 0 0 1 0 PCI-MSI-edge xhci_hcd
49: 0 0 0 0 PCI-MSI-edge xhci_hcd
50: 0 0 0 0 PCI-MSI-edge xhci_hcd
51: 0 0 0 0 PCI-MSI-edge xhci_hcd
52: 0 0 0 0 PCI-MSI-edge xhci_hcd
53: 0 196 0 0 PCI-MSI-edge hda_intel
54: 0 0 28 0 PCI-MSI-edge hda_intel
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 268 229 137 45 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 70 43 38 32 Rescheduling interrupts
CAL: 29 60 51 52 Function call interrupts
TLB: 212 169 116 9 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 2 2 2 Machine check polls
ERR: 3
MIS: 0
[-- Attachment #5: ethtool-stats --]
[-- Type: text/plain, Size: 4706 bytes --]
# while true; do echo .; ethtool -S eth0; sleep 0.5; done
.
NIC statistics:
tx_packets: 1068
rx_packets: 879
tx_errors: 0
rx_errors: 0
rx_missed: 0
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 822
broadcast: 56
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 1073
rx_packets: 882
tx_errors: 0
rx_errors: 0
rx_missed: 0
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 825
broadcast: 56
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 1081
rx_packets: 889
tx_errors: 0
rx_errors: 0
rx_missed: 0
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 832
broadcast: 56
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 1086
rx_packets: 892
tx_errors: 0
rx_errors: 0
rx_missed: 0
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 835
broadcast: 56
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 1094
rx_packets: 898
tx_errors: 0
rx_errors: 0
rx_missed: 0
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 841
broadcast: 56
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 1100
rx_packets: 902
tx_errors: 0
rx_errors: 0
rx_missed: 0
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 845
broadcast: 56
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 1397
rx_packets: 2086
tx_errors: 0
rx_errors: 0
rx_missed: 8
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 2028
broadcast: 57
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 2434
rx_packets: 6544
tx_errors: 0
rx_errors: 0
rx_missed: 34
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 6486
broadcast: 57
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 4361
rx_packets: 14479
tx_errors: 0
rx_errors: 0
rx_missed: 80
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 14421
broadcast: 57
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 5853
rx_packets: 20659
tx_errors: 0
rx_errors: 0
rx_missed: 113
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 20601
broadcast: 57
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 7485
rx_packets: 28008
tx_errors: 0
rx_errors: 0
rx_missed: 151
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 27950
broadcast: 57
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 7787
rx_packets: 29253
tx_errors: 0
rx_errors: 0
rx_missed: 161
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 29195
broadcast: 57
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 8031
rx_packets: 29904
tx_errors: 0
rx_errors: 0
rx_missed: 170
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 29846
broadcast: 57
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 9031
rx_packets: 34321
tx_errors: 0
rx_errors: 0
rx_missed: 186
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 34263
broadcast: 57
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 10261
rx_packets: 39805
tx_errors: 0
rx_errors: 0
rx_missed: 221
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 39747
broadcast: 57
multicast: 1
tx_aborted: 0
tx_underrun: 0
.
NIC statistics:
tx_packets: 10261
rx_packets: 39805
tx_errors: 0
rx_errors: 0
rx_missed: 221
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 39747
broadcast: 57
multicast: 1
tx_aborted: 0
tx_underrun: 0
[-- Attachment #6: vmstat --]
[-- Type: text/plain, Size: 722 bytes --]
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 8066536 4876 28220 0 0 65 3 1010 45 0 0 98 1
0 0 0 8066536 4876 28188 0 0 0 0 4036 144 0 0 100 0
0 0 0 8065040 4892 28428 0 0 388 0 4051 194 0 0 99 1
0 0 0 8049916 4892 42896 0 0 0 0 8419 3320 0 5 95 0
0 0 0 8030112 4892 62188 0 0 0 0 9749 4420 0 7 93 0
0 0 0 8024664 4900 67920 0 0 0 24 5659 1356 0 2 98 0
0 0 0 8014876 4900 77140 0 0 0 0 6953 2442 0 4 96 0
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
2011-02-22 17:33 Hans Nieser
@ 2011-02-23 9:55 ` Francois Romieu
2011-02-23 12:21 ` Hans Nieser
0 siblings, 1 reply; 16+ messages in thread
From: Francois Romieu @ 2011-02-23 9:55 UTC (permalink / raw)
To: Hans Nieser; +Cc: netdev, linux-kernel
Hans Nieser <hnsr@xs4all.nl> :
[...]
> With your patches applied to 2.6.38-rc6, I have gathered some of the
> info you requested from Seblu as well, I hope it's helpful:
>
> 1: see attachment
Ok.
The chipset requires no trivial last minute regression fix (yet).
> 2: I'm not sure how to check the size of the packets, but I'm just
> fetching a (large) file over http/tcp, so I guess they are mostly of the
> size of my MTU which is 1500 looking at ifconfig output
Fine.
Your testcases are always based on a real download, whence including some
disk activity, as opposed to a pure network test, right ?
> For the other vmstat/ethtool/interrupts output, I started the following
> commands remotely via ssh a second or two before starting the download,
> and the machine locked up a few seconds later:
SysRq is enabled (/etc/sysctl.conf::kernel.sysrq = 1), the computer was
switched back on a no-X console before the test. Then the keyboard leds
ignore keypresses and the sysrq keys don't display anything in the
console, right ?
You may enable PCIEASPM_DEBUG, force 'pcie_aspm=off' and switch from
SLUB to SLAB but it's a bit cargo-cultish.
A bisection could help. Bisecting 2.6.35 .. 2.6.35.9 may be enough if
2.6.35.9 works well.
--
Ueimor
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
2011-02-23 9:55 ` Francois Romieu
@ 2011-02-23 12:21 ` Hans Nieser
2011-02-23 18:31 ` Hans Nieser
0 siblings, 1 reply; 16+ messages in thread
From: Hans Nieser @ 2011-02-23 12:21 UTC (permalink / raw)
To: Francois Romieu; +Cc: netdev, linux-kernel
On Wed, 2011-02-23 at 10:55 +0100, Francois Romieu wrote:
> Hans Nieser <hnsr@xs4all.nl> :
> [...]
> > With your patches applied to 2.6.38-rc6, I have gathered some of the
> > info you requested from Seblu as well, I hope it's helpful:
> >
> > 1: see attachment
>
> Ok.
>
> The chipset requires no trivial last minute regression fix (yet).
>
> > 2: I'm not sure how to check the size of the packets, but I'm just
> > fetching a (large) file over http/tcp, so I guess they are mostly of the
> > size of my MTU which is 1500 looking at ifconfig output
>
> Fine.
>
> Your testcases are always based on a real download, whence including some
> disk activity, as opposed to a pure network test, right ?
Yeah, I just had a little script that wgetted a file from a webserver in
my LAN and saved it to separate (non-root) fs, then removed it - in a
loop. When testing on the 2.6.35 and 2.6.35.9 kernels it did max out at
about 107MiB/s, sometimes falling down a little presumably when disk was
being touched.
> > For the other vmstat/ethtool/interrupts output, I started the following
> > commands remotely via ssh a second or two before starting the download,
> > and the machine locked up a few seconds later:
>
> SysRq is enabled (/etc/sysctl.conf::kernel.sysrq = 1), the computer was
> switched back on a no-X console before the test. Then the keyboard leds
> ignore keypresses and the sysrq keys don't display anything in the
> console, right ?
Yep I had X shutdown and switched to VT1, after lock up the LEDs can't
be toggled anymore and sysrq key combo was nonresponsive (it works if I
do it before it locks up)
> You may enable PCIEASPM_DEBUG, force 'pcie_aspm=off' and switch from
> SLUB to SLAB but it's a bit cargo-cultish.
I'll give that a try this evening
> A bisection could help. Bisecting 2.6.35 .. 2.6.35.9 may be enough if
> 2.6.35.9 works well.
Hmm did you mean bisecting 2.6.36 - 2.6.35.9 ? Since with 2.6.36 and
above I can get the machine to hang within seconds and performance is
really bad (10-20MiB/s with wget), while with 2.6.35.9 and 2.6.35
performance was really good (reaching 107MiB/s most of the time) and
lock up took 5-10 minutes instead of seconds (I guess I didn't mention
this in my last e-mail but I managed to get both 2.6.35 and 2.6.35.9 to
lock up eventually) - but I guess something changed between .35 and .36
that made the issue easier to trigger.
I can also try even older kernels to see if there is one that doesn't
lock up at all
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
2011-02-23 12:21 ` Hans Nieser
@ 2011-02-23 18:31 ` Hans Nieser
2011-02-25 13:45 ` Hans Nieser
0 siblings, 1 reply; 16+ messages in thread
From: Hans Nieser @ 2011-02-23 18:31 UTC (permalink / raw)
To: Francois Romieu; +Cc: netdev, linux-kernel
On Wed, 2011-02-23 at 13:21 +0100, Hans Nieser wrote:
> On Wed, 2011-02-23 at 10:55 +0100, Francois Romieu wrote:
> > Hans Nieser <hnsr@xs4all.nl> :
> > [...]
> > > With your patches applied to 2.6.38-rc6, I have gathered some of the
> > > info you requested from Seblu as well, I hope it's helpful:
> > >
> > > 1: see attachment
> >
> > Ok.
> >
> > The chipset requires no trivial last minute regression fix (yet).
> >
> > > 2: I'm not sure how to check the size of the packets, but I'm just
> > > fetching a (large) file over http/tcp, so I guess they are mostly of the
> > > size of my MTU which is 1500 looking at ifconfig output
> >
> > Fine.
> >
> > Your testcases are always based on a real download, whence including some
> > disk activity, as opposed to a pure network test, right ?
>
> Yeah, I just had a little script that wgetted a file from a webserver in
> my LAN and saved it to separate (non-root) fs, then removed it - in a
> loop. When testing on the 2.6.35 and 2.6.35.9 kernels it did max out at
> about 107MiB/s, sometimes falling down a little presumably when disk was
> being touched.
>
> > > For the other vmstat/ethtool/interrupts output, I started the following
> > > commands remotely via ssh a second or two before starting the download,
> > > and the machine locked up a few seconds later:
> >
> > SysRq is enabled (/etc/sysctl.conf::kernel.sysrq = 1), the computer was
> > switched back on a no-X console before the test. Then the keyboard leds
> > ignore keypresses and the sysrq keys don't display anything in the
> > console, right ?
>
> Yep I had X shutdown and switched to VT1, after lock up the LEDs can't
> be toggled anymore and sysrq key combo was nonresponsive (it works if I
> do it before it locks up)
>
> > You may enable PCIEASPM_DEBUG, force 'pcie_aspm=off' and switch from
> > SLUB to SLAB but it's a bit cargo-cultish.
>
> I'll give that a try this evening
>
> > A bisection could help. Bisecting 2.6.35 .. 2.6.35.9 may be enough if
> > 2.6.35.9 works well.
>
> Hmm did you mean bisecting 2.6.36 - 2.6.35.9 ? Since with 2.6.36 and
> above I can get the machine to hang within seconds and performance is
> really bad (10-20MiB/s with wget), while with 2.6.35.9 and 2.6.35
> performance was really good (reaching 107MiB/s most of the time) and
> lock up took 5-10 minutes instead of seconds (I guess I didn't mention
> this in my last e-mail but I managed to get both 2.6.35 and 2.6.35.9 to
> lock up eventually) - but I guess something changed between .35 and .36
> that made the issue easier to trigger.
>
> I can also try even older kernels to see if there is one that doesn't
> lock up at all
>
Ok, I just tried 2.6.34, and after over 5 hours of running my script,
the system is still up and running, with only 24 'link up' messages on
dmesg, and having transferred 2.1TiB of data (1428042421 rx_packets, 45
rx_missed). So I'm going to assume the problem isn't present with this
kernel and try a bisect between it and 2.6.35
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
2011-02-23 18:31 ` Hans Nieser
@ 2011-02-25 13:45 ` Hans Nieser
2011-03-03 19:53 ` Hans Nieser
0 siblings, 1 reply; 16+ messages in thread
From: Hans Nieser @ 2011-02-25 13:45 UTC (permalink / raw)
To: Francois Romieu; +Cc: netdev, linux-kernel
On Wed, 2011-02-23 at 19:31 +0100, Hans Nieser wrote:
> On Wed, 2011-02-23 at 13:21 +0100, Hans Nieser wrote:
> > On Wed, 2011-02-23 at 10:55 +0100, Francois Romieu wrote:
> > You may enable PCIEASPM_DEBUG, force 'pcie_aspm=off' and switch from
> > SLUB to SLAB but it's a bit cargo-cultish.
This seemed to have no effect sadly
> Ok, I just tried 2.6.34, and after over 5 hours of running my script,
> the system is still up and running, with only 24 'link up' messages on
> dmesg, and having transferred 2.1TiB of data (1428042421 rx_packets, 45
> rx_missed). So I'm going to assume the problem isn't present with this
> kernel and try a bisect between it and 2.6.35
After spending the entire day yesterday and this morning bisecting this,
I haven't gotten anywhere :/ I ended up at an unrelated commit as being
the first known bad commit (84d4db0e22965334ae8272f324d31fb4657465aa), I
think I may have marked a bad commit as good.. To properly bisect this
issue I probably need to test each commit for a several hours across
multiple reboots, but that is going to be too much time. I've at least
been able to establish that following v2.6.34, the following commits are
bad:
c222fb2efaf1a421f5bf74403df40a9384ccf516
4a973f2495fba8775d1c408b3ee7f2c19b19f13f
84d4db0e22965334ae8272f324d31fb4657465aa
After that I've been trying other various things (on 2.6.38-rc6+) and
made some interesting and confusing discoveries;
- Setting pci=nomsi causes instant reboot when I start my test script
- Enabling only one CPU core in the BIOS seems to solve the whole lock
up problem, I have not been able to reproduce it after a few hours of
testing (nor on 2.6.35). (Normally on 2.6.38-rc6 it would crash in
just a few seconds.)
Additionally, when I force wget to use IPv4 with only one core
enabled, I'm suddenly getting a solid 112MB/s instead of the lousy
9-12MB/s I have been getting since 2.6.36 - but only when using one
core. With all 4 cores enabled, performance is bad again even when
forcing wget to use IPv4..
Using only one CPU core also reduces the 'link up' messages a lot, I
only got a couple instead of hundreds/thousands.
- Enabling Tickless System (NO_HZ) kernel option seems to make lock up
occur less frequently (but it still happens), also much less 'link up'
messages, but also causes an occasional "NOHZ: local_softirq_pending
08" to appear on dmesg.
- Enabling HyperThreading (I disable it by default due to an issue with
VirtualBox) in BIOS causes performance to get even worse, just 2-3MB/s
instead of 9-12MB/s
I've also attempted to bisect the issue I have been having with slow
transfer speed (I don't know if its related to the hang, but I figure if
the hang ever gets fixed, this will have to be fixed as well to make
r8169 usable for me), which started somewhere between v2.6.35 (good) and
v2.6.36 (bad), unfortunately this too ended up at a seemingly unrelated
commit:
af5ab277ded04bd9bc6b048c5a2f0e7d70ef0867 - clockevents: Remove the per
cpu tick skew
Just for kicks I attempted to revert this change on 2.6.38-rc6+, which
seemed to reduce the frequency of 'link up' messages, but no other real
change noticed.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Mass udp flow reboot linux with RealTek RTL-8169 Gigabit
2011-02-25 13:45 ` Hans Nieser
@ 2011-03-03 19:53 ` Hans Nieser
0 siblings, 0 replies; 16+ messages in thread
From: Hans Nieser @ 2011-03-03 19:53 UTC (permalink / raw)
To: Francois Romieu; +Cc: netdev, linux-kernel
On Fri, 2011-02-25 at 14:45 +0100, Hans Nieser wrote:
...
> I've also attempted to bisect the issue I have been having with slow
> transfer speed (I don't know if its related to the hang, but I figure if
> the hang ever gets fixed, this will have to be fixed as well to make
> r8169 usable for me)
...
Just wanted to let you know that after installing a new Intel gbit NIC I
ordered, I sadly still get poor performance, so this is definitely a
separate issue from the r8169 lock up. (lock ups are gone now)
I've given up on getting r8169 working at this point but if you have any
patches you need tested or need more info in the future please do let me
know
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2011-03-10 12:08 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <AANLkTin7GBSTcfZgr_9sNZ8CPMkW7Vstni+fs2v1-ink@mail.gmail.com>
2011-02-13 7:17 ` Mass udp flow reboot linux with RealTek RTL-8169 Gigabit Eric Dumazet
2011-02-13 13:56 ` Francois Romieu
2011-02-13 17:27 ` Seblu
2011-02-13 18:02 ` Seblu
2011-02-13 20:34 ` Francois Romieu
2011-02-18 2:54 ` Seblu
2011-02-18 9:30 ` Francois Romieu
2011-03-06 0:29 ` Seblu
2011-03-10 12:08 ` Francois Romieu
2011-02-21 11:56 Hans Nieser
-- strict thread matches above, loose matches on Subject: below --
2011-02-22 17:33 Hans Nieser
2011-02-23 9:55 ` Francois Romieu
2011-02-23 12:21 ` Hans Nieser
2011-02-23 18:31 ` Hans Nieser
2011-02-25 13:45 ` Hans Nieser
2011-03-03 19:53 ` Hans Nieser
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).