* bnx2_poll panicking kernel
@ 2008-06-16 12:57 Josip Rodin
2008-06-16 14:04 ` Ben Hutchings
0 siblings, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-06-16 12:57 UTC (permalink / raw)
To: netdev; +Cc: mirrors
Hi,
Has anyone seen the below bug before? I'm already upgrading in hope that it
might be already fixed in a newer kernel, but anyway.
It started happening a few days ago, on a load of a few tens of MB/s (Debian
rsync mirroring). It doesn't happen on every mirror pulse, so it's not
exactly reproducible...
I'm not sure what the perl process could be. I have one script running
as part of the RRD-graphing setup, it connects to apache's server-status
page every minute.
It certainly doesn't sound like something that should affect the network
driver or the kernel task scheduler...
Unable to handle kernel NULL pointer dereference at 0000000000000
0bc RIP:
[<ffffffff88042eee>] :bnx2:bnx2_poll+0xfe/0xd10
PGD 194c69067 PUD 19844c067 PMD 0
Oops: 0000 [1] SMP
CPU 4
Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler nfs lockd nfs_acl sunrpc
thermal fan button processor ac battery cls_u32 sch_sfq sch_htb ip6t_REJECT ip6
t_LOG ip6table_filter ip6_tables ipt_REJECT xt_tcpudp iptable_filter ip_tables x
_tables ipv6 dm_snapshot dm_mirror dm_mod usbhid ehci_hcd bnx2 zlib_inflate uhci
_hcd crc32 usbcore shpchp
Pid: 5854, comm: perl Not tainted 2.6.22.18 #3
IP: 0010:[<ffffffff88042eee>] [<ffffffff88042eee>] :bnx2:bnx2_poll+0xfe/0xd10
RSP: 0018:ffff81022fe57e38 EFLAGS: 00010282
RAX: 00000000000096c7 RBX: ffff81022d6209e0 RCX: 00000000000096c6
RDX: 0000000000000080 RSI: 000000000000009e RDI: ffff81007f1dc7d8
RBP: 0000000096c6969e R08: ffff810198cef900 R09: 0000000000000003
R10: 00000000dfdac3ce R11: 0000000000000003 R12: 000000000000009c
R13: 0000000000000001 R14: 0000000000000000 R15: ffff81022fcd6780
FS: 00002b444ada1ae0(0000) GS:ffff81022fe25c40(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000bc CR3: 00000002155c0000 CR4: 00000000000006e0
Process perl (pid: 5854, threadinfo ffff81020fe74000, task ffff81011b864e80)
Stack: 000000020e524810 ffff81022f348070 ffff81022d63f000 ffff81022fe57f4c
ffff81022fcd6000 0000000400000040 ffff810200000001 ffffffff805968d9
0000005600000004 00000000a576e010 ffffc20000012000 ffff81022d102000
Call Trace:
<IRQ> [<ffffffff802291f0>] run_rebalance_domains+0x170/0x460
[<ffffffff8022802e>] task_running_tick+0x11e/0x320
[<ffffffff803dcff0>] net_rx_action+0xc0/0x210
[<ffffffff802373ec>] run_timer_softirq+0x3c/0x1d0
[<ffffffff80233e09>] __do_softirq+0x69/0xe0
[<ffffffff8020abbc>] call_softirq+0x1c/0x30
<EOI> [<ffffffff8020c975>] do_softirq+0x35/0x90
[<ffffffff80233c5a>] local_bh_enable+0x5a/0xa0
[<ffffffff80296e67>] seq_read+0x267/0x2e0
[<ffffffff8027a5e5>] vfs_read+0xc5/0x160
[<ffffffff8027aac3>] sys_read+0x53/0x90
[<ffffffff80209a2e>] system_call+0x7e/0x83
Code: 41 8b 86 bc 00 00 00 49 03 86 c0 00 00 00 66 83 78 06 00 0f
RIP [<ffffffff88042eee>] :bnx2:bnx2_poll+0xfe/0xd10
RSP <ffff81022fe57e38>
CR2: 00000000000000bc
Kernel panic - not syncing: Aiee, killing interrupt handler!
--
2. That which causes joy or happiness.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-16 12:57 bnx2_poll panicking kernel Josip Rodin
@ 2008-06-16 14:04 ` Ben Hutchings
2008-06-16 15:52 ` Michael Chan
0 siblings, 1 reply; 50+ messages in thread
From: Ben Hutchings @ 2008-06-16 14:04 UTC (permalink / raw)
To: Josip Rodin; +Cc: netdev, mirrors
Josip Rodin wrote:
> Hi,
>
> Has anyone seen the below bug before? I'm already upgrading in hope that it
> might be already fixed in a newer kernel, but anyway.
>
> It started happening a few days ago, on a load of a few tens of MB/s (Debian
> rsync mirroring). It doesn't happen on every mirror pulse, so it's not
> exactly reproducible...
That might be a result of the bug fixed by this commit, which would be
triggered by heavy traffic:
commit 6f535763165331bb91277d7519b507fed22034e5
Author: David S. Miller <davem@sunset.davemloft.net>
Date: Thu Oct 11 18:08:29 2007 -0700
[NET]: Fix NAPI completion handling in some drivers.
In order for the list handling in net_rx_action() to be
correct, drivers must follow certain rules as stated by
this comment in net_rx_action():
/* Drivers must not modify the NAPI state if they
* consume the entire weight. In such cases this code
* still "owns" the NAPI instance and therefore can
* move the instance around on the list at-will.
*/
A few drivers do not do this because they mix the budget checks
with reading hardware state, resulting in crashes like the one
reported by takano@axe-inc.co.jp.
BNX2 and TG3 are taken care of here, SKY2 fix is from Stephen
Hemminger.
This fix went into 2.6.24.
Ben.
--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-16 14:04 ` Ben Hutchings
@ 2008-06-16 15:52 ` Michael Chan
2008-06-16 19:13 ` Josip Rodin
0 siblings, 1 reply; 50+ messages in thread
From: Michael Chan @ 2008-06-16 15:52 UTC (permalink / raw)
To: Ben Hutchings, Josip Rodin; +Cc: netdev, mirrors
Ben Hutchings wrote:
> Josip Rodin wrote:
> > Hi,
> >
> > Has anyone seen the below bug before? I'm already upgrading
> in hope that it
> > might be already fixed in a newer kernel, but anyway.
> >
> > It started happening a few days ago, on a load of a few
> tens of MB/s (Debian
> > rsync mirroring). It doesn't happen on every mirror pulse,
> so it's not
> > exactly reproducible...
>
> That might be a result of the bug fixed by this commit, which would be
> triggered by heavy traffic:
>
> commit 6f535763165331bb91277d7519b507fed22034e5
> Author: David S. Miller <davem@sunset.davemloft.net>
> Date: Thu Oct 11 18:08:29 2007 -0700
>
> [NET]: Fix NAPI completion handling in some drivers.
>
This commit fixed a bug that existed for a very short time after
the patch to make the NAPI struct independent of the netdevice was
committed.
So I think it is unlikely that Josip's panic is caused by this.
Josip, please provide more information such as driver version from
ethtool -i, lspci, which version did you first see the problem, and
whether the latest in 2.6.25 has the same problem. Thanks.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-16 15:52 ` Michael Chan
@ 2008-06-16 19:13 ` Josip Rodin
2008-06-16 21:38 ` Josip Rodin
0 siblings, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-06-16 19:13 UTC (permalink / raw)
To: Michael Chan; +Cc: Ben Hutchings, netdev, mirrors
On Mon, Jun 16, 2008 at 08:52:13AM -0700, Michael Chan wrote:
> Josip, please provide more information such as driver version from
> ethtool -i, lspci, which version did you first see the problem, and
> whether the latest in 2.6.25 has the same problem. Thanks.
The version with the problem is 2.6.22.18. The machine is fairly new
(HP DL360 G5), it only ever ran the .22 kernel. It was running this kernel
just fine for 112 days prior to the first incident on 2008-06-03, and since
then, the crash happened a total of 17 times. Our big rsync runs happen
twice a day, so it's a pretty bad record.
I can't verify that the crash was caused by the same thing every time
because I don't have console logs, but the symptoms are always the same - a
few minutes into the rsync run, it goes down, logging nothing in syslog.
I'll be rebooting into 2.6.25.6 shortly, and then I'll have information
relevant to that in about a day or two.
% sudo ethtool -i eth0
driver: bnx2
version: 1.5.11
firmware-version: 1.9.6
bus-info: 0000:03:00.0
% lspci
00:00.0 Host bridge: Intel Corporation 5000P Chipset Memory Controller Hub (rev b1)
00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 2 (rev b1)
00:03.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 3 (rev b1)
00:04.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x8 Port 4-5 (rev b1)
00:05.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 5 (rev b1)
00:06.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 6 (rev b1)
00:07.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 7 (rev b1)
00:10.0 Host bridge: Intel Corporation 5000 Series Chipset Error Reporting Registers (rev b1)
00:10.1 Host bridge: Intel Corporation 5000 Series Chipset Error Reporting Registers (rev b1)
00:10.2 Host bridge: Intel Corporation 5000 Series Chipset Error Reporting Registers (rev b1)
00:11.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev b1)
00:13.0 Host bridge: Intel Corporation 5000 Series Chipset Reserved Registers (rev b1)
00:15.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev b1)
00:16.0 Host bridge: Intel Corporation 5000 Series Chipset FBD Registers (rev b1)
00:1d.0 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #1 (rev 09)
00:1d.1 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #2 (rev 09)
00:1d.2 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #3 (rev 09)
00:1d.3 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset UHCI USB Controller #4 (rev 09)
00:1d.7 USB Controller: Intel Corporation 631xESB/632xESB/3100 Chipset EHCI USB2 Controller (rev 09)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d9)
00:1f.0 ISA bridge: Intel Corporation 631xESB/632xESB/3100 Chipset LPC Interface Controller (rev 09)
00:1f.1 IDE interface: Intel Corporation 631xESB/632xESB IDE Controller (rev 09)
01:03.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)
01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03)
01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 03)
01:04.4 USB Controller: Hewlett-Packard Company Unknown device 3300
01:04.6 Serial bus controller [0c07]: Hewlett-Packard Company Unknown device 3302
02:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c3)
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
04:00.0 PCI bridge: Broadcom EPB PCI-Express to PCI-X Bridge (rev c3)
05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
06:00.0 RAID bus controller: Hewlett-Packard Company Hewlett-Packard Smart Array Controller (rev 01)
09:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01)
09:00.3 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express to PCI-X Bridge (rev 01)
0a:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E1 (rev 01)
0a:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E2 (rev 01)
0a:02.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E3 (rev 01)
--
2. That which causes joy or happiness.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-16 19:13 ` Josip Rodin
@ 2008-06-16 21:38 ` Josip Rodin
2008-06-16 21:48 ` Josip Rodin
0 siblings, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-06-16 21:38 UTC (permalink / raw)
To: Michael Chan; +Cc: Ben Hutchings, netdev, mirrors
On Mon, Jun 16, 2008 at 09:13:54PM +0200, joy wrote:
> On Mon, Jun 16, 2008 at 08:52:13AM -0700, Michael Chan wrote:
> > Josip, please provide more information such as driver version from
> > ethtool -i, lspci, which version did you first see the problem, and
> > whether the latest in 2.6.25 has the same problem. Thanks.
>
> The version with the problem is 2.6.22.18. The machine is fairly new
> (HP DL360 G5), it only ever ran the .22 kernel. It was running this kernel
> just fine for 112 days prior to the first incident on 2008-06-03, and since
> then, the crash happened a total of 17 times. Our big rsync runs happen
> twice a day, so it's a pretty bad record.
>
> I can't verify that the crash was caused by the same thing every time
> because I don't have console logs, but the symptoms are always the same - a
> few minutes into the rsync run, it goes down, logging nothing in syslog.
>
> I'll be rebooting into 2.6.25.6 shortly, and then I'll have information
> relevant to that in about a day or two.
Crap, it just crashed again, with 2.6.25.6.
BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8
IP: [<ffffffff88084c9a>] :bnx2:bnx2_tx_int+0x6a/0x330
PGD 22e8c2067 PUD 1bfa48067 PMD 0
Oops: 0000 [1] SMP
CPU 2
Modules linked in: ipmi_devintf nfs lockd nfs_acl sunrpc fan ac battery cls_u32 sch_sfq sch_htb ip6t_REJECT ip6t_LOG ip6table_filter ip6_tables ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables ipv6 dm_snapshot dm_mirror dm_mod usbhid bnx2 zlib_inflate container ehci_hcd ipmi_si crc32 thermal uhci_hcd ipmi_msghandler button usbcore processor shpchp
Pid: 0, comm: swapper Not tainted 2.6.25.6 #1
RIP: 0010:[<ffffffff88084c9a>] [<ffffffff88084c9a>] :bnx2:bnx2_tx_int+0x6a/0x330
RSP: 0018:ffff81022f8cfd00 EFLAGS: 00010286
RAX: 00000000000005ba RBX: ffff81022f9faaf0 RCX: 00000000000000af
RDX: 00000000000005ba RSI: ffff8100865f85bf RDI: ffff8100cf974198
RBP: 0000000005a505af R08: ffff8100867e2100 R09: ffff8101bf8019a8
R10: 0000000000000000 R11: ffffffff80419a50 R12: 00000000000000ae
R13: 0000000000000001 R14: 0000000000000000 R15: ffff81022f92a700
FS: 0000000000000000(0000) GS:ffff81022f875940(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000000000000b8 CR3: 00000001756e1000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff81022f8ca000, task ffff81022f8837d0)
Stack: ffff81022e97f000 000000002eb51600 ffff81022f92a800 05baffff880d9767
0000000100000004 ffff81022f92a800 0000000000000040 0000000000000000
ffff810001016ac0 ffff81022f92a700 000000000000012c ffffffff88085422
Call Trace:
<IRQ> [<ffffffff88085422>] ? :bnx2:bnx2_poll+0xd2/0x12d0
[<ffffffff8022d7e4>] ? run_rebalance_domains+0x194/0x4c0
[<ffffffff803edaa8>] ? net_rx_action+0xf8/0x200
[<ffffffff8023a5b9>] ? __do_softirq+0x69/0xe0
[<ffffffff8020c4ac>] ? call_softirq+0x1c/0x30
[<ffffffff8020e995>] ? do_softirq+0x35/0x70
[<ffffffff8020ec10>] ? do_IRQ+0x80/0x100
[<ffffffff8020a410>] ? mwait_idle+0x0/0x50
[<ffffffff8020b831>] ? ret_from_intr+0x0/0xa
<EOI> [<ffffffff80419a50>] ? tcp_poll+0x0/0x180
[<ffffffff8020a451>] ? mwait_idle+0x41/0x50
[<ffffffff8020a378>] ? cpu_idle+0x48/0x90
Code: 24 1e 8b 69 68 c7 44 24 20 00 00 00 00 66 39 ea 0f 84 6c 01 00 00 40 0f b6 cd 0f b7 d9 48 c1 e3 04 49 03 9f e0 02 00 00 4c 8b 33 <41> 8b 86 b8 00 00 00 49 03 86 c0 00 00 00 66 83 78 06 00 0f 85
RIP [<ffffffff88084c9a>] :bnx2:bnx2_tx_int+0x6a/0x330
RSP <ffff81022f8cfd00>
CR2: 00000000000000b8
---[ end trace 7fde2b18cd9ffdbf ]---
Kernel panic - not syncing: Aiee, killing interrupt handler!
--
2. That which causes joy or happiness.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-16 21:38 ` Josip Rodin
@ 2008-06-16 21:48 ` Josip Rodin
2008-06-16 23:45 ` Michael Chan
0 siblings, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-06-16 21:48 UTC (permalink / raw)
To: Michael Chan; +Cc: Ben Hutchings, netdev, mirrors
On Mon, Jun 16, 2008 at 11:38:21PM +0200, Josip Rodin wrote:
> > > Josip, please provide more information such as driver version from
> > > ethtool -i, lspci, which version did you first see the problem, and
> > > whether the latest in 2.6.25 has the same problem. Thanks.
> >
> > The version with the problem is 2.6.22.18. The machine is fairly new
> > (HP DL360 G5), it only ever ran the .22 kernel. It was running this kernel
> > just fine for 112 days prior to the first incident on 2008-06-03, and since
> > then, the crash happened a total of 17 times. Our big rsync runs happen
> > twice a day, so it's a pretty bad record.
> >
> > I can't verify that the crash was caused by the same thing every time
> > because I don't have console logs, but the symptoms are always the same - a
> > few minutes into the rsync run, it goes down, logging nothing in syslog.
> >
> > I'll be rebooting into 2.6.25.6 shortly, and then I'll have information
> > relevant to that in about a day or two.
>
> Crap, it just crashed again, with 2.6.25.6.
I should mention that I might have a clue - around the time the crashes
started, I started using HTB with the following configuration:
tc qdisc add dev eth0 root handle 1: htb default 100
tc class add dev eth0 parent 1: classid 1:1 htb rate 500mbit ceil 1gbit burst 5m
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 100mbit ceil 1gbit burst 1m prio 0
tc qdisc add dev eth0 parent 1:2 handle 2: sfq
tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip dport 22 0xffff flowid 1:2
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 100mbit ceil 1gbit burst 1m prio 1
tc qdisc add dev eth0 parent 1:3 handle 3: sfq
tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip dport 873 0xffff flowid 1:3
tc class add dev eth0 parent 1:1 classid 1:16 htb rate 50mbit ceil 100mbit burst 512k prio 3
tc qdisc add dev eth0 parent 1:16 handle 16: sfq
tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip dst 161.53.160.16 flowid 1:16
tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip src 161.53.160.16 flowid 1:16
tc class add dev eth0 parent 1:1 classid 1:100 htb rate 100mbit ceil 300mbit burst 1m prio 2
tc qdisc add dev eth0 parent 1:100 handle 100: sfq
HTB actually complains a little bit about r2q/quantum (too big), but that's
not supposed to be so harmful.
--
2. That which causes joy or happiness.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-16 21:48 ` Josip Rodin
@ 2008-06-16 23:45 ` Michael Chan
2008-06-17 22:37 ` Josip Rodin
0 siblings, 1 reply; 50+ messages in thread
From: Michael Chan @ 2008-06-16 23:45 UTC (permalink / raw)
To: Josip Rodin; +Cc: Ben Hutchings, netdev, mirrors
On Mon, 2008-06-16 at 23:48 +0200, Josip Rodin wrote:
> > Crap, it just crashed again, with 2.6.25.6.
Looking at the panic dmesg, I believe it was crashing because skb was
NULL in bnx2_tx_int().
>
> I should mention that I might have a clue - around the time the crashes
> started, I started using HTB with the following configuration:
This rings a bell. A similar crash was reported by a user using HTB on
a tg3 device. After some analysis, my suspicion was that the shinfo
(skb)->nr_frags on the TX packet was corrupted before the SKB was freed
by the driver. In both the tg3 and bnx2 drivers, we rely on the
nr_frags to locate the TX packet boundaries on the ring to free the
packets.
Please try this debug patch below. If the theory is correct, the patch
should avoid the crash and will print something to the dmesg log every
time a crash is avoided. Please run it again with the HTB rules and
send me the dmesg log containing:
bnx2: skb->nr_frags ...
Here's the debug patch below. It saves the nr_frags from the SKB and
uses it to locate the packet boundaries instead. It will also compare
the saved value with the one in SKB and print a warning when they don't
match. Please apply to 2.6.25.6.
diff --git a/drivers/net/bnx2.c b/drivers/net/bnx2.c
index 4b46e68..f7ecd07 100644
--- a/drivers/net/bnx2.c
+++ b/drivers/net/bnx2.c
@@ -2495,14 +2495,20 @@ bnx2_tx_int(struct bnx2 *bp, struct bnx2_napi *bnapi, int budget)
tx_buf = &bp->tx_buf_ring[sw_ring_cons];
skb = tx_buf->skb;
+ if (tx_buf->nr_frags != skb_shinfo(skb)->nr_frags) {
+ printk(KERN_ALERT "bnx2: skb->nr_frags=%d is corrupted,"
+ " should be %d\n",
+ skb_shinfo(skb)->nr_frags,
+ tx_buf->nr_frags);
+ }
/* partial BD completions possible with TSO packets */
if (skb_is_gso(skb)) {
u16 last_idx, last_ring_idx;
last_idx = sw_cons +
- skb_shinfo(skb)->nr_frags + 1;
+ tx_buf->nr_frags + 1;
last_ring_idx = sw_ring_cons +
- skb_shinfo(skb)->nr_frags + 1;
+ tx_buf->nr_frags + 1;
if (unlikely(last_ring_idx >= MAX_TX_DESC_CNT)) {
last_idx++;
}
@@ -2515,7 +2521,7 @@ bnx2_tx_int(struct bnx2 *bp, struct bnx2_napi *bnapi, int budget)
skb_headlen(skb), PCI_DMA_TODEVICE);
tx_buf->skb = NULL;
- last = skb_shinfo(skb)->nr_frags;
+ last = tx_buf->nr_frags;
for (i = 0; i < last; i++) {
sw_cons = NEXT_TX_BD(sw_cons);
@@ -4806,7 +4812,7 @@ bnx2_free_tx_skbs(struct bnx2 *bp)
tx_buf->skb = NULL;
- last = skb_shinfo(skb)->nr_frags;
+ last = tx_buf->nr_frags;
for (j = 0; j < last; j++) {
tx_buf = &bp->tx_buf_ring[i + j + 1];
pci_unmap_page(bp->pdev,
@@ -5859,6 +5865,8 @@ bnx2_start_xmit(struct sk_buff *skb, struct net_device *dev)
last_frag = skb_shinfo(skb)->nr_frags;
+ tx_buf->nr_frags = last_frag;
+
for (i = 0; i < last_frag; i++) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
diff --git a/drivers/net/bnx2.h b/drivers/net/bnx2.h
index 1eaf5bb..aa9fa6f 100644
--- a/drivers/net/bnx2.h
+++ b/drivers/net/bnx2.h
@@ -6485,6 +6485,7 @@ struct l2_fhdr {
struct sw_bd {
struct sk_buff *skb;
DECLARE_PCI_UNMAP_ADDR(mapping)
+ unsigned short nr_frags;
};
struct sw_pg {
^ permalink raw reply related [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-16 23:45 ` Michael Chan
@ 2008-06-17 22:37 ` Josip Rodin
2008-06-17 22:47 ` Michael Chan
0 siblings, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-06-17 22:37 UTC (permalink / raw)
To: Michael Chan; +Cc: Ben Hutchings, netdev, mirrors
On Mon, Jun 16, 2008 at 04:45:08PM -0700, Michael Chan wrote:
> On Mon, 2008-06-16 at 23:48 +0200, Josip Rodin wrote:
> > > Crap, it just crashed again, with 2.6.25.6.
>
> Looking at the panic dmesg, I believe it was crashing because skb was
> NULL in bnx2_tx_int().
>
> >
> > I should mention that I might have a clue - around the time the crashes
> > started, I started using HTB with the following configuration:
>
> This rings a bell. A similar crash was reported by a user using HTB on
> a tg3 device. After some analysis, my suspicion was that the shinfo
> (skb)->nr_frags on the TX packet was corrupted before the SKB was freed
> by the driver. In both the tg3 and bnx2 drivers, we rely on the
> nr_frags to locate the TX packet boundaries on the ring to free the
> packets.
>
> Please try this debug patch below. If the theory is correct, the patch
> should avoid the crash and will print something to the dmesg log every
> time a crash is avoided. Please run it again with the HTB rules and
> send me the dmesg log containing:
>
> bnx2: skb->nr_frags ...
>
> Here's the debug patch below. It saves the nr_frags from the SKB and
> uses it to locate the packet boundaries instead. It will also compare
> the saved value with the one in SKB and print a warning when they don't
> match. Please apply to 2.6.25.6.
After applying this patch, two rsync runs went through without any problem,
yay.
Yet, there were no alerts from the new printk in the patch...?
--
2. That which causes joy or happiness.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-17 22:37 ` Josip Rodin
@ 2008-06-17 22:47 ` Michael Chan
2008-06-21 11:18 ` Josip Rodin
0 siblings, 1 reply; 50+ messages in thread
From: Michael Chan @ 2008-06-17 22:47 UTC (permalink / raw)
To: Josip Rodin; +Cc: Ben Hutchings, netdev, mirrors
On Wed, 2008-06-18 at 00:37 +0200, Josip Rodin wrote:
> After applying this patch, two rsync runs went through without any problem,
> yay.
>
> Yet, there were no alerts from the new printk in the patch...?
Please keep monitoring the printks during future rsyncs with those HTB
rules. May be we're just lucky and not hitting the problem this time.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-17 22:47 ` Michael Chan
@ 2008-06-21 11:18 ` Josip Rodin
2008-06-21 15:34 ` Bill Fink
0 siblings, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-06-21 11:18 UTC (permalink / raw)
To: Michael Chan; +Cc: Ben Hutchings, netdev, mirrors
On Tue, Jun 17, 2008 at 03:47:22PM -0700, Michael Chan wrote:
> On Wed, 2008-06-18 at 00:37 +0200, Josip Rodin wrote:
> > After applying this patch, two rsync runs went through without any problem,
> > yay.
> >
> > Yet, there were no alerts from the new printk in the patch...?
>
> Please keep monitoring the printks during future rsyncs with those HTB
> rules. May be we're just lucky and not hitting the problem this time.
Still nothing after four days. I'm starting to wonder...
The only thing that I did that was slightly out of the ordinary was that
I applied your patch, ran make modules and then modules_install, and then
modprobe -r bnx2, modprobe bnx2; IOW I didn't reboot the whole kernel or
reinitialize HTB. Should I try to do any of that?
--
2. That which causes joy or happiness.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-21 11:18 ` Josip Rodin
@ 2008-06-21 15:34 ` Bill Fink
2008-06-21 16:11 ` Michael Chan
0 siblings, 1 reply; 50+ messages in thread
From: Bill Fink @ 2008-06-21 15:34 UTC (permalink / raw)
To: Josip Rodin; +Cc: Michael Chan, Ben Hutchings, netdev, mirrors
On Sat, 21 Jun 2008, Josip Rodin wrote:
> On Tue, Jun 17, 2008 at 03:47:22PM -0700, Michael Chan wrote:
> > On Wed, 2008-06-18 at 00:37 +0200, Josip Rodin wrote:
> > > After applying this patch, two rsync runs went through without any problem,
> > > yay.
> > >
> > > Yet, there were no alerts from the new printk in the patch...?
> >
> > Please keep monitoring the printks during future rsyncs with those HTB
> > rules. May be we're just lucky and not hitting the problem this time.
>
> Still nothing after four days. I'm starting to wonder...
>
> The only thing that I did that was slightly out of the ordinary was that
> I applied your patch, ran make modules and then modules_install, and then
> modprobe -r bnx2, modprobe bnx2; IOW I didn't reboot the whole kernel or
> reinitialize HTB. Should I try to do any of that?
Wouldn't the "modprobe -r bnx2" destroy the HTB info for that device?
-Bill
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-21 15:34 ` Bill Fink
@ 2008-06-21 16:11 ` Michael Chan
2008-06-23 18:04 ` Josip Rodin
0 siblings, 1 reply; 50+ messages in thread
From: Michael Chan @ 2008-06-21 16:11 UTC (permalink / raw)
To: 'Bill Fink', Josip Rodin
Cc: Ben Hutchings, netdev, mirrors@debian.org
Bill Fink wrote:
>
> Wouldn't the "modprobe -r bnx2" destroy the HTB info for that device?
>
>
I think it would. Josip, you can do "tc -d class show dev eth0" to
check.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-21 16:11 ` Michael Chan
@ 2008-06-23 18:04 ` Josip Rodin
2008-06-23 21:36 ` Josip Rodin
0 siblings, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-06-23 18:04 UTC (permalink / raw)
To: Michael Chan
Cc: 'Bill Fink', Ben Hutchings, netdev, mirrors@debian.org
On Sat, Jun 21, 2008 at 09:11:05AM -0700, Michael Chan wrote:
> Bill Fink wrote:
>
> >
> > Wouldn't the "modprobe -r bnx2" destroy the HTB info for that device?
> >
> >
>
> I think it would. Josip, you can do "tc -d class show dev eth0" to
> check.
>
Oh, duh, yes, I'm a moron. It's back on now, sorry about that.
--
2. That which causes joy or happiness.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-23 18:04 ` Josip Rodin
@ 2008-06-23 21:36 ` Josip Rodin
2008-06-23 22:48 ` Michael Chan
0 siblings, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-06-23 21:36 UTC (permalink / raw)
To: Michael Chan
Cc: 'Bill Fink', Ben Hutchings, netdev, mirrors@debian.org
On Mon, Jun 23, 2008 at 08:04:39PM +0200, Josip Rodin wrote:
> Oh, duh, yes, I'm a moron. It's back on now, sorry about that.
There we go, I got the debugging messages:
[...]
Jun 23 19:53:18 arrakis kernel: HTB: quantum of class 10100 is big. Consider r2q change.
Jun 23 22:57:55 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 4
Jun 23 22:58:32 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 23 22:59:02 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 3
Jun 23 22:59:23 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 9
Jun 23 22:59:36 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 3
Jun 23 23:08:19 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 3
--
2. That which causes joy or happiness.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-23 21:36 ` Josip Rodin
@ 2008-06-23 22:48 ` Michael Chan
2008-06-24 22:58 ` Michael Chan
0 siblings, 1 reply; 50+ messages in thread
From: Michael Chan @ 2008-06-23 22:48 UTC (permalink / raw)
To: Josip Rodin
Cc: 'Bill Fink', Ben Hutchings, netdev, mirrors@debian.org
Josip Rodin wrote:
> On Mon, Jun 23, 2008 at 08:04:39PM +0200, Josip Rodin wrote:
>> Oh, duh, yes, I'm a moron. It's back on now, sorry about that.
>
> There we go, I got the debugging messages:
>
> [...]
> Jun 23 19:53:18 arrakis kernel: HTB: quantum of class 10100 is big. Consider r2q change.
> Jun 23 22:57:55 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 4
> Jun 23 22:58:32 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
> Jun 23 22:59:02 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 3
> Jun 23 22:59:23 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 9
> Jun 23 22:59:36 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 3
> Jun 23 23:08:19 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 3
>
OK, this definitely confirms the theory that the skb->nr_frags is changed
between ->hard_start_xmit() and tx completion. Since we rely on nr_frags to
locate the packet boundaries in the tx ring, it would definitely crash.
One possibility is that it is corrupted by the driver and only happens when
there are HTB rules. I think this is unlikely.
TG3 which operates the same way has also been reported to crash in the
presence of HTB rules. We were not able to pinpoint the problem at that
time.
Can anyone think of a scenario where the stack can modify the SKB this way?
These SKBs look like they are TSO packets.
If not, I will send Josip another patch to print more SKB fields. I can
even save all the SKB fields and see which other ones are modified besides
the nr_frags. May be that will give us a better clue.
Thanks.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-23 22:48 ` Michael Chan
@ 2008-06-24 22:58 ` Michael Chan
2008-06-25 0:04 ` David Miller
2008-06-26 11:01 ` Josip Rodin
0 siblings, 2 replies; 50+ messages in thread
From: Michael Chan @ 2008-06-24 22:58 UTC (permalink / raw)
To: Josip Rodin
Cc: 'Bill Fink', Ben Hutchings, netdev, mirrors@debian.org
Michael Chan wrote:
> I will send Josip another patch to print more SKB fields. I can
> even save all the SKB fields and see which other ones are modified besides
> the nr_frags. May be that will give us a better clue.
>
Please try this patch, which should go on top of the last debug patch.
Thanks.
diff --git a/drivers/net/bnx2.c b/drivers/net/bnx2.c
index f7ecd07..52756b6 100644
--- a/drivers/net/bnx2.c
+++ b/drivers/net/bnx2.c
@@ -2476,6 +2476,22 @@ bnx2_get_hw_tx_cons(struct bnx2_napi *bnapi)
return cons;
}
+static void
+bnx2_dump_mem(void *ptr, int size)
+{
+ u32 *p = ptr;
+ int i;
+
+ for (i = 0; i < size; i += 4) {
+ if ((i % 32) == 0)
+ printk(KERN_ALERT);
+ printk("%08x ", *p++);
+ if ((i % 32) == 28 || i == (size - 4))
+ printk("\n");
+ }
+ printk(KERN_ALERT "\n");
+}
+
static int
bnx2_tx_int(struct bnx2 *bp, struct bnx2_napi *bnapi, int budget)
{
@@ -2500,6 +2516,15 @@ bnx2_tx_int(struct bnx2 *bp, struct bnx2_napi *bnapi, int budget)
" should be %d\n",
skb_shinfo(skb)->nr_frags,
tx_buf->nr_frags);
+ printk(KERN_ALERT "original skb:\n");
+ bnx2_dump_mem(&bp->tx_skb_ring[sw_ring_cons],
+ sizeof(struct sk_buff));
+ bnx2_dump_mem(&bp->tx_skb_sh_ring[sw_ring_cons],
+ sizeof(struct skb_shared_info));
+ printk(KERN_ALERT "Corrupted skb:\n");
+ bnx2_dump_mem(skb, sizeof(*skb));
+ bnx2_dump_mem(skb_shinfo(skb),
+ sizeof(struct skb_shared_info));
}
/* partial BD completions possible with TSO packets */
if (skb_is_gso(skb)) {
@@ -5866,6 +5891,8 @@ bnx2_start_xmit(struct sk_buff *skb, struct net_device *dev)
last_frag = skb_shinfo(skb)->nr_frags;
tx_buf->nr_frags = last_frag;
+ bp->tx_skb_ring[ring_prod] = *skb;
+ bp->tx_skb_sh_ring[ring_prod] = *skb_shinfo(skb);
for (i = 0; i < last_frag; i++) {
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
diff --git a/drivers/net/bnx2.h b/drivers/net/bnx2.h
index aa9fa6f..91569dd 100644
--- a/drivers/net/bnx2.h
+++ b/drivers/net/bnx2.h
@@ -6649,6 +6649,9 @@ struct bnx2 {
int tx_ring_size;
u32 tx_wake_thresh;
+ struct sk_buff tx_skb_ring[TX_DESC_CNT];
+ struct skb_shared_info tx_skb_sh_ring[TX_DESC_CNT];
+
/* End of fields used in the performance code paths. */
char *name;
^ permalink raw reply related [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-24 22:58 ` Michael Chan
@ 2008-06-25 0:04 ` David Miller
2008-06-26 11:01 ` Josip Rodin
1 sibling, 0 replies; 50+ messages in thread
From: David Miller @ 2008-06-25 0:04 UTC (permalink / raw)
To: mchan; +Cc: joy, billfink, bhutchings, netdev, mirrors
From: "Michael Chan" <mchan@broadcom.com>
Date: Tue, 24 Jun 2008 15:58:33 -0700
> Michael Chan wrote:
> > I will send Josip another patch to print more SKB fields. I can
> > even save all the SKB fields and see which other ones are modified besides
> > the nr_frags. May be that will give us a better clue.
> >
> Please try this patch, which should go on top of the last debug patch.
> Thanks.
Thanks for taking the time to track this down Michael.
Regardless of whether there is some bug in HTB or similar,
the trush is that currently nr_frags can change on us even
if the SKB is cloned.
For example, when using TSO, the TCP ACK processing path can mince up
a TSO packet into two parts, changing nr_frags in the original packet.
This is done by tcp_tso_fragment().
This would rarely hit an SKB still on a device queue, but perhaps HTB
makes this situation much more likely.
This behavior I consider a bug, because it could trigger while the
driver is initially queueing the packet and thus result in strange
OOPS's as the driver ends up trying to derefernce page slots in
skb_shared_info() which suddenly become not value.
Therefore, I suggest we do a few things:
1) Add an assertion to skb_split() such that it BUG()'s if
the SKB it is asked to operate on is cloned/shared.
2) Make tcp_tso_fragment() make a copy if the SKB is cloned.
Similarly to how code pathes leading up to tcp_fragment()
behave.
I'll try to get to this after I finish up some TX path reworking
I'm deeply into at the moment, but if someone beats me to it,
all the better :-)
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-24 22:58 ` Michael Chan
2008-06-25 0:04 ` David Miller
@ 2008-06-26 11:01 ` Josip Rodin
2008-06-26 18:04 ` Michael Chan
1 sibling, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-06-26 11:01 UTC (permalink / raw)
To: Michael Chan
Cc: 'Bill Fink', Ben Hutchings, netdev, mirrors@debian.org
[-- Attachment #1: Type: text/plain, Size: 499 bytes --]
On Tue, Jun 24, 2008 at 03:58:33PM -0700, Michael Chan wrote:
> Michael Chan wrote:
> > I will send Josip another patch to print more SKB fields. I can
> > even save all the SKB fields and see which other ones are modified besides
> > the nr_frags. May be that will give us a better clue.
> >
> Please try this patch, which should go on top of the last debug patch.
> Thanks.
Here we go, this is the first batch of debug output. I hope it helps.
--
2. That which causes joy or happiness.
[-- Attachment #2: arrakis.bnx2.debug.log --]
[-- Type: text/plain, Size: 99007 bytes --]
Jun 25 23:09:01 arrakis kernel: bnx2: skb->nr_frags=2 is corrupted, should be 3
Jun 25 23:09:01 arrakis kernel: original skb:
Jun 25 23:09:01 arrakis kernel: 00000000 00000000 00000000 00000000 15c17980 ffff8100 00000000 00000000
Jun 25 23:09:01 arrakis kernel: 15c80000 ffff8100 cf0eae80 ffff8100 00000000 00000000 00000000 00000000
Jun 25 23:09:01 arrakis kernel: 00000000 00000000 00000000 00000000 51301f6b 5130415b 0aecbf6f 00000010
Jun 25 23:09:01 arrakis kernel: 00000000 00000000 00002232 000021f0 01000000 001000e0 00000000 0008100e
Jun 25 23:09:01 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:09:01 arrakis kernel: 00000021 00000000 000000e0 000000cc 00000000 00000100 00000100 00000000
Jun 25 23:09:01 arrakis kernel: 57c93000 ffff8100 57c930be ffff8100 000023c8 00000001
Jun 25 23:09:01 arrakis kernel:
Jun 25 23:09:01 arrakis kernel: 00010002 05a80003 00010006 00000000 00000000 00000000 0295cd00 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000b48 000004b8 01d0fa58 ffffe200 00000000 00001000 009cdaa0 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00000d38 014b63d8 ffffe200 00000000 00001000 00e67358 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00001000 022d0d50 ffffe200 00000000 00001000 00c51640 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00000b70 0274b658 ffffe200 00000000 00001000 02d37718 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00001000 02d2dd18 ffffe200 00000000 00001000 0115daf0 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00001000 0199cbb0 ffffe200 00000000 00001000 0113ce60 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00001000 00adf130 ffffe200 00000000 00001000 0139f830 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00001000 019318b0 ffffe200 00000000 00000440 07865428 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 000008a8 30fd90cd d3c22e87 58270676 8a776b71
Jun 25 23:09:01 arrakis kernel:
Jun 25 23:09:01 arrakis kernel: Corrupted skb:
Jun 25 23:09:01 arrakis kernel: 00000000 00000000 00000000 00000000 15c17980 ffff8100 00000000 00000000
Jun 25 23:09:01 arrakis kernel: 15c80000 ffff8100 cf0eae80 ffff8100 00000000 00000000 00000000 00000000
Jun 25 23:09:01 arrakis kernel: 00000000 00000000 00000000 00000000 51301f6b 5130415b 0aecbf6f 00000010
Jun 25 23:09:01 arrakis kernel: 00000000 00000000 00002232 000021f0 01000000 001000e0 00000000 0008100e
Jun 25 23:09:01 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:09:01 arrakis kernel: 00000021 00000000 000000e0 000000cc 00000000 00000100 00000100 00000000
Jun 25 23:09:01 arrakis kernel: 57c93000 ffff8100 57c930be ffff8100 000023c8 00000001
Jun 25 23:09:01 arrakis kernel:
Jun 25 23:09:01 arrakis kernel: 00010002 00000002 00000001 00000000 00000000 00000000 0295cd00 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000b48 000004b8 01d0fa58 ffffe200 00000000 000000f0 009cdaa0 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00000d38 014b63d8 ffffe200 00000000 00001000 00e67358 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00001000 022d0d50 ffffe200 00000000 00001000 00c51640 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00000b70 0274b658 ffffe200 00000000 00001000 02d37718 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00001000 02d2dd18 ffffe200 00000000 00001000 0115daf0 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00001000 0199cbb0 ffffe200 00000000 00001000 0113ce60 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00001000 00adf130 ffffe200 00000000 00001000 0139f830 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 00001000 019318b0 ffffe200 00000000 00000440 07865428 ffffe200
Jun 25 23:09:01 arrakis kernel: 00000000 000008a8 30fd90cd d3c22e87 58270676 8a776b71
Jun 25 23:09:01 arrakis kernel:
Jun 25 23:11:29 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 25 23:11:29 arrakis kernel: original skb:
Jun 25 23:11:29 arrakis kernel: 00000000 00000000 00000000 00000000 15c17980 ffff8100 00000000 00000000
Jun 25 23:11:29 arrakis kernel: 15c80000 ffff8100 cf0eae80 ffff8100 00000000 00000000 00000000 00000000
Jun 25 23:11:29 arrakis kernel: 00000000 00000000 00000000 00000000 8b5b5afe 8b5b719e 0aed5000 00000010
Jun 25 23:11:29 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008500e
Jun 25 23:11:29 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:11:29 arrakis kernel: d576f236 00000000 000000e0 000000cc 00000000 00000100 00000100 c6639f68
Jun 25 23:11:29 arrakis kernel: 055bbc00 ffff8102 055bbcbe ffff8102 00001878 00000001
Jun 25 23:11:29 arrakis kernel:
Jun 25 23:11:29 arrakis kernel: 00010002 05a80002 00010004 00000000 00000000 00000000 07992ce8 ffffe200
Jun 25 23:11:29 arrakis kernel: 000006db 00000925 07a38710 ffffe200 00000000 00000d7b 079968d8 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 00001000 056ba278 ffffe200 00000000 00001000 07802ba0 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 00001000 0798e330 ffffe200 00000000 00001000 079022f8 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 00001000 078b8a50 ffffe200 00000000 00001000 078fcbe8 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 00001000 056b31d0 ffffe200 00000000 00001000 06287878 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 00001000 078b0c88 ffffe200 00000000 00001000 06480100 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 00001000 07a4a4c0 ffffe200 00000000 00000823 077f3518 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 0000026b 00b1e948 ffffe200 00000000 00000b3b 07877da8 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 000001d0 ac3ffbe2 507445f7 d9e89c78 6dd1b209
Jun 25 23:11:29 arrakis kernel:
Jun 25 23:11:29 arrakis kernel: Corrupted skb:
Jun 25 23:11:29 arrakis kernel: 00000000 00000000 00000000 00000000 15c17980 ffff8100 00000000 00000000
Jun 25 23:11:29 arrakis kernel: 15c80000 ffff8100 cf0eae80 ffff8100 00000000 00000000 00000000 00000000
Jun 25 23:11:29 arrakis kernel: 00000000 00000000 00000000 00000000 8b5b5afe 8b5b719e 0aed5000 00000010
Jun 25 23:11:29 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008500e
Jun 25 23:11:29 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:11:29 arrakis kernel: d576f236 00000000 000000e0 000000cc 00000000 00000100 00000100 c6639f68
Jun 25 23:11:29 arrakis kernel: 055bbc00 ffff8102 055bbcbe ffff8102 00001878 00000001
Jun 25 23:11:29 arrakis kernel:
Jun 25 23:11:29 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 07992ce8 ffffe200
Jun 25 23:11:29 arrakis kernel: 000006db 000005a8 07a38710 ffffe200 00000000 00000d7b 079968d8 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 00001000 056ba278 ffffe200 00000000 00001000 07802ba0 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 00001000 0798e330 ffffe200 00000000 00001000 079022f8 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 00001000 078b8a50 ffffe200 00000000 00001000 078fcbe8 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 00001000 056b31d0 ffffe200 00000000 00001000 06287878 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 00001000 078b0c88 ffffe200 00000000 00001000 06480100 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 00001000 07a4a4c0 ffffe200 00000000 00000823 077f3518 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 0000026b 00b1e948 ffffe200 00000000 00000b3b 07877da8 ffffe200
Jun 25 23:11:29 arrakis kernel: 00000000 000001d0 ac3ffbe2 507445f7 d9e89c78 6dd1b209
Jun 25 23:11:29 arrakis kernel:
Jun 25 23:13:31 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 25 23:13:31 arrakis kernel: original skb:
Jun 25 23:13:31 arrakis kernel: 00000000 00000000 00000000 00000000 265d33c0 ffff8102 00000000 00000000
Jun 25 23:13:31 arrakis kernel: 15c80000 ffff8100 278b2a80 ffff8102 00000000 00000000 00000000 00000000
Jun 25 23:13:31 arrakis kernel: 00000000 00000000 00000000 00000000 5eead675 5eeae76d 0aedc6b1 00000010
Jun 25 23:13:31 arrakis kernel: 00000000 00000000 0000113a 000010f8 01000000 001000e0 00000000 0008500e
Jun 25 23:13:31 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:13:31 arrakis kernel: d913664c 00000000 000000e0 000000cc 00000000 00000100 00000100 fd99e2bd
Jun 25 23:13:31 arrakis kernel: 75992c00 ffff8100 75992cbe ffff8100 000012d0 00000001
Jun 25 23:13:31 arrakis kernel:
Jun 25 23:13:31 arrakis kernel: 00010002 05a80002 00010003 00000000 00000000 00000000 029147d0 ffffe200
Jun 25 23:13:31 arrakis kernel: 000001e5 00000e1b 02465380 ffffe200 00000000 000002dd 019869c0 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00001000 00c68ef0 ffffe200 00000000 00001000 01501f48 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00001000 015f3fd0 ffffe200 00000000 00001000 009f3468 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00000a95 0137cbe8 ffffe200 00000000 00001000 0215cbd0 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00001000 020e0840 ffffe200 00000000 00001000 02d37718 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00001000 017c3070 ffffe200 00000000 00001000 01303bc0 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00001000 01239aa0 ffffe200 00000000 00001000 00d502a8 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00001000 02d47b30 ffffe200 00000000 00001000 022a4a50 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00000e2a 2c836a7e 798ceb1a 7c86f808 56cf4cda
Jun 25 23:13:31 arrakis kernel:
Jun 25 23:13:31 arrakis kernel: Corrupted skb:
Jun 25 23:13:31 arrakis kernel: 00000000 00000000 00000000 00000000 265d33c0 ffff8102 00000000 00000000
Jun 25 23:13:31 arrakis kernel: 15c80000 ffff8100 278b2a80 ffff8102 00000000 00000000 00000000 00000000
Jun 25 23:13:31 arrakis kernel: 00000000 00000000 00000000 00000000 5eead675 5eeae76d 0aedc6b1 00000010
Jun 25 23:13:31 arrakis kernel: 00000000 00000000 0000113a 000010f8 01000000 001000e0 00000000 0008500e
Jun 25 23:13:31 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:13:31 arrakis kernel: d913664c 00000000 000000e0 000000cc 00000000 00000100 00000100 fd99e2bd
Jun 25 23:13:31 arrakis kernel: 75992c00 ffff8100 75992cbe ffff8100 000012d0 00000001
Jun 25 23:13:31 arrakis kernel:
Jun 25 23:13:31 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 029147d0 ffffe200
Jun 25 23:13:31 arrakis kernel: 000001e5 000005a8 02465380 ffffe200 00000000 000002dd 019869c0 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00001000 00c68ef0 ffffe200 00000000 00001000 01501f48 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00001000 015f3fd0 ffffe200 00000000 00001000 009f3468 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00000a95 0137cbe8 ffffe200 00000000 00001000 0215cbd0 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00001000 020e0840 ffffe200 00000000 00001000 02d37718 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00001000 017c3070 ffffe200 00000000 00001000 01303bc0 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00001000 01239aa0 ffffe200 00000000 00001000 00d502a8 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00001000 02d47b30 ffffe200 00000000 00001000 022a4a50 ffffe200
Jun 25 23:13:31 arrakis kernel: 00000000 00000e2a 2c836a7e 798ceb1a 7c86f808 56cf4cda
Jun 25 23:13:31 arrakis kernel:
Jun 25 23:14:32 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 25 23:14:32 arrakis kernel: original skb:
Jun 25 23:14:32 arrakis kernel: 00000000 00000000 00000000 00000000 004fd380 ffff8101 00000000 00000000
Jun 25 23:14:32 arrakis kernel: 15c80000 ffff8100 83d27e80 ffff8100 00000000 00000000 00000000 00000000
Jun 25 23:14:32 arrakis kernel: 00000000 00000000 00000000 00000000 61d2d45f 61d2dfaf 0aee0233 00000010
Jun 25 23:14:32 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008f00e
Jun 25 23:14:32 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:14:32 arrakis kernel: b3af4a41 00000000 000000e0 000000cc 00000000 00000100 00000100 d5d4b767
Jun 25 23:14:32 arrakis kernel: 7a0cac00 ffff8100 7a0cacbe ffff8100 00000d28 00000001
Jun 25 23:14:32 arrakis kernel:
Jun 25 23:14:32 arrakis kernel: 00010002 05a80002 00010002 00000000 00000000 00000000 0797d268 ffffe200
Jun 25 23:14:32 arrakis kernel: 000006bd 00000943 07900d88 ffffe200 00000000 0000020d 04841218 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00001000 079c6368 ffffe200 00000000 00001000 07901878 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00001000 0647dff8 ffffe200 00000000 00000045 078b79b0 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00001000 04e7c728 ffffe200 00000000 00001000 07893168 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00001000 06f0f018 ffffe200 00000000 00001000 0786e958 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 000001bd 05b5d610 ffffe200 00000000 00001000 079c9008 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00001000 077c6e60 ffffe200 00000000 00001000 04950ab0 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00001000 0494e4d8 ffffe200 00000000 00001000 078883f0 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00000d21 3b357dbf edfa717d 422122ec b17d7ce3
Jun 25 23:14:32 arrakis kernel:
Jun 25 23:14:32 arrakis kernel: Corrupted skb:
Jun 25 23:14:32 arrakis kernel: 00000000 00000000 00000000 00000000 004fd380 ffff8101 00000000 00000000
Jun 25 23:14:32 arrakis kernel: 15c80000 ffff8100 83d27e80 ffff8100 00000000 00000000 00000000 00000000
Jun 25 23:14:32 arrakis kernel: 00000000 00000000 00000000 00000000 61d2d45f 61d2dfaf 0aee0233 00000010
Jun 25 23:14:32 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008f00e
Jun 25 23:14:32 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:14:32 arrakis kernel: b3af4a41 00000000 000000e0 000000cc 00000000 00000100 00000100 d5d4b767
Jun 25 23:14:32 arrakis kernel: 7a0cac00 ffff8100 7a0cacbe ffff8100 00000d28 00000001
Jun 25 23:14:32 arrakis kernel:
Jun 25 23:14:32 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 0797d268 ffffe200
Jun 25 23:14:32 arrakis kernel: 000006bd 000005a8 07900d88 ffffe200 00000000 0000020d 04841218 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00001000 079c6368 ffffe200 00000000 00001000 07901878 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00001000 0647dff8 ffffe200 00000000 00000045 078b79b0 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00001000 04e7c728 ffffe200 00000000 00001000 07893168 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00001000 06f0f018 ffffe200 00000000 00001000 0786e958 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 000001bd 05b5d610 ffffe200 00000000 00001000 079c9008 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00001000 077c6e60 ffffe200 00000000 00001000 04950ab0 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00001000 0494e4d8 ffffe200 00000000 00001000 078883f0 ffffe200
Jun 25 23:14:32 arrakis kernel: 00000000 00000d21 3b357dbf edfa717d 422122ec b17d7ce3
Jun 25 23:14:32 arrakis kernel:
Jun 25 23:17:21 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 3
Jun 25 23:17:21 arrakis kernel: original skb:
Jun 25 23:17:21 arrakis kernel: 00000000 00000000 00000000 00000000 1c403340 ffff8102 00000000 00000000
Jun 25 23:17:21 arrakis kernel: 15c80000 ffff8100 2b4cad00 ffff8102 00000000 00000000 00000000 00000000
Jun 25 23:17:21 arrakis kernel: 00000000 00000000 00000000 00000000 65581399 6558301d 0aeea7a6 00000010
Jun 25 23:17:21 arrakis kernel: 00000000 00000000 00001cba 00001c84 01000000 001000ec 00000000 0008300e
Jun 25 23:17:21 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:17:21 arrakis kernel: ef45bfe1 00000000 000000ec 000000d8 00000000 00000100 00000100 46f000f8
Jun 25 23:17:21 arrakis kernel: 247d5000 ffff8102 247d50ca ffff8102 00001e5c 00000001
Jun 25 23:17:21 arrakis kernel:
Jun 25 23:17:21 arrakis kernel: 00010002 05b40003 00010005 00000000 00000000 00000000 013deec0 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000619 000009e7 02d67b48 ffffe200 00000000 00001000 02226f90 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 0000029d 01a001c8 ffffe200 00000000 00001000 016c9610 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 00001000 01de23f8 ffffe200 00000000 00001000 00b758f8 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 0000070d 0177a398 ffffe200 00000000 00001000 00c76d30 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 00001000 00d74c78 ffffe200 00000000 00001000 026767c0 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 00001000 02b6a128 ffffe200 00000000 000002c1 07866810 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 00001000 07783278 ffffe200 00000000 00001000 07843c70 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 00001000 078b7908 ffffe200 00000000 00000fa9 0331559f d8ad1d6b
Jun 25 23:17:21 arrakis kernel: 4d879a79 62b135c1 985d5ad9 736b9b56 03fbbf93 76becc4b
Jun 25 23:17:21 arrakis kernel:
Jun 25 23:17:21 arrakis kernel: Corrupted skb:
Jun 25 23:17:21 arrakis kernel: 00000000 00000000 00000000 00000000 1c403340 ffff8102 00000000 00000000
Jun 25 23:17:21 arrakis kernel: 15c80000 ffff8100 2b4cad00 ffff8102 00000000 00000000 00000000 00000000
Jun 25 23:17:21 arrakis kernel: 00000000 00000000 00000000 00000000 65581399 6558301d 0aeea7a6 00000010
Jun 25 23:17:21 arrakis kernel: 00000000 00000000 00001cba 00001c84 01000000 001000ec 00000000 0008300e
Jun 25 23:17:21 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:17:21 arrakis kernel: ef45bfe1 00000000 000000ec 000000d8 00000000 00000100 00000100 46f000f8
Jun 25 23:17:21 arrakis kernel: 247d5000 ffff8102 247d50ca ffff8102 00001e5c 00000001
Jun 25 23:17:21 arrakis kernel:
Jun 25 23:17:21 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 013deec0 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000619 000005b4 02d67b48 ffffe200 00000000 00001000 02226f90 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 0000029d 01a001c8 ffffe200 00000000 00001000 016c9610 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 00001000 01de23f8 ffffe200 00000000 00001000 00b758f8 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 0000070d 0177a398 ffffe200 00000000 00001000 00c76d30 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 00001000 00d74c78 ffffe200 00000000 00001000 026767c0 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 00001000 02b6a128 ffffe200 00000000 000002c1 07866810 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 00001000 07783278 ffffe200 00000000 00001000 07843c70 ffffe200
Jun 25 23:17:21 arrakis kernel: 00000000 00001000 078b7908 ffffe200 00000000 00000fa9 0331559f d8ad1d6b
Jun 25 23:17:21 arrakis kernel: 4d879a79 62b135c1 985d5ad9 736b9b56 03fbbf93 76becc4b
Jun 25 23:17:21 arrakis kernel:
Jun 25 23:21:16 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 25 23:21:16 arrakis kernel: original skb:
Jun 25 23:21:16 arrakis kernel: 00000000 00000000 00000000 00000000 004fd380 ffff8101 00000000 00000000
Jun 25 23:21:16 arrakis kernel: 15c80000 ffff8100 83d27e80 ffff8100 00000000 00000000 00000000 00000000
Jun 25 23:21:16 arrakis kernel: 00000000 00000000 00000000 00000000 7c613e8f 7c61552f 0aef8d2e 00000010
Jun 25 23:21:16 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008f00e
Jun 25 23:21:16 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:21:16 arrakis kernel: e63ebeb3 00000000 000000e0 000000cc 00000000 00000100 00000100 21b1f4b4
Jun 25 23:21:16 arrakis kernel: 82da7c00 ffff8100 82da7cbe ffff8100 00001878 00000001
Jun 25 23:21:16 arrakis kernel:
Jun 25 23:21:16 arrakis kernel: 00010002 05a80002 00010004 00000000 00000000 00000000 078ac308 ffffe200
Jun 25 23:21:16 arrakis kernel: 000000ed 00000f13 0793d820 ffffe200 00000000 0000078d 07982d30 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 00001000 0777e8f8 ffffe200 00000000 00001000 056ba320 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 00001000 0778b548 ffffe200 00000000 00001000 07943f28 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 00001000 07922118 ffffe200 00000000 00001000 077b31d8 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 00001000 07869bb0 ffffe200 00000000 00001000 078e9430 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 000004f5 0776ff20 ffffe200 00000000 00001000 077d4060 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 00001000 078aa468 ffffe200 00000000 00001000 07858768 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 0000037d 07090a98 ffffe200 00000000 00001000 07306b40 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 000003a5 eeeaa36c c2338b4b 05eeaa36 43561c98
Jun 25 23:21:16 arrakis kernel:
Jun 25 23:21:16 arrakis kernel: Corrupted skb:
Jun 25 23:21:16 arrakis kernel: 00000000 00000000 00000000 00000000 004fd380 ffff8101 00000000 00000000
Jun 25 23:21:16 arrakis kernel: 15c80000 ffff8100 83d27e80 ffff8100 00000000 00000000 00000000 00000000
Jun 25 23:21:16 arrakis kernel: 00000000 00000000 00000000 00000000 7c613e8f 7c61552f 0aef8d2e 00000010
Jun 25 23:21:16 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008f00e
Jun 25 23:21:16 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:21:16 arrakis kernel: e63ebeb3 00000000 000000e0 000000cc 00000000 00000100 00000100 21b1f4b4
Jun 25 23:21:16 arrakis kernel: 82da7c00 ffff8100 82da7cbe ffff8100 00001878 00000001
Jun 25 23:21:16 arrakis kernel:
Jun 25 23:21:16 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 078ac308 ffffe200
Jun 25 23:21:16 arrakis kernel: 000000ed 000005a8 0793d820 ffffe200 00000000 0000078d 07982d30 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 00001000 0777e8f8 ffffe200 00000000 00001000 056ba320 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 00001000 0778b548 ffffe200 00000000 00001000 07943f28 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 00001000 07922118 ffffe200 00000000 00001000 077b31d8 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 00001000 07869bb0 ffffe200 00000000 00001000 078e9430 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 000004f5 0776ff20 ffffe200 00000000 00001000 077d4060 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 00001000 078aa468 ffffe200 00000000 00001000 07858768 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 0000037d 07090a98 ffffe200 00000000 00001000 07306b40 ffffe200
Jun 25 23:21:16 arrakis kernel: 00000000 000003a5 eeeaa36c c2338b4b 05eeaa36 43561c98
Jun 25 23:21:16 arrakis kernel:
Jun 25 23:21:56 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 25 23:21:56 arrakis kernel: original skb:
Jun 25 23:21:56 arrakis kernel: 00000000 00000000 00000000 00000000 004fd380 ffff8101 00000000 00000000
Jun 25 23:21:56 arrakis kernel: 15c80000 ffff8100 cf928e80 ffff8100 00000000 00000000 00000000 00000000
Jun 25 23:21:56 arrakis kernel: 00000000 00000000 00000000 00000000 808093af 8080aa4f 0aefb3ab 00000010
Jun 25 23:21:56 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008900e
Jun 25 23:21:56 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:21:56 arrakis kernel: 6ff82ce7 00000000 000000e0 000000cc 00000000 00000100 00000100 fc02ee95
Jun 25 23:21:56 arrakis kernel: 7a601400 ffff8101 7a6014be ffff8101 00001878 00000001
Jun 25 23:21:56 arrakis kernel:
Jun 25 23:21:56 arrakis kernel: 00010002 05a80002 00010004 00000000 00000000 00000000 077f3a90 ffffe200
Jun 25 23:21:56 arrakis kernel: 0000060d 000009f3 0787e280 ffffe200 00000000 00000cad 078d5188 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 00001000 05b5f408 ffffe200 00000000 00000e9d 07a36410 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 000008f0 04fe33d0 ffffe200 00000000 00001000 05b54038 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 000004c0 068fc218 ffffe200 00000000 00001000 0574c3d0 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 00001000 0579a780 ffffe200 00000000 00001000 07a20568 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 00001000 078e3498 ffffe200 00000000 00001000 07826498 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 00001000 07865b60 ffffe200 00000000 0000091b 077d8cf0 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 00001000 05b5ad60 ffffe200 00000000 00001000 071807e8 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 0000053d 003f4b84 c54298bd cb8138da dd376270
Jun 25 23:21:56 arrakis kernel:
Jun 25 23:21:56 arrakis kernel: Corrupted skb:
Jun 25 23:21:56 arrakis kernel: 00000000 00000000 00000000 00000000 004fd380 ffff8101 00000000 00000000
Jun 25 23:21:56 arrakis kernel: 15c80000 ffff8100 cf928e80 ffff8100 00000000 00000000 00000000 00000000
Jun 25 23:21:56 arrakis kernel: 00000000 00000000 00000000 00000000 808093af 8080aa4f 0aefb3ab 00000010
Jun 25 23:21:56 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008900e
Jun 25 23:21:56 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:21:56 arrakis kernel: 6ff82ce7 00000000 000000e0 000000cc 00000000 00000100 00000100 fc02ee95
Jun 25 23:21:56 arrakis kernel: 7a601400 ffff8101 7a6014be ffff8101 00001878 00000001
Jun 25 23:21:56 arrakis kernel:
Jun 25 23:21:56 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 077f3a90 ffffe200
Jun 25 23:21:56 arrakis kernel: 0000060d 000005a8 0787e280 ffffe200 00000000 00000cad 078d5188 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 00001000 05b5f408 ffffe200 00000000 00000e9d 07a36410 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 000008f0 04fe33d0 ffffe200 00000000 00001000 05b54038 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 000004c0 068fc218 ffffe200 00000000 00001000 0574c3d0 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 00001000 0579a780 ffffe200 00000000 00001000 07a20568 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 00001000 078e3498 ffffe200 00000000 00001000 07826498 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 00001000 07865b60 ffffe200 00000000 0000091b 077d8cf0 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 00001000 05b5ad60 ffffe200 00000000 00001000 071807e8 ffffe200
Jun 25 23:21:56 arrakis kernel: 00000000 0000053d 003f4b84 c54298bd cb8138da dd376270
Jun 25 23:21:56 arrakis kernel:
Jun 25 23:24:31 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 25 23:24:31 arrakis kernel: original skb:
Jun 25 23:24:31 arrakis kernel: 00000000 00000000 00000000 00000000 15c35880 ffff8100 00000000 00000000
Jun 25 23:24:31 arrakis kernel: 15c80000 ffff8100 21d5e700 ffff8102 00000000 00000000 00000000 00000000
Jun 25 23:24:31 arrakis kernel: 00000000 00000000 00000000 00000000 6b75421c 6b754d6c 0af04b0a 00000010
Jun 25 23:24:31 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008500e
Jun 25 23:24:31 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:24:31 arrakis kernel: 615ea3c3 00000000 000000e0 000000cc 00000000 00000100 00000100 abe78958
Jun 25 23:24:31 arrakis kernel: a47f0000 ffff8100 a47f00be ffff8100 00000d28 00000001
Jun 25 23:24:31 arrakis kernel:
Jun 25 23:24:31 arrakis kernel: 00010002 05a80002 00010002 00000000 00000000 00000000 027d34b8 ffffe200
Jun 25 23:24:31 arrakis kernel: 000007ad 00000853 00c54d98 ffffe200 00000000 000002fd 010bd500 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00001000 0231cdc8 ffffe200 00000000 00001000 02834c68 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00001000 00f71630 ffffe200 00000000 00000135 0231cdc8 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00001000 02834c68 ffffe200 00000000 00001000 00f71630 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00000135 0231cdc8 ffffe200 00000000 00001000 02834c68 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00001000 00f71630 ffffe200 00000000 00000135 0231cdc8 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00001000 02834c68 ffffe200 00000000 00001000 00f71630 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00000135 000543b8 ffffe200 00000000 00001000 00cf8178 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 0000052d f90f3df9 297f46c9 833e7d59 65ff089c
Jun 25 23:24:31 arrakis kernel:
Jun 25 23:24:31 arrakis kernel: Corrupted skb:
Jun 25 23:24:31 arrakis kernel: 00000000 00000000 00000000 00000000 15c35880 ffff8100 00000000 00000000
Jun 25 23:24:31 arrakis kernel: 15c80000 ffff8100 21d5e700 ffff8102 00000000 00000000 00000000 00000000
Jun 25 23:24:31 arrakis kernel: 00000000 00000000 00000000 00000000 6b75421c 6b754d6c 0af04b0a 00000010
Jun 25 23:24:31 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008500e
Jun 25 23:24:31 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:24:31 arrakis kernel: 615ea3c3 00000000 000000e0 000000cc 00000000 00000100 00000100 abe78958
Jun 25 23:24:31 arrakis kernel: a47f0000 ffff8100 a47f00be ffff8100 00000d28 00000001
Jun 25 23:24:31 arrakis kernel:
Jun 25 23:24:31 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 027d34b8 ffffe200
Jun 25 23:24:31 arrakis kernel: 000007ad 000005a8 00c54d98 ffffe200 00000000 000002fd 010bd500 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00001000 0231cdc8 ffffe200 00000000 00001000 02834c68 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00001000 00f71630 ffffe200 00000000 00000135 0231cdc8 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00001000 02834c68 ffffe200 00000000 00001000 00f71630 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00000135 0231cdc8 ffffe200 00000000 00001000 02834c68 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00001000 00f71630 ffffe200 00000000 00000135 0231cdc8 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00001000 02834c68 ffffe200 00000000 00001000 00f71630 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 00000135 000543b8 ffffe200 00000000 00001000 00cf8178 ffffe200
Jun 25 23:24:31 arrakis kernel: 00000000 0000052d f90f3df9 297f46c9 833e7d59 65ff089c
Jun 25 23:24:31 arrakis kernel:
Jun 25 23:28:10 arrakis kernel: bnx2: skb->nr_frags=2 is corrupted, should be 3
Jun 25 23:28:10 arrakis kernel: original skb:
Jun 25 23:28:10 arrakis kernel: 00000000 00000000 00000000 00000000 1c403340 ffff8102 00000000 00000000
Jun 25 23:28:10 arrakis kernel: 15c80000 ffff8100 289d9ac0 ffff8102 00000000 00000000 00000000 00000000
Jun 25 23:28:10 arrakis kernel: 00000000 00000000 00000000 00000000 8b6f7c11 8b6f9895 0af12192 00000010
Jun 25 23:28:10 arrakis kernel: 00000000 00000000 00001cba 00001c84 01000000 001000ec 00000000 0008900e
Jun 25 23:28:10 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:28:10 arrakis kernel: 7f63ab1b 00000000 000000ec 000000d8 00000000 00000100 00000100 600a594c
Jun 25 23:28:10 arrakis kernel: 28a80800 ffff8102 28a808ca ffff8102 00001e5c 00000001
Jun 25 23:28:10 arrakis kernel:
Jun 25 23:28:10 arrakis kernel: 00010002 05b40003 00010005 00000000 00000000 00000000 077cd4c0 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000e91 0000016f 0793b718 ffffe200 00000000 00001000 0274fda8 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00000b15 019c1d98 ffffe200 00000000 00001000 02d315c0 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00001000 02d6c308 ffffe200 00000000 00001000 0792b680 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 0000041d 078d7bc0 ffffe200 00000000 00001000 07980b80 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00001000 078fa2c8 ffffe200 00000000 00001000 07920828 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00001000 07788e90 ffffe200 00000000 00000555 056b5f18 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00001000 0791e838 ffffe200 00000000 00001000 04e812d8 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00001000 0785eb98 ffffe200 00000000 00000845 06283dd8 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00000040 a8dbaf9e 1035f3c8 3c91e710 e1e4de63
Jun 25 23:28:10 arrakis kernel:
Jun 25 23:28:10 arrakis kernel: Corrupted skb:
Jun 25 23:28:10 arrakis kernel: 00000000 00000000 00000000 00000000 1c403340 ffff8102 00000000 00000000
Jun 25 23:28:10 arrakis kernel: 15c80000 ffff8100 289d9ac0 ffff8102 00000000 00000000 00000000 00000000
Jun 25 23:28:10 arrakis kernel: 00000000 00000000 00000000 00000000 8b6f7c11 8b6f9895 0af12192 00000010
Jun 25 23:28:10 arrakis kernel: 00000000 00000000 00001cba 00001c84 01000000 001000ec 00000000 0008900e
Jun 25 23:28:10 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 25 23:28:10 arrakis kernel: 7f63ab1b 00000000 000000ec 000000d8 00000000 00000100 00000100 600a594c
Jun 25 23:28:10 arrakis kernel: 28a80800 ffff8102 28a808ca ffff8102 00001e5c 00000001
Jun 25 23:28:10 arrakis kernel:
Jun 25 23:28:10 arrakis kernel: 00010002 00000002 00000001 00000000 00000000 00000000 077cd4c0 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000e91 0000016f 0793b718 ffffe200 00000000 00000445 0274fda8 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00000b15 019c1d98 ffffe200 00000000 00001000 02d315c0 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00001000 02d6c308 ffffe200 00000000 00001000 0792b680 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 0000041d 078d7bc0 ffffe200 00000000 00001000 07980b80 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00001000 078fa2c8 ffffe200 00000000 00001000 07920828 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00001000 07788e90 ffffe200 00000000 00000555 056b5f18 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00001000 0791e838 ffffe200 00000000 00001000 04e812d8 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00001000 0785eb98 ffffe200 00000000 00000845 06283dd8 ffffe200
Jun 25 23:28:10 arrakis kernel: 00000000 00000040 a8dbaf9e 1035f3c8 3c91e710 e1e4de63
Jun 25 23:28:10 arrakis kernel:
Jun 25 23:30:01 arrakis kernel: TCP: Treason uncloaked! Peer 80.58.205.43:47069/80 shrinks window 231443135:231491211. Repaired.
Jun 26 03:58:50 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 26 03:58:50 arrakis kernel: original skb:
Jun 26 03:58:50 arrakis kernel: 00000000 00000000 00000000 00000000 2e52ed80 ffff8102 00000000 00000000
Jun 26 03:58:50 arrakis kernel: 15c80000 ffff8100 cf0ea980 ffff8100 00000000 00000000 00000000 00000000
Jun 26 03:58:50 arrakis kernel: 00000000 00000000 00000000 00000000 ce92008d ce920bf5 0b2f1455 00000010
Jun 26 03:58:50 arrakis kernel: 00000000 00000000 00000b9e 00000b68 01000000 001000ec 00000000 0008d00e
Jun 26 03:58:50 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 03:58:50 arrakis kernel: e29e9d62 00000000 000000ec 000000d8 00000000 00000100 00000100 a7058e70
Jun 26 03:58:50 arrakis kernel: 248e7000 ffff8102 248e70ca ffff8102 00000d40 00000001
Jun 26 03:58:50 arrakis kernel:
Jun 26 03:58:50 arrakis kernel: 00010002 05b40002 00010002 00000000 00000000 00000000 005d50d0 ffffe200
Jun 26 03:58:50 arrakis kernel: 0000065d 000009a3 026865f0 ffffe200 00000000 000001c5 02573a60 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00001000 015d4760 ffffe200 00000000 000003fd 07932338 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00001000 077f6fb8 ffffe200 00000000 00001000 0797dd90 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00001000 077c1d00 ffffe200 00000000 00001000 07822d08 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00001000 06f078a8 ffffe200 00000000 00001000 06f0ee20 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00001000 078d1448 ffffe200 00000000 00001000 0411f8d8 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00000c84 06f051f0 ffffe200 00000000 00001000 0411f398 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00001000 07987448 ffffe200 00000000 000001fc 05d76ee0 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 000001da d6e3f713 062dba4d a7b3cdce 3462b5c5
Jun 26 03:58:50 arrakis kernel:
Jun 26 03:58:50 arrakis kernel: Corrupted skb:
Jun 26 03:58:50 arrakis kernel: 00000000 00000000 00000000 00000000 2e52ed80 ffff8102 00000000 00000000
Jun 26 03:58:50 arrakis kernel: 15c80000 ffff8100 cf0ea980 ffff8100 00000000 00000000 00000000 00000000
Jun 26 03:58:50 arrakis kernel: 00000000 00000000 00000000 00000000 ce92008d ce920bf5 0b2f1455 00000010
Jun 26 03:58:50 arrakis kernel: 00000000 00000000 00000b9e 00000b68 01000000 001000ec 00000000 0008d00e
Jun 26 03:58:50 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 03:58:50 arrakis kernel: e29e9d62 00000000 000000ec 000000d8 00000000 00000100 00000100 a7058e70
Jun 26 03:58:50 arrakis kernel: 248e7000 ffff8102 248e70ca ffff8102 00000d40 00000001
Jun 26 03:58:50 arrakis kernel:
Jun 26 03:58:50 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 005d50d0 ffffe200
Jun 26 03:58:50 arrakis kernel: 0000065d 000005b4 026865f0 ffffe200 00000000 000001c5 02573a60 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00001000 015d4760 ffffe200 00000000 000003fd 07932338 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00001000 077f6fb8 ffffe200 00000000 00001000 0797dd90 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00001000 077c1d00 ffffe200 00000000 00001000 07822d08 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00001000 06f078a8 ffffe200 00000000 00001000 06f0ee20 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00001000 078d1448 ffffe200 00000000 00001000 0411f8d8 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00000c84 06f051f0 ffffe200 00000000 00001000 0411f398 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 00001000 07987448 ffffe200 00000000 000001fc 05d76ee0 ffffe200
Jun 26 03:58:50 arrakis kernel: 00000000 000001da d6e3f713 062dba4d a7b3cdce 3462b5c5
Jun 26 03:58:50 arrakis kernel:
Jun 26 03:58:55 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 26 03:58:55 arrakis kernel: original skb:
Jun 26 03:58:55 arrakis kernel: 00000000 00000000 00000000 00000000 23ea18c0 ffff8102 00000000 00000000
Jun 26 03:58:55 arrakis kernel: 15c80000 ffff8100 b3a80840 ffff8100 00000000 00000000 00000000 00000000
Jun 26 03:58:55 arrakis kernel: 00000000 00000000 00000000 00000000 7701cf69 7701dab9 0b2f18e7 00000010
Jun 26 03:58:55 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008100e
Jun 26 03:58:55 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 03:58:55 arrakis kernel: 84a3c676 00000000 000000e0 000000cc 00000000 00000100 00000100 02214f64
Jun 26 03:58:55 arrakis kernel: 728e4800 ffff8100 728e48be ffff8100 00000d28 00000001
Jun 26 03:58:55 arrakis kernel:
Jun 26 03:58:55 arrakis kernel: 00010002 05a80002 00010002 00000000 00000000 00000000 078e19b0 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000521 00000adf 078810a8 ffffe200 00000000 00000071 04e86eb8 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 078e0bb0 ffffe200 00000000 00001000 07947db8 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 0797fcd8 ffffe200 00000000 00001000 0783de98 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 079494e8 ffffe200 00000000 00001000 077c5180 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 077da3e8 ffffe200 00000000 00001000 0484a4a8 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 07941720 ffffe200 00000000 00001000 07636ea0 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 078442c8 ffffe200 00000000 00001000 0792f6d0 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 078cfa08 ffffe200 00000000 00001000 06285b60 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 000003a9 00000000 00000000 00000000 00000000
Jun 26 03:58:55 arrakis kernel:
Jun 26 03:58:55 arrakis kernel: Corrupted skb:
Jun 26 03:58:55 arrakis kernel: 00000000 00000000 00000000 00000000 23ea18c0 ffff8102 00000000 00000000
Jun 26 03:58:55 arrakis kernel: 15c80000 ffff8100 b3a80840 ffff8100 00000000 00000000 00000000 00000000
Jun 26 03:58:55 arrakis kernel: 00000000 00000000 00000000 00000000 7701cf69 7701dab9 0b2f18e7 00000010
Jun 26 03:58:55 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008100e
Jun 26 03:58:55 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 03:58:55 arrakis kernel: 84a3c676 00000000 000000e0 000000cc 00000000 00000100 00000100 02214f64
Jun 26 03:58:55 arrakis kernel: 728e4800 ffff8100 728e48be ffff8100 00000d28 00000001
Jun 26 03:58:55 arrakis kernel:
Jun 26 03:58:55 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 078e19b0 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000521 000005a8 078810a8 ffffe200 00000000 00000071 04e86eb8 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 078e0bb0 ffffe200 00000000 00001000 07947db8 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 0797fcd8 ffffe200 00000000 00001000 0783de98 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 079494e8 ffffe200 00000000 00001000 077c5180 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 077da3e8 ffffe200 00000000 00001000 0484a4a8 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 07941720 ffffe200 00000000 00001000 07636ea0 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 078442c8 ffffe200 00000000 00001000 0792f6d0 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 00001000 078cfa08 ffffe200 00000000 00001000 06285b60 ffffe200
Jun 26 03:58:55 arrakis kernel: 00000000 000003a9 00000000 00000000 00000000 00000000
Jun 26 03:58:55 arrakis kernel:
Jun 26 04:00:08 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 26 04:00:08 arrakis kernel: original skb:
Jun 26 04:00:08 arrakis kernel: 00000000 00000000 00000000 00000000 23ea18c0 ffff8102 00000000 00000000
Jun 26 04:00:08 arrakis kernel: 15c80000 ffff8100 b3a80840 ffff8100 00000000 00000000 00000000 00000000
Jun 26 04:00:08 arrakis kernel: 00000000 00000000 00000000 00000000 78e8b089 78e8bbd9 0b2f606b 00000010
Jun 26 04:00:08 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008100e
Jun 26 04:00:08 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:00:08 arrakis kernel: 9ad91d3e 00000000 000000e0 000000cc 00000000 00000100 00000100 dd7bd86a
Jun 26 04:00:08 arrakis kernel: 2e88e800 ffff8102 2e88e8be ffff8102 00000d28 00000001
Jun 26 04:00:08 arrakis kernel:
Jun 26 04:00:08 arrakis kernel: 00010002 05a80002 00010002 00000000 00000000 00000000 0799f580 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000641 000009bf 079817f8 ffffe200 00000000 00000191 077b17d0 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 07863630 ffffe200 00000000 00001000 078cc5c0 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 07633860 ffffe200 00000000 00000571 0791cb90 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 0647d6c8 ffffe200 00000000 00001000 078cdf90 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 07832ef0 ffffe200 00000000 00001000 07821370 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 0411acf0 ffffe200 00000000 00001000 0484ae80 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 05c1d6d8 ffffe200 00000000 00001000 076373a8 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 07984888 ffffe200 00000000 000007e4 0628b890 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 000007e4 00000000 00000000 00000000 00000000
Jun 26 04:00:08 arrakis kernel:
Jun 26 04:00:08 arrakis kernel: Corrupted skb:
Jun 26 04:00:08 arrakis kernel: 00000000 00000000 00000000 00000000 23ea18c0 ffff8102 00000000 00000000
Jun 26 04:00:08 arrakis kernel: 15c80000 ffff8100 b3a80840 ffff8100 00000000 00000000 00000000 00000000
Jun 26 04:00:08 arrakis kernel: 00000000 00000000 00000000 00000000 78e8b089 78e8bbd9 0b2f606b 00000010
Jun 26 04:00:08 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008100e
Jun 26 04:00:08 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:00:08 arrakis kernel: 9ad91d3e 00000000 000000e0 000000cc 00000000 00000100 00000100 dd7bd86a
Jun 26 04:00:08 arrakis kernel: 2e88e800 ffff8102 2e88e8be ffff8102 00000d28 00000001
Jun 26 04:00:08 arrakis kernel:
Jun 26 04:00:08 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 0799f580 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000641 000005a8 079817f8 ffffe200 00000000 00000191 077b17d0 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 07863630 ffffe200 00000000 00001000 078cc5c0 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 07633860 ffffe200 00000000 00000571 0791cb90 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 0647d6c8 ffffe200 00000000 00001000 078cdf90 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 07832ef0 ffffe200 00000000 00001000 07821370 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 0411acf0 ffffe200 00000000 00001000 0484ae80 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 05c1d6d8 ffffe200 00000000 00001000 076373a8 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 00001000 07984888 ffffe200 00000000 000007e4 0628b890 ffffe200
Jun 26 04:00:08 arrakis kernel: 00000000 000007e4 00000000 00000000 00000000 00000000
Jun 26 04:00:08 arrakis kernel:
Jun 26 04:00:21 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 26 04:00:21 arrakis kernel: original skb:
Jun 26 04:00:21 arrakis kernel: 00000000 00000000 00000000 00000000 23ea18c0 ffff8102 00000000 00000000
Jun 26 04:00:21 arrakis kernel: 15c80000 ffff8100 b3a80840 ffff8100 00000000 00000000 00000000 00000000
Jun 26 04:00:21 arrakis kernel: 00000000 00000000 00000000 00000000 793db021 793dbb71 0b2f6cfc 00000010
Jun 26 04:00:21 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008300e
Jun 26 04:00:21 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:00:21 arrakis kernel: c33a205b 00000000 000000e0 000000cc 00000000 00000100 00000100 d6b500cb
Jun 26 04:00:21 arrakis kernel: 28afd000 ffff8102 28afd0be ffff8102 00000d28 00000001
Jun 26 04:00:21 arrakis kernel:
Jun 26 04:00:21 arrakis kernel: 00010002 05a80002 00010002 00000000 00000000 00000000 07906038 ffffe200
Jun 26 04:00:21 arrakis kernel: 000005d9 00000a27 04fe5628 ffffe200 00000000 00000129 07884758 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00001000 079728a8 ffffe200 00000000 00001000 078c6820 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00001000 06480d08 ffffe200 00000000 00001000 0647ae88 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00001000 07932d80 ffffe200 00000000 00001000 04842e50 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00001000 04848d78 ffffe200 00000000 00001000 0792f6d0 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00001000 07893398 ffffe200 00000000 00001000 078d0648 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00001000 07a484d0 ffffe200 00000000 00000721 079989e0 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00000354 04114f50 ffffe200 00000000 0000041c 07910950 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 0000006c 00000000 00000000 00000000 00000000
Jun 26 04:00:21 arrakis kernel:
Jun 26 04:00:21 arrakis kernel: Corrupted skb:
Jun 26 04:00:21 arrakis kernel: 00000000 00000000 00000000 00000000 23ea18c0 ffff8102 00000000 00000000
Jun 26 04:00:21 arrakis kernel: 15c80000 ffff8100 b3a80840 ffff8100 00000000 00000000 00000000 00000000
Jun 26 04:00:21 arrakis kernel: 00000000 00000000 00000000 00000000 793db021 793dbb71 0b2f6cfc 00000010
Jun 26 04:00:21 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008300e
Jun 26 04:00:21 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:00:21 arrakis kernel: c33a205b 00000000 000000e0 000000cc 00000000 00000100 00000100 d6b500cb
Jun 26 04:00:21 arrakis kernel: 28afd000 ffff8102 28afd0be ffff8102 00000d28 00000001
Jun 26 04:00:21 arrakis kernel:
Jun 26 04:00:21 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 07906038 ffffe200
Jun 26 04:00:21 arrakis kernel: 000005d9 000005a8 04fe5628 ffffe200 00000000 00000129 07884758 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00001000 079728a8 ffffe200 00000000 00001000 078c6820 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00001000 06480d08 ffffe200 00000000 00001000 0647ae88 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00001000 07932d80 ffffe200 00000000 00001000 04842e50 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00001000 04848d78 ffffe200 00000000 00001000 0792f6d0 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00001000 07893398 ffffe200 00000000 00001000 078d0648 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00001000 07a484d0 ffffe200 00000000 00000721 079989e0 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 00000354 04114f50 ffffe200 00000000 0000041c 07910950 ffffe200
Jun 26 04:00:21 arrakis kernel: 00000000 0000006c 00000000 00000000 00000000 00000000
Jun 26 04:00:21 arrakis kernel:
Jun 26 04:01:52 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 3
Jun 26 04:01:52 arrakis kernel: original skb:
Jun 26 04:01:52 arrakis kernel: 00000000 00000000 00000000 00000000 a53d2d00 ffff8101 00000000 00000000
Jun 26 04:01:52 arrakis kernel: 15c80000 ffff8100 2bfe6ac0 ffff8102 00000000 00000000 00000000 00000000
Jun 26 04:01:52 arrakis kernel: 00000000 00000000 00000000 00000000 442a1a03 442a3bf3 0b2fc621 00000010
Jun 26 04:01:52 arrakis kernel: 00000000 00000000 00002232 000021f0 01000000 001000e0 00000000 0008100e
Jun 26 04:01:52 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:01:52 arrakis kernel: 0000000a 00000000 000000e0 000000cc 00000000 00000100 00000100 00000000
Jun 26 04:01:52 arrakis kernel: cb76e000 ffff8101 cb76e0be ffff8101 000023c8 00000001
Jun 26 04:01:52 arrakis kernel:
Jun 26 04:01:52 arrakis kernel: 00010002 05a80003 00010006 00000000 00000000 00000000 07985458 ffffe200
Jun 26 04:01:52 arrakis kernel: 0000034e 00000cb2 077b0e30 ffffe200 00000000 00001000 07982a58 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 0000053e 0777e038 ffffe200 00000000 00001000 05c217d0 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 00001000 0647beb8 ffffe200 00000000 0000027e 07922348 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 0000099e 07933a30 ffffe200 00000000 00000efe 077ad1d0 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 0000061e 078fbbf0 ffffe200 00000000 00001000 07892f38 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 00000bed 0259fba0 ffffe200 00000000 00001000 007c5180 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 00001000 00c8b160 ffffe200 00000000 0000050d 078c3480 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 00000b45 077e9e98 ffffe200 00000000 000005b5 07938340 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 00000c55 8b913da7 c0172940 ff900fc3 38812a75
Jun 26 04:01:52 arrakis kernel:
Jun 26 04:01:52 arrakis kernel: Corrupted skb:
Jun 26 04:01:52 arrakis kernel: 00000000 00000000 00000000 00000000 a53d2d00 ffff8101 00000000 00000000
Jun 26 04:01:52 arrakis kernel: 15c80000 ffff8100 2bfe6ac0 ffff8102 00000000 00000000 00000000 00000000
Jun 26 04:01:52 arrakis kernel: 00000000 00000000 00000000 00000000 442a1a03 442a3bf3 0b2fc621 00000010
Jun 26 04:01:52 arrakis kernel: 00000000 00000000 00002232 000021f0 01000000 001000e0 00000000 0008100e
Jun 26 04:01:52 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:01:52 arrakis kernel: 0000000a 00000000 000000e0 000000cc 00000000 00000100 00000100 00000000
Jun 26 04:01:52 arrakis kernel: cb76e000 ffff8101 cb76e0be ffff8101 000023c8 00000001
Jun 26 04:01:52 arrakis kernel:
Jun 26 04:01:52 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 07985458 ffffe200
Jun 26 04:01:52 arrakis kernel: 0000034e 000005a8 077b0e30 ffffe200 00000000 00001000 07982a58 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 0000053e 0777e038 ffffe200 00000000 00001000 05c217d0 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 00001000 0647beb8 ffffe200 00000000 0000027e 07922348 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 0000099e 07933a30 ffffe200 00000000 00000efe 077ad1d0 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 0000061e 078fbbf0 ffffe200 00000000 00001000 07892f38 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 00000bed 0259fba0 ffffe200 00000000 00001000 007c5180 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 00001000 00c8b160 ffffe200 00000000 0000050d 078c3480 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 00000b45 077e9e98 ffffe200 00000000 000005b5 07938340 ffffe200
Jun 26 04:01:52 arrakis kernel: 00000000 00000c55 8b913da7 c0172940 ff900fc3 38812a75
Jun 26 04:01:52 arrakis kernel:
Jun 26 04:02:08 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 26 04:02:08 arrakis kernel: original skb:
Jun 26 04:02:08 arrakis kernel: 00000000 00000000 00000000 00000000 23ea18c0 ffff8102 00000000 00000000
Jun 26 04:02:08 arrakis kernel: 15c80000 ffff8100 b3a80840 ffff8100 00000000 00000000 00000000 00000000
Jun 26 04:02:08 arrakis kernel: 00000000 00000000 00000000 00000000 7c06f331 7c0709d1 0b2fd5e5 00000010
Jun 26 04:02:08 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008300e
Jun 26 04:02:08 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:02:08 arrakis kernel: 206e754a 00000000 000000e0 000000cc 00000000 00000100 00000100 2020203a
Jun 26 04:02:08 arrakis kernel: cee61400 ffff8100 cee614be ffff8100 00001878 00000001
Jun 26 04:02:08 arrakis kernel:
Jun 26 04:02:08 arrakis kernel: 00010002 05a80002 00010004 00000000 00000000 00000000 046efa40 ffffe200
Jun 26 04:02:08 arrakis kernel: 000008e9 00000717 04e7c108 ffffe200 00000000 00000f89 077687e8 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00001000 077d1900 ffffe200 00000000 00001000 04114e00 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00001000 0799f580 ffffe200 00000000 00001000 0786c968 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00001000 07902330 ffffe200 00000000 00000a09 04fe4438 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00001000 05b59160 ffffe200 00000000 00001000 04952058 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00001000 077e2808 ffffe200 00000000 00001000 07907650 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 000004a1 078587d8 ffffe200 00000000 00001000 077de860 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00001000 079bef40 ffffe200 00000000 00001000 0792f548 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00000909 0f0adaab 3ca68f3f 5c23f4d8 7fbe4f3c
Jun 26 04:02:08 arrakis kernel:
Jun 26 04:02:08 arrakis kernel: Corrupted skb:
Jun 26 04:02:08 arrakis kernel: 00000000 00000000 00000000 00000000 23ea18c0 ffff8102 00000000 00000000
Jun 26 04:02:08 arrakis kernel: 15c80000 ffff8100 b3a80840 ffff8100 00000000 00000000 00000000 00000000
Jun 26 04:02:08 arrakis kernel: 00000000 00000000 00000000 00000000 7c06f331 7c0709d1 0b2fd5e5 00000010
Jun 26 04:02:08 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008300e
Jun 26 04:02:08 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:02:08 arrakis kernel: 206e754a 00000000 000000e0 000000cc 00000000 00000100 00000100 2020203a
Jun 26 04:02:08 arrakis kernel: cee61400 ffff8100 cee614be ffff8100 00001878 00000001
Jun 26 04:02:08 arrakis kernel:
Jun 26 04:02:08 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 046efa40 ffffe200
Jun 26 04:02:08 arrakis kernel: 000008e9 000005a8 04e7c108 ffffe200 00000000 00000f89 077687e8 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00001000 077d1900 ffffe200 00000000 00001000 04114e00 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00001000 0799f580 ffffe200 00000000 00001000 0786c968 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00001000 07902330 ffffe200 00000000 00000a09 04fe4438 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00001000 05b59160 ffffe200 00000000 00001000 04952058 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00001000 077e2808 ffffe200 00000000 00001000 07907650 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 000004a1 078587d8 ffffe200 00000000 00001000 077de860 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00001000 079bef40 ffffe200 00000000 00001000 0792f548 ffffe200
Jun 26 04:02:08 arrakis kernel: 00000000 00000909 0f0adaab 3ca68f3f 5c23f4d8 7fbe4f3c
Jun 26 04:02:08 arrakis kernel:
Jun 26 04:03:53 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 26 04:03:53 arrakis kernel: original skb:
Jun 26 04:03:53 arrakis kernel: 00000000 00000000 00000000 00000000 23ea18c0 ffff8102 00000000 00000000
Jun 26 04:03:53 arrakis kernel: 15c80000 ffff8100 b3a80840 ffff8100 00000000 00000000 00000000 00000000
Jun 26 04:03:53 arrakis kernel: 00000000 00000000 00000000 00000000 7ebce109 7ebcec59 0b303bf2 00000010
Jun 26 04:03:53 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008700e
Jun 26 04:03:53 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:03:53 arrakis kernel: 58de8e9b 00000000 000000e0 000000cc 00000000 00000100 00000100 4f52b489
Jun 26 04:03:53 arrakis kernel: ce9fec00 ffff8100 ce9fecbe ffff8100 00000d28 00000001
Jun 26 04:03:53 arrakis kernel:
Jun 26 04:03:53 arrakis kernel: 00010002 05a80002 00010002 00000000 00000000 00000000 06283978 ffffe200
Jun 26 04:03:53 arrakis kernel: 000006c1 0000093f 079689d8 ffffe200 00000000 00000211 0791e100 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 00001000 07824a20 ffffe200 00000000 00001000 0792f548 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 00001000 07932d80 ffffe200 00000000 00001000 0785c1d0 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 00001000 079498d8 ffffe200 00000000 00001000 07860ed0 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 000008d9 0274ab30 ffffe200 00000000 00001000 02c47cd8 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 00001000 00276700 ffffe200 00000000 00000901 07892ad8 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 00001000 078f86c8 ffffe200 00000000 00000979 077c7870 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 00001000 0793b328 ffffe200 00000000 00000de1 04fe0490 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 000004b8 00000000 00000000 00000000 00000000
Jun 26 04:03:53 arrakis kernel:
Jun 26 04:03:53 arrakis kernel: Corrupted skb:
Jun 26 04:03:53 arrakis kernel: 00000000 00000000 00000000 00000000 23ea18c0 ffff8102 00000000 00000000
Jun 26 04:03:53 arrakis kernel: 15c80000 ffff8100 b3a80840 ffff8100 00000000 00000000 00000000 00000000
Jun 26 04:03:53 arrakis kernel: 00000000 00000000 00000000 00000000 7ebce109 7ebcec59 0b303bf2 00000010
Jun 26 04:03:53 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008700e
Jun 26 04:03:53 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:03:53 arrakis kernel: 58de8e9b 00000000 000000e0 000000cc 00000000 00000100 00000100 4f52b489
Jun 26 04:03:53 arrakis kernel: ce9fec00 ffff8100 ce9fecbe ffff8100 00000d28 00000001
Jun 26 04:03:53 arrakis kernel:
Jun 26 04:03:53 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 06283978 ffffe200
Jun 26 04:03:53 arrakis kernel: 000006c1 000005a8 079689d8 ffffe200 00000000 00000211 0791e100 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 00001000 07824a20 ffffe200 00000000 00001000 0792f548 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 00001000 07932d80 ffffe200 00000000 00001000 0785c1d0 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 00001000 079498d8 ffffe200 00000000 00001000 07860ed0 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 000008d9 0274ab30 ffffe200 00000000 00001000 02c47cd8 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 00001000 00276700 ffffe200 00000000 00000901 07892ad8 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 00001000 078f86c8 ffffe200 00000000 00000979 077c7870 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 00001000 0793b328 ffffe200 00000000 00000de1 04fe0490 ffffe200
Jun 26 04:03:53 arrakis kernel: 00000000 000004b8 00000000 00000000 00000000 00000000
Jun 26 04:03:53 arrakis kernel:
Jun 26 04:03:57 arrakis kernel: bnx2: skb->nr_frags=2 is corrupted, should be 3
Jun 26 04:03:57 arrakis kernel: original skb:
Jun 26 04:03:57 arrakis kernel: 00000000 00000000 00000000 00000000 23ea18c0 ffff8102 00000000 00000000
Jun 26 04:03:57 arrakis kernel: 15c80000 ffff8100 b3a80840 ffff8100 00000000 00000000 00000000 00000000
Jun 26 04:03:57 arrakis kernel: 00000000 00000000 00000000 00000000 7edc0859 7edc1ef9 0b3040ad 00000010
Jun 26 04:03:57 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008900e
Jun 26 04:03:57 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:03:57 arrakis kernel: 7db159ab 00000000 000000e0 000000cc 00000000 00000100 00000100 63ce096f
Jun 26 04:03:57 arrakis kernel: cfac9000 ffff8100 cfac90be ffff8100 00001878 00000001
Jun 26 04:03:57 arrakis kernel:
Jun 26 04:03:57 arrakis kernel: 00010002 05a80003 00010004 00000000 00000000 00000000 07933a30 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000e11 000001ef 04952790 ffffe200 00000000 00001000 0792aa78 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 000004b1 07918670 ffffe200 00000000 00000b51 0647d5e8 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00001000 04fdac30 ffffe200 00000000 00001000 079279e8 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00001000 079c58b0 ffffe200 00000000 00001000 077b1108 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00001000 06f0cca8 ffffe200 00000000 00001000 079728a8 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00001000 078035b0 ffffe200 00000000 00001000 077cd418 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00000e41 023b0260 ffffe200 00000000 00001000 029dbe10 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00001000 00bfe718 ffffe200 00000000 00000159 07939370 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00000009 3b9dc031 63eed13a 27689a0a fb700f22
Jun 26 04:03:57 arrakis kernel:
Jun 26 04:03:57 arrakis kernel: Corrupted skb:
Jun 26 04:03:57 arrakis kernel: 00000000 00000000 00000000 00000000 23ea18c0 ffff8102 00000000 00000000
Jun 26 04:03:57 arrakis kernel: 15c80000 ffff8100 b3a80840 ffff8100 00000000 00000000 00000000 00000000
Jun 26 04:03:57 arrakis kernel: 00000000 00000000 00000000 00000000 7edc0859 7edc1ef9 0b3040ad 00000010
Jun 26 04:03:57 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008900e
Jun 26 04:03:57 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:03:57 arrakis kernel: 7db159ab 00000000 000000e0 000000cc 00000000 00000100 00000100 63ce096f
Jun 26 04:03:57 arrakis kernel: cfac9000 ffff8100 cfac90be ffff8100 00001878 00000001
Jun 26 04:03:57 arrakis kernel:
Jun 26 04:03:57 arrakis kernel: 00010002 00000002 00000001 00000000 00000000 00000000 07933a30 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000e11 000001ef 04952790 ffffe200 00000000 000003b9 0792aa78 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 000004b1 07918670 ffffe200 00000000 00000b51 0647d5e8 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00001000 04fdac30 ffffe200 00000000 00001000 079279e8 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00001000 079c58b0 ffffe200 00000000 00001000 077b1108 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00001000 06f0cca8 ffffe200 00000000 00001000 079728a8 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00001000 078035b0 ffffe200 00000000 00001000 077cd418 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00000e41 023b0260 ffffe200 00000000 00001000 029dbe10 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00001000 00bfe718 ffffe200 00000000 00000159 07939370 ffffe200
Jun 26 04:03:57 arrakis kernel: 00000000 00000009 3b9dc031 63eed13a 27689a0a fb700f22
Jun 26 04:03:57 arrakis kernel:
Jun 26 04:04:32 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 26 04:04:32 arrakis kernel: original skb:
Jun 26 04:04:32 arrakis kernel: 00000000 00000000 00000000 00000000 a53d2d00 ffff8101 00000000 00000000
Jun 26 04:04:32 arrakis kernel: 15c80000 ffff8100 21d5e340 ffff8102 00000000 00000000 00000000 00000000
Jun 26 04:04:32 arrakis kernel: 00000000 00000000 00000000 00000000 4c49fd1b 4c4a086b 0b3062ab 00000010
Jun 26 04:04:32 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008b00e
Jun 26 04:04:32 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:04:32 arrakis kernel: 3adf9a26 00000000 000000e0 000000cc 00000000 00000100 00000100 7140523b
Jun 26 04:04:32 arrakis kernel: cfbbc800 ffff8100 cfbbc8be ffff8100 00000d28 00000001
Jun 26 04:04:32 arrakis kernel:
Jun 26 04:04:32 arrakis kernel: 00010002 05a80002 00010002 00000000 00000000 00000000 02d2e920 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000666 0000099a 02d3a9d8 ffffe200 00000000 000001b6 02749cc0 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 00001000 02751d28 ffffe200 00000000 00001000 01b9b1d8 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 00001000 012b3820 ffffe200 00000000 00001000 019c1900 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 0000068e 077f8368 ffffe200 00000000 00001000 0793ba98 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 000004ce 07876a68 ffffe200 00000000 00001000 0777eab8 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 00001000 07636d88 ffffe200 00000000 00001000 07933a30 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 00001000 079683f0 ffffe200 00000000 00001000 079383b0 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 00000b38 02844130 ffffe200 00000000 00001000 021472d8 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 00000e30 7af1713b 4fc25aba 5ce4081f 24b970c8
Jun 26 04:04:32 arrakis kernel:
Jun 26 04:04:32 arrakis kernel: Corrupted skb:
Jun 26 04:04:32 arrakis kernel: 00000000 00000000 00000000 00000000 a53d2d00 ffff8101 00000000 00000000
Jun 26 04:04:32 arrakis kernel: 15c80000 ffff8100 21d5e340 ffff8102 00000000 00000000 00000000 00000000
Jun 26 04:04:32 arrakis kernel: 00000000 00000000 00000000 00000000 4c49fd1b 4c4a086b 0b3062ab 00000010
Jun 26 04:04:32 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008b00e
Jun 26 04:04:32 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:04:32 arrakis kernel: 3adf9a26 00000000 000000e0 000000cc 00000000 00000100 00000100 7140523b
Jun 26 04:04:32 arrakis kernel: cfbbc800 ffff8100 cfbbc8be ffff8100 00000d28 00000001
Jun 26 04:04:32 arrakis kernel:
Jun 26 04:04:32 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 02d2e920 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000666 000005a8 02d3a9d8 ffffe200 00000000 000001b6 02749cc0 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 00001000 02751d28 ffffe200 00000000 00001000 01b9b1d8 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 00001000 012b3820 ffffe200 00000000 00001000 019c1900 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 0000068e 077f8368 ffffe200 00000000 00001000 0793ba98 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 000004ce 07876a68 ffffe200 00000000 00001000 0777eab8 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 00001000 07636d88 ffffe200 00000000 00001000 07933a30 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 00001000 079683f0 ffffe200 00000000 00001000 079383b0 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 00000b38 02844130 ffffe200 00000000 00001000 021472d8 ffffe200
Jun 26 04:04:32 arrakis kernel: 00000000 00000e30 7af1713b 4fc25aba 5ce4081f 24b970c8
Jun 26 04:04:32 arrakis kernel:
Jun 26 04:37:41 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 26 04:37:41 arrakis kernel: original skb:
Jun 26 04:37:41 arrakis kernel: 00000000 00000000 00000000 00000000 a53d2d00 ffff8101 00000000 00000000
Jun 26 04:37:41 arrakis kernel: 15c80000 ffff8100 25fd1980 ffff8102 00000000 00000000 00000000 00000000
Jun 26 04:37:41 arrakis kernel: 00000000 00000000 00000000 00000000 b642ddbb b642e90b 0b37f93e 00000010
Jun 26 04:37:41 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008100e
Jun 26 04:37:41 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:37:41 arrakis kernel: 76fdf047 00000000 000000e0 000000cc 00000000 00000100 00000100 6debfca8
Jun 26 04:37:41 arrakis kernel: 27d8dc00 ffff8102 27d8dcbe ffff8102 00000d28 00000001
Jun 26 04:37:41 arrakis kernel:
Jun 26 04:37:41 arrakis kernel: 00010002 05a80002 00010002 00000000 00000000 00000000 01017e58 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000706 000008fa 008fd4e0 ffffe200 00000000 00000256 0198bca8 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00001000 021776d0 ffffe200 00000000 00001000 02d376e0 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00001000 02d39970 ffffe200 00000000 00000bde 02d307c0 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00001000 02d33000 ffffe200 00000000 000002fe 02750080 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00000a1e 02c43828 ffffe200 00000000 00001000 02d4e388 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00001000 0274bcb0 ffffe200 00000000 00001000 02d48498 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00001000 019c0160 ffffe200 00000000 00001000 02d412d8 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00001000 021bd4d8 ffffe200 00000000 00001000 02d46d30 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00000380 00000000 00000000 00000000 00000000
Jun 26 04:37:41 arrakis kernel:
Jun 26 04:37:41 arrakis kernel: Corrupted skb:
Jun 26 04:37:41 arrakis kernel: 00000000 00000000 00000000 00000000 a53d2d00 ffff8101 00000000 00000000
Jun 26 04:37:41 arrakis kernel: 15c80000 ffff8100 25fd1980 ffff8102 00000000 00000000 00000000 00000000
Jun 26 04:37:41 arrakis kernel: 00000000 00000000 00000000 00000000 b642ddbb b642e90b 0b37f93e 00000010
Jun 26 04:37:41 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008100e
Jun 26 04:37:41 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:37:41 arrakis kernel: 76fdf047 00000000 000000e0 000000cc 00000000 00000100 00000100 6debfca8
Jun 26 04:37:41 arrakis kernel: 27d8dc00 ffff8102 27d8dcbe ffff8102 00000d28 00000001
Jun 26 04:37:41 arrakis kernel:
Jun 26 04:37:41 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 01017e58 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000706 000005a8 008fd4e0 ffffe200 00000000 00000256 0198bca8 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00001000 021776d0 ffffe200 00000000 00001000 02d376e0 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00001000 02d39970 ffffe200 00000000 00000bde 02d307c0 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00001000 02d33000 ffffe200 00000000 000002fe 02750080 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00000a1e 02c43828 ffffe200 00000000 00001000 02d4e388 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00001000 0274bcb0 ffffe200 00000000 00001000 02d48498 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00001000 019c0160 ffffe200 00000000 00001000 02d412d8 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00001000 021bd4d8 ffffe200 00000000 00001000 02d46d30 ffffe200
Jun 26 04:37:41 arrakis kernel: 00000000 00000380 00000000 00000000 00000000 00000000
Jun 26 04:37:41 arrakis kernel:
Jun 26 04:39:14 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 26 04:39:14 arrakis kernel: original skb:
Jun 26 04:39:14 arrakis kernel: 00000000 00000000 00000000 00000000 a53d2d00 ffff8101 00000000 00000000
Jun 26 04:39:14 arrakis kernel: 15c80000 ffff8100 25fd1980 ffff8102 00000000 00000000 00000000 00000000
Jun 26 04:39:14 arrakis kernel: 00000000 00000000 00000000 00000000 ba9edeeb ba9eea3b 0b3853e0 00000010
Jun 26 04:39:14 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008700e
Jun 26 04:39:14 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:39:14 arrakis kernel: 7afdd72a 00000000 000000e0 000000cc 00000000 00000100 00000100 ff9402d7
Jun 26 04:39:14 arrakis kernel: 66f73c00 ffff8101 66f73cbe ffff8101 00000d28 00000001
Jun 26 04:39:14 arrakis kernel:
Jun 26 04:39:14 arrakis kernel: 00010002 05a80002 00010002 00000000 00000000 00000000 07874618 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000836 000007ca 0785f688 ffffe200 00000000 00000386 05b5c420 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 00001000 078b56b0 ffffe200 00000000 00001000 06480528 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 00001000 07937f18 ffffe200 00000000 00001000 0787c8e8 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 0000085e 077800d0 ffffe200 00000000 00001000 077fd6c0 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 0000030e 029d9100 ffffe200 00000000 00001000 005bab80 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 00001000 0155cae8 ffffe200 00000000 00001000 0238acc0 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 00001000 01f1ad78 ffffe200 00000000 00001000 0142c550 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 00001000 0219cdc0 ffffe200 00000000 00000a88 02a0ccc0 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 00000958 d173ae1d 3f562267 87ea2984 b7e4518d
Jun 26 04:39:14 arrakis kernel:
Jun 26 04:39:14 arrakis kernel: Corrupted skb:
Jun 26 04:39:14 arrakis kernel: 00000000 00000000 00000000 00000000 a53d2d00 ffff8101 00000000 00000000
Jun 26 04:39:14 arrakis kernel: 15c80000 ffff8100 25fd1980 ffff8102 00000000 00000000 00000000 00000000
Jun 26 04:39:14 arrakis kernel: 00000000 00000000 00000000 00000000 ba9edeeb ba9eea3b 0b3853e0 00000010
Jun 26 04:39:14 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008700e
Jun 26 04:39:14 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:39:14 arrakis kernel: 7afdd72a 00000000 000000e0 000000cc 00000000 00000100 00000100 ff9402d7
Jun 26 04:39:14 arrakis kernel: 66f73c00 ffff8101 66f73cbe ffff8101 00000d28 00000001
Jun 26 04:39:14 arrakis kernel:
Jun 26 04:39:14 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 07874618 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000836 000005a8 0785f688 ffffe200 00000000 00000386 05b5c420 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 00001000 078b56b0 ffffe200 00000000 00001000 06480528 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 00001000 07937f18 ffffe200 00000000 00001000 0787c8e8 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 0000085e 077800d0 ffffe200 00000000 00001000 077fd6c0 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 0000030e 029d9100 ffffe200 00000000 00001000 005bab80 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 00001000 0155cae8 ffffe200 00000000 00001000 0238acc0 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 00001000 01f1ad78 ffffe200 00000000 00001000 0142c550 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 00001000 0219cdc0 ffffe200 00000000 00000a88 02a0ccc0 ffffe200
Jun 26 04:39:14 arrakis kernel: 00000000 00000958 d173ae1d 3f562267 87ea2984 b7e4518d
Jun 26 04:39:14 arrakis kernel:
Jun 26 04:39:27 arrakis kernel: bnx2: skb->nr_frags=2 is corrupted, should be 3
Jun 26 04:39:27 arrakis kernel: original skb:
Jun 26 04:39:27 arrakis kernel: 00000000 00000000 00000000 00000000 a53d2d00 ffff8101 00000000 00000000
Jun 26 04:39:27 arrakis kernel: 15c80000 ffff8100 25fd1980 ffff8102 00000000 00000000 00000000 00000000
Jun 26 04:39:27 arrakis kernel: 00000000 00000000 00000000 00000000 bb3785f3 bb3796eb 0b386039 00000010
Jun 26 04:39:27 arrakis kernel: 00000000 00000000 0000113a 000010f8 01000000 001000e0 00000000 0008300e
Jun 26 04:39:27 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:39:27 arrakis kernel: 342c702e 00000000 000000e0 000000cc 00000000 00000100 00000100 12f7f8df
Jun 26 04:39:27 arrakis kernel: 2eb94c00 ffff8102 2eb94cbe ffff8102 000012d0 00000001
Jun 26 04:39:27 arrakis kernel:
Jun 26 04:39:27 arrakis kernel: 00010002 05a80003 00010003 00000000 00000000 00000000 019bc298 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000f3e 000000c2 019c1af8 ffffe200 00000000 00001000 02d30520 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00000036 02d46cf8 ffffe200 00000000 00001000 02d32f90 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00001000 02d4a028 ffffe200 00000000 00001000 01607cc8 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 000009be 07887430 ffffe200 00000000 00000159 07839c88 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00001000 04fdf150 ffffe200 00000000 00001000 077c8ad0 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00001000 079c8b70 ffffe200 00000000 00001000 07639248 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00001000 06f0ae40 ffffe200 00000000 00000181 01fbf578 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00000e58 02d26180 ffffe200 00000000 00001000 00fbe518 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00000228 00000000 00000000 00000000 00000000
Jun 26 04:39:27 arrakis kernel:
Jun 26 04:39:27 arrakis kernel: Corrupted skb:
Jun 26 04:39:27 arrakis kernel: 00000000 00000000 00000000 00000000 a53d2d00 ffff8101 00000000 00000000
Jun 26 04:39:27 arrakis kernel: 15c80000 ffff8100 25fd1980 ffff8102 00000000 00000000 00000000 00000000
Jun 26 04:39:27 arrakis kernel: 00000000 00000000 00000000 00000000 bb3785f3 bb3796eb 0b386039 00000010
Jun 26 04:39:27 arrakis kernel: 00000000 00000000 0000113a 000010f8 01000000 001000e0 00000000 0008300e
Jun 26 04:39:27 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 04:39:27 arrakis kernel: 342c702e 00000000 000000e0 000000cc 00000000 00000100 00000100 12f7f8df
Jun 26 04:39:27 arrakis kernel: 2eb94c00 ffff8102 2eb94cbe ffff8102 000012d0 00000001
Jun 26 04:39:27 arrakis kernel:
Jun 26 04:39:27 arrakis kernel: 00010002 00000002 00000001 00000000 00000000 00000000 019bc298 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000f3e 000000c2 019c1af8 ffffe200 00000000 000004e6 02d30520 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00000036 02d46cf8 ffffe200 00000000 00001000 02d32f90 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00001000 02d4a028 ffffe200 00000000 00001000 01607cc8 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 000009be 07887430 ffffe200 00000000 00000159 07839c88 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00001000 04fdf150 ffffe200 00000000 00001000 077c8ad0 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00001000 079c8b70 ffffe200 00000000 00001000 07639248 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00001000 06f0ae40 ffffe200 00000000 00000181 01fbf578 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00000e58 02d26180 ffffe200 00000000 00001000 00fbe518 ffffe200
Jun 26 04:39:27 arrakis kernel: 00000000 00000228 00000000 00000000 00000000 00000000
Jun 26 04:39:27 arrakis kernel:
Jun 26 10:50:49 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 26 10:50:49 arrakis kernel: original skb:
Jun 26 10:50:49 arrakis kernel: 00000000 00000000 00000000 00000000 a53d3900 ffff8101 00000000 00000000
Jun 26 10:50:49 arrakis kernel: 15c80000 ffff8100 2828d6c0 ffff8102 00000000 00000000 00000000 00000000
Jun 26 10:50:49 arrakis kernel: 00000000 00000000 00000000 00000000 7e5cb1b4 7e5cc854 0b8d6089 00000010
Jun 26 10:50:49 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008b00e
Jun 26 10:50:49 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 10:50:49 arrakis kernel: 30a97065 00000000 000000e0 000000cc 00000000 00000100 00000100 4ac981eb
Jun 26 10:50:49 arrakis kernel: 286d7000 ffff8102 286d70be ffff8102 00001878 00000001
Jun 26 10:50:49 arrakis kernel:
Jun 26 10:50:49 arrakis kernel: 00010002 05a80002 00010004 00000000 00000000 00000000 019e11e0 ffffe200
Jun 26 10:50:49 arrakis kernel: 000002fd 00000d03 007e3988 ffffe200 00000000 0000099d 005c9670 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 01ba1058 ffffe200 00000000 00001000 02d54e80 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 021fedc0 ffffe200 00000000 0000022d 04e87cf0 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 079c0f30 ffffe200 00000000 00001000 0792d750 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 078533d8 ffffe200 00000000 00001000 049552a8 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 0785f768 ffffe200 00000000 00000c75 013dab98 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 01d51998 ffffe200 00000000 00001000 02669588 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 0260e550 ffffe200 00000000 00001000 00a15c50 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00000c96 00000000 00000000 00000000 00000000
Jun 26 10:50:49 arrakis kernel:
Jun 26 10:50:49 arrakis kernel: Corrupted skb:
Jun 26 10:50:49 arrakis kernel: 00000000 00000000 00000000 00000000 a53d3900 ffff8101 00000000 00000000
Jun 26 10:50:49 arrakis kernel: 15c80000 ffff8100 2828d6c0 ffff8102 00000000 00000000 00000000 00000000
Jun 26 10:50:49 arrakis kernel: 00000000 00000000 00000000 00000000 7e5cb1b4 7e5cc854 0b8d6089 00000010
Jun 26 10:50:49 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008b00e
Jun 26 10:50:49 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 10:50:49 arrakis kernel: 30a97065 00000000 000000e0 000000cc 00000000 00000100 00000100 4ac981eb
Jun 26 10:50:49 arrakis kernel: 286d7000 ffff8102 286d70be ffff8102 00001878 00000001
Jun 26 10:50:49 arrakis kernel:
Jun 26 10:50:49 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 019e11e0 ffffe200
Jun 26 10:50:49 arrakis kernel: 000002fd 000005a8 007e3988 ffffe200 00000000 0000099d 005c9670 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 01ba1058 ffffe200 00000000 00001000 02d54e80 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 021fedc0 ffffe200 00000000 0000022d 04e87cf0 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 079c0f30 ffffe200 00000000 00001000 0792d750 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 078533d8 ffffe200 00000000 00001000 049552a8 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 0785f768 ffffe200 00000000 00000c75 013dab98 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 01d51998 ffffe200 00000000 00001000 02669588 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00001000 0260e550 ffffe200 00000000 00001000 00a15c50 ffffe200
Jun 26 10:50:49 arrakis kernel: 00000000 00000c96 00000000 00000000 00000000 00000000
Jun 26 10:50:49 arrakis kernel:
Jun 26 10:53:21 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 26 10:53:21 arrakis kernel: original skb:
Jun 26 10:53:21 arrakis kernel: 00000000 00000000 00000000 00000000 0fc4f880 ffff8101 00000000 00000000
Jun 26 10:53:21 arrakis kernel: 15c80000 ffff8100 c26cb080 ffff8101 00000000 00000000 00000000 00000000
Jun 26 10:53:21 arrakis kernel: 00000000 00000000 00000000 00000000 94c99b11 94c9b795 0b8df4c7 00000010
Jun 26 10:53:21 arrakis kernel: 00000000 00000000 00001cba 00001c84 01000000 001000ec 00000000 0008100e
Jun 26 10:53:21 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 10:53:21 arrakis kernel: 642e3436 00000000 000000ec 000000d8 00000000 00000100 00000100 00000000
Jun 26 10:53:21 arrakis kernel: de3fdc00 ffff8101 de3fdcca ffff8101 00001e5c 00000001
Jun 26 10:53:21 arrakis kernel:
Jun 26 10:53:21 arrakis kernel: 00010002 05b40002 00010005 00000000 00000000 00000000 077ea9c0 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000287 00000d79 05c1b790 ffffe200 00000000 00000f0b 05c200d8 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 00001000 02d69cf8 ffffe200 00000000 00001000 02d54470 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 000006f7 077d1d98 ffffe200 00000000 00001000 078c65f0 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 0000090b 079068f8 ffffe200 00000000 00001000 079793a0 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 000004db 077b3210 ffffe200 00000000 000000ab 077dc790 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 00001000 078cee38 ffffe200 00000000 00001000 04846110 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 00001000 06478e98 ffffe200 00000000 00001000 07968cb0 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 00001000 078b02e8 ffffe200 00000000 00001000 04e87158 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 000002cc 37201156 e75e8e69 202b3562 13e07842
Jun 26 10:53:21 arrakis kernel:
Jun 26 10:53:21 arrakis kernel: Corrupted skb:
Jun 26 10:53:21 arrakis kernel: 00000000 00000000 00000000 00000000 0fc4f880 ffff8101 00000000 00000000
Jun 26 10:53:21 arrakis kernel: 15c80000 ffff8100 c26cb080 ffff8101 00000000 00000000 00000000 00000000
Jun 26 10:53:21 arrakis kernel: 00000000 00000000 00000000 00000000 94c99b11 94c9b795 0b8df4c7 00000010
Jun 26 10:53:21 arrakis kernel: 00000000 00000000 00001cba 00001c84 01000000 001000ec 00000000 0008100e
Jun 26 10:53:21 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 10:53:21 arrakis kernel: 642e3436 00000000 000000ec 000000d8 00000000 00000100 00000100 00000000
Jun 26 10:53:21 arrakis kernel: de3fdc00 ffff8101 de3fdcca ffff8101 00001e5c 00000001
Jun 26 10:53:21 arrakis kernel:
Jun 26 10:53:21 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 077ea9c0 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000287 000005b4 05c1b790 ffffe200 00000000 00000f0b 05c200d8 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 00001000 02d69cf8 ffffe200 00000000 00001000 02d54470 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 000006f7 077d1d98 ffffe200 00000000 00001000 078c65f0 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 0000090b 079068f8 ffffe200 00000000 00001000 079793a0 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 000004db 077b3210 ffffe200 00000000 000000ab 077dc790 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 00001000 078cee38 ffffe200 00000000 00001000 04846110 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 00001000 06478e98 ffffe200 00000000 00001000 07968cb0 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 00001000 078b02e8 ffffe200 00000000 00001000 04e87158 ffffe200
Jun 26 10:53:21 arrakis kernel: 00000000 000002cc 37201156 e75e8e69 202b3562 13e07842
Jun 26 10:53:21 arrakis kernel:
Jun 26 10:54:15 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 26 10:54:15 arrakis kernel: original skb:
Jun 26 10:54:15 arrakis kernel: 00000000 00000000 00000000 00000000 15c1b980 ffff8100 00000000 00000000
Jun 26 10:54:15 arrakis kernel: 15c80000 ffff8100 cfa03340 ffff8100 00000000 00000000 00000000 00000000
Jun 26 10:54:15 arrakis kernel: 00000000 00000000 00000000 00000000 ac63b2b2 ac63c952 0b8e28d2 00000010
Jun 26 10:54:15 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008b00e
Jun 26 10:54:15 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 10:54:15 arrakis kernel: f3de7445 00000000 000000e0 000000cc 00000000 00000100 00000100 c14e92c2
Jun 26 10:54:15 arrakis kernel: 16f95000 ffff8100 16f950be ffff8100 00001878 00000001
Jun 26 10:54:15 arrakis kernel:
Jun 26 10:54:15 arrakis kernel: 00010002 05a80002 00010004 00000000 00000000 00000000 005dbf48 ffffe200
Jun 26 10:54:15 arrakis kernel: 000004a3 00000b5d 01fe3730 ffffe200 00000000 00000b43 02842808 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 00001000 013cd8b8 ffffe200 00000000 00001000 0782f028 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 00001000 062844a0 ffffe200 00000000 00001000 078d8d78 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 00001000 077cd6b8 ffffe200 00000000 00001000 077fc888 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 000006bb 078e09f0 ffffe200 00000000 00001000 078ca020 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 00001000 077de320 ffffe200 00000000 000002b5 006a6a78 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 00001000 0126ca90 ffffe200 00000000 00000545 02bad990 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 00001000 019aa018 ffffe200 00000000 00000105 06479020 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 0000017d ba79763f 192e5c4d 5d49e7f0 e450cbca
Jun 26 10:54:15 arrakis kernel:
Jun 26 10:54:15 arrakis kernel: Corrupted skb:
Jun 26 10:54:15 arrakis kernel: 00000000 00000000 00000000 00000000 15c1b980 ffff8100 00000000 00000000
Jun 26 10:54:15 arrakis kernel: 15c80000 ffff8100 cfa03340 ffff8100 00000000 00000000 00000000 00000000
Jun 26 10:54:15 arrakis kernel: 00000000 00000000 00000000 00000000 ac63b2b2 ac63c952 0b8e28d2 00000010
Jun 26 10:54:15 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008b00e
Jun 26 10:54:15 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 10:54:15 arrakis kernel: f3de7445 00000000 000000e0 000000cc 00000000 00000100 00000100 c14e92c2
Jun 26 10:54:15 arrakis kernel: 16f95000 ffff8100 16f950be ffff8100 00001878 00000001
Jun 26 10:54:15 arrakis kernel:
Jun 26 10:54:15 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 005dbf48 ffffe200
Jun 26 10:54:15 arrakis kernel: 000004a3 000005a8 01fe3730 ffffe200 00000000 00000b43 02842808 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 00001000 013cd8b8 ffffe200 00000000 00001000 0782f028 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 00001000 062844a0 ffffe200 00000000 00001000 078d8d78 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 00001000 077cd6b8 ffffe200 00000000 00001000 077fc888 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 000006bb 078e09f0 ffffe200 00000000 00001000 078ca020 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 00001000 077de320 ffffe200 00000000 000002b5 006a6a78 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 00001000 0126ca90 ffffe200 00000000 00000545 02bad990 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 00001000 019aa018 ffffe200 00000000 00000105 06479020 ffffe200
Jun 26 10:54:15 arrakis kernel: 00000000 0000017d ba79763f 192e5c4d 5d49e7f0 e450cbca
Jun 26 10:54:15 arrakis kernel:
Jun 26 11:01:59 arrakis kernel: bnx2: skb->nr_frags=2 is corrupted, should be 3
Jun 26 11:01:59 arrakis kernel: original skb:
Jun 26 11:01:59 arrakis kernel: 00000000 00000000 00000000 00000000 26da53c0 ffff8102 00000000 00000000
Jun 26 11:01:59 arrakis kernel: 15c80000 ffff8100 95a4d980 ffff8100 00000000 00000000 00000000 00000000
Jun 26 11:01:59 arrakis kernel: 00000000 00000000 00000000 00000000 8ca83560 8ca84c00 0b8fee0e 00000010
Jun 26 11:01:59 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008900e
Jun 26 11:01:59 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 11:01:59 arrakis kernel: 2de35755 00000000 000000e0 000000cc 00000000 00000100 00000100 8ce0ccdd
Jun 26 11:01:59 arrakis kernel: 0d6f3800 ffff8101 0d6f38be ffff8101 00001878 00000001
Jun 26 11:01:59 arrakis kernel:
Jun 26 11:01:59 arrakis kernel: 00010002 05a80003 00010004 00000000 00000000 00000000 00683d18 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000fa8 00000058 01a41c38 ffffe200 00000000 00001000 0186e758 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00000648 0260a340 ffffe200 00000000 00001000 00fd4c48 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00001000 0176b0c8 ffffe200 00000000 00001000 01838208 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00000fd0 02c81910 ffffe200 00000000 00001000 025fddb8 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00001000 01efa2a8 ffffe200 00000000 00001000 00593e78 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00001000 0263ec20 ffffe200 00000000 00001000 00bf0910 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00001000 01019a90 ffffe200 00000000 00001000 01f93ba8 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00001000 027036d8 ffffe200 00000000 00000438 01724760 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00000cfa 00000000 00000000 00000000 00000000
Jun 26 11:01:59 arrakis kernel:
Jun 26 11:01:59 arrakis kernel: Corrupted skb:
Jun 26 11:01:59 arrakis kernel: 00000000 00000000 00000000 00000000 26da53c0 ffff8102 00000000 00000000
Jun 26 11:01:59 arrakis kernel: 15c80000 ffff8100 95a4d980 ffff8100 00000000 00000000 00000000 00000000
Jun 26 11:01:59 arrakis kernel: 00000000 00000000 00000000 00000000 8ca83560 8ca84c00 0b8fee0e 00000010
Jun 26 11:01:59 arrakis kernel: 00000000 00000000 000016e2 000016a0 01000000 001000e0 00000000 0008900e
Jun 26 11:01:59 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 11:01:59 arrakis kernel: 2de35755 00000000 000000e0 000000cc 00000000 00000100 00000100 8ce0ccdd
Jun 26 11:01:59 arrakis kernel: 0d6f3800 ffff8101 0d6f38be ffff8101 00001878 00000001
Jun 26 11:01:59 arrakis kernel:
Jun 26 11:01:59 arrakis kernel: 00010002 00000002 00000001 00000000 00000000 00000000 00683d18 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000fa8 00000058 01a41c38 ffffe200 00000000 00000550 0186e758 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00000648 0260a340 ffffe200 00000000 00001000 00fd4c48 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00001000 0176b0c8 ffffe200 00000000 00001000 01838208 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00000fd0 02c81910 ffffe200 00000000 00001000 025fddb8 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00001000 01efa2a8 ffffe200 00000000 00001000 00593e78 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00001000 0263ec20 ffffe200 00000000 00001000 00bf0910 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00001000 01019a90 ffffe200 00000000 00001000 01f93ba8 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00001000 027036d8 ffffe200 00000000 00000438 01724760 ffffe200
Jun 26 11:01:59 arrakis kernel: 00000000 00000cfa 00000000 00000000 00000000 00000000
Jun 26 11:01:59 arrakis kernel:
Jun 26 11:42:55 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jun 26 11:42:55 arrakis kernel: original skb:
Jun 26 11:42:55 arrakis kernel: 00000000 00000000 00000000 00000000 25e866c0 ffff8102 00000000 00000000
Jun 26 11:42:55 arrakis kernel: 15c80000 ffff8100 95a4d340 ffff8100 00000000 00000000 00000000 00000000
Jun 26 11:42:55 arrakis kernel: 00000000 00000000 00000000 00000000 6c9f62d2 6c9f6e22 0b994d30 00000010
Jun 26 11:42:55 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008500e
Jun 26 11:42:55 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 11:42:55 arrakis kernel: 71bdb092 00000000 000000e0 000000cc 00000000 00000100 00000100 0fd6f2e2
Jun 26 11:42:55 arrakis kernel: cf170000 ffff8100 cf1700be ffff8100 00000d28 00000001
Jun 26 11:42:55 arrakis kernel:
Jun 26 11:42:55 arrakis kernel: 00010002 05a80002 00010002 00000000 00000000 00000000 07767080 ffffe200
Jun 26 11:42:55 arrakis kernel: 000008cc 00000734 07920828 ffffe200 00000000 0000041c 07858960 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00000514 02c4aa90 ffffe200 00000000 00001000 078bc988 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00001000 07a23c50 ffffe200 00000000 00001000 079843f0 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00001000 06b15fb8 ffffe200 00000000 0000068c 078bcfe0 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00001000 078ce818 ffffe200 00000000 00001000 04fd88c0 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00001000 0790bb38 ffffe200 00000000 00001000 07968ce8 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00000804 04bea990 ffffe200 00000000 00000390 0793f5a8 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00001000 0788e350 ffffe200 00000000 00001000 07808668 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00000250 a993b471 b053abf0 a7211990 fca423b0
Jun 26 11:42:55 arrakis kernel:
Jun 26 11:42:55 arrakis kernel: Corrupted skb:
Jun 26 11:42:55 arrakis kernel: 00000000 00000000 00000000 00000000 25e866c0 ffff8102 00000000 00000000
Jun 26 11:42:55 arrakis kernel: 15c80000 ffff8100 95a4d340 ffff8100 00000000 00000000 00000000 00000000
Jun 26 11:42:55 arrakis kernel: 00000000 00000000 00000000 00000000 6c9f62d2 6c9f6e22 0b994d30 00000010
Jun 26 11:42:55 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008500e
Jun 26 11:42:55 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jun 26 11:42:55 arrakis kernel: 71bdb092 00000000 000000e0 000000cc 00000000 00000100 00000100 0fd6f2e2
Jun 26 11:42:55 arrakis kernel: cf170000 ffff8100 cf1700be ffff8100 00000d28 00000001
Jun 26 11:42:55 arrakis kernel:
Jun 26 11:42:55 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 07767080 ffffe200
Jun 26 11:42:55 arrakis kernel: 000008cc 000005a8 07920828 ffffe200 00000000 0000041c 07858960 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00000514 02c4aa90 ffffe200 00000000 00001000 078bc988 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00001000 07a23c50 ffffe200 00000000 00001000 079843f0 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00001000 06b15fb8 ffffe200 00000000 0000068c 078bcfe0 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00001000 078ce818 ffffe200 00000000 00001000 04fd88c0 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00001000 0790bb38 ffffe200 00000000 00001000 07968ce8 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00000804 04bea990 ffffe200 00000000 00000390 0793f5a8 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00001000 0788e350 ffffe200 00000000 00001000 07808668 ffffe200
Jun 26 11:42:55 arrakis kernel: 00000000 00000250 a993b471 b053abf0 a7211990 fca423b0
Jun 26 11:42:55 arrakis kernel:
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-26 11:01 ` Josip Rodin
@ 2008-06-26 18:04 ` Michael Chan
2008-07-09 16:46 ` Josip Rodin
0 siblings, 1 reply; 50+ messages in thread
From: Michael Chan @ 2008-06-26 18:04 UTC (permalink / raw)
To: Josip Rodin
Cc: 'Bill Fink', Ben Hutchings, netdev, mirrors@debian.org
Josip Rodin wrote:
>
> Here we go, this is the first batch of debug output. I hope it helps.
>
Thanks for the data. I've looked at a few of these SKB dumps and they
appear to be all modified in the same way:
1. SKB is unchanged and the clone bit is set.
2. skb_shared_info(SKB) has dataref 0x00010002 and these fields are
changed:
nr_frags changed from n to 1
gso_size changed from 0x5a8 to 0
gso_type changed from 1 (TCP_V4) to 0
gso_segs changed from m to 1
1 of the frags' length is modified to split the TSO packet into 1
segment
I think this is consistent with what David said about tso_fragment()
splitting the cloned TSO SKB into 2.
Thanks.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-06-26 18:04 ` Michael Chan
@ 2008-07-09 16:46 ` Josip Rodin
2008-07-09 16:57 ` Michael Chan
0 siblings, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-07-09 16:46 UTC (permalink / raw)
To: Michael Chan
Cc: 'Bill Fink', Ben Hutchings, netdev, mirrors@debian.org
On Thu, Jun 26, 2008 at 11:04:48AM -0700, Michael Chan wrote:
> Josip Rodin wrote:
> >
> > Here we go, this is the first batch of debug output. I hope it helps.
> >
>
> Thanks for the data. I've looked at a few of these SKB dumps and they
> appear to be all modified in the same way:
>
> 1. SKB is unchanged and the clone bit is set.
>
> 2. skb_shared_info(SKB) has dataref 0x00010002 and these fields are
> changed:
>
> nr_frags changed from n to 1
> gso_size changed from 0x5a8 to 0
> gso_type changed from 1 (TCP_V4) to 0
> gso_segs changed from m to 1
> 1 of the frags' length is modified to split the TSO packet into 1
> segment
>
> I think this is consistent with what David said about tso_fragment()
> splitting the cloned TSO SKB into 2.
Okay, thanks, can you tell me if the previous patch is a proper fix or just
a workaround?
Recently we've been seeing rsync errors with some of the sites which used to
be pretty fast - the connection unexpectedly closes, or it times out.
Something is still fishy about it...
--
Josip Rodin
mirrors@debian.org
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-09 16:46 ` Josip Rodin
@ 2008-07-09 16:57 ` Michael Chan
2008-07-09 23:46 ` David Miller
0 siblings, 1 reply; 50+ messages in thread
From: Michael Chan @ 2008-07-09 16:57 UTC (permalink / raw)
To: Josip Rodin
Cc: 'Bill Fink', Ben Hutchings, netdev, mirrors@debian.org
On Wed, 2008-07-09 at 09:46 -0700, Josip Rodin wrote:
> On Thu, Jun 26, 2008 at 11:04:48AM -0700, Michael Chan wrote:
> > 1. SKB is unchanged and the clone bit is set.
> >
> > 2. skb_shared_info(SKB) has dataref 0x00010002 and these fields are
> > changed:
> >
> > nr_frags changed from n to 1
> > gso_size changed from 0x5a8 to 0
> > gso_type changed from 1 (TCP_V4) to 0
> > gso_segs changed from m to 1
> > 1 of the frags' length is modified to split the TSO packet into 1
> > segment
> >
> > I think this is consistent with what David said about tso_fragment()
> > splitting the cloned TSO SKB into 2.
>
> Okay, thanks, can you tell me if the previous patch is a proper fix or just
> a workaround?
That was just a debug patch. The proper fix needs to go into
tso_fragment() as David outlined earlier.
>
> Recently we've been seeing rsync errors with some of the sites which used to
> be pretty fast - the connection unexpectedly closes, or it times out.
> Something is still fishy about it...
>
The tso_fragment() issue will always lead to a crash in the bnx2 driver
(before applying any of the debug patches I provided). If it doesn't
crash, it's probably something else.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-09 16:57 ` Michael Chan
@ 2008-07-09 23:46 ` David Miller
2008-07-10 9:45 ` Aviv Greenberg
2008-07-10 21:00 ` Michael Chan
0 siblings, 2 replies; 50+ messages in thread
From: David Miller @ 2008-07-09 23:46 UTC (permalink / raw)
To: mchan; +Cc: joy, billfink, bhutchings, netdev, mirrors
From: "Michael Chan" <mchan@broadcom.com>
Date: Wed, 09 Jul 2008 09:57:05 -0700
> That was just a debug patch. The proper fix needs to go into
> tso_fragment() as David outlined earlier.
Actually I went investigating this and all the code paths check for
skb_cloned() and if true they make a copy of the data area (and thus
the skb_shared_info()) and this should ensure that the driver doesn't
see changing nr_frags values.
So it has to be something else.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-09 23:46 ` David Miller
@ 2008-07-10 9:45 ` Aviv Greenberg
2008-07-10 10:09 ` David Miller
2008-07-10 21:00 ` Michael Chan
1 sibling, 1 reply; 50+ messages in thread
From: Aviv Greenberg @ 2008-07-10 9:45 UTC (permalink / raw)
To: David Miller; +Cc: mchan, joy, billfink, bhutchings, netdev, mirrors
On Thu, Jul 10, 2008 at 02:46, David Miller <davem@davemloft.net> wrote:
>
> So it has to be something else.
What mechanism prevents the clone from happening between
the skb_clone() and skb_split() calls in tcp_fragment?
Maybe there's a path that doesn't take a lock...
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-10 9:45 ` Aviv Greenberg
@ 2008-07-10 10:09 ` David Miller
0 siblings, 0 replies; 50+ messages in thread
From: David Miller @ 2008-07-10 10:09 UTC (permalink / raw)
To: avivgnet; +Cc: mchan, joy, billfink, bhutchings, netdev, mirrors
From: "Aviv Greenberg" <avivgnet@gmail.com>
Date: Thu, 10 Jul 2008 12:45:54 +0300
> On Thu, Jul 10, 2008 at 02:46, David Miller <davem@davemloft.net> wrote:
> >
> > So it has to be something else.
>
> What mechanism prevents the clone from happening between
> the skb_clone() and skb_split() calls in tcp_fragment?
The TCP packet is guarenteed to be cloned when the driver
first gets it. That's fundamentally how TCP's retransmission
handling works.
It makes a clone, and sends it down to the driver.
You can't make clones of clones, so in such situations
code makes a full copy instead.
TCP always had the whole state of the retransmit queue,
and thus the base SKB state, locked when it makes all
of these operations.
When TCP is looking at an SKB on the retransmit queue it can only go
from cloned to uncloned, not the other way around.
> Maybe there's a path that doesn't take a lock...
I doubt it.
I think it's some piece of code not making careful enough
checks on the way down to an SKB modification.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-10 21:00 ` Michael Chan
@ 2008-07-10 21:00 ` David Miller
2008-07-10 21:23 ` Josip Rodin
1 sibling, 0 replies; 50+ messages in thread
From: David Miller @ 2008-07-10 21:00 UTC (permalink / raw)
To: mchan; +Cc: joy, billfink, bhutchings, netdev, mirrors
From: "Michael Chan" <mchan@broadcom.com>
Date: Thu, 10 Jul 2008 14:00:17 -0700
> On Wed, 2008-07-09 at 16:46 -0700, David Miller wrote:
> > Actually I went investigating this and all the code paths check for
> > skb_cloned() and if true they make a copy of the data area (and thus
> > the skb_shared_info()) and this should ensure that the driver doesn't
> > see changing nr_frags values.
>
> Since Josip can readily reproduce this problem, let's confirm if the SKB
> is split while it is cloned. Please try this debug patch:
Another good place to check would be __pskb_trim_head() in
net/ipv4/tcp_output.c but the skb_cloned() check in one of that
function's only callers (tcp_trim_head) is rather explicit :-)
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-09 23:46 ` David Miller
2008-07-10 9:45 ` Aviv Greenberg
@ 2008-07-10 21:00 ` Michael Chan
2008-07-10 21:00 ` David Miller
2008-07-10 21:23 ` Josip Rodin
1 sibling, 2 replies; 50+ messages in thread
From: Michael Chan @ 2008-07-10 21:00 UTC (permalink / raw)
To: David Miller
Cc: joy@debian.org, billfink@mindspring.com,
bhutchings@solarflare.com, netdev, mirrors@debian.org
On Wed, 2008-07-09 at 16:46 -0700, David Miller wrote:
> Actually I went investigating this and all the code paths check for
> skb_cloned() and if true they make a copy of the data area (and thus
> the skb_shared_info()) and this should ensure that the driver doesn't
> see changing nr_frags values.
Since Josip can readily reproduce this problem, let's confirm if the SKB
is split while it is cloned. Please try this debug patch:
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 5c459f2..03ec3b8 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -1960,6 +1960,10 @@ void skb_split(struct sk_buff *skb, struct sk_buff *skb1, const u32 len)
{
int pos = skb_headlen(skb);
+ if (skb_cloned(skb)) {
+ printk(KERN_ALERT "Splitting cloned skb\n")
+ dump_stack();
+ }
if (len < pos) /* Split line is inside header. */
skb_split_inside_header(skb, skb1, len, pos);
else /* Second chunk has no header, nothing to copy. */
^ permalink raw reply related [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-10 21:00 ` Michael Chan
2008-07-10 21:00 ` David Miller
@ 2008-07-10 21:23 ` Josip Rodin
2008-07-10 21:38 ` Michael Chan
1 sibling, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-07-10 21:23 UTC (permalink / raw)
To: Michael Chan
Cc: David Miller, billfink@mindspring.com, bhutchings@solarflare.com,
netdev, mirrors@debian.org
On Thu, Jul 10, 2008 at 02:00:17PM -0700, Michael Chan wrote:
> On Wed, 2008-07-09 at 16:46 -0700, David Miller wrote:
> > Actually I went investigating this and all the code paths check for
> > skb_cloned() and if true they make a copy of the data area (and thus
> > the skb_shared_info()) and this should ensure that the driver doesn't
> > see changing nr_frags values.
>
> Since Josip can readily reproduce this problem, let's confirm if the SKB
> is split while it is cloned. Please try this debug patch:
>
> diff --git a/net/core/skbuff.c b/net/core/skbuff.c
> index 5c459f2..03ec3b8 100644
> --- a/net/core/skbuff.c
> +++ b/net/core/skbuff.c
> @@ -1960,6 +1960,10 @@ void skb_split(struct sk_buff *skb, struct sk_buff *skb1, const u32 len)
> {
> int pos = skb_headlen(skb);
>
> + if (skb_cloned(skb)) {
> + printk(KERN_ALERT "Splitting cloned skb\n")
> + dump_stack();
> + }
> if (len < pos) /* Split line is inside header. */
> skb_split_inside_header(skb, skb1, len, pos);
> else /* Second chunk has no header, nothing to copy. */
>
I'll try it - but just as I was reading this e-mail, the machine managed to
crash :(
The log was:
BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8
IP: [<ffffffff8807ad7e>] :bnx2:bnx2_tx_int+0x7e/0x3f0
PGD 22bdaf067 PUD 221fef067 PMD 0
Oops: 0000 [1] SMP
CPU 7
Modules linked in: bnx2 zlib_inflate crc32 ipmi_devintf nfs lockd nfs_acl sunrpc fan ac battery cls_u32 sch_sfq sch_htb ip6t_R
EJECT ip6t_LOG ip6table_filter ip6_tables ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables ipv6 dm_snapshot dm_mirror dm
_mod usbhid ehci_hcd ipmi_si uhci_hcd ipmi_msghandler thermal usbcore shpchp container button processor [last unloaded: crc32]
Pid: 0, comm: swapper Not tainted 2.6.25.6 #1
RIP: 0010:[<ffffffff8807ad7e>] [<ffffffff8807ad7e>] :bnx2:bnx2_tx_int+0x7e/0x3f0
RSP: 0018:ffff81022fa0fd10 EFLAGS: 00010286
RAX: 0000000000000620 RBX: ffff810100427260 RCX: 0000000000010002
RDX: 0000000000000001 RSI: ffff810043c3c000 RDI: ffff8101cb537858
RBP: 00000000000000c4 R08: ffff81022a6ba100 R09: ffff8101005999a8
R10: 0000000000000000 R11: ffffffff80419a50 R12: 00000000000000c4
R13: 00000000fbbffbc4 R14: 0000000000000000 R15: ffff810015c80700
FS: 0000000000000000(0000) GS:ffff81022f9877c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000000000000b8 CR3: 000000022602d000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff81022fa0a000, task ffff81022f9997d0)
Stack: ffff810100540000 000000004909a600 ffff810015c80800 fbc3ffff880d7767
0000000200000002 ffff810015c80800 0000000000000040 0000000000000000
ffff810015c80700 0000000000000000 000000000000012c ffffffff8807b5b4
Call Trace:
<IRQ> [<ffffffff8807b5b4>] ? :bnx2:bnx2_poll+0xd4/0x1330
[<ffffffff8025e52a>] ? handle_edge_irq+0x7a/0x150
[<ffffffff8020ec10>] ? do_IRQ+0x80/0x100
[<ffffffff8025e52a>] ? handle_edge_irq+0x7a/0x150
[<ffffffff88065a5f>] ? :ipmi_si:kcs_event+0x13f/0x800
[<ffffffff8023e7a4>] ? lock_timer_base+0x34/0x70
[<ffffffff80220fd6>] ? send_IPI_self+0x6/0x30
[<ffffffff803edaa8>] ? net_rx_action+0xf8/0x200
[<ffffffff8023a5b9>] ? __do_softirq+0x69/0xe0
[<ffffffff8020c4ac>] ? call_softirq+0x1c/0x30
[<ffffffff8020e995>] ? do_softirq+0x35/0x70
[<ffffffff8020ec10>] ? do_IRQ+0x80/0x100
[<ffffffff8020a410>] ? mwait_idle+0x0/0x50
[<ffffffff8020b831>] ? ret_from_intr+0x0/0xa
<EOI> [<ffffffff80419a50>] ? tcp_poll+0x0/0x180
[<ffffffff8020a451>] ? mwait_idle+0x41/0x50
[<ffffffff8020a378>] ? cpu_idle+0x48/0x90
Code: 24 02 00 00 45 0f b6 e5 41 0f b7 ec 48 8d 04 ed 00 00 00 00 48 89 eb 48 c1 e3 05 48 29 c3 49 03 9f e0 02 00 00 4c 8b 33
8b 53 10 <41> 8b 8e b8 00 00 00 49 8b be c0 00 00 00 89 c8 8b 44 07 04 66
RIP [<ffffffff8807ad7e>] :bnx2:bnx2_tx_int+0x7e/0x3f0
RSP <ffff81022fa0fd10>
CR2: 00000000000000b8
---[ end trace 786b1aa9b912c54e ]---
Kernel panic - not syncing: Aiee, killing interrupt handler!
Hope that helps. I'll go reboot it and apply the new patch.
--
Josip Rodin
mirrors@debian.org
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-10 21:23 ` Josip Rodin
@ 2008-07-10 21:38 ` Michael Chan
2008-07-10 22:00 ` Josip Rodin
0 siblings, 1 reply; 50+ messages in thread
From: Michael Chan @ 2008-07-10 21:38 UTC (permalink / raw)
To: Josip Rodin
Cc: David Miller, billfink@mindspring.com, bhutchings@solarflare.com,
netdev, mirrors@debian.org
On Thu, 2008-07-10 at 14:23 -0700, Josip Rodin wrote:
> I'll try it - but just as I was reading this e-mail, the machine
> managed to
> crash :(
>
> The log was:
>
> BUG: unable to handle kernel NULL pointer dereference at
> 00000000000000b8
> IP: [<ffffffff8807ad7e>] :bnx2:bnx2_tx_int+0x7e/0x3f0
This is the same crash that you originally reported. The driver was
referencing a NULL SKB because the skb_shinfo(skb)->nr_frags was
modified on the previous packet in the TX ring.
If you had applied my debug patch #1 that I posted several weeks ago, it
should have avoided this because we should have saved away the nr_frags
for every packet on the TX ring.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-10 21:38 ` Michael Chan
@ 2008-07-10 22:00 ` Josip Rodin
2008-07-10 22:26 ` Michael Chan
0 siblings, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-07-10 22:00 UTC (permalink / raw)
To: Michael Chan
Cc: David Miller, billfink@mindspring.com, bhutchings@solarflare.com,
netdev, mirrors@debian.org
On Thu, Jul 10, 2008 at 02:38:41PM -0700, Michael Chan wrote:
> On Thu, 2008-07-10 at 14:23 -0700, Josip Rodin wrote:
> > I'll try it - but just as I was reading this e-mail, the machine
> > managed to
> > crash :(
> >
> > The log was:
> >
> > BUG: unable to handle kernel NULL pointer dereference at
> > 00000000000000b8
> > IP: [<ffffffff8807ad7e>] :bnx2:bnx2_tx_int+0x7e/0x3f0
>
> This is the same crash that you originally reported. The driver was
> referencing a NULL SKB because the skb_shinfo(skb)->nr_frags was
> modified on the previous packet in the TX ring.
>
> If you had applied my debug patch #1 that I posted several weeks ago, it
> should have avoided this because we should have saved away the nr_frags
> for every packet on the TX ring.
It was that same patched kernel that crashed nevertheless.
--
Josip Rodin
mirrors@debian.org
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-10 22:00 ` Josip Rodin
@ 2008-07-10 22:26 ` Michael Chan
2008-07-10 22:31 ` Josip Rodin
0 siblings, 1 reply; 50+ messages in thread
From: Michael Chan @ 2008-07-10 22:26 UTC (permalink / raw)
To: Josip Rodin
Cc: David Miller, billfink@mindspring.com, bhutchings@solarflare.com,
netdev, mirrors@debian.org
On Thu, 2008-07-10 at 15:00 -0700, Josip Rodin wrote:
> > This is the same crash that you originally reported. The driver was
> > referencing a NULL SKB because the skb_shinfo(skb)->nr_frags was
> > modified on the previous packet in the TX ring.
> >
> > If you had applied my debug patch #1 that I posted several weeks ago, it
> > should have avoided this because we should have saved away the nr_frags
> > for every packet on the TX ring.
>
> It was that same patched kernel that crashed nevertheless.
>
That's strange. You also saw lots of "Corrupted SKB..." dmesg as well,
right? May be the debug patch is not bullet-proof although I cannot see
how we can still get a NULL SKB in the TX completion path.
Anyway, please try today's patch as we need to find the source of the
problem.
Thanks.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-10 22:26 ` Michael Chan
@ 2008-07-10 22:31 ` Josip Rodin
2008-07-10 23:20 ` David Miller
0 siblings, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-07-10 22:31 UTC (permalink / raw)
To: Michael Chan
Cc: David Miller, billfink@mindspring.com, bhutchings@solarflare.com,
netdev, mirrors@debian.org
On Thu, Jul 10, 2008 at 03:26:03PM -0700, Michael Chan wrote:
> > > This is the same crash that you originally reported. The driver was
> > > referencing a NULL SKB because the skb_shinfo(skb)->nr_frags was
> > > modified on the previous packet in the TX ring.
> > >
> > > If you had applied my debug patch #1 that I posted several weeks ago, it
> > > should have avoided this because we should have saved away the nr_frags
> > > for every packet on the TX ring.
> >
> > It was that same patched kernel that crashed nevertheless.
>
> That's strange. You also saw lots of "Corrupted SKB..." dmesg as well,
> right? May be the debug patch is not bullet-proof although I cannot see
> how we can still get a NULL SKB in the TX completion path.
Yes, there were gobs of those, as usual. JFTR, this was the last one
recorded before the crash, though I can't be sure if it was responsible or not:
Jul 10 23:13:20 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jul 10 23:13:20 arrakis kernel: original skb:
Jul 10 23:13:20 arrakis kernel: 00000000 00000000 00000000 00000000 15c45900 ffff8100 00000000 00000000
Jul 10 23:13:20 arrakis kernel: 15c80000 ffff8100 258d4d00 ffff8102 00000000 00000000 00000000 00000000
Jul 10 23:13:20 arrakis kernel: 00000000 00000000 00000000 00000000 5383c74e 5383d29e 1e3d909e 00000010
Jul 10 23:13:20 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008d00e
Jul 10 23:13:20 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jul 10 23:13:20 arrakis kernel: 6d8b2897 00000000 000000e0 000000cc 00000000 00000100 00000100 0a8ab8a0
Jul 10 23:13:20 arrakis kernel: fbd14000 ffff8101 fbd140be ffff8101 00000d28 00000001
Jul 10 23:13:20 arrakis kernel:
Jul 10 23:13:20 arrakis kernel: 00010002 05a80002 00010002 00000000 00000000 00000000 0097e340 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000823 000007dd 0134e620 ffffe200 00000000 00000373 026e4338 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00001000 008fd160 ffffe200 00000000 00001000 02447898 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00001000 004e5070 ffffe200 00000000 000001ab 00857ff8 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00000d7b 00898ca0 ffffe200 00000000 0000094b 020068b8 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 0000051b 078e1fd0 ffffe200 00000000 00001000 0790b710 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00001000 07907d18 ffffe200 00000000 00001000 05c1f540 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00001000 07924a38 ffffe200 00000000 00001000 078d79c8 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00001000 0782ae18 ffffe200 00000000 00001000 078c8308 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00000169 00000021 00000000 50545448 654d3a3a
Jul 10 23:13:20 arrakis kernel:
Jul 10 23:13:20 arrakis kernel: Corrupted skb:
Jul 10 23:13:20 arrakis kernel: 00000000 00000000 00000000 00000000 15c45900 ffff8100 00000000 00000000
Jul 10 23:13:20 arrakis kernel: 15c80000 ffff8100 258d4d00 ffff8102 00000000 00000000 00000000 00000000
Jul 10 23:13:20 arrakis kernel: 00000000 00000000 00000000 00000000 5383c74e 5383d29e 1e3d909e 00000010
Jul 10 23:13:20 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008d00e
Jul 10 23:13:20 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jul 10 23:13:20 arrakis kernel: 6d8b2897 00000000 000000e0 000000cc 00000000 00000100 00000100 0a8ab8a0
Jul 10 23:13:20 arrakis kernel: fbd14000 ffff8101 fbd140be ffff8101 00000d28 00000001
Jul 10 23:13:20 arrakis kernel:
Jul 10 23:13:20 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 0097e340 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000823 000005a8 0134e620 ffffe200 00000000 00000373 026e4338 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00001000 008fd160 ffffe200 00000000 00001000 02447898 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00001000 004e5070 ffffe200 00000000 000001ab 00857ff8 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00000d7b 00898ca0 ffffe200 00000000 0000094b 020068b8 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 0000051b 078e1fd0 ffffe200 00000000 00001000 0790b710 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00001000 07907d18 ffffe200 00000000 00001000 05c1f540 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00001000 07924a38 ffffe200 00000000 00001000 078d79c8 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00001000 0782ae18 ffffe200 00000000 00001000 078c8308 ffffe200
Jul 10 23:13:20 arrakis kernel: 00000000 00000169 00000021 00000000 50545448 654d3a3a
Jul 10 23:13:20 arrakis kernel:
> Anyway, please try today's patch as we need to find the source of the
> problem.
It's running with it now, no messages yet. I'll report when if/when they
happen.
--
Josip Rodin
mirrors@debian.org
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-10 22:31 ` Josip Rodin
@ 2008-07-10 23:20 ` David Miller
2008-07-11 9:24 ` Josip Rodin
0 siblings, 1 reply; 50+ messages in thread
From: David Miller @ 2008-07-10 23:20 UTC (permalink / raw)
To: joy; +Cc: mchan, billfink, bhutchings, netdev, mirrors
From: Josip Rodin <joy@entuzijast.net>
Date: Fri, 11 Jul 2008 00:31:42 +0200
> It's running with it now, no messages yet. I'll report when if/when they
> happen.
Thanks Josip.
I'll try to whip up some other debug patches if this one Michael gave
you doesn't trigger.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-10 23:20 ` David Miller
@ 2008-07-11 9:24 ` Josip Rodin
2008-07-11 9:56 ` David Miller
0 siblings, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-07-11 9:24 UTC (permalink / raw)
To: David Miller; +Cc: mchan, billfink, bhutchings, netdev, mirrors
On Thu, Jul 10, 2008 at 04:20:07PM -0700, David Miller wrote:
> From: Josip Rodin <joy@entuzijast.net>
> Date: Fri, 11 Jul 2008 00:31:42 +0200
>
> > It's running with it now, no messages yet. I'll report when if/when they
> > happen.
>
> Thanks Josip.
>
> I'll try to whip up some other debug patches if this one Michael gave
> you doesn't trigger.
Here we go, it triggered, here are the first few, tell me if you need more:
Jul 11 02:15:10 arrakis kernel: Splitting cloned skb
Jul 11 02:15:10 arrakis kernel: Pid: 0, comm: swapper Not tainted 2.6.25.6 #2
Jul 11 02:15:10 arrakis kernel:
Jul 11 02:15:10 arrakis kernel: Call Trace:
Jul 11 02:15:10 arrakis kernel: <IRQ> [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 02:15:10 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 02:15:10 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 02:15:10 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 02:15:10 arrakis kernel: [<ffffffff80424596>] tcp_rcv_established+0x596/0x9d0
Jul 11 02:15:10 arrakis kernel: [<ffffffff8042a77c>] tcp_v4_do_rcv+0xdc/0x220
Jul 11 02:15:10 arrakis kernel: [<ffffffff8042c213>] tcp_v4_rcv+0x5e3/0x810
Jul 11 02:15:10 arrakis kernel: [<ffffffff8040e58f>] ip_local_deliver_finish+0x9f/0x200
Jul 11 02:15:10 arrakis kernel: [<ffffffff8040e2a0>] ip_rcv_finish+0x130/0x380
Jul 11 02:15:10 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 02:15:10 arrakis kernel: [<ffffffff88083b56>] :bnx2:bnx2_poll+0x676/0x1330
Jul 11 02:15:10 arrakis kernel: [<ffffffff8025e52a>] handle_edge_irq+0x7a/0x150
Jul 11 02:15:10 arrakis kernel: [<ffffffff80225076>] __change_page_attr_set_clr+0x8a6/0x920
Jul 11 02:15:10 arrakis kernel: [<ffffffff803edad8>] net_rx_action+0xf8/0x200
Jul 11 02:15:10 arrakis kernel: [<ffffffff8023a5b9>] __do_softirq+0x69/0xe0
Jul 11 02:15:10 arrakis kernel: [<ffffffff8020c4ac>] call_softirq+0x1c/0x30
Jul 11 02:15:10 arrakis kernel: [<ffffffff8020e995>] do_softirq+0x35/0x70
Jul 11 02:15:10 arrakis kernel: [<ffffffff8020ec10>] do_IRQ+0x80/0x100
Jul 11 02:15:10 arrakis kernel: [<ffffffff8020a410>] mwait_idle+0x0/0x50
Jul 11 02:15:10 arrakis kernel: [<ffffffff8020b831>] ret_from_intr+0x0/0xa
Jul 11 02:15:10 arrakis kernel: <EOI> [<ffffffff80419a80>] tcp_poll+0x0/0x180
Jul 11 02:15:10 arrakis kernel: [<ffffffff8020a451>] mwait_idle+0x41/0x50
Jul 11 02:15:10 arrakis kernel: [<ffffffff8020a378>] cpu_idle+0x48/0x90
Jul 11 02:15:10 arrakis kernel:
Jul 11 02:15:33 arrakis kernel: Splitting cloned skb
Jul 11 02:15:33 arrakis kernel: Pid: 4350, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 02:15:33 arrakis kernel:
Jul 11 02:15:33 arrakis kernel: Call Trace:
Jul 11 02:15:33 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 02:15:33 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 02:15:33 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 02:15:33 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 02:15:33 arrakis kernel: [<ffffffff80265f7b>] __alloc_pages+0x5b/0x390
Jul 11 02:15:33 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 02:15:33 arrakis kernel: [<ffffffff802a4f44>] __mark_inode_dirty+0x34/0x1a0
Jul 11 02:15:33 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 02:15:33 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 02:15:33 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 02:15:33 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 02:15:33 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 02:15:33 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 02:15:33 arrakis kernel:
Jul 11 02:16:11 arrakis kernel: Splitting cloned skb
Jul 11 02:16:11 arrakis kernel: Pid: 0, comm: swapper Not tainted 2.6.25.6 #2
Jul 11 02:16:11 arrakis kernel:
Jul 11 02:16:11 arrakis kernel: Call Trace:
Jul 11 02:16:11 arrakis kernel: <IRQ> [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 02:16:11 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 02:16:11 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 02:16:11 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 02:16:11 arrakis kernel: [<ffffffff80424596>] tcp_rcv_established+0x596/0x9d0
Jul 11 02:16:11 arrakis kernel: [<ffffffff8042a77c>] tcp_v4_do_rcv+0xdc/0x220
Jul 11 02:16:11 arrakis kernel: [<ffffffff8042c213>] tcp_v4_rcv+0x5e3/0x810
Jul 11 02:16:11 arrakis kernel: [<ffffffff8040e58f>] ip_local_deliver_finish+0x9f/0x200
Jul 11 02:16:11 arrakis kernel: [<ffffffff8040e2a0>] ip_rcv_finish+0x130/0x380
Jul 11 02:16:11 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 02:16:11 arrakis kernel: [<ffffffff88083b56>] :bnx2:bnx2_poll+0x676/0x1330
Jul 11 02:16:11 arrakis kernel: [<ffffffff8023e7a4>] lock_timer_base+0x34/0x70
Jul 11 02:16:11 arrakis kernel: [<ffffffff8023e910>] __mod_timer+0xb0/0xd0
Jul 11 02:16:11 arrakis kernel: [<ffffffff88085c00>] :bnx2:bnx2_timer+0x0/0x3d0
Jul 11 02:16:11 arrakis kernel: [<ffffffff803edad8>] net_rx_action+0xf8/0x200
Jul 11 02:16:11 arrakis kernel: [<ffffffff8023a5b9>] __do_softirq+0x69/0xe0
Jul 11 02:16:11 arrakis kernel: [<ffffffff8020c4ac>] call_softirq+0x1c/0x30
Jul 11 02:16:11 arrakis kernel: [<ffffffff8020e995>] do_softirq+0x35/0x70
Jul 11 02:16:11 arrakis kernel: [<ffffffff8020ec10>] do_IRQ+0x80/0x100
Jul 11 02:16:11 arrakis kernel: [<ffffffff8020a410>] mwait_idle+0x0/0x50
Jul 11 02:16:11 arrakis kernel: [<ffffffff8020b831>] ret_from_intr+0x0/0xa
Jul 11 02:16:11 arrakis kernel: <EOI> [<ffffffff80419a80>] tcp_poll+0x0/0x180
Jul 11 02:16:11 arrakis kernel: [<ffffffff8020a451>] mwait_idle+0x41/0x50
Jul 11 02:16:11 arrakis kernel: [<ffffffff8020a378>] cpu_idle+0x48/0x90
Jul 11 02:16:11 arrakis kernel:
Jul 11 02:16:11 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jul 11 02:16:11 arrakis kernel: original skb:
Jul 11 02:16:11 arrakis kernel: 00000000 00000000 00000000 00000000 8e964080 ffff8101 00000000 00000000
Jul 11 02:16:11 arrakis kernel: 2db80000 ffff8102 2e54e700 ffff8102 00000000 00000000 00000000 00000000
Jul 11 02:16:11 arrakis kernel: 00000000 00000000 00000000 00000000 6616cee6 6616da36 0024c8af 00000010
Jul 11 02:16:11 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008d00e
Jul 11 02:16:11 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jul 11 02:16:11 arrakis kernel: 56255203 00000000 000000e0 000000cc 00000000 00000100 00000100 1eadfa52
Jul 11 02:16:11 arrakis kernel: 62c55800 ffff8100 62c558be ffff8100 00000d28 00000001
Jul 11 02:16:11 arrakis kernel:
Jul 11 02:16:11 arrakis kernel: 00010002 05a80002 00010002 00000000 00000000 00000000 016aaeb0 ffffe200
Jul 11 02:16:11 arrakis kernel: 0000071d 000008e3 018b4b80 ffffe200 00000000 0000026d 01487eb8 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00001000 0159ebb0 ffffe200 00000000 00001000 01bb2ac0 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00001000 014dbe10 ffffe200 00000000 00000bf5 014b1fd0 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00001000 015c29b0 ffffe200 00000000 00001000 01598588 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00001000 015b48d0 ffffe200 00000000 00001000 015c51b8 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00001000 01e26de0 ffffe200 00000000 00001000 0148a998 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 0000005d 01594fb8 ffffe200 00000000 00001000 016d8f00 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00001000 00733028 ffffe200 00000000 00001000 01baaa20 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00000b9f a958b6d7 05df7a82 05ddd09a 312d8da0
Jul 11 02:16:11 arrakis kernel:
Jul 11 02:16:11 arrakis kernel: Corrupted skb:
Jul 11 02:16:11 arrakis kernel: 00000000 00000000 00000000 00000000 8e964080 ffff8101 00000000 00000000
Jul 11 02:16:11 arrakis kernel: 2db80000 ffff8102 2e54e700 ffff8102 00000000 00000000 00000000 00000000
Jul 11 02:16:11 arrakis kernel: 00000000 00000000 00000000 00000000 6616cee6 6616da36 0024c8af 00000010
Jul 11 02:16:11 arrakis kernel: 00000000 00000000 00000b92 00000b50 01000000 001000e0 00000000 0008d00e
Jul 11 02:16:11 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jul 11 02:16:11 arrakis kernel: 56255203 00000000 000000e0 000000cc 00000000 00000100 00000100 1eadfa52
Jul 11 02:16:11 arrakis kernel: 62c55800 ffff8100 62c558be ffff8100 00000d28 00000001
Jul 11 02:16:11 arrakis kernel:
Jul 11 02:16:11 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 016aaeb0 ffffe200
Jul 11 02:16:11 arrakis kernel: 0000071d 000005a8 018b4b80 ffffe200 00000000 0000026d 01487eb8 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00001000 0159ebb0 ffffe200 00000000 00001000 01bb2ac0 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00001000 014dbe10 ffffe200 00000000 00000bf5 014b1fd0 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00001000 015c29b0 ffffe200 00000000 00001000 01598588 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00001000 015b48d0 ffffe200 00000000 00001000 015c51b8 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00001000 01e26de0 ffffe200 00000000 00001000 0148a998 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 0000005d 01594fb8 ffffe200 00000000 00001000 016d8f00 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00001000 00733028 ffffe200 00000000 00001000 01baaa20 ffffe200
Jul 11 02:16:11 arrakis kernel: 00000000 00000b9f a958b6d7 05df7a82 05ddd09a 312d8da0
Jul 11 02:16:11 arrakis kernel:
Jul 11 02:17:23 arrakis kernel: Splitting cloned skb
Jul 11 02:17:23 arrakis kernel: Pid: 4350, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 02:17:23 arrakis kernel:
Jul 11 02:17:23 arrakis kernel: Call Trace:
Jul 11 02:17:23 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 02:17:23 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 02:17:23 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 02:17:23 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 02:17:23 arrakis kernel: [<ffffffff80265f7b>] __alloc_pages+0x5b/0x390
Jul 11 02:17:23 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 02:17:23 arrakis kernel: [<ffffffff802a4f44>] __mark_inode_dirty+0x34/0x1a0
Jul 11 02:17:23 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 02:17:23 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 02:17:23 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 02:17:23 arrakis kernel: [<ffffffff802299c3>] set_next_entity+0x23/0x50
Jul 11 02:17:23 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 02:17:23 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 02:17:23 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 02:17:23 arrakis kernel:
Jul 11 02:17:46 arrakis kernel: Splitting cloned skb
Jul 11 02:17:46 arrakis kernel: Pid: 4350, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 02:17:46 arrakis kernel:
Jul 11 02:17:46 arrakis kernel: Call Trace:
Jul 11 02:17:46 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 02:17:46 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 02:17:46 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 02:17:46 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 02:17:46 arrakis kernel: [<ffffffff80265f7b>] __alloc_pages+0x5b/0x390
Jul 11 02:17:46 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 02:17:46 arrakis kernel: [<ffffffff802a4f44>] __mark_inode_dirty+0x34/0x1a0
Jul 11 02:17:46 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 02:17:46 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 02:17:46 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 02:17:46 arrakis kernel: [<ffffffff802299c3>] set_next_entity+0x23/0x50
Jul 11 02:17:46 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 02:17:46 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 02:17:46 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 02:17:46 arrakis kernel:
Jul 11 02:18:28 arrakis kernel: Splitting cloned skb
Jul 11 02:18:28 arrakis kernel: Pid: 4350, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 02:18:28 arrakis kernel:
Jul 11 02:18:28 arrakis kernel: Call Trace:
Jul 11 02:18:28 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 02:18:28 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 02:18:28 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 02:18:28 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 02:18:28 arrakis kernel: [<ffffffff80265f7b>] __alloc_pages+0x5b/0x390
Jul 11 02:18:28 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 02:18:28 arrakis kernel: [<ffffffff802a4f44>] __mark_inode_dirty+0x34/0x1a0
Jul 11 02:18:28 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 02:18:28 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 02:18:28 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 02:18:28 arrakis kernel: [<ffffffff802299c3>] set_next_entity+0x23/0x50
Jul 11 02:18:28 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 02:18:28 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 02:18:28 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 02:18:28 arrakis kernel:
Jul 11 03:57:08 arrakis kernel: Splitting cloned skb
Jul 11 03:57:08 arrakis kernel: Pid: 0, comm: swapper Not tainted 2.6.25.6 #2
Jul 11 03:57:08 arrakis kernel:
Jul 11 03:57:08 arrakis kernel: Call Trace:
Jul 11 03:57:08 arrakis kernel: <IRQ> [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 03:57:08 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 03:57:08 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 03:57:08 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 03:57:08 arrakis kernel: [<ffffffff80424596>] tcp_rcv_established+0x596/0x9d0
Jul 11 03:57:08 arrakis kernel: [<ffffffff8042a77c>] tcp_v4_do_rcv+0xdc/0x220
Jul 11 03:57:08 arrakis kernel: [<ffffffff8042c213>] tcp_v4_rcv+0x5e3/0x810
Jul 11 03:57:08 arrakis kernel: [<ffffffff8040e58f>] ip_local_deliver_finish+0x9f/0x200
Jul 11 03:57:08 arrakis kernel: [<ffffffff8040e2a0>] ip_rcv_finish+0x130/0x380
Jul 11 03:57:08 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 03:57:08 arrakis kernel: [<ffffffff88083b56>] :bnx2:bnx2_poll+0x676/0x1330
Jul 11 03:57:08 arrakis kernel: [<ffffffff8022d746>] run_rebalance_domains+0xf6/0x4c0
Jul 11 03:57:08 arrakis kernel: [<ffffffff80226787>] sys32_mmap2+0x47/0x100
Jul 11 03:57:08 arrakis kernel: [<ffffffff803edad8>] net_rx_action+0xf8/0x200
Jul 11 03:57:08 arrakis kernel: [<ffffffff8023a5b9>] __do_softirq+0x69/0xe0
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020c4ac>] call_softirq+0x1c/0x30
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020e995>] do_softirq+0x35/0x70
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020ec10>] do_IRQ+0x80/0x100
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020a410>] mwait_idle+0x0/0x50
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020b831>] ret_from_intr+0x0/0xa
Jul 11 03:57:08 arrakis kernel: <EOI> [<ffffffff80419a80>] tcp_poll+0x0/0x180
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020a451>] mwait_idle+0x41/0x50
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020a378>] cpu_idle+0x48/0x90
Jul 11 03:57:08 arrakis kernel:
Jul 11 03:57:08 arrakis kernel: Splitting cloned skb
Jul 11 03:57:08 arrakis kernel: Pid: 0, comm: swapper Not tainted 2.6.25.6 #2
Jul 11 03:57:08 arrakis kernel:
Jul 11 03:57:08 arrakis kernel: Call Trace:
Jul 11 03:57:08 arrakis kernel: <IRQ> [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 03:57:08 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 03:57:08 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 03:57:08 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 03:57:08 arrakis kernel: [<ffffffff80424596>] tcp_rcv_established+0x596/0x9d0
Jul 11 03:57:08 arrakis kernel: [<ffffffff8042a77c>] tcp_v4_do_rcv+0xdc/0x220
Jul 11 03:57:08 arrakis kernel: [<ffffffff8042c213>] tcp_v4_rcv+0x5e3/0x810
Jul 11 03:57:08 arrakis kernel: [<ffffffff8040e58f>] ip_local_deliver_finish+0x9f/0x200
Jul 11 03:57:08 arrakis kernel: [<ffffffff8040e2a0>] ip_rcv_finish+0x130/0x380
Jul 11 03:57:08 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 03:57:08 arrakis kernel: [<ffffffff88083b56>] :bnx2:bnx2_poll+0x676/0x1330
Jul 11 03:57:08 arrakis kernel: [<ffffffff8022d746>] run_rebalance_domains+0xf6/0x4c0
Jul 11 03:57:08 arrakis kernel: [<ffffffff802267da>] sys32_mmap2+0x9a/0x100
Jul 11 03:57:08 arrakis kernel: [<ffffffff803edad8>] net_rx_action+0xf8/0x200
Jul 11 03:57:08 arrakis kernel: [<ffffffff8023a5b9>] __do_softirq+0x69/0xe0
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020c4ac>] call_softirq+0x1c/0x30
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020e995>] do_softirq+0x35/0x70
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020ec10>] do_IRQ+0x80/0x100
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020a410>] mwait_idle+0x0/0x50
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020b831>] ret_from_intr+0x0/0xa
Jul 11 03:57:08 arrakis kernel: <EOI> [<ffffffff80419a80>] tcp_poll+0x0/0x180
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020a451>] mwait_idle+0x41/0x50
Jul 11 03:57:08 arrakis kernel: [<ffffffff8020a378>] cpu_idle+0x48/0x90
Jul 11 03:57:08 arrakis kernel:
Jul 11 03:57:08 arrakis kernel: bnx2: skb->nr_frags=1 is corrupted, should be 2
Jul 11 03:57:08 arrakis kernel: original skb:
Jul 11 03:57:08 arrakis kernel: 00000000 00000000 00000000 00000000 964321c0 ffff8101 00000000 00000000
Jul 11 03:57:08 arrakis kernel: 2db80000 ffff8102 2e8a4800 ffff8102 00000000 00000000 00000000 00000000
Jul 11 03:57:08 arrakis kernel: 00000000 00000000 00000000 00000000 7207adad 7207beb1 003be3a4 00000010
Jul 11 03:57:08 arrakis kernel: 00000000 00000000 0000113a 00001104 01000000 001000ec 00000000 0008300e
Jul 11 03:57:08 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jul 11 03:57:08 arrakis kernel: 2d2e3cdb 00000000 000000ec 000000d8 00000000 00000100 00000100 a6f7e35a
Jul 11 03:57:08 arrakis kernel: 94e95000 ffff8101 94e950ca ffff8101 000012dc 00000001
Jul 11 03:57:08 arrakis kernel:
Jul 11 03:57:08 arrakis kernel: 00010002 05ac0002 00010003 00000000 00000000 00000000 05f972a8 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000275 00000d8b 05f8f828 ffffe200 00000000 00000379 05f20388 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00001000 054167a0 ffffe200 00000000 00001000 0540e000 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00000c31 05f8f828 ffffe200 00000000 00001000 05f20388 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00001000 054167a0 ffffe200 00000000 00001000 0540e000 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00000c31 06ff9af0 ffffe200 00000000 00001000 06d055b0 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00001000 04683f60 ffffe200 00000000 000000ab 014864e8 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00000f83 038daff0 ffffe200 00000000 00001000 038dae30 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00001000 038da1f0 ffffe200 00000000 00001000 038dd910 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00000ce3 115fe1a6 8119c99d 52de821b 31d7c6c7
Jul 11 03:57:08 arrakis kernel:
Jul 11 03:57:08 arrakis kernel: Corrupted skb:
Jul 11 03:57:08 arrakis kernel: 00000000 00000000 00000000 00000000 964321c0 ffff8101 00000000 00000000
Jul 11 03:57:08 arrakis kernel: 2db80000 ffff8102 2e8a4800 ffff8102 00000000 00000000 00000000 00000000
Jul 11 03:57:08 arrakis kernel: 00000000 00000000 00000000 00000000 7207adad 7207beb1 003be3a4 00000010
Jul 11 03:57:08 arrakis kernel: 00000000 00000000 0000113a 00001104 01000000 001000ec 00000000 0008300e
Jul 11 03:57:08 arrakis kernel: 803e3440 ffffffff 00000000 00000000 00000000 00000000 00000000 20000000
Jul 11 03:57:08 arrakis kernel: 2d2e3cdb 00000000 000000ec 000000d8 00000000 00000100 00000100 a6f7e35a
Jul 11 03:57:08 arrakis kernel: 94e95000 ffff8101 94e950ca ffff8101 000012dc 00000001
Jul 11 03:57:08 arrakis kernel:
Jul 11 03:57:08 arrakis kernel: 00010002 00000001 00000001 00000000 00000000 00000000 05f972a8 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000275 000005ac 05f8f828 ffffe200 00000000 00000379 05f20388 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00001000 054167a0 ffffe200 00000000 00001000 0540e000 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00000c31 05f8f828 ffffe200 00000000 00001000 05f20388 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00001000 054167a0 ffffe200 00000000 00001000 0540e000 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00000c31 06ff9af0 ffffe200 00000000 00001000 06d055b0 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00001000 04683f60 ffffe200 00000000 000000ab 014864e8 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00000f83 038daff0 ffffe200 00000000 00001000 038dae30 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00001000 038da1f0 ffffe200 00000000 00001000 038dd910 ffffe200
Jul 11 03:57:08 arrakis kernel: 00000000 00000ce3 115fe1a6 8119c99d 52de821b 31d7c6c7
Jul 11 03:57:08 arrakis kernel:
Jul 11 03:59:02 arrakis kernel: Splitting cloned skb
Jul 11 03:59:02 arrakis kernel: Pid: 2082, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 03:59:02 arrakis kernel:
Jul 11 03:59:02 arrakis kernel: Call Trace:
Jul 11 03:59:02 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 03:59:02 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 03:59:02 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 03:59:02 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 03:59:02 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 03:59:02 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 03:59:02 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 03:59:02 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 03:59:02 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 03:59:02 arrakis kernel: [<ffffffff802299c3>] set_next_entity+0x23/0x50
Jul 11 03:59:02 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 03:59:02 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 03:59:02 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 03:59:02 arrakis kernel:
Jul 11 04:00:02 arrakis kernel: Splitting cloned skb
Jul 11 04:00:02 arrakis kernel: Pid: 2082, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 04:00:02 arrakis kernel:
Jul 11 04:00:02 arrakis kernel: Call Trace:
Jul 11 04:00:02 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 04:00:02 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 04:00:02 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 04:00:02 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 04:00:02 arrakis kernel: [<ffffffff80265f7b>] __alloc_pages+0x5b/0x390
Jul 11 04:00:02 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 04:00:02 arrakis kernel: [<ffffffff802a4f75>] __mark_inode_dirty+0x65/0x1a0
Jul 11 04:00:02 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 04:00:02 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 04:00:02 arrakis kernel: [<ffffffff80268ba2>] __pagevec_lru_add_active+0xd2/0xf0
Jul 11 04:00:02 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 04:00:02 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 04:00:02 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 04:00:02 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 04:00:02 arrakis kernel:
Jul 11 04:00:18 arrakis kernel: Splitting cloned skb
Jul 11 04:00:18 arrakis kernel: Pid: 2082, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 04:00:18 arrakis kernel:
Jul 11 04:00:18 arrakis kernel: Call Trace:
Jul 11 04:00:18 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 04:00:18 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 04:00:18 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 04:00:18 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 04:00:18 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 04:00:18 arrakis kernel: [<ffffffff802a4f75>] __mark_inode_dirty+0x65/0x1a0
Jul 11 04:00:18 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 04:00:18 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 04:00:18 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 04:00:18 arrakis kernel: [<ffffffff802299c3>] set_next_entity+0x23/0x50
Jul 11 04:00:18 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 04:00:18 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 04:00:18 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 04:00:18 arrakis kernel:
Jul 11 04:00:34 arrakis kernel: Splitting cloned skb
Jul 11 04:00:34 arrakis kernel: Pid: 2082, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 04:00:34 arrakis kernel:
Jul 11 04:00:34 arrakis kernel: Call Trace:
Jul 11 04:00:34 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 04:00:34 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 04:00:34 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 04:00:34 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 04:00:34 arrakis kernel: [<ffffffff80265f7b>] __alloc_pages+0x5b/0x390
Jul 11 04:00:34 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 04:00:34 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 04:00:34 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 04:00:34 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 04:00:34 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 04:00:34 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 04:00:34 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 04:00:34 arrakis kernel:
Jul 11 04:01:54 arrakis kernel: Splitting cloned skb
Jul 11 04:01:54 arrakis kernel: Pid: 2082, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 04:01:54 arrakis kernel:
Jul 11 04:01:54 arrakis kernel: Call Trace:
Jul 11 04:01:54 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 04:01:54 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 04:01:54 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 04:01:54 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 04:01:54 arrakis kernel: [<ffffffff80265f7b>] __alloc_pages+0x5b/0x390
Jul 11 04:01:54 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 04:01:54 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 04:01:54 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 04:01:54 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 04:01:54 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 04:01:54 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 04:01:54 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 04:01:54 arrakis kernel:
Jul 11 04:02:14 arrakis kernel: Splitting cloned skb
Jul 11 04:02:14 arrakis kernel: Pid: 2082, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 04:02:14 arrakis kernel:
Jul 11 04:02:14 arrakis kernel: Call Trace:
Jul 11 04:02:14 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 04:02:14 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 04:02:14 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 04:02:14 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 04:02:14 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 04:02:14 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 04:02:14 arrakis kernel: [<ffffffff802a4f44>] __mark_inode_dirty+0x34/0x1a0
Jul 11 04:02:14 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 04:02:14 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 04:02:14 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 04:02:14 arrakis kernel: [<ffffffff802299c3>] set_next_entity+0x23/0x50
Jul 11 04:02:14 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 04:02:14 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 04:02:14 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 04:02:14 arrakis kernel:
Jul 11 04:02:34 arrakis kernel: Splitting cloned skb
Jul 11 04:02:34 arrakis kernel: Pid: 2082, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 04:02:34 arrakis kernel:
Jul 11 04:02:34 arrakis kernel: Call Trace:
Jul 11 04:02:34 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 04:02:34 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 04:02:34 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 04:02:34 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 04:02:34 arrakis kernel: [<ffffffff80265f7b>] __alloc_pages+0x5b/0x390
Jul 11 04:02:34 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 04:02:34 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 04:02:34 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 04:02:34 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 04:02:34 arrakis kernel: [<ffffffff802299c3>] set_next_entity+0x23/0x50
Jul 11 04:02:34 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 04:02:34 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 04:02:34 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 04:02:34 arrakis kernel:
Jul 11 04:02:41 arrakis kernel: Splitting cloned skb
Jul 11 04:02:41 arrakis kernel: Pid: 2082, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 04:02:41 arrakis kernel:
Jul 11 04:02:41 arrakis kernel: Call Trace:
Jul 11 04:02:41 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 04:02:41 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 04:02:41 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 04:02:41 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 04:02:41 arrakis kernel: [<ffffffff80265f7b>] __alloc_pages+0x5b/0x390
Jul 11 04:02:41 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 04:02:41 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 04:02:41 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 04:02:41 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 04:02:41 arrakis kernel: [<ffffffff802299c3>] set_next_entity+0x23/0x50
Jul 11 04:02:41 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 04:02:41 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 04:02:41 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 04:02:41 arrakis kernel:
Jul 11 04:02:57 arrakis kernel: Splitting cloned skb
Jul 11 04:02:57 arrakis kernel: Pid: 2082, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 04:02:57 arrakis kernel:
Jul 11 04:02:57 arrakis kernel: Call Trace:
Jul 11 04:02:57 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 04:02:57 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 04:02:57 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 04:02:57 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 04:02:57 arrakis kernel: [<ffffffff80265f7b>] __alloc_pages+0x5b/0x390
Jul 11 04:02:57 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 04:02:57 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 04:02:57 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 04:02:57 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 04:02:57 arrakis kernel: [<ffffffff802299c3>] set_next_entity+0x23/0x50
Jul 11 04:02:57 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 04:02:57 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 04:02:57 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 04:02:57 arrakis kernel:
Jul 11 04:03:02 arrakis kernel: Splitting cloned skb
Jul 11 04:03:02 arrakis kernel: Pid: 2082, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 04:03:02 arrakis kernel:
Jul 11 04:03:02 arrakis kernel: Call Trace:
Jul 11 04:03:02 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 04:03:02 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 04:03:02 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 04:03:02 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 04:03:02 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 04:03:02 arrakis kernel: [<ffffffff802a4f44>] __mark_inode_dirty+0x34/0x1a0
Jul 11 04:03:02 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 04:03:02 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 04:03:02 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 04:03:02 arrakis kernel: [<ffffffff802299c3>] set_next_entity+0x23/0x50
Jul 11 04:03:02 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 04:03:02 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 04:03:02 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 04:03:02 arrakis kernel:
Jul 11 04:04:24 arrakis kernel: Splitting cloned skb
Jul 11 04:04:24 arrakis kernel: Pid: 2082, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 04:04:24 arrakis kernel:
Jul 11 04:04:24 arrakis kernel: Call Trace:
Jul 11 04:04:24 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 04:04:24 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 04:04:24 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 04:04:24 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 04:04:24 arrakis kernel: [<ffffffff80265f7b>] __alloc_pages+0x5b/0x390
Jul 11 04:04:24 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 04:04:24 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 04:04:24 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 04:04:24 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 04:04:24 arrakis kernel: [<ffffffff802299c3>] set_next_entity+0x23/0x50
Jul 11 04:04:24 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 04:04:24 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 04:04:24 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 04:04:24 arrakis kernel:
Jul 11 04:05:03 arrakis kernel: Splitting cloned skb
Jul 11 04:05:03 arrakis kernel: Pid: 2082, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 04:05:03 arrakis kernel:
Jul 11 04:05:03 arrakis kernel: Call Trace:
Jul 11 04:05:03 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 04:05:03 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 04:05:03 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 04:05:03 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 04:05:03 arrakis kernel: [<ffffffff80265f7b>] __alloc_pages+0x5b/0x390
Jul 11 04:05:03 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 04:05:03 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 04:05:03 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 04:05:03 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 04:05:03 arrakis kernel: [<ffffffff802299c3>] set_next_entity+0x23/0x50
Jul 11 04:05:03 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 04:05:03 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 04:05:03 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 04:05:03 arrakis kernel:
Jul 11 04:05:19 arrakis kernel: Splitting cloned skb
Jul 11 04:05:19 arrakis kernel: Pid: 2082, comm: rsync Not tainted 2.6.25.6 #2
Jul 11 04:05:19 arrakis kernel:
Jul 11 04:05:19 arrakis kernel: Call Trace:
Jul 11 04:05:19 arrakis kernel: [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
Jul 11 04:05:19 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
Jul 11 04:05:19 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
Jul 11 04:05:19 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
Jul 11 04:05:19 arrakis kernel: [<ffffffff80265f7b>] __alloc_pages+0x5b/0x390
Jul 11 04:05:19 arrakis kernel: [<ffffffff8041b150>] tcp_sendmsg+0x880/0xbf0
Jul 11 04:05:19 arrakis kernel: [<ffffffff802a4f75>] __mark_inode_dirty+0x65/0x1a0
Jul 11 04:05:19 arrakis kernel: [<ffffffff803ddf70>] sock_aio_write+0x110/0x120
Jul 11 04:05:19 arrakis kernel: [<ffffffff80285309>] do_sync_write+0xd9/0x120
Jul 11 04:05:19 arrakis kernel: [<ffffffff80249590>] autoremove_wake_function+0x0/0x30
Jul 11 04:05:19 arrakis kernel: [<ffffffff802299c3>] set_next_entity+0x23/0x50
Jul 11 04:05:19 arrakis kernel: [<ffffffff80285c5a>] vfs_write+0x15a/0x170
Jul 11 04:05:19 arrakis kernel: [<ffffffff802862a3>] sys_write+0x53/0x90
Jul 11 04:05:19 arrakis kernel: [<ffffffff8020b30b>] system_call_after_swapgs+0x7b/0x80
Jul 11 04:05:19 arrakis kernel:
--
Josip Rodin
mirrors@debian.org
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-11 9:24 ` Josip Rodin
@ 2008-07-11 9:56 ` David Miller
2008-07-11 12:19 ` Patrick McHardy
2008-07-12 9:49 ` Jarek Poplawski
0 siblings, 2 replies; 50+ messages in thread
From: David Miller @ 2008-07-11 9:56 UTC (permalink / raw)
To: joy; +Cc: mchan, billfink, bhutchings, netdev, mirrors, kaber, devik
From: Josip Rodin <joy@entuzijast.net>
Date: Fri, 11 Jul 2008 11:24:16 +0200
[ Patrick/Martin, you can simply skip to the final paragraph. ]
> Here we go, it triggered, here are the first few, tell me if you need more:
Thanks for the trace:
> Jul 11 02:15:10 arrakis kernel: Splitting cloned skb
> Jul 11 02:15:10 arrakis kernel: Pid: 0, comm: swapper Not tainted 2.6.25.6 #2
> Jul 11 02:15:10 arrakis kernel:
> Jul 11 02:15:10 arrakis kernel: Call Trace:
> Jul 11 02:15:10 arrakis kernel: <IRQ> [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
> Jul 11 02:15:10 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
> Jul 11 02:15:10 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
> Jul 11 02:15:10 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
> Jul 11 02:15:10 arrakis kernel: [<ffffffff80424596>] tcp_rcv_established+0x596/0x9d0
So it's splitting a frame up which should be new data, but for
some reason made it to the device previously.
The comment above tso_fragment() reads:
/* Trim TSO SKB to LEN bytes, put the remaining data into a new packet
* which is put after SKB on the list. It is very much like
* tcp_fragment() except that it may make several kinds of assumptions
* in order to speed up the splitting operation. In particular, we
* know that all the data is in scatter-gather pages, and that the
* packet has never been sent out before (and thus is not cloned).
*/
Note in particular the final phrase inside the parens. :-)))
There is only one way this situation seen in the trace can develop.
That is if the queueing discipline gave the packet to the device, yet
returned a value that made TCP believe the packet was not.
When TCP sees such a return value, it does not advance the head of the
write queue. It will retry to send that head packet again later. And
that's what we seem to be seeing here.
TCP treats any non-zero return value other than NET_XMIT_CN
in this way (see tcp_transmit_skb and how it uses net_xmit_eval).
I notice that HTB does a lot of very queer things wrt. return
values.
For example, it seems that if the class's leaf queue ->enqueue()
returns any non-success value, it gives NET_XMIT_DROP back down to the
call chain.
But what if that leaf ->enqueue() is something that passes back
NET_XMIT_CN? NET_XMIT_CN can be signalled for things like RED, in
cases where some "other" packet in the same class got dropped but not
necessarily the one you enqueued.
NET_XMIT_CN means backoff, but it does not indicate that the specific
packet being enqueued was dropped. It just means "some" packet from
the same flow was dropped, and therefore there is congestion on this
flow.
Even more simpler qdiscs such as SFQ use the NET_XMIT_CN return value
when it does a drop.
So this return value munging being done by HTB creates the illegal
situation.
I'm not sure how to fix this, because I'm not sure how these
NET_XMIT_CN situations should be handled wrt. maintaining a proper
parent queue length value.
Patrick/Martin, in HTB's ->enqueue() and ->requeue() we need to
propagate NET_XMIT_CN to the caller if that's what the leaf qdisc
signals to us. But the question is, should sch->q.qlen be
incremented in that case? NET_XMIT_CN means that some packet got
dropped, but not necessarily this one. If, for example, RED
drops another packet already in the queue does it somehow adjust
the parent sch->q.qlen back down? If not, it's pretty clear how
this bug got created in the first place :) Below is my idiotic
attempt to cure this, but this whole situation needs an audit:
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
index 3fb58f4..aa20b47 100644
--- a/net/sched/sch_htb.c
+++ b/net/sched/sch_htb.c
@@ -583,6 +583,7 @@ static int htb_enqueue(struct sk_buff *skb, struct Qdisc *sch)
if (q->direct_queue.qlen < q->direct_qlen) {
__skb_queue_tail(&q->direct_queue, skb);
q->direct_pkts++;
+ ret = NET_XMIT_SUCCESS;
} else {
kfree_skb(skb);
sch->qstats.drops++;
@@ -595,22 +596,26 @@ static int htb_enqueue(struct sk_buff *skb, struct Qdisc *sch)
kfree_skb(skb);
return ret;
#endif
- } else if (cl->un.leaf.q->enqueue(skb, cl->un.leaf.q) !=
- NET_XMIT_SUCCESS) {
- sch->qstats.drops++;
- cl->qstats.drops++;
- return NET_XMIT_DROP;
} else {
- cl->bstats.packets +=
- skb_is_gso(skb)?skb_shinfo(skb)->gso_segs:1;
- cl->bstats.bytes += skb->len;
- htb_activate(q, cl);
+ ret = cl->un.leaf.q->enqueue(skb, cl->un.leaf.q);
+ if (ret == NET_XMIT_DROP) {
+ sch->qstats.drops++;
+ cl->qstats.drops++;
+ } else {
+ cl->bstats.packets +=
+ skb_is_gso(skb)?skb_shinfo(skb)->gso_segs:1;
+ cl->bstats.bytes += skb->len;
+ htb_activate(q, cl);
+ }
}
- sch->q.qlen++;
- sch->bstats.packets += skb_is_gso(skb)?skb_shinfo(skb)->gso_segs:1;
- sch->bstats.bytes += skb->len;
- return NET_XMIT_SUCCESS;
+ if (ret == NET_XMIT_SUCCESS) {
+ sch->q.qlen++;
+ sch->bstats.packets +=
+ skb_is_gso(skb)?skb_shinfo(skb)->gso_segs:1;
+ sch->bstats.bytes += skb->len;
+ }
+ return ret;
}
/* TODO: requeuing packet charges it to policers again !! */
@@ -625,6 +630,7 @@ static int htb_requeue(struct sk_buff *skb, struct Qdisc *sch)
/* enqueue to helper queue */
if (q->direct_queue.qlen < q->direct_qlen) {
__skb_queue_head(&q->direct_queue, skb);
+ ret = NET_XMIT_SUCCESS;
} else {
__skb_queue_head(&q->direct_queue, skb);
tskb = __skb_dequeue_tail(&q->direct_queue);
@@ -639,17 +645,21 @@ static int htb_requeue(struct sk_buff *skb, struct Qdisc *sch)
kfree_skb(skb);
return ret;
#endif
- } else if (cl->un.leaf.q->ops->requeue(skb, cl->un.leaf.q) !=
- NET_XMIT_SUCCESS) {
- sch->qstats.drops++;
- cl->qstats.drops++;
- return NET_XMIT_DROP;
- } else
- htb_activate(q, cl);
+ } else {
+ ret = cl->un.leaf.q->ops->requeue(skb, cl->un.leaf.q);
+ if (ret == NET_XMIT_DROP) {
+ sch->qstats.drops++;
+ cl->qstats.drops++;
+ } else {
+ htb_activate(q, cl);
+ }
+ }
- sch->q.qlen++;
- sch->qstats.requeues++;
- return NET_XMIT_SUCCESS;
+ if (ret == NET_XMIT_SUCCESS) {
+ sch->q.qlen++;
+ sch->qstats.requeues++;
+ }
+ return ret;
}
/**
^ permalink raw reply related [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-11 9:56 ` David Miller
@ 2008-07-11 12:19 ` Patrick McHardy
2008-07-12 9:49 ` Jarek Poplawski
1 sibling, 0 replies; 50+ messages in thread
From: Patrick McHardy @ 2008-07-11 12:19 UTC (permalink / raw)
To: David Miller; +Cc: joy, mchan, billfink, bhutchings, netdev, mirrors, devik
David Miller wrote:
> From: Josip Rodin <joy@entuzijast.net>
> Date: Fri, 11 Jul 2008 11:24:16 +0200
>
> [ Patrick/Martin, you can simply skip to the final paragraph. ]
>
>> Here we go, it triggered, here are the first few, tell me if you need more:
>
> Thanks for the trace:
>
>> Jul 11 02:15:10 arrakis kernel: Splitting cloned skb
>> Jul 11 02:15:10 arrakis kernel: Pid: 0, comm: swapper Not tainted 2.6.25.6 #2
>> Jul 11 02:15:10 arrakis kernel:
>> Jul 11 02:15:10 arrakis kernel: Call Trace:
>> Jul 11 02:15:10 arrakis kernel: <IRQ> [<ffffffff803e5e75>] __alloc_skb+0x85/0x150
>> Jul 11 02:15:10 arrakis kernel: [<ffffffff803e66aa>] skb_split+0x4a/0x300
>> Jul 11 02:15:10 arrakis kernel: [<ffffffff8042700b>] tso_fragment+0xfb/0x180
>> Jul 11 02:15:10 arrakis kernel: [<ffffffff8042716e>] __tcp_push_pending_frames+0xde/0x860
>> Jul 11 02:15:10 arrakis kernel: [<ffffffff80424596>] tcp_rcv_established+0x596/0x9d0
>
> So it's splitting a frame up which should be new data, but for
> some reason made it to the device previously.
>
> The comment above tso_fragment() reads:
>
> /* Trim TSO SKB to LEN bytes, put the remaining data into a new packet
> * which is put after SKB on the list. It is very much like
> * tcp_fragment() except that it may make several kinds of assumptions
> * in order to speed up the splitting operation. In particular, we
> * know that all the data is in scatter-gather pages, and that the
> * packet has never been sent out before (and thus is not cloned).
> */
>
> Note in particular the final phrase inside the parens. :-)))
>
> There is only one way this situation seen in the trace can develop.
> That is if the queueing discipline gave the packet to the device, yet
> returned a value that made TCP believe the packet was not.
>
> When TCP sees such a return value, it does not advance the head of the
> write queue. It will retry to send that head packet again later. And
> that's what we seem to be seeing here.
>
> TCP treats any non-zero return value other than NET_XMIT_CN
> in this way (see tcp_transmit_skb and how it uses net_xmit_eval).
>
> I notice that HTB does a lot of very queer things wrt. return
> values.
>
> For example, it seems that if the class's leaf queue ->enqueue()
> returns any non-success value, it gives NET_XMIT_DROP back down to the
> call chain.
>
> But what if that leaf ->enqueue() is something that passes back
> NET_XMIT_CN? NET_XMIT_CN can be signalled for things like RED, in
> cases where some "other" packet in the same class got dropped but not
> necessarily the one you enqueued.
>
> NET_XMIT_CN means backoff, but it does not indicate that the specific
> packet being enqueued was dropped. It just means "some" packet from
> the same flow was dropped, and therefore there is congestion on this
> flow.
>
> Even more simpler qdiscs such as SFQ use the NET_XMIT_CN return value
> when it does a drop.
>
> So this return value munging being done by HTB creates the illegal
> situation.
>
> I'm not sure how to fix this, because I'm not sure how these
> NET_XMIT_CN situations should be handled wrt. maintaining a proper
> parent queue length value.
>
> Patrick/Martin, in HTB's ->enqueue() and ->requeue() we need to
> propagate NET_XMIT_CN to the caller if that's what the leaf qdisc
> signals to us. But the question is, should sch->q.qlen be
> incremented in that case? NET_XMIT_CN means that some packet got
> dropped, but not necessarily this one. If, for example, RED
> drops another packet already in the queue does it somehow adjust
> the parent sch->q.qlen back down? If not, it's pretty clear how
> this bug got created in the first place :)
Usually we only increment q.qlen on NET_XMIT_SUCCESS, in
all other cases it stays untouched.
> Below is my idiotic
> attempt to cure this, but this whole situation needs an audit:
Yes, this also reminded me of another related bug, when actions
steel a packet, qdiscs return NET_XMIT_SUCCESS, which causes upper
qdiscs to perform incorrect qlen adjustments.
I'll see if I can audit all these paths sometime this weekend.
> diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
> index 3fb58f4..aa20b47 100644
> --- a/net/sched/sch_htb.c
> +++ b/net/sched/sch_htb.c
> + ret = cl->un.leaf.q->enqueue(skb, cl->un.leaf.q);
> + if (ret == NET_XMIT_DROP) {
> + sch->qstats.drops++;
> + cl->qstats.drops++;
> + } else {
> + cl->bstats.packets +=
> + skb_is_gso(skb)?skb_shinfo(skb)->gso_segs:1;
> + cl->bstats.bytes += skb->len;
> + htb_activate(q, cl);
> + }
> }
The propagation of the leaf qdiscs return value is definitely
correct. The patch looks fine to me.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-11 9:56 ` David Miller
2008-07-11 12:19 ` Patrick McHardy
@ 2008-07-12 9:49 ` Jarek Poplawski
2008-07-12 13:21 ` Jarek Poplawski
1 sibling, 1 reply; 50+ messages in thread
From: Jarek Poplawski @ 2008-07-12 9:49 UTC (permalink / raw)
To: David Miller
Cc: joy, mchan, billfink, bhutchings, netdev, mirrors, kaber, devik
David Miller wrote, On 07/11/2008 11:56 AM:
...
> diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
> index 3fb58f4..aa20b47 100644
> --- a/net/sched/sch_htb.c
> +++ b/net/sched/sch_htb.c
> @@ -583,6 +583,7 @@ static int htb_enqueue(struct sk_buff *skb, struct Qdisc *sch)
> if (q->direct_queue.qlen < q->direct_qlen) {
> __skb_queue_tail(&q->direct_queue, skb);
> q->direct_pkts++;
> + ret = NET_XMIT_SUCCESS;
> } else {
> kfree_skb(skb);
> sch->qstats.drops++;
> @@ -595,22 +596,26 @@ static int htb_enqueue(struct sk_buff *skb, struct Qdisc *sch)
> kfree_skb(skb);
> return ret;
> #endif
> - } else if (cl->un.leaf.q->enqueue(skb, cl->un.leaf.q) !=
> - NET_XMIT_SUCCESS) {
> - sch->qstats.drops++;
> - cl->qstats.drops++;
> - return NET_XMIT_DROP;
> } else {
> - cl->bstats.packets +=
> - skb_is_gso(skb)?skb_shinfo(skb)->gso_segs:1;
> - cl->bstats.bytes += skb->len;
> - htb_activate(q, cl);
> + ret = cl->un.leaf.q->enqueue(skb, cl->un.leaf.q);
> + if (ret == NET_XMIT_DROP) {
> + sch->qstats.drops++;
> + cl->qstats.drops++;
> + } else {
> + cl->bstats.packets +=
> + skb_is_gso(skb)?skb_shinfo(skb)->gso_segs:1;
> + cl->bstats.bytes += skb->len;
> + htb_activate(q, cl);
This isn't correct: htb_activate() needs cl->un.leaf.q->q.qlen > 0, and
only NET_XMIT_SUCCESS can guarantee this.
Jarek P.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-12 9:49 ` Jarek Poplawski
@ 2008-07-12 13:21 ` Jarek Poplawski
2008-07-14 15:27 ` Patrick McHardy
0 siblings, 1 reply; 50+ messages in thread
From: Jarek Poplawski @ 2008-07-12 13:21 UTC (permalink / raw)
Cc: David Miller, joy, mchan, billfink, bhutchings, netdev, mirrors,
kaber, devik
Jarek Poplawski wrote, On 07/12/2008 11:49 AM:
> David Miller wrote, On 07/11/2008 11:56 AM:
> ...
>
>> diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
>> index 3fb58f4..aa20b47 100644
>> --- a/net/sched/sch_htb.c
>> +++ b/net/sched/sch_htb.c
>> @@ -583,6 +583,7 @@ static int htb_enqueue(struct sk_buff *skb, struct Qdisc *sch)
>> if (q->direct_queue.qlen < q->direct_qlen) {
>> __skb_queue_tail(&q->direct_queue, skb);
>> q->direct_pkts++;
>> + ret = NET_XMIT_SUCCESS;
>> } else {
>> kfree_skb(skb);
>> sch->qstats.drops++;
>> @@ -595,22 +596,26 @@ static int htb_enqueue(struct sk_buff *skb, struct Qdisc *sch)
>> kfree_skb(skb);
>> return ret;
>> #endif
>> - } else if (cl->un.leaf.q->enqueue(skb, cl->un.leaf.q) !=
>> - NET_XMIT_SUCCESS) {
>> - sch->qstats.drops++;
>> - cl->qstats.drops++;
>> - return NET_XMIT_DROP;
>> } else {
>> - cl->bstats.packets +=
>> - skb_is_gso(skb)?skb_shinfo(skb)->gso_segs:1;
>> - cl->bstats.bytes += skb->len;
>> - htb_activate(q, cl);
>> + ret = cl->un.leaf.q->enqueue(skb, cl->un.leaf.q);
>> + if (ret == NET_XMIT_DROP) {
>> + sch->qstats.drops++;
>> + cl->qstats.drops++;
>> + } else {
>> + cl->bstats.packets +=
>> + skb_is_gso(skb)?skb_shinfo(skb)->gso_segs:1;
>> + cl->bstats.bytes += skb->len;
>> + htb_activate(q, cl);
>
>
> This isn't correct: htb_activate() needs cl->un.leaf.q->q.qlen > 0, and
> only NET_XMIT_SUCCESS can guarantee this.
...Actually, it looks like even NET_XMIT_SUCCESS can't guarantee this
(when returned after TC_ACT_QUEUED or TC_ACT_STOLEN), so additional qlen
test is needed here.
Jarek P.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-12 13:21 ` Jarek Poplawski
@ 2008-07-14 15:27 ` Patrick McHardy
2008-07-14 17:20 ` Jarek Poplawski
0 siblings, 1 reply; 50+ messages in thread
From: Patrick McHardy @ 2008-07-14 15:27 UTC (permalink / raw)
To: Jarek Poplawski
Cc: David Miller, joy, mchan, billfink, bhutchings, netdev, mirrors,
devik
Jarek Poplawski wrote:
> Jarek Poplawski wrote, On 07/12/2008 11:49 AM:
>
>> This isn't correct: htb_activate() needs cl->un.leaf.q->q.qlen > 0, and
>> only NET_XMIT_SUCCESS can guarantee this.
>
>
> ...Actually, it looks like even NET_XMIT_SUCCESS can't guarantee this
> (when returned after TC_ACT_QUEUED or TC_ACT_STOLEN), so additional qlen
> test is needed here.
The TC action code translation needs to be fixed instead.
I didn't manage to work on this this weekend as I intended, but
I'll take care of it soon unless someone beats me to it.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-14 15:27 ` Patrick McHardy
@ 2008-07-14 17:20 ` Jarek Poplawski
2008-07-14 17:25 ` Jarek Poplawski
2008-07-14 20:21 ` Josip Rodin
0 siblings, 2 replies; 50+ messages in thread
From: Jarek Poplawski @ 2008-07-14 17:20 UTC (permalink / raw)
To: Patrick McHardy
Cc: David Miller, joy, mchan, billfink, bhutchings, netdev, mirrors,
devik
On Mon, Jul 14, 2008 at 05:27:29PM +0200, Patrick McHardy wrote:
> Jarek Poplawski wrote:
>> Jarek Poplawski wrote, On 07/12/2008 11:49 AM:
>>
>>> This isn't correct: htb_activate() needs cl->un.leaf.q->q.qlen > 0, and
>>> only NET_XMIT_SUCCESS can guarantee this.
>>
>>
>> ...Actually, it looks like even NET_XMIT_SUCCESS can't guarantee this
>> (when returned after TC_ACT_QUEUED or TC_ACT_STOLEN), so additional qlen
>> test is needed here.
>
> The TC action code translation needs to be fixed instead.
>
> I didn't manage to work on this this weekend as I intended, but
> I'll take care of it soon unless someone beats me to it.
IMHO, since this TC_ACT_ issue looks quite rare, David's patch, with
"htb_activate()" moved to the last "if (ret == NET_XMIT_SUCCESS)"
blocks, looks like necessary.
BTW, I wonder how Josip's testing?
Jarek P.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-14 17:20 ` Jarek Poplawski
@ 2008-07-14 17:25 ` Jarek Poplawski
2008-07-14 20:21 ` Josip Rodin
1 sibling, 0 replies; 50+ messages in thread
From: Jarek Poplawski @ 2008-07-14 17:25 UTC (permalink / raw)
To: Patrick McHardy
Cc: David Miller, joy, mchan, billfink, bhutchings, netdev, mirrors,
devik
On Mon, Jul 14, 2008 at 07:20:55PM +0200, Jarek Poplawski wrote:
...
> IMHO, since this TC_ACT_ issue looks quite rare, David's patch, with
> "htb_activate()" moved to the last "if (ret == NET_XMIT_SUCCESS)"
> blocks, looks like necessary.
...OOOPS! Sorry, IMHO, these htb_activate() simply need their own
"if (ret == NET_XMIT_SUCCESS)" right where they are.
Jarek P.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-14 17:20 ` Jarek Poplawski
2008-07-14 17:25 ` Jarek Poplawski
@ 2008-07-14 20:21 ` Josip Rodin
2008-07-14 21:22 ` Jarek Poplawski
1 sibling, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-07-14 20:21 UTC (permalink / raw)
To: Jarek Poplawski
Cc: Patrick McHardy, David Miller, mchan, billfink, bhutchings,
netdev, mirrors, devik
On Mon, Jul 14, 2008 at 07:20:55PM +0200, Jarek Poplawski wrote:
> On Mon, Jul 14, 2008 at 05:27:29PM +0200, Patrick McHardy wrote:
> > Jarek Poplawski wrote:
> >> Jarek Poplawski wrote, On 07/12/2008 11:49 AM:
> >>
> >>> This isn't correct: htb_activate() needs cl->un.leaf.q->q.qlen > 0, and
> >>> only NET_XMIT_SUCCESS can guarantee this.
> >>
> >>
> >> ...Actually, it looks like even NET_XMIT_SUCCESS can't guarantee this
> >> (when returned after TC_ACT_QUEUED or TC_ACT_STOLEN), so additional qlen
> >> test is needed here.
> >
> > The TC action code translation needs to be fixed instead.
> >
> > I didn't manage to work on this this weekend as I intended, but
> > I'll take care of it soon unless someone beats me to it.
>
> IMHO, since this TC_ACT_ issue looks quite rare, David's patch, with
> "htb_activate()" moved to the last "if (ret == NET_XMIT_SUCCESS)"
> blocks, looks like necessary.
>
> BTW, I wonder how Josip's testing?
Do you want me to try out David's patch? I was hoping to get a definite okay
from someone before applying it, given that this is a production machine
for us and we do need it in *some* kind of an operational state.
I've got many more tens of kilobytes of logs from the previous debugging
patches, if you want I can send them over.
--
2. That which causes joy or happiness.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-14 20:21 ` Josip Rodin
@ 2008-07-14 21:22 ` Jarek Poplawski
2008-07-14 21:26 ` Josip Rodin
0 siblings, 1 reply; 50+ messages in thread
From: Jarek Poplawski @ 2008-07-14 21:22 UTC (permalink / raw)
To: Josip Rodin
Cc: Patrick McHardy, David Miller, mchan, billfink, bhutchings,
netdev, mirrors, devik
On Mon, Jul 14, 2008 at 10:21:51PM +0200, Josip Rodin wrote:
> On Mon, Jul 14, 2008 at 07:20:55PM +0200, Jarek Poplawski wrote:
...
> > BTW, I wonder how Josip's testing?
>
> Do you want me to try out David's patch? I was hoping to get a definite okay
> from someone before applying it, given that this is a production machine
> for us and we do need it in *some* kind of an operational state.
Hmm... if you trust me more than David?! As a matter of fact, I thought
it's under testing already, but since I've some doubts, I attach below
my version of David's patch. IMHO, it looks safe, but never knows...
> I've got many more tens of kilobytes of logs from the previous debugging
> patches, if you want I can send them over.
This debugging, I guess, shows corrupted data, but the reason is hard
to find. David found one of possible reasons and it needs checking. I
don't think this patch in any version can do more damage than doing
nothing - unless you prefer to use Michael's patch to next kernels
(BTW, don't remove this patch yet).
Thanks,
Jarek P.
---
diff -Nurp 2.6.25.4-/net/sched/sch_htb.c 2.6.25.4+/net/sched/sch_htb.c
--- 2.6.25.4-/net/sched/sch_htb.c 2008-05-17 19:23:44.000000000 +0200
+++ 2.6.25.4+/net/sched/sch_htb.c 2008-07-14 22:39:28.000000000 +0200
@@ -576,6 +576,7 @@ static int htb_enqueue(struct sk_buff *s
if (q->direct_queue.qlen < q->direct_qlen) {
__skb_queue_tail(&q->direct_queue, skb);
q->direct_pkts++;
+ ret = NET_XMIT_SUCCESS;
} else {
kfree_skb(skb);
sch->qstats.drops++;
@@ -588,22 +589,27 @@ static int htb_enqueue(struct sk_buff *s
kfree_skb(skb);
return ret;
#endif
- } else if (cl->un.leaf.q->enqueue(skb, cl->un.leaf.q) !=
- NET_XMIT_SUCCESS) {
- sch->qstats.drops++;
- cl->qstats.drops++;
- return NET_XMIT_DROP;
} else {
- cl->bstats.packets +=
- skb_is_gso(skb)?skb_shinfo(skb)->gso_segs:1;
- cl->bstats.bytes += skb->len;
- htb_activate(q, cl);
+ ret = cl->un.leaf.q->enqueue(skb, cl->un.leaf.q);
+ if (ret == NET_XMIT_DROP) {
+ sch->qstats.drops++;
+ cl->qstats.drops++;
+ } else {
+ cl->bstats.packets +=
+ skb_is_gso(skb)?skb_shinfo(skb)->gso_segs:1;
+ cl->bstats.bytes += skb->len;
+ if (ret == NET_XMIT_SUCCESS)
+ htb_activate(q, cl);
+ }
}
- sch->q.qlen++;
- sch->bstats.packets += skb_is_gso(skb)?skb_shinfo(skb)->gso_segs:1;
- sch->bstats.bytes += skb->len;
- return NET_XMIT_SUCCESS;
+ if (ret == NET_XMIT_SUCCESS) {
+ sch->q.qlen++;
+ sch->bstats.packets +=
+ skb_is_gso(skb)?skb_shinfo(skb)->gso_segs:1;
+ sch->bstats.bytes += skb->len;
+ }
+ return ret;
}
/* TODO: requeuing packet charges it to policers again !! */
@@ -618,6 +624,7 @@ static int htb_requeue(struct sk_buff *s
/* enqueue to helper queue */
if (q->direct_queue.qlen < q->direct_qlen) {
__skb_queue_head(&q->direct_queue, skb);
+ ret = NET_XMIT_SUCCESS;
} else {
__skb_queue_head(&q->direct_queue, skb);
tskb = __skb_dequeue_tail(&q->direct_queue);
@@ -632,17 +639,21 @@ static int htb_requeue(struct sk_buff *s
kfree_skb(skb);
return ret;
#endif
- } else if (cl->un.leaf.q->ops->requeue(skb, cl->un.leaf.q) !=
- NET_XMIT_SUCCESS) {
- sch->qstats.drops++;
- cl->qstats.drops++;
- return NET_XMIT_DROP;
- } else
- htb_activate(q, cl);
-
- sch->q.qlen++;
- sch->qstats.requeues++;
- return NET_XMIT_SUCCESS;
+ } else {
+ ret = cl->un.leaf.q->ops->requeue(skb, cl->un.leaf.q);
+ if (ret == NET_XMIT_DROP) {
+ sch->qstats.drops++;
+ cl->qstats.drops++;
+ } else if (ret == NET_XMIT_SUCCESS) {
+ htb_activate(q, cl);
+ }
+ }
+
+ if (ret == NET_XMIT_SUCCESS) {
+ sch->q.qlen++;
+ sch->qstats.requeues++;
+ }
+ return ret;
}
/**
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-14 21:22 ` Jarek Poplawski
@ 2008-07-14 21:26 ` Josip Rodin
2008-07-14 21:48 ` Jarek Poplawski
2008-07-14 22:05 ` bnx2_poll panicking kernel Jarek Poplawski
0 siblings, 2 replies; 50+ messages in thread
From: Josip Rodin @ 2008-07-14 21:26 UTC (permalink / raw)
To: Jarek Poplawski
Cc: Patrick McHardy, David Miller, mchan, billfink, bhutchings,
netdev, mirrors, devik
On Mon, Jul 14, 2008 at 11:22:23PM +0200, Jarek Poplawski wrote:
> On Mon, Jul 14, 2008 at 10:21:51PM +0200, Josip Rodin wrote:
> > On Mon, Jul 14, 2008 at 07:20:55PM +0200, Jarek Poplawski wrote:
> ...
> > > BTW, I wonder how Josip's testing?
> >
> > Do you want me to try out David's patch? I was hoping to get a definite okay
> > from someone before applying it, given that this is a production machine
> > for us and we do need it in *some* kind of an operational state.
>
> Hmm... if you trust me more than David?! As a matter of fact, I thought
> it's under testing already, but since I've some doubts, I attach below
> my version of David's patch. IMHO, it looks safe, but never knows...
No (sorry :) but he did call his patch an "idiotic attempt", and asked you
guys for input, so I held back :)
> > I've got many more tens of kilobytes of logs from the previous debugging
> > patches, if you want I can send them over.
>
> This debugging, I guess, shows corrupted data, but the reason is hard
> to find. David found one of possible reasons and it needs checking. I
> don't think this patch in any version can do more damage than doing
> nothing - unless you prefer to use Michael's patch to next kernels
> (BTW, don't remove this patch yet).
OK, I'll give it a whirl.
(Should I update to .26 at the same time, now that it went live in the
meantime?)
--
2. That which causes joy or happiness.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-14 21:26 ` Josip Rodin
@ 2008-07-14 21:48 ` Jarek Poplawski
2008-07-17 21:30 ` Josip Rodin
2008-07-14 22:05 ` bnx2_poll panicking kernel Jarek Poplawski
1 sibling, 1 reply; 50+ messages in thread
From: Jarek Poplawski @ 2008-07-14 21:48 UTC (permalink / raw)
To: Josip Rodin
Cc: Patrick McHardy, David Miller, mchan, billfink, bhutchings,
netdev, mirrors, devik
On Mon, Jul 14, 2008 at 11:26:20PM +0200, Josip Rodin wrote:
...
> No (sorry :) but he did call his patch an "idiotic attempt", and asked you
> guys for input, so I held back :)
And, IMHO, you were right. With more complex config (filters with
actions) there could be some difference, but you didn't mention such.
...
> OK, I'll give it a whirl.
>
> (Should I update to .26 at the same time, now that it went live in the
> meantime?)
My proposal is to try .25 first.
Jarek P.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-14 21:26 ` Josip Rodin
2008-07-14 21:48 ` Jarek Poplawski
@ 2008-07-14 22:05 ` Jarek Poplawski
1 sibling, 0 replies; 50+ messages in thread
From: Jarek Poplawski @ 2008-07-14 22:05 UTC (permalink / raw)
To: Josip Rodin
Cc: Patrick McHardy, mchan, billfink, bhutchings, netdev, mirrors,
devik
On Mon, Jul 14, 2008 at 11:26:20PM +0200, Josip Rodin wrote:
...
> No (sorry :) but he did call his patch an "idiotic attempt", and asked you
> guys for input, so I held back :)
FYI: David's "idiotic attempt" usually equals 2x average "brigtest idea"
here. (It's not a :)
Jarek P.
PS: but don't repeat this (I removed him from CC)
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-14 21:48 ` Jarek Poplawski
@ 2008-07-17 21:30 ` Josip Rodin
2008-07-17 21:44 ` David Miller
0 siblings, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-07-17 21:30 UTC (permalink / raw)
To: Jarek Poplawski
Cc: Patrick McHardy, David Miller, mchan, billfink, bhutchings,
netdev, mirrors, devik
On Mon, Jul 14, 2008 at 11:48:51PM +0200, Jarek Poplawski wrote:
> > OK, I'll give it a whirl.
> >
> > (Should I update to .26 at the same time, now that it went live in the
> > meantime?)
>
> My proposal is to try .25 first.
It looks like the patch fixed my issues - no relevant printk's appeared
since I've applied it, so apparently there is no more cloned skb splitting
or corrupted data getting to bnx2.
I used the latest variant, from Jarek's mail
<20080714212223.GA4849@ami.dom.local>, on the 2.6.25.6 kernel.
Thanks all!
(Now if only someone could make sure this propagates into the stable kernel
branches, since those were pretty nasty crashes...)
--
2. That which causes joy or happiness.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-17 21:30 ` Josip Rodin
@ 2008-07-17 21:44 ` David Miller
2008-07-18 5:12 ` Jarek Poplawski
2008-08-02 12:28 ` bad htb_{en,re}queue return codes causing corrupt data in drivers [was Re: bnx2_poll panicking kernel] Josip Rodin
0 siblings, 2 replies; 50+ messages in thread
From: David Miller @ 2008-07-17 21:44 UTC (permalink / raw)
To: joy; +Cc: jarkao2, kaber, mchan, billfink, bhutchings, netdev, mirrors,
devik
From: Josip Rodin <joy@entuzijast.net>
Date: Thu, 17 Jul 2008 23:30:58 +0200
> It looks like the patch fixed my issues - no relevant printk's appeared
> since I've applied it, so apparently there is no more cloned skb splitting
> or corrupted data getting to bnx2.
>
> I used the latest variant, from Jarek's mail
> <20080714212223.GA4849@ami.dom.local>, on the 2.6.25.6 kernel.
>
> Thanks all!
>
> (Now if only someone could make sure this propagates into the stable kernel
> branches, since those were pretty nasty crashes...)
We will make sure the fix propagates in some form, thanks for testing.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bnx2_poll panicking kernel
2008-07-17 21:44 ` David Miller
@ 2008-07-18 5:12 ` Jarek Poplawski
2008-08-02 12:28 ` bad htb_{en,re}queue return codes causing corrupt data in drivers [was Re: bnx2_poll panicking kernel] Josip Rodin
1 sibling, 0 replies; 50+ messages in thread
From: Jarek Poplawski @ 2008-07-18 5:12 UTC (permalink / raw)
To: David Miller
Cc: joy, kaber, mchan, billfink, bhutchings, netdev, mirrors, devik
On Thu, Jul 17, 2008 at 02:44:07PM -0700, David Miller wrote:
> From: Josip Rodin <joy@entuzijast.net>
> Date: Thu, 17 Jul 2008 23:30:58 +0200
>
> > It looks like the patch fixed my issues - no relevant printk's appeared
> > since I've applied it, so apparently there is no more cloned skb splitting
> > or corrupted data getting to bnx2.
> >
> > I used the latest variant, from Jarek's mail
> > <20080714212223.GA4849@ami.dom.local>, on the 2.6.25.6 kernel.
> >
> > Thanks all!
> >
> > (Now if only someone could make sure this propagates into the stable kernel
> > branches, since those were pretty nasty crashes...)
>
> We will make sure the fix propagates in some form, thanks for testing.
What an ... attempt!
Congratulations to everybody here,
Jarek P.
^ permalink raw reply [flat|nested] 50+ messages in thread
* bad htb_{en,re}queue return codes causing corrupt data in drivers [was Re: bnx2_poll panicking kernel]
2008-07-17 21:44 ` David Miller
2008-07-18 5:12 ` Jarek Poplawski
@ 2008-08-02 12:28 ` Josip Rodin
2008-08-03 7:06 ` bad htb_{en,re}queue return codes causing corrupt data in drivers David Miller
1 sibling, 1 reply; 50+ messages in thread
From: Josip Rodin @ 2008-08-02 12:28 UTC (permalink / raw)
To: netdev
On Thu, Jul 17, 2008 at 02:44:07PM -0700, David Miller wrote:
> From: Josip Rodin <joy@entuzijast.net>
> Date: Thu, 17 Jul 2008 23:30:58 +0200
>
> > It looks like the patch fixed my issues - no relevant printk's appeared
> > since I've applied it, so apparently there is no more cloned skb splitting
> > or corrupted data getting to bnx2.
> >
> > I used the latest variant, from Jarek's mail
> > <20080714212223.GA4849@ami.dom.local>, on the 2.6.25.6 kernel.
> >
> > Thanks all!
> >
> > (Now if only someone could make sure this propagates into the stable kernel
> > branches, since those were pretty nasty crashes...)
>
> We will make sure the fix propagates in some form, thanks for testing.
Hi again, sorry to bother you, but it's been a few weeks now and I can't
seem see that fix in any of stable/linux-2.6.2[56].y.git, davem/net-2.6.git,
sfr/linux-next.git, so I just wanted to check that it hasn't fallen through
the cracks :)
--
2. That which causes joy or happiness.
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: bad htb_{en,re}queue return codes causing corrupt data in drivers
2008-08-02 12:28 ` bad htb_{en,re}queue return codes causing corrupt data in drivers [was Re: bnx2_poll panicking kernel] Josip Rodin
@ 2008-08-03 7:06 ` David Miller
0 siblings, 0 replies; 50+ messages in thread
From: David Miller @ 2008-08-03 7:06 UTC (permalink / raw)
To: joy; +Cc: netdev
From: Josip Rodin <joy@entuzijast.net>
Date: Sat, 2 Aug 2008 14:28:22 +0200
> On Thu, Jul 17, 2008 at 02:44:07PM -0700, David Miller wrote:
> > From: Josip Rodin <joy@entuzijast.net>
> > Date: Thu, 17 Jul 2008 23:30:58 +0200
> >
> > > It looks like the patch fixed my issues - no relevant printk's appeared
> > > since I've applied it, so apparently there is no more cloned skb splitting
> > > or corrupted data getting to bnx2.
> > >
> > > I used the latest variant, from Jarek's mail
> > > <20080714212223.GA4849@ami.dom.local>, on the 2.6.25.6 kernel.
> > >
> > > Thanks all!
> > >
> > > (Now if only someone could make sure this propagates into the stable kernel
> > > branches, since those were pretty nasty crashes...)
> >
> > We will make sure the fix propagates in some form, thanks for testing.
>
> Hi again, sorry to bother you, but it's been a few weeks now and I can't
> seem see that fix in any of stable/linux-2.6.2[56].y.git, davem/net-2.6.git,
> sfr/linux-next.git, so I just wanted to check that it hasn't fallen through
> the cracks :)
It hasn't, we're doing an audit of the entire packet scheduler layer
to make sure we fix this bug everywhere it occurs.
Thanks for pinging us.
^ permalink raw reply [flat|nested] 50+ messages in thread
end of thread, other threads:[~2008-08-03 7:06 UTC | newest]
Thread overview: 50+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-06-16 12:57 bnx2_poll panicking kernel Josip Rodin
2008-06-16 14:04 ` Ben Hutchings
2008-06-16 15:52 ` Michael Chan
2008-06-16 19:13 ` Josip Rodin
2008-06-16 21:38 ` Josip Rodin
2008-06-16 21:48 ` Josip Rodin
2008-06-16 23:45 ` Michael Chan
2008-06-17 22:37 ` Josip Rodin
2008-06-17 22:47 ` Michael Chan
2008-06-21 11:18 ` Josip Rodin
2008-06-21 15:34 ` Bill Fink
2008-06-21 16:11 ` Michael Chan
2008-06-23 18:04 ` Josip Rodin
2008-06-23 21:36 ` Josip Rodin
2008-06-23 22:48 ` Michael Chan
2008-06-24 22:58 ` Michael Chan
2008-06-25 0:04 ` David Miller
2008-06-26 11:01 ` Josip Rodin
2008-06-26 18:04 ` Michael Chan
2008-07-09 16:46 ` Josip Rodin
2008-07-09 16:57 ` Michael Chan
2008-07-09 23:46 ` David Miller
2008-07-10 9:45 ` Aviv Greenberg
2008-07-10 10:09 ` David Miller
2008-07-10 21:00 ` Michael Chan
2008-07-10 21:00 ` David Miller
2008-07-10 21:23 ` Josip Rodin
2008-07-10 21:38 ` Michael Chan
2008-07-10 22:00 ` Josip Rodin
2008-07-10 22:26 ` Michael Chan
2008-07-10 22:31 ` Josip Rodin
2008-07-10 23:20 ` David Miller
2008-07-11 9:24 ` Josip Rodin
2008-07-11 9:56 ` David Miller
2008-07-11 12:19 ` Patrick McHardy
2008-07-12 9:49 ` Jarek Poplawski
2008-07-12 13:21 ` Jarek Poplawski
2008-07-14 15:27 ` Patrick McHardy
2008-07-14 17:20 ` Jarek Poplawski
2008-07-14 17:25 ` Jarek Poplawski
2008-07-14 20:21 ` Josip Rodin
2008-07-14 21:22 ` Jarek Poplawski
2008-07-14 21:26 ` Josip Rodin
2008-07-14 21:48 ` Jarek Poplawski
2008-07-17 21:30 ` Josip Rodin
2008-07-17 21:44 ` David Miller
2008-07-18 5:12 ` Jarek Poplawski
2008-08-02 12:28 ` bad htb_{en,re}queue return codes causing corrupt data in drivers [was Re: bnx2_poll panicking kernel] Josip Rodin
2008-08-03 7:06 ` bad htb_{en,re}queue return codes causing corrupt data in drivers David Miller
2008-07-14 22:05 ` bnx2_poll panicking kernel Jarek Poplawski
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).