netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Re: Fw: [Bug 82471] New: net/core/dev.c skb_war_bad_offload
@ 2014-12-17  8:44 Richard Laager
  2014-12-17  9:55 ` Michal Kubecek
  2014-12-31  6:51 ` Richard Laager
  0 siblings, 2 replies; 7+ messages in thread
From: Richard Laager @ 2014-12-17  8:44 UTC (permalink / raw)
  To: netdev

[-- Attachment #1: Type: text/plain, Size: 24114 bytes --]

Previous history of this thread:
http://thread.gmane.org/gmane.linux.network/326672

On 2014-11-04 22:57:19, Tom Herbert wrote:
> Using vlan and bonding? vlan_dev_hard_start_xmit called. A possible
> cause is that bonding interface is out of sync with slave interface
> w.r.t. GSO features. Do we know if this worked in 3.14, 3.15?

I'm seeing the same sort of crash/warning (skb_war_bad_offload). It's
happening on Intel 10 Gig NICs using the ixgbe driver. I'm using bridges
(for virtual machines) on top of VLANs on top of 802.3ad bonding. I'm
using an MTU of 9000 on the bond0 interface, but 1500 everywhere else.

I'm always bonding two ports: one one system, I'm bonding two ports on
identical one-port NICs; on another system, I'm bonding two ports on a
single two-port NIC. Both systems exhibit the same behavior.

Everything has worked fine for a couple years on Ubuntu 12.04 Precise
(Linux 3.2.0). It immediately broke when I upgraded to Ubuntu 14.04
Trusty (Linux 3.13.0). I can also reproduce this using the packaged
version of Linux 3.16.0 on Trusty.

In contrast to other reports of this bug, disabling scatter gather on
the physical interfaces (e.g. eth0) does *not* stop the crashes
(assuming I disabled it correctly).

I currently have two systems (one with Precise, one with Trusty)
available to do any testing that you'd find helpful.

Here's a first pass at getting some debugging data.

The broken system (Ubuntu 14.04 Trusty):

rlaager@BROKEN:~$ uname -a
Linux BROKEN 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux

rlaager@BROKEN:~$ ethtool -k p6p1
Features for p6p1:
rx-checksumming: on
tx-checksumming: on
	tx-checksum-ipv4: on
	tx-checksum-ip-generic: off [fixed]
	tx-checksum-ipv6: on
	tx-checksum-fcoe-crc: on [fixed]
	tx-checksum-sctp: on
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off

rlaager@BROKEN:~$ ethtool -k bond0
Features for bond0:
rx-checksumming: off [fixed]
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: on
	tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: on
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off [requested on]
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]

rlaager@BROKEN:~$ ethtool -k br7
Features for br7:
rx-checksumming: off [fixed]
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: on
	tx-tcp6-segmentation: on
udp-fragmentation-offload: off [requested on]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [requested on]
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: on
tx-ipip-segmentation: on
tx-sit-segmentation: on
tx-udp_tnl-segmentation: on
tx-mpls-segmentation: on
fcoe-mtu: off [fixed]
tx-nocache-copy: off [requested on]
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]

rlaager@BROKEN:~$ lspci
00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22)
00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 22)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22)
00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 22)
00:0d.0 Host bridge: Intel Corporation Device 343a (rev 22)
00:0d.1 Host bridge: Intel Corporation Device 343b (rev 22)
00:0d.2 Host bridge: Intel Corporation Device 343c (rev 22)
00:0d.3 Host bridge: Intel Corporation Device 343d (rev 22)
00:0d.4 Host bridge: Intel Corporation 7500/5520/5500/X58 Physical Layer Port 0 (rev 22)
00:0d.5 Host bridge: Intel Corporation 7500/5520/5500 Physical Layer Port 1 (rev 22)
00:0d.6 Host bridge: Intel Corporation Device 341a (rev 22)
00:0e.0 Host bridge: Intel Corporation Device 341c (rev 22)
00:0e.1 Host bridge: Intel Corporation Device 341d (rev 22)
00:0e.2 Host bridge: Intel Corporation Device 341e (rev 22)
00:0e.4 Host bridge: Intel Corporation Device 3439 (rev 22)
00:13.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 22)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 22)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22)
00:14.3 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
01:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)
05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
fe:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)
fe:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)
fe:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)
fe:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)
fe:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)
fe:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)
fe:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)
fe:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)
fe:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)
fe:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)
fe:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)
fe:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)
fe:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)
fe:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)
fe:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)
fe:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)
fe:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)
fe:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)
fe:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)
fe:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)
fe:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)
fe:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)
fe:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)
fe:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02)
ff:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)
ff:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)
ff:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)
ff:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)
ff:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)
ff:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)
ff:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)
ff:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)
ff:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)
ff:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)
ff:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)
ff:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)
ff:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)
ff:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)
ff:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)
ff:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)
ff:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)
ff:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)
ff:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)
ff:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)
ff:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)
ff:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)
ff:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)
ff:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02)


The working system (Ubuntu 12.04 Precise):

rlaager@WORKING:~$ uname -a
Linux WORKING 3.2.0-74-generic #109-Ubuntu SMP Tue Dec 9 16:45:49 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux

rlaager@WORKING:~$ ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on

rlaager@WORKING:~$ ethtool -k bond0
Offload parameters for bond0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

rlaager@WORKING:~$ ethtool -k br7
Offload parameters for br7:
rx-checksumming: on
tx-checksumming: on
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: on
ntuple-filters: off




A stack trace from 3.13.0 (the default kernel in Ubuntu Trusty):

[ 1161.275007] WARNING: CPU: 7 PID: 0 at /build/buildd/linux-3.13.0/net/core/dev.c:2224 skb_warn_bad_offload+0xcd/0xda()
[ 1161.275011] : caps=(0x00000022000048c1, 0x0000000000000000) len=1514 data_len=1460 gso_size=1460 gso_type=1 ip_summed=1
[ 1161.275012] Modules linked in: nfsv3 ipmi_devintf ipmi_si vhost_net vhost macvtap macvlan bridge ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_comment xt_mul
 mrp xt_addrtype llc bonding nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_
ch intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd serio_raw joydev i7core_eda
id nfs_acl lp parport nfs lockd sunrpc fscache ses enclosure raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor ixgbe raid6_pq dca hid_generic raid1 ptp mpt2sas
smouse hid libahci scsi_transport_sas mdio linear
[ 1161.275077] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G        W     3.13.0-43-generic #72-Ubuntu
[ 1161.275079] Hardware name: Supermicro X8DT6/X8DT6, BIOS 2.0a    09/14/2010
[ 1161.275080]  0000000000000009 ffff880c3fc239d8 ffffffff81720bf6 ffff880c3fc23a20
[ 1161.275085]  ffff880c3fc23a10 ffffffff810677cd ffff880c1d3b9600 ffff880618e08000
[ 1161.275089]  0000000000000001 0000000000000001 ffff880c1d3b9600 ffff880c3fc23a70
[ 1161.275092] Call Trace:
[ 1161.275094]  <IRQ>  [<ffffffff81720bf6>] dump_stack+0x45/0x56
[ 1161.275101]  [<ffffffff810677cd>] warn_slowpath_common+0x7d/0xa0
[ 1161.275105]  [<ffffffff8106783c>] warn_slowpath_fmt+0x4c/0x50
[ 1161.275109]  [<ffffffff8136a0a3>] ? ___ratelimit+0x93/0x100
[ 1161.275113]  [<ffffffff81723afe>] skb_warn_bad_offload+0xcd/0xda
[ 1161.275118]  [<ffffffff81626489>] __skb_gso_segment+0x79/0xb0
[ 1161.275122]  [<ffffffff8162677a>] dev_hard_start_xmit+0x18a/0x560
[ 1161.275126]  [<ffffffff81098209>] ? ttwu_do_wakeup+0x19/0xc0
[ 1161.275129]  [<ffffffff8164594e>] sch_direct_xmit+0xee/0x1c0
[ 1161.275133]  [<ffffffff81626d80>] __dev_queue_xmit+0x230/0x500
[ 1161.275137]  [<ffffffff81627060>] dev_queue_xmit+0x10/0x20
[ 1161.275143]  [<ffffffffa04ab31b>] br_dev_queue_push_xmit+0x7b/0xc0 [bridge]
[ 1161.275149]  [<ffffffffa04ab532>] br_forward_finish+0x22/0x60 [bridge]
[ 1161.275155]  [<ffffffffa04ab710>] __br_forward+0x80/0xf0 [bridge]
[ 1161.275161]  [<ffffffffa04ab9bb>] br_forward+0x8b/0xa0 [bridge]
[ 1161.275167]  [<ffffffffa04ac6d9>] br_handle_frame_finish+0x149/0x3d0 [bridge]
[ 1161.275173]  [<ffffffffa04acad5>] br_handle_frame+0x175/0x250 [bridge]
[ 1161.275177]  [<ffffffff81624ac2>] __netif_receive_skb_core+0x262/0x840
[ 1161.275181]  [<ffffffff8101b700>] ? check_tsc_unstable+0x10/0x10
[ 1161.275184]  [<ffffffff816250b8>] __netif_receive_skb+0x18/0x60
[ 1161.275188]  [<ffffffff81625123>] netif_receive_skb+0x23/0x90
[ 1161.275192]  [<ffffffff81625b70>] napi_gro_receive+0x80/0xb0
[ 1161.275202]  [<ffffffffa014009c>] ixgbe_clean_rx_irq+0x7ac/0xb10 [ixgbe]
[ 1161.275211]  [<ffffffffa0141140>] ixgbe_poll+0x460/0x800 [ixgbe]
[ 1161.275216]  [<ffffffff816254a2>] net_rx_action+0x152/0x250
[ 1161.275220]  [<ffffffff8106cc1c>] __do_softirq+0xec/0x2c0
[ 1161.275223]  [<ffffffff8106d165>] irq_exit+0x105/0x110
[ 1161.275227]  [<ffffffff817339e6>] do_IRQ+0x56/0xc0
[ 1161.275231]  [<ffffffff817290ed>] common_interrupt+0x6d/0x6d
[ 1161.275232]  <EOI>  [<ffffffff815d361f>] ? cpuidle_enter_state+0x4f/0xc0
[ 1161.275240]  [<ffffffff815d3749>] cpuidle_idle_call+0xb9/0x1f0
[ 1161.275244]  [<ffffffff8101d35e>] arch_cpu_idle+0xe/0x30
[ 1161.275247]  [<ffffffff810bef35>] cpu_startup_entry+0xc5/0x290
[ 1161.275251]  [<ffffffff810413ed>] start_secondary+0x21d/0x2d0


A stack trace from 3.16.0 (still on Ubuntu Trusty):

[  120.376026] WARNING: CPU: 6 PID: 0 at /build/buildd/linux-lts-utopic-3.16.0/net/core/dev.c:2246 skb_warn_bad_offload+0xcd/0xda()
[  120.376029] : caps=(0x00000080000048c1, 0x0000000000000000) len=1514 data_len=1460 gso_size=1460 gso_type=1 ip_summed=1
[  120.376030] Modules linked in: nfsv3 ipmi_devintf ipmi_si ipmi_msghandler vhost_net vhost macvtap macvlan bridge 8021q garp stp mrp llc bonding ip6t_REJECT xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT xt_comment xt_multiport xt_recent xt_limit xt_tcpudp xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack iptable_filter ip_tables x_tables intel_powerclamp coretemp kvm_intel gpio_ich kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd serio_raw lpc_ich joydev i7core_edac ioatdma edac_core nfsd auth_rpcgss mac_hid nfs_acl lp parport nfs lockd sunrpc fscache ses enclosure raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic raid6_pq ixgbe usbhid raid1 mpt2sas dca ahci raid0 ptp raid_class pps_core scsi_transport_sas multipath hid mdio libahci linear
[  120.376085] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 3.16.0-28-generic #37-Ubuntu
[  120.376086] Hardware name: Supermicro X8DT6/X8DT6, BIOS 2.0a    09/14/2010
[  120.376088]  0000000000000009 ffff880c3fc039b8 ffffffff81762220 ffff880c3fc03a00
[  120.376090]  ffff880c3fc039f0 ffffffff8106dd2d ffff880c1ac99a00 ffff88061c2fc000
[  120.376092]  0000000000000001 0000000000000001 ffff880c1ac99a00 ffff880c3fc03a50
[  120.376094] Call Trace:
[  120.376096]  <IRQ>  [<ffffffff81762220>] dump_stack+0x45/0x56
[  120.376105]  [<ffffffff8106dd2d>] warn_slowpath_common+0x7d/0xa0
[  120.376107]  [<ffffffff8106dd9c>] warn_slowpath_fmt+0x4c/0x50
[  120.376111]  [<ffffffff8138b153>] ? ___ratelimit+0x93/0x100
[  120.376114]  [<ffffffff817654da>] skb_warn_bad_offload+0xcd/0xda
[  120.376119]  [<ffffffff81661d29>] __skb_gso_segment+0x79/0xb0
[  120.376122]  [<ffffffff81662052>] dev_hard_start_xmit+0x182/0x5c0
[  120.376125]  [<ffffffff8168337e>] sch_direct_xmit+0xee/0x1c0
[  120.376127]  [<ffffffff81662690>] __dev_queue_xmit+0x200/0x4d0
[  120.376129]  [<ffffffff81662970>] dev_queue_xmit+0x10/0x20
[  120.376135]  [<ffffffffc0796ac8>] br_dev_queue_push_xmit+0x68/0xa0 [bridge]
[  120.376138]  [<ffffffffc0796cd2>] br_forward_finish+0x22/0x60 [bridge]
[  120.376142]  [<ffffffffc0796e90>] __br_forward+0x80/0xf0 [bridge]
[  120.376145]  [<ffffffffc079713b>] br_forward+0x8b/0xa0 [bridge]
[  120.376149]  [<ffffffffc0797fb9>] br_handle_frame_finish+0x139/0x3c0 [bridge]
[  120.376153]  [<ffffffffc079838e>] br_handle_frame+0x14e/0x240 [bridge]
[  120.376155]  [<ffffffff81660102>] __netif_receive_skb_core+0x1b2/0x790
[  120.376158]  [<ffffffff8101bcd9>] ? read_tsc+0x9/0x20
[  120.376161]  [<ffffffff816606f8>] __netif_receive_skb+0x18/0x60
[  120.376163]  [<ffffffff81660763>] netif_receive_skb_internal+0x23/0x90
[  120.376165]  [<ffffffff816612c0>] napi_gro_receive+0xc0/0xf0
[  120.376174]  [<ffffffffc03007ac>] ixgbe_clean_rx_irq+0x7bc/0xb40 [ixgbe]
[  120.376180]  [<ffffffffc03018a2>] ixgbe_poll+0x482/0x850 [ixgbe]
[  120.376183]  [<ffffffff8109e9e9>] ? ttwu_do_wakeup+0x19/0xc0
[  120.376186]  [<ffffffff81660b52>] net_rx_action+0x152/0x250
[  120.376189]  [<ffffffff81073055>] __do_softirq+0xf5/0x2e0
[  120.376191]  [<ffffffff81073515>] irq_exit+0x105/0x110
[  120.376194]  [<ffffffff8176d748>] do_IRQ+0x58/0xf0
[  120.376198]  [<ffffffff8176b5ed>] common_interrupt+0x6d/0x6d
[  120.376199]  <EOI>  [<ffffffff815fb83f>] ? cpuidle_enter_state+0x4f/0xc0
[  120.376204]  [<ffffffff815fb838>] ? cpuidle_enter_state+0x48/0xc0
[  120.376206]  [<ffffffff815fb967>] cpuidle_enter+0x17/0x20
[  120.376209]  [<ffffffff810b527d>] cpu_startup_entry+0x31d/0x450
[  120.376213]  [<ffffffff810e028d>] ? tick_check_new_device+0xdd/0xf0
[  120.376216]  [<ffffffff8104520d>] start_secondary+0x21d/0x2e0
[  120.376217] ---[ end trace 90d53a2c9c47f360 ]---

-- 
Richard

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread
* Fw: [Bug 82471] New: net/core/dev.c skb_war_bad_offload
@ 2014-08-15 21:26 Stephen Hemminger
  2014-11-04 21:59 ` Jesse Brandeburg
  0 siblings, 1 reply; 7+ messages in thread
From: Stephen Hemminger @ 2014-08-15 21:26 UTC (permalink / raw)
  To: netdev



Begin forwarded message:

Date: Fri, 15 Aug 2014 13:21:00 -0700
From: "bugzilla-daemon@bugzilla.kernel.org" <bugzilla-daemon@bugzilla.kernel.org>
To: "stephen@networkplumber.org" <stephen@networkplumber.org>
Subject: [Bug 82471] New: net/core/dev.c skb_war_bad_offload


https://bugzilla.kernel.org/show_bug.cgi?id=82471

            Bug ID: 82471
           Summary: net/core/dev.c skb_war_bad_offload
           Product: Networking
           Version: 2.5
    Kernel Version: 3.16.1
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: IPV4
          Assignee: shemminger@linux-foundation.org
          Reporter: vladi@aresgate.net
        Regression: No

Created attachment 146761
  --> https://bugzilla.kernel.org/attachment.cgi?id=146761&action=edit
kernel config

Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973225] ------------[
cut here ]------------
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973236] WARNING: CPU:
2 PID: 0 at net/core/dev.c:2246 skb_warn_bad_offload+0xc8/0xd5()
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973238] :
caps=(0x000000000419fba9, 0x00000000001b583b) len=2962 data_len=2896
gso_size=1448 gso_type=1 ip_summed=3
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973239] Modules
linked in: ntfs msdos xfs libcrc32c ipmi_devintf intel_rapl
x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper
cryptd sb_edac edac_core 8021q garp ioatdma stp ipmi_si mrp llc bonding
hid_generic ixgbe usbhid hid ahci dca libahci mdio
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973257] CPU: 2 PID: 0
Comm: swapper/2 Tainted: G        W     3.16.0 #2
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973259] Hardware
name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973260]
0000000000000009 ffff88046fd036b0 ffffffff815c4096 ffff88046fd036f8
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973262]
ffff88046fd036e8 ffffffff8103f633 ffff880018b7c4e0 ffff8804687df000
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973264]
0000000000000001 0000000000000003 ffffffffa0193320 ffff88046fd03748
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973266] Call Trace:
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973268]  <IRQ>
[<ffffffff815c4096>] dump_stack+0x45/0x56
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973280]
[<ffffffff8103f633>] warn_slowpath_common+0x73/0x90
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973286]
[<ffffffff8103f697>] warn_slowpath_fmt+0x47/0x50
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973288]
[<ffffffff812edadc>] ? ___ratelimit+0x7c/0xf0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973291]
[<ffffffff815c5e19>] skb_warn_bad_offload+0xc8/0xd5
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973294]
[<ffffffff814e57fe>] skb_checksum_help+0x16e/0x180
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973297]
[<ffffffff814e9ecc>] dev_hard_start_xmit+0x42c/0x4b0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973299]
[<ffffffff814ea154>] ? __dev_queue_xmit+0x204/0x440
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973301]
[<ffffffff814ea232>] __dev_queue_xmit+0x2e2/0x440
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973302]
[<ffffffff814ea39b>] ? dev_queue_xmit+0xb/0x10
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973304]
[<ffffffff814ea39b>] dev_queue_xmit+0xb/0x10
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973308]
[<ffffffffa012b758>] vlan_dev_hard_start_xmit+0x88/0x100 [8021q]
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973317]
[<ffffffff814e9d9a>] dev_hard_start_xmit+0x2fa/0x4b0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973321]
[<ffffffff814ea232>] __dev_queue_xmit+0x2e2/0x440
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973323]
[<ffffffff814ea39b>] dev_queue_xmit+0xb/0x10
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973325]
[<ffffffff814f1192>] neigh_connected_output+0xb2/0xf0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973327]
[<ffffffff815192dc>] ip_finish_output+0x4ec/0x890
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973329]
[<ffffffff8151ac03>] ip_output+0x53/0x90
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973331]
[<ffffffff8151a39b>] ip_local_out_sk+0x2b/0x30
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973333]
[<ffffffff8151a6fa>] ip_queue_xmit+0x13a/0x3c0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973335]
[<ffffffff815309fa>] tcp_transmit_skb+0x42a/0x8f0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973337]
[<ffffffff81530ffa>] tcp_write_xmit+0x13a/0xc00
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973347]
[<ffffffff8152f033>] ? tcp_established_options+0x33/0xd0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973350]
[<ffffffff81531d09>] __tcp_push_pending_frames+0x29/0xc0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973353]
[<ffffffff8152da77>] tcp_rcv_established+0x1f7/0x5e0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973356]
[<ffffffff81535fc5>] tcp_v4_do_rcv+0x215/0x4a0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973369]
[<ffffffff810651f8>] ? ttwu_do_activate.constprop.64+0x58/0x60
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973374]
[<ffffffff81295d31>] ? security_sock_rcv_skb+0x11/0x20
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973377]
[<ffffffff815392ad>] tcp_v4_rcv+0x73d/0x7c0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973380]
[<ffffffff8106fafc>] ? update_group_capacity+0x16c/0x270
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973386]
[<ffffffff812e8300>] ? cpumask_next_and+0x30/0x50
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973388]
[<ffffffff81514b50>] ip_local_deliver_finish+0x80/0x1c0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973390]
[<ffffffff81515154>] ip_local_deliver+0x34/0x90
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973392]
[<ffffffff81514d99>] ip_rcv_finish+0x109/0x350
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973399]
[<ffffffff815153d2>] ip_rcv+0x222/0x370
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973403]
[<ffffffff814e5eb6>] __netif_receive_skb_core+0x416/0x570
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973407]
[<ffffffff814e73f3>] __netif_receive_skb+0x13/0x60
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973410]
[<ffffffff814e745e>] netif_receive_skb_internal+0x1e/0x90
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973415]
[<ffffffff814e7b40>] napi_gro_receive+0x70/0xa0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973422]
[<ffffffffa0134f4c>] ixgbe_clean_rx_irq+0x75c/0xb20 [ixgbe]
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973427]
[<ffffffffa0136172>] ixgbe_poll+0x522/0x850 [ixgbe]
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973430]
[<ffffffff8105e986>] ? hrtimer_get_next_event+0xb6/0xc0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973437]
[<ffffffff814e8dc1>] net_rx_action+0x101/0x1a0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973443]
[<ffffffff810430ab>] __do_softirq+0xdb/0x240
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973447]
[<ffffffff8104349e>] irq_exit+0xee/0x110
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973450]
[<ffffffff81004913>] do_IRQ+0x53/0xf0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973453]
[<ffffffff815caaaa>] common_interrupt+0x6a/0x6a
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973454]  <EOI>
[<ffffffff814af007>] ? cpuidle_enter_state+0x47/0xc0
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973463]
[<ffffffff814af132>] cpuidle_enter+0x12/0x20
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973468]
[<ffffffff81075fbf>] cpu_startup_entry+0x24f/0x280
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973477]
[<ffffffff810905e3>] ? clockevents_config_and_register+0x23/0x30
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973482]
[<ffffffff810282be>] start_secondary+0x1be/0x270
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.973486] ---[ end
trace de552357488766e8 ]---
Aug  6 06:46:50 prod-ent-ceph03.dc2.ec.loc kernel: [29530.974181] ------------[
cut here ]------------

--
You are receiving this mail because:
You are the assignee for the bug.

^ permalink raw reply	[flat|nested] 7+ messages in thread
end of thread, other threads:[~2014-12-31  6:51 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-12-17  8:44 Fw: [Bug 82471] New: net/core/dev.c skb_war_bad_offload Richard Laager
2014-12-17  9:55 ` Michal Kubecek
2014-12-17 14:52   ` Richard Laager
2014-12-31  6:51 ` Richard Laager
  -- strict thread matches above, loose matches on Subject: below --
2014-08-15 21:26 Stephen Hemminger
2014-11-04 21:59 ` Jesse Brandeburg
2014-11-04 22:57   ` Tom Herbert

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).