* kernel BUG at net/core/skbuff.c:2839 RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 @ 2013-11-17 19:17 Sander Eikelenboom 2013-11-17 19:59 ` Cong Wang 2013-11-21 12:00 ` Sander Eikelenboom 0 siblings, 2 replies; 11+ messages in thread From: Sander Eikelenboom @ 2013-11-17 19:17 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev Hi Eric, With the linux-net changes from this merge window i get the kernel panic below (not with 3.12.0). It's on a machine running Xen, 2x rtl8169 nic, and using a bridge for guest networking. This panic in the host kernel only seems to occur when generating a lot of network traffic to and from a guest. I tried reverting "tcp: gso: fix truesize tracking" 0d08c42cf9a71530fef5ebcfe368f38f2dd0476f, but that didn't help. -- Sander [ 1164.511712] ------------[ cut here ]------------ [ 1164.518446] kernel BUG at net/core/skbuff.c:2839! [ 1164.525226] invalid opcode: 0000 [#2] PREEMPT SMP [ 1164.532024] Modules linked in: [ 1164.538713] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G D 3.12.0-mw-20131117+ #1 [ 1164.545649] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640) , BIOS V1.8B1 09/13/2010 [ 1164.552659] task: ffff880059b8a180 ti: ffff880059b96000 task.ti: ffff880059b96000 [ 1164.559649] RIP: e030:[<ffffffff819109a2>] [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 [ 1164.566860] RSP: e02b:ffff880059b97448 EFLAGS: 00010216 [ 1164.574023] RAX: 0000000000000011 RBX: 0000000000006612 RCX: 0000000000006612 [ 1164.581169] RDX: 00000000000005a8 RSI: 0000000000006612 RDI: ffff8800478ff682 [ 1164.588115] RBP: ffff880059b97518 R08: ffff88004ca06a00 R09: 0000000000000011 [ 1164.595169] R10: 000000000000606a R11: 0000000000000011 R12: 0000000000000000 [ 1164.602214] R13: ffff88004ca06900 R14: ffff88004ca06a00 R15: ffff88004bb57f00 [ 1164.609274] FS: 00007eff7dc67700(0000) GS:ffff88005f600000(0000) knlGS:0000000000000000 [ 1164.616394] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1164.623562] CR2: 00007faa868fff30 CR3: 000000005877e000 CR4: 0000000000000660 [ 1164.630807] Stack: [ 1164.637959] ffff880059b97458 ffffffff810d90dd ffff880059b97478 ffffffff8109f69a [ 1164.645296] ffff88004b824628 ffff88004b824400 ffff880059b974e8 ffff88004ca06a00 [ 1164.652626] 0000001100000001 0000000000000040 0000000000000042 ffffffffffffffbe [ 1164.659816] Call Trace: [ 1164.667034] [<ffffffff810d90dd>] ? trace_hardirqs_on+0xd/0x10 [ 1164.674313] [<ffffffff8109f69a>] ? local_bh_enable+0xaa/0x110 [ 1164.681543] [<ffffffff819dafc2>] tcp_gso_segment+0x102/0x3e0 [ 1164.688691] [<ffffffff819b8224>] ? ip_queue_xmit+0x194/0x480 [ 1164.695741] [<ffffffff819e9fe4>] inet_gso_segment+0x124/0x350 [ 1164.702836] [<ffffffff8191cb25>] skb_mac_gso_segment+0xd5/0x1d0 [ 1164.709735] [<ffffffff8191ca92>] ? skb_mac_gso_segment+0x42/0x1d0 [ 1164.716739] [<ffffffff8191cc7b>] __skb_gso_segment+0x5b/0xc0 [ 1164.723802] [<ffffffff8191ce80>] dev_hard_start_xmit+0x1a0/0x500 [ 1164.730744] [<ffffffff81939780>] sch_direct_xmit+0x100/0x280 [ 1164.737531] [<ffffffff8191d404>] dev_queue_xmit+0x224/0x600 [ 1164.744403] [<ffffffff8191d1e0>] ? dev_hard_start_xmit+0x500/0x500 [ 1164.751317] [<ffffffff8194765e>] ? nf_hook_slow+0x11e/0x160 [ 1164.758332] [<ffffffff81a14040>] ? deliver_clone+0x60/0x60 [ 1164.765264] [<ffffffff81a140d7>] br_dev_queue_push_xmit+0x97/0x140 [ 1164.772082] [<ffffffff81a1419d>] br_forward_finish+0x1d/0x60 [ 1164.778925] [<ffffffff81a12490>] ? br_dev_free+0x30/0x30 [ 1164.785714] [<ffffffff81a142f2>] __br_deliver+0x52/0x180 [ 1164.792355] [<ffffffff81a146cd>] br_deliver+0x3d/0x50 [ 1164.798950] [<ffffffff81a126be>] br_dev_xmit+0x22e/0x290 [ 1164.805576] [<ffffffff81a12490>] ? br_dev_free+0x30/0x30 [ 1164.812106] [<ffffffff8191d18d>] dev_hard_start_xmit+0x4ad/0x500 [ 1164.818729] [<ffffffff8191d55e>] dev_queue_xmit+0x37e/0x600 [ 1164.825314] [<ffffffff8191d1e0>] ? dev_hard_start_xmit+0x500/0x500 [ 1164.831898] [<ffffffff819b7043>] ip_finish_output+0x293/0x610 [ 1164.838483] [<ffffffff819b8a44>] ? ip_output+0x54/0xf0 [ 1164.845055] [<ffffffff819b8a44>] ip_output+0x54/0xf0 [ 1164.851400] [<ffffffff819b3e71>] ip_forward_finish+0x71/0x1a0 [ 1164.857725] [<ffffffff819b42d8>] ip_forward+0x338/0x420 [ 1164.864167] [<ffffffff819b1ca0>] ip_rcv_finish+0x150/0x660 [ 1164.870477] [<ffffffff819b275b>] ip_rcv+0x22b/0x370 [ 1164.876707] [<ffffffff81a0de22>] ? packet_rcv_spkt+0x42/0x190 [ 1164.883040] [<ffffffff8191a3a2>] __netif_receive_skb_core+0x6e2/0x8b0 [ 1164.889193] [<ffffffff81919dd4>] ? __netif_receive_skb_core+0x114/0x8b0 [ 1164.895039] [<ffffffff810f28b9>] ? getnstimeofday+0x9/0x30 [ 1164.900704] [<ffffffff8191a58c>] __netif_receive_skb+0x1c/0x70 [ 1164.906327] [<ffffffff8191a7af>] netif_receive_skb+0x3f/0x50 [ 1164.911892] [<ffffffff8191a8d4>] napi_gro_complete+0x114/0x140 [ 1164.917459] [<ffffffff8191a7e0>] ? napi_gro_complete+0x20/0x140 [ 1164.923048] [<ffffffff810dcf3a>] ? lock_release+0x12a/0x240 [ 1164.928595] [<ffffffff819e9cb7>] ? inet_gro_receive+0x57/0x260 [ 1164.934042] [<ffffffff8191b552>] dev_gro_receive+0x2b2/0x3f0 [ 1164.939384] [<ffffffff8191b48b>] ? dev_gro_receive+0x1eb/0x3f0 [ 1164.944704] [<ffffffff8191b849>] napi_gro_receive+0x29/0xc0 [ 1164.949906] [<ffffffff816d9253>] rtl8169_poll+0x2d3/0x680 [ 1164.955036] [<ffffffff8191aba1>] net_rx_action+0x171/0x270 [ 1164.960180] [<ffffffff8109f27d>] __do_softirq+0xed/0x210 [ 1164.965285] [<ffffffff8109f3f5>] run_ksoftirqd+0x55/0x90 [ 1164.970334] [<ffffffff810c1e29>] smpboot_thread_fn+0x199/0x2a0 [ 1164.975402] [<ffffffff810c1c90>] ? SyS_setgroups+0x150/0x150 [ 1164.980438] [<ffffffff810bb00f>] kthread+0xdf/0x100 [ 1164.985309] [<ffffffff810baf30>] ? __init_kthread_worker+0x70/0x70 [ 1164.990221] [<ffffffff81a8a9cc>] ret_from_fork+0x7c/0xb0 [ 1164.995085] [<ffffffff810baf30>] ? __init_kthread_worker+0x70/0x70 [ 1164.999925] Code: ff 4c 8b 85 68 ff ff ff 44 8b 8d 50 ff ff ff 44 8b 95 48 ff ff ff 44 8b 9d 40 ff ff ff 0f 85 2c fe ff ff e9 23 fe ff ff 90 0f 0b <0f> 0b 48 c7 45 b0 ea ff ff ff e9 cf fc ff ff 0f 0b 0f 0b 66 66 [ 1165.010399] RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 [ 1165.015512] RSP <ffff880059b97448> [ 1165.020980] ---[ end trace 88f75f0c791ac25c ]--- [ 1165.026033] Kernel panic - not syncing: Fatal exception in interrupt ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at net/core/skbuff.c:2839 RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 2013-11-17 19:17 kernel BUG at net/core/skbuff.c:2839 RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 Sander Eikelenboom @ 2013-11-17 19:59 ` Cong Wang 2013-11-21 12:00 ` Sander Eikelenboom 1 sibling, 0 replies; 11+ messages in thread From: Cong Wang @ 2013-11-17 19:59 UTC (permalink / raw) To: netdev On Sun, 17 Nov 2013 at 19:17 GMT, Sander Eikelenboom <linux@eikelenboom.it> wrote: > Hi Eric, > > With the linux-net changes from this merge window i get the kernel panic below (not with 3.12.0). > > It's on a machine running Xen, 2x rtl8169 nic, and using a bridge for guest networking. > This panic in the host kernel only seems to occur when generating a lot of network traffic to and from a guest. > > I tried reverting "tcp: gso: fix truesize tracking" 0d08c42cf9a71530fef5ebcfe368f38f2dd0476f, but that didn't help. > > -- > Sander > > [ 1164.511712] ------------[ cut here ]------------ > [ 1164.518446] kernel BUG at net/core/skbuff.c:2839! This is discussed in another long thread: http://marc.info/?t=138419721200049&r=1&w=2 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at net/core/skbuff.c:2839 RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 2013-11-17 19:17 kernel BUG at net/core/skbuff.c:2839 RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 Sander Eikelenboom 2013-11-17 19:59 ` Cong Wang @ 2013-11-21 12:00 ` Sander Eikelenboom 2013-11-21 14:10 ` Eric Dumazet 1 sibling, 1 reply; 11+ messages in thread From: Sander Eikelenboom @ 2013-11-21 12:00 UTC (permalink / raw) To: Eric Dumazet, Francois Romieu; +Cc: netdev, David S. Miller Hello Sander, Sunday, November 17, 2013, 8:17:44 PM, you wrote: > Hi Eric, > With the linux-net changes from this merge window i get the kernel panic below (not with 3.12.0). > It's on a machine running Xen, 2x rtl8169 nic, and using a bridge for guest networking. > This panic in the host kernel only seems to occur when generating a lot of network traffic to and from a guest. > I tried reverting "tcp: gso: fix truesize tracking" 0d08c42cf9a71530fef5ebcfe368f38f2dd0476f, but that didn't help. > -- > Sander > [ 1164.511712] ------------[ cut here ]------------ > [ 1164.518446] kernel BUG at net/core/skbuff.c:2839! > [ 1164.525226] invalid opcode: 0000 [#2] PREEMPT SMP > [ 1164.532024] Modules linked in: > [ 1164.538713] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G D 3.12.0-mw-20131117+ #1 > [ 1164.545649] Hardware name: MSI MS-7640/890FXA-GD70 (MS-7640) , BIOS V1.8B1 09/13/2010 > [ 1164.552659] task: ffff880059b8a180 ti: ffff880059b96000 task.ti: ffff880059b96000 > [ 1164.559649] RIP: e030:[<ffffffff819109a2>] [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 > [ 1164.566860] RSP: e02b:ffff880059b97448 EFLAGS: 00010216 > [ 1164.574023] RAX: 0000000000000011 RBX: 0000000000006612 RCX: 0000000000006612 > [ 1164.581169] RDX: 00000000000005a8 RSI: 0000000000006612 RDI: ffff8800478ff682 > [ 1164.588115] RBP: ffff880059b97518 R08: ffff88004ca06a00 R09: 0000000000000011 > [ 1164.595169] R10: 000000000000606a R11: 0000000000000011 R12: 0000000000000000 > [ 1164.602214] R13: ffff88004ca06900 R14: ffff88004ca06a00 R15: ffff88004bb57f00 > [ 1164.609274] FS: 00007eff7dc67700(0000) GS:ffff88005f600000(0000) knlGS:0000000000000000 > [ 1164.616394] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 1164.623562] CR2: 00007faa868fff30 CR3: 000000005877e000 CR4: 0000000000000660 > [ 1164.630807] Stack: > [ 1164.637959] ffff880059b97458 ffffffff810d90dd ffff880059b97478 ffffffff8109f69a > [ 1164.645296] ffff88004b824628 ffff88004b824400 ffff880059b974e8 ffff88004ca06a00 > [ 1164.652626] 0000001100000001 0000000000000040 0000000000000042 ffffffffffffffbe > [ 1164.659816] Call Trace: > [ 1164.667034] [<ffffffff810d90dd>] ? trace_hardirqs_on+0xd/0x10 > [ 1164.674313] [<ffffffff8109f69a>] ? local_bh_enable+0xaa/0x110 > [ 1164.681543] [<ffffffff819dafc2>] tcp_gso_segment+0x102/0x3e0 > [ 1164.688691] [<ffffffff819b8224>] ? ip_queue_xmit+0x194/0x480 > [ 1164.695741] [<ffffffff819e9fe4>] inet_gso_segment+0x124/0x350 > [ 1164.702836] [<ffffffff8191cb25>] skb_mac_gso_segment+0xd5/0x1d0 > [ 1164.709735] [<ffffffff8191ca92>] ? skb_mac_gso_segment+0x42/0x1d0 > [ 1164.716739] [<ffffffff8191cc7b>] __skb_gso_segment+0x5b/0xc0 > [ 1164.723802] [<ffffffff8191ce80>] dev_hard_start_xmit+0x1a0/0x500 > [ 1164.730744] [<ffffffff81939780>] sch_direct_xmit+0x100/0x280 > [ 1164.737531] [<ffffffff8191d404>] dev_queue_xmit+0x224/0x600 > [ 1164.744403] [<ffffffff8191d1e0>] ? dev_hard_start_xmit+0x500/0x500 > [ 1164.751317] [<ffffffff8194765e>] ? nf_hook_slow+0x11e/0x160 > [ 1164.758332] [<ffffffff81a14040>] ? deliver_clone+0x60/0x60 > [ 1164.765264] [<ffffffff81a140d7>] br_dev_queue_push_xmit+0x97/0x140 > [ 1164.772082] [<ffffffff81a1419d>] br_forward_finish+0x1d/0x60 > [ 1164.778925] [<ffffffff81a12490>] ? br_dev_free+0x30/0x30 > [ 1164.785714] [<ffffffff81a142f2>] __br_deliver+0x52/0x180 > [ 1164.792355] [<ffffffff81a146cd>] br_deliver+0x3d/0x50 > [ 1164.798950] [<ffffffff81a126be>] br_dev_xmit+0x22e/0x290 > [ 1164.805576] [<ffffffff81a12490>] ? br_dev_free+0x30/0x30 > [ 1164.812106] [<ffffffff8191d18d>] dev_hard_start_xmit+0x4ad/0x500 > [ 1164.818729] [<ffffffff8191d55e>] dev_queue_xmit+0x37e/0x600 > [ 1164.825314] [<ffffffff8191d1e0>] ? dev_hard_start_xmit+0x500/0x500 > [ 1164.831898] [<ffffffff819b7043>] ip_finish_output+0x293/0x610 > [ 1164.838483] [<ffffffff819b8a44>] ? ip_output+0x54/0xf0 > [ 1164.845055] [<ffffffff819b8a44>] ip_output+0x54/0xf0 > [ 1164.851400] [<ffffffff819b3e71>] ip_forward_finish+0x71/0x1a0 > [ 1164.857725] [<ffffffff819b42d8>] ip_forward+0x338/0x420 > [ 1164.864167] [<ffffffff819b1ca0>] ip_rcv_finish+0x150/0x660 > [ 1164.870477] [<ffffffff819b275b>] ip_rcv+0x22b/0x370 > [ 1164.876707] [<ffffffff81a0de22>] ? packet_rcv_spkt+0x42/0x190 > [ 1164.883040] [<ffffffff8191a3a2>] __netif_receive_skb_core+0x6e2/0x8b0 > [ 1164.889193] [<ffffffff81919dd4>] ? __netif_receive_skb_core+0x114/0x8b0 > [ 1164.895039] [<ffffffff810f28b9>] ? getnstimeofday+0x9/0x30 > [ 1164.900704] [<ffffffff8191a58c>] __netif_receive_skb+0x1c/0x70 > [ 1164.906327] [<ffffffff8191a7af>] netif_receive_skb+0x3f/0x50 > [ 1164.911892] [<ffffffff8191a8d4>] napi_gro_complete+0x114/0x140 > [ 1164.917459] [<ffffffff8191a7e0>] ? napi_gro_complete+0x20/0x140 > [ 1164.923048] [<ffffffff810dcf3a>] ? lock_release+0x12a/0x240 > [ 1164.928595] [<ffffffff819e9cb7>] ? inet_gro_receive+0x57/0x260 > [ 1164.934042] [<ffffffff8191b552>] dev_gro_receive+0x2b2/0x3f0 > [ 1164.939384] [<ffffffff8191b48b>] ? dev_gro_receive+0x1eb/0x3f0 > [ 1164.944704] [<ffffffff8191b849>] napi_gro_receive+0x29/0xc0 > [ 1164.949906] [<ffffffff816d9253>] rtl8169_poll+0x2d3/0x680 > [ 1164.955036] [<ffffffff8191aba1>] net_rx_action+0x171/0x270 > [ 1164.960180] [<ffffffff8109f27d>] __do_softirq+0xed/0x210 > [ 1164.965285] [<ffffffff8109f3f5>] run_ksoftirqd+0x55/0x90 > [ 1164.970334] [<ffffffff810c1e29>] smpboot_thread_fn+0x199/0x2a0 > [ 1164.975402] [<ffffffff810c1c90>] ? SyS_setgroups+0x150/0x150 > [ 1164.980438] [<ffffffff810bb00f>] kthread+0xdf/0x100 > [ 1164.985309] [<ffffffff810baf30>] ? __init_kthread_worker+0x70/0x70 > [ 1164.990221] [<ffffffff81a8a9cc>] ret_from_fork+0x7c/0xb0 > [ 1164.995085] [<ffffffff810baf30>] ? __init_kthread_worker+0x70/0x70 > [ 1164.999925] Code: ff 4c 8b 85 68 ff ff ff 44 8b 8d 50 ff ff ff 44 8b 95 48 ff ff ff 44 8b 9d 40 ff ff ff 0f 85 2c fe ff ff e9 23 fe ff ff 90 0f 0b <0f> 0b 48 c7 45 b0 ea ff ff ff e9 cf fc ff ff 0f 0b 0f 0b 66 66 > [ 1165.010399] RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 > [ 1165.015512] RSP <ffff880059b97448> > [ 1165.020980] ---[ end trace 88f75f0c791ac25c ]--- > [ 1165.026033] Kernel panic - not syncing: Fatal exception in interrupt Hi Eric and Francois, I have tested some more: First tried with switching off GSO and GRO on the bridge, this didn't help. Then i only switched off GRO on eth0 (r8169) and left the bridge alone. That helped to prevent the oops. Below the output of ethtool -k for the bridge and eth0 after boot (so the default situation) with which the oops occurs. And the part of dmesg where the r8169 get initialized on boot (there are 2, eth0 and eth1). -- Sander ~# ethtool -k eth0 Features for eth0: rx-checksumming: on tx-checksumming: off tx-checksum-ipv4: off tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off [fixed] udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off [requested on] generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: off [fixed] rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] tx-mpls-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off rx-all: off tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] ~# ethtool -k xen_bridge Features for xen_bridge: rx-checksumming: off [fixed] tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: on tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: on tx-tcp6-segmentation: on udp-fragmentation-offload: on generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: on ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: on rx-vlan-filter: off [fixed] vlan-challenged: off [fixed] tx-lockless: on [fixed] netns-local: on [fixed] tx-gso-robust: on tx-fcoe-segmentation: on tx-gre-segmentation: on tx-ipip-segmentation: on tx-sit-segmentation: on tx-udp_tnl-segmentation: on tx-mpls-segmentation: on fcoe-mtu: off [fixed] tx-nocache-copy: on loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] [ 12.356379] r8169 0000:0b:00.0 eth0: RTL8168d/8111d at 0xffffc90000334000, 40:61:86:f4:67:d9, XID 081000c0 IRQ 128 [ 12.361803] r8169 0000:0b:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko] [ 12.367291] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded [ 12.372748] xen: registering gsi 51 triggering 0 polarity 1 [ 12.378225] xen: --> pirq=51 -> irq=51 (gsi=51) [ 12.383612] r8169 0000:0a:00.0: enabling Mem-Wr-Inval [ 12.389505] r8169 0000:0a:00.0 eth1: RTL8168d/8111d at 0xffffc90000336000, 40:61:86:f4:67:d8, XID 081000c0 IRQ 129 [ 12.395056] r8169 0000:0a:00.0 eth1: jumbo features [frames: 9200 bytes, tx checksumming: ko] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at net/core/skbuff.c:2839 RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 2013-11-21 12:00 ` Sander Eikelenboom @ 2013-11-21 14:10 ` Eric Dumazet 2013-11-21 14:14 ` Sander Eikelenboom ` (2 more replies) 0 siblings, 3 replies; 11+ messages in thread From: Eric Dumazet @ 2013-11-21 14:10 UTC (permalink / raw) To: Sander Eikelenboom; +Cc: Eric Dumazet, Francois Romieu, netdev, David S. Miller On Thu, 2013-11-21 at 13:00 +0100, Sander Eikelenboom wrote: > Hello Sander, > > Sunday, November 17, 2013, 8:17:44 PM, you wrote: > > > Hi Eric, > > > With the linux-net changes from this merge window i get the kernel panic below (not with 3.12.0). > > > It's on a machine running Xen, 2x rtl8169 nic, and using a bridge for guest networking. > > This panic in the host kernel only seems to occur when generating a lot of network traffic to and from a guest. > > > I tried reverting "tcp: gso: fix truesize tracking" 0d08c42cf9a71530fef5ebcfe368f38f2dd0476f, but that didn't help. > Hi Eric and Francois, > > I have tested some more: > > First tried with switching off GSO and GRO on the bridge, this didn't help. > Then i only switched off GRO on eth0 (r8169) and left the bridge alone. That helped to prevent the oops. > > Below the output of ethtool -k for the bridge and eth0 after boot (so the default situation) with which the oops occurs. > And the part of dmesg where the r8169 get initialized on boot (there are 2, eth0 and eth1). As mentioned earlier, this problem is known and a patch is under review. http://marc.info/?l=linux-netdev&m=138419594024851&w=2 Thanks ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at net/core/skbuff.c:2839 RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 2013-11-21 14:10 ` Eric Dumazet @ 2013-11-21 14:14 ` Sander Eikelenboom 2013-11-21 15:46 ` Sander Eikelenboom 2013-11-21 18:17 ` David Miller 2 siblings, 0 replies; 11+ messages in thread From: Sander Eikelenboom @ 2013-11-21 14:14 UTC (permalink / raw) To: Eric Dumazet; +Cc: Francois Romieu, netdev, David S. Miller Thursday, November 21, 2013, 3:10:04 PM, you wrote: > On Thu, 2013-11-21 at 13:00 +0100, Sander Eikelenboom wrote: >> Hello Sander, >> >> Sunday, November 17, 2013, 8:17:44 PM, you wrote: >> >> > Hi Eric, >> >> > With the linux-net changes from this merge window i get the kernel panic below (not with 3.12.0). >> >> > It's on a machine running Xen, 2x rtl8169 nic, and using a bridge for guest networking. >> > This panic in the host kernel only seems to occur when generating a lot of network traffic to and from a guest. >> >> > I tried reverting "tcp: gso: fix truesize tracking" 0d08c42cf9a71530fef5ebcfe368f38f2dd0476f, but that didn't help. >> Hi Eric and Francois, >> >> I have tested some more: >> >> First tried with switching off GSO and GRO on the bridge, this didn't help. >> Then i only switched off GRO on eth0 (r8169) and left the bridge alone. That helped to prevent the oops. >> >> Below the output of ethtool -k for the bridge and eth0 after boot (so the default situation) with which the oops occurs. >> And the part of dmesg where the r8169 get initialized on boot (there are 2, eth0 and eth1). > As mentioned earlier, this problem is known and a patch is under review. > http://marc.info/?l=linux-netdev&m=138419594024851&w=2 > Thanks Ah ok, sorry but never received that message. Will test the patch. Thanks, Sander ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at net/core/skbuff.c:2839 RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 2013-11-21 14:10 ` Eric Dumazet 2013-11-21 14:14 ` Sander Eikelenboom @ 2013-11-21 15:46 ` Sander Eikelenboom 2013-11-21 18:17 ` David Miller 2 siblings, 0 replies; 11+ messages in thread From: Sander Eikelenboom @ 2013-11-21 15:46 UTC (permalink / raw) To: Eric Dumazet; +Cc: Francois Romieu, netdev, David S. Miller Thursday, November 21, 2013, 3:10:04 PM, you wrote: > On Thu, 2013-11-21 at 13:00 +0100, Sander Eikelenboom wrote: >> Hello Sander, >> >> Sunday, November 17, 2013, 8:17:44 PM, you wrote: >> >> > Hi Eric, >> >> > With the linux-net changes from this merge window i get the kernel panic below (not with 3.12.0). >> >> > It's on a machine running Xen, 2x rtl8169 nic, and using a bridge for guest networking. >> > This panic in the host kernel only seems to occur when generating a lot of network traffic to and from a guest. >> >> > I tried reverting "tcp: gso: fix truesize tracking" 0d08c42cf9a71530fef5ebcfe368f38f2dd0476f, but that didn't help. >> Hi Eric and Francois, >> >> I have tested some more: >> >> First tried with switching off GSO and GRO on the bridge, this didn't help. >> Then i only switched off GRO on eth0 (r8169) and left the bridge alone. That helped to prevent the oops. >> >> Below the output of ethtool -k for the bridge and eth0 after boot (so the default situation) with which the oops occurs. >> And the part of dmesg where the r8169 get initialized on boot (there are 2, eth0 and eth1). > As mentioned earlier, this problem is known and a patch is under review. > http://marc.info/?l=linux-netdev&m=138419594024851&w=2 > Thanks Ok seems to work for me! Thx, Sander ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at net/core/skbuff.c:2839 RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 2013-11-21 14:10 ` Eric Dumazet 2013-11-21 14:14 ` Sander Eikelenboom 2013-11-21 15:46 ` Sander Eikelenboom @ 2013-11-21 18:17 ` David Miller 2013-11-21 18:23 ` Eric Dumazet 2 siblings, 1 reply; 11+ messages in thread From: David Miller @ 2013-11-21 18:17 UTC (permalink / raw) To: eric.dumazet, herbert; +Cc: linux, edumazet, romieu, netdev From: Eric Dumazet <eric.dumazet@gmail.com> Date: Thu, 21 Nov 2013 06:10:04 -0800 > As mentioned earlier, this problem is known and a patch is under review. > > http://marc.info/?l=linux-netdev&m=138419594024851&w=2 It seems this patch still requires some improvements, can you guys sort out what needs to be done so that I can apply a fix soon? Thanks! ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at net/core/skbuff.c:2839 RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 2013-11-21 18:17 ` David Miller @ 2013-11-21 18:23 ` Eric Dumazet 2013-11-21 18:38 ` David Miller 0 siblings, 1 reply; 11+ messages in thread From: Eric Dumazet @ 2013-11-21 18:23 UTC (permalink / raw) To: David Miller; +Cc: herbert, linux, edumazet, romieu, netdev On Thu, 2013-11-21 at 13:17 -0500, David Miller wrote: > From: Eric Dumazet <eric.dumazet@gmail.com> > Date: Thu, 21 Nov 2013 06:10:04 -0800 > > > As mentioned earlier, this problem is known and a patch is under review. > > > > http://marc.info/?l=linux-netdev&m=138419594024851&w=2 > > It seems this patch still requires some improvements, can you guys sort > out what needs to be done so that I can apply a fix soon? I think we might apply this patch as is, and do the TSO optimization when net-next reopens ? I was planning to resend Herbert patch today anyway, after a new round of testing on my hosts. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: kernel BUG at net/core/skbuff.c:2839 RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 2013-11-21 18:23 ` Eric Dumazet @ 2013-11-21 18:38 ` David Miller 2013-11-21 19:10 ` [PATCH v2] gso: handle new frag_list of frags GRO packets Eric Dumazet 0 siblings, 1 reply; 11+ messages in thread From: David Miller @ 2013-11-21 18:38 UTC (permalink / raw) To: eric.dumazet; +Cc: herbert, linux, edumazet, romieu, netdev From: Eric Dumazet <eric.dumazet@gmail.com> Date: Thu, 21 Nov 2013 10:23:36 -0800 > I was planning to resend Herbert patch today anyway, after a new round > of testing on my hosts. Please do, it would be nice to have your tested-by or similar tag on it at least. Thanks! ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v2] gso: handle new frag_list of frags GRO packets 2013-11-21 18:38 ` David Miller @ 2013-11-21 19:10 ` Eric Dumazet 2013-11-21 19:12 ` David Miller 0 siblings, 1 reply; 11+ messages in thread From: Eric Dumazet @ 2013-11-21 19:10 UTC (permalink / raw) To: David Miller; +Cc: herbert, linux, edumazet, romieu, netdev From: Herbert Xu <herbert@gondor.apana.org.au> Recently GRO started generating packets with frag_lists of frags. This was not handled by GSO, thus leading to a crash. Thankfully these packets are of a regular form and are easy to handle. This patch handles them in two ways. For completely non-linear frag_list entries, we simply continue to iterate over the frag_list frags once we exhaust the normal frags. For frag_list entries with linear parts, we call pskb_trim on the first part of the frag_list skb, and then process the rest of the frags in the usual way. This patch also kills a chunk of dead frag_list code that has obviously never ever been run since it ends up generating a bogus GSO-segmented packet with a frag_list entry. Future work is planned to split super big packets into TSO ones. Fixes: 8a29111c7ca6 ("net: gro: allow to build full sized skb") Reported-by: Christoph Paasch <christoph.paasch@uclouvain.be> Reported-by: Jerry Chu <hkchu@google.com> Reported-by: Sander Eikelenboom <linux@eikelenboom.it> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Signed-off-by: Eric Dumazet <edumazet@google.com> Tested-by: Sander Eikelenboom <linux@eikelenboom.it> Tested-by: Eric Dumazet <edumazet@google.com> --- net/core/skbuff.c | 75 +++++++++++++++++++++++++++++--------------- 1 file changed, 50 insertions(+), 25 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 8cec1e6b844d..2718fed53d8c 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -2796,6 +2796,7 @@ struct sk_buff *skb_segment(struct sk_buff *skb, netdev_features_t features) struct sk_buff *segs = NULL; struct sk_buff *tail = NULL; struct sk_buff *fskb = skb_shinfo(skb)->frag_list; + skb_frag_t *skb_frag = skb_shinfo(skb)->frags; unsigned int mss = skb_shinfo(skb)->gso_size; unsigned int doffset = skb->data - skb_mac_header(skb); unsigned int offset = doffset; @@ -2835,16 +2836,38 @@ struct sk_buff *skb_segment(struct sk_buff *skb, netdev_features_t features) if (hsize > len || !sg) hsize = len; - if (!hsize && i >= nfrags) { - BUG_ON(fskb->len != len); + if (!hsize && i >= nfrags && skb_headlen(fskb) && + (skb_headlen(fskb) == len || sg)) { + BUG_ON(skb_headlen(fskb) > len); + + i = 0; + nfrags = skb_shinfo(fskb)->nr_frags; + skb_frag = skb_shinfo(fskb)->frags; + pos += skb_headlen(fskb); + + while (pos < offset + len) { + BUG_ON(i >= nfrags); + + size = skb_frag_size(skb_frag); + if (pos + size > offset + len) + break; + + i++; + pos += size; + skb_frag++; + } - pos += len; nskb = skb_clone(fskb, GFP_ATOMIC); fskb = fskb->next; if (unlikely(!nskb)) goto err; + if (unlikely(pskb_trim(nskb, len))) { + kfree_skb(nskb); + goto err; + } + hsize = skb_end_offset(nskb); if (skb_cow_head(nskb, doffset + headroom)) { kfree_skb(nskb); @@ -2881,7 +2904,7 @@ struct sk_buff *skb_segment(struct sk_buff *skb, netdev_features_t features) nskb->data - tnl_hlen, doffset + tnl_hlen); - if (fskb != skb_shinfo(skb)->frag_list) + if (nskb->len == len + doffset) goto perform_csum_check; if (!sg) { @@ -2899,8 +2922,28 @@ struct sk_buff *skb_segment(struct sk_buff *skb, netdev_features_t features) skb_shinfo(nskb)->tx_flags = skb_shinfo(skb)->tx_flags & SKBTX_SHARED_FRAG; - while (pos < offset + len && i < nfrags) { - *frag = skb_shinfo(skb)->frags[i]; + while (pos < offset + len) { + if (i >= nfrags) { + BUG_ON(skb_headlen(fskb)); + + i = 0; + nfrags = skb_shinfo(fskb)->nr_frags; + skb_frag = skb_shinfo(fskb)->frags; + + BUG_ON(!nfrags); + + fskb = fskb->next; + } + + if (unlikely(skb_shinfo(nskb)->nr_frags >= + MAX_SKB_FRAGS)) { + net_warn_ratelimited( + "skb_segment: too many frags: %u %u\n", + pos, mss); + goto err; + } + + *frag = *skb_frag; __skb_frag_ref(frag); size = skb_frag_size(frag); @@ -2913,6 +2956,7 @@ struct sk_buff *skb_segment(struct sk_buff *skb, netdev_features_t features) if (pos + size <= offset + len) { i++; + skb_frag++; pos += size; } else { skb_frag_size_sub(frag, pos + size - (offset + len)); @@ -2922,25 +2966,6 @@ struct sk_buff *skb_segment(struct sk_buff *skb, netdev_features_t features) frag++; } - if (pos < offset + len) { - struct sk_buff *fskb2 = fskb; - - BUG_ON(pos + fskb->len != offset + len); - - pos += fskb->len; - fskb = fskb->next; - - if (fskb2->next) { - fskb2 = skb_clone(fskb2, GFP_ATOMIC); - if (!fskb2) - goto err; - } else - skb_get(fskb2); - - SKB_FRAG_ASSERT(nskb); - skb_shinfo(nskb)->frag_list = fskb2; - } - skip_fraglist: nskb->data_len = len - hsize; nskb->len += nskb->data_len; ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH v2] gso: handle new frag_list of frags GRO packets 2013-11-21 19:10 ` [PATCH v2] gso: handle new frag_list of frags GRO packets Eric Dumazet @ 2013-11-21 19:12 ` David Miller 0 siblings, 0 replies; 11+ messages in thread From: David Miller @ 2013-11-21 19:12 UTC (permalink / raw) To: eric.dumazet; +Cc: herbert, linux, edumazet, romieu, netdev From: Eric Dumazet <eric.dumazet@gmail.com> Date: Thu, 21 Nov 2013 11:10:04 -0800 > From: Herbert Xu <herbert@gondor.apana.org.au> > > Recently GRO started generating packets with frag_lists of frags. > This was not handled by GSO, thus leading to a crash. > > Thankfully these packets are of a regular form and are easy to > handle. This patch handles them in two ways. For completely > non-linear frag_list entries, we simply continue to iterate over > the frag_list frags once we exhaust the normal frags. For frag_list > entries with linear parts, we call pskb_trim on the first part > of the frag_list skb, and then process the rest of the frags in > the usual way. > > This patch also kills a chunk of dead frag_list code that has > obviously never ever been run since it ends up generating a bogus > GSO-segmented packet with a frag_list entry. > > Future work is planned to split super big packets into TSO > ones. > > Fixes: 8a29111c7ca6 ("net: gro: allow to build full sized skb") > Reported-by: Christoph Paasch <christoph.paasch@uclouvain.be> > Reported-by: Jerry Chu <hkchu@google.com> > Reported-by: Sander Eikelenboom <linux@eikelenboom.it> > Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> > Signed-off-by: Eric Dumazet <edumazet@google.com> > Tested-by: Sander Eikelenboom <linux@eikelenboom.it> > Tested-by: Eric Dumazet <edumazet@google.com> Applied and queued up for 3.12-stable, thanks! ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2013-11-21 19:12 UTC | newest] Thread overview: 11+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-11-17 19:17 kernel BUG at net/core/skbuff.c:2839 RIP [<ffffffff819109a2>] skb_segment+0x6b2/0x6d0 Sander Eikelenboom 2013-11-17 19:59 ` Cong Wang 2013-11-21 12:00 ` Sander Eikelenboom 2013-11-21 14:10 ` Eric Dumazet 2013-11-21 14:14 ` Sander Eikelenboom 2013-11-21 15:46 ` Sander Eikelenboom 2013-11-21 18:17 ` David Miller 2013-11-21 18:23 ` Eric Dumazet 2013-11-21 18:38 ` David Miller 2013-11-21 19:10 ` [PATCH v2] gso: handle new frag_list of frags GRO packets Eric Dumazet 2013-11-21 19:12 ` David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox