From: Christoph Paasch <christoph.paasch@uclouvain.be>
To: Eric Dumazet <eric.dumazet@gmail.com>,
Herbert Xu <herbert@gondor.apana.org.au>
Cc: netdev <netdev@vger.kernel.org>
Subject: Bug in skb_segment: fskb->len != len
Date: Mon, 28 Oct 2013 12:55:52 +0100 [thread overview]
Message-ID: <20131028115552.GC4408@cpaasch-mac> (raw)
Hello,
I have been seeing the below BUG in skb_segment with the latest net-next
head on my router.
I am forwarding Multipath TCP-traffic on this router. The MPTCP-sender is simply
doing an iperf-session. Strangely, I cannot reproduce the bug when sending
regular TCP-traffic across the router.
Note: The crash happens on a vanilla net-next kernel. It does not has any
MPTCP-code in it.
I bisected it down to 8a29111c7c (net: gro: allow to build full sized skb),
but I guess 8a29111c7c is just revealing a more fundamental bug in skb_segment.
Some info I found:
In skb_segment, when the bug happens, fskb->len is 4284 but the mss and len is 1428.
Shortly before the bug happens, skb_gro_receive is building a packet where
lp->len is equal to 4284 inside the frag_list.
Seems like skb_segment cannot handle those bigger skb's in the frag_list.
Cheers,
Christoph
Here the crash-dump:
[ 399.832854] ------------[ cut here ]------------
[ 399.888048] kernel BUG at /home/cpaasch/builder/net-next/net/core/skbuff.c:2796!
[ 399.976504] invalid opcode: 0000 [#1] SMP
[ 400.025675] Modules linked in:
[ 400.062270] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 3.12.0-rc6-mptcp #231
[ 400.145531] Hardware name: HP ProLiant DL120 G6/ProLiant DL120 G6, BIOS O26 09/06/2010
[ 400.243342] task: ffff88042d8a4680 ti: ffff88042d8ce000 task.ti: ffff88042d8ce000
[ 400.332841] RIP: 0010:[<ffffffff81447d21>] [<ffffffff81447d21>] skb_segment+0x1aa/0x5fa
[ 400.429722] RSP: 0018:ffff88043fd03770 EFLAGS: 00010212
[ 400.493231] RAX: 0000000000000594 RBX: ffff8800ba89ac00 RCX: 00000000000064be
[ 400.578574] RDX: 0000000000000000 RSI: 0000000000000011 RDI: ffff8804273a7080
[ 400.663918] RBP: ffff88043fd03820 R08: 0000000000000000 R09: ffff88042c4d4600
[ 400.749259] R10: 0000000000010000 R11: ffff88042d801900 R12: ffff88042c7ca000
[ 400.834596] R13: ffff88042c5d5400 R14: 0000000000001650 R15: 0000000000000056
[ 400.919934] FS: 0000000000000000(0000) GS:ffff88043fd00000(0000) knlGS:0000000000000000
[ 401.016711] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 401.085422] CR2: ffffffffff600400 CR3: 000000042c86b000 CR4: 00000000000007e0
[ 401.170765] Stack:
[ 401.194780] ffff88042d94e900 ffff88042c4d46f0 0000000000000000 0000000000000042
[ 401.283663] 0100000000000000 0000000000000001 0000001100000594 0000000000000056
[ 401.372555] 0000000000000000 0000004200000098 ffffffffffffffaa 0000001100000001
[ 401.461445] Call Trace:
[ 401.490658] <IRQ>
[ 401.513631] [<ffffffff8149b077>] tcp_gso_segment+0x168/0x395
[ 401.584644] [<ffffffff814a5ba1>] inet_gso_segment+0x175/0x2a9
[ 401.654396] [<ffffffff8144fb40>] skb_mac_gso_segment+0x10a/0x16a
[ 401.727264] [<ffffffff81451062>] __skb_gso_segment+0xaf/0xb4
[ 401.795977] [<ffffffff814515ae>] dev_hard_start_xmit+0x215/0x40a
[ 401.868846] [<ffffffff814689ed>] sch_direct_xmit+0x6b/0x195
[ 401.936519] [<ffffffff81451988>] dev_queue_xmit+0x1e5/0x3ac
[ 402.004193] [<ffffffff814b6461>] ? iptable_filter_hook+0x41/0x4c
[ 402.077061] [<ffffffff8148039d>] ip_finish_output+0x2f6/0x351
[ 402.146812] [<ffffffff8147c6dc>] ? ip_frag_mem+0x34/0x34
[ 402.211366] [<ffffffff81480470>] ip_output+0x78/0x7f
[ 402.271765] [<ffffffff8147c71c>] ip_forward_finish+0x40/0x44
[ 402.340475] [<ffffffff8147c9c5>] ip_forward+0x2a5/0x300
[ 402.403993] [<ffffffff8147b104>] ip_rcv_finish+0x214/0x22c
[ 402.470625] [<ffffffff8147b3cd>] ip_rcv+0x2b1/0x2e9
[ 402.529983] [<ffffffff81446a19>] ? skb_gro_receive+0x562/0x582
[ 402.600773] [<ffffffff8144dcd8>] __netif_receive_skb_core+0x49a/0x4cd
[ 402.678840] [<ffffffff8144dd60>] __netif_receive_skb+0x55/0x5a
[ 402.749631] [<ffffffff81450190>] netif_receive_skb+0x71/0x78
[ 402.818344] [<ffffffff8149af07>] ? tcp4_gro_receive+0xf4/0xfc
[ 402.888095] [<ffffffff81450249>] napi_gro_complete+0xb2/0xba
[ 402.956808] [<ffffffff8145045f>] dev_gro_receive+0x20e/0x34d
[ 403.025519] [<ffffffff81450ae5>] napi_gro_receive+0x92/0xf1
[ 403.093195] [<ffffffff813acfe2>] netxen_process_rcv_ring+0x1b0/0x767
[ 403.170222] [<ffffffff810b3ae8>] ? kmem_cache_free+0xef/0xf3
[ 403.238931] [<ffffffff81450fb1>] ? dev_kfree_skb_any+0x2e/0x30
[ 403.309723] [<ffffffff813acc42>] ? netxen_process_cmd_ring+0x33/0x223
[ 403.387790] [<ffffffff813a8f70>] netxen_nic_poll+0x35/0x9a
[ 403.454423] [<ffffffff814506dc>] net_rx_action+0xa7/0x1d2
[ 403.520017] [<ffffffff8103605d>] __do_softirq+0xbd/0x17e
[ 403.584572] [<ffffffff815289bc>] call_softirq+0x1c/0x26
[ 403.648085] [<ffffffff81003bbb>] do_softirq+0x33/0x68
[ 403.709523] [<ffffffff81035efb>] irq_exit+0x40/0x4e
[ 403.768880] [<ffffffff81003423>] do_IRQ+0x98/0xaf
[ 403.826158] [<ffffffff8152716a>] common_interrupt+0x6a/0x6a
[ 403.893829] <EOI>
[ 403.916800] [<ffffffff8100933d>] ? default_idle+0x6/0x8
[ 403.982604] [<ffffffff81009542>] arch_cpu_idle+0x13/0x18
[ 404.047159] [<ffffffff8105ea2b>] cpu_startup_entry+0xa4/0xf1
[ 404.115873] [<ffffffff8102320b>] start_secondary+0x1b2/0x1b7
[ 404.184582] Code: bd 7f ff ff ff 00 74 04 44 8b 75 c0 45 85 f6 0f 85 e5 00 00 00 8b 75 84 39 75 ac 0f 8c d9 00 00 00 45 8b 75 68 44 3b 75 c0 74 04 <0f> 0b eb fe 4c 89 ef be 20 00 00 00 e8 08 f1 ff ff 48 85 c0 48
[ 404.417106] RIP [<ffffffff81447d21>] skb_segment+0x1aa/0x5fa
[ 404.485928] RSP <ffff88043fd03770>
[ 404.527614] ---[ end trace 32152a68c7bdc3ac ]---
next reply other threads:[~2013-10-28 11:56 UTC|newest]
Thread overview: 163+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-10-28 11:55 Christoph Paasch [this message]
2013-10-28 13:21 ` Bug in skb_segment: fskb->len != len Eric Dumazet
2013-10-28 13:28 ` Christoph Paasch
2013-10-29 1:15 ` Eric Dumazet
2013-10-29 9:08 ` Christoph Paasch
2013-10-29 12:57 ` Eric Dumazet
2013-10-29 13:06 ` [PATCH net-next] net: introduce gro_frag_list_enable sysctl Eric Dumazet
2013-10-29 13:48 ` Christoph Paasch
2013-10-29 15:12 ` [PATCH v2 " Eric Dumazet
2013-10-29 23:44 ` David Miller
2013-10-30 0:06 ` Ben Hutchings
2013-11-02 14:01 ` [PATCH v3 net-next] net: introduce dev_set_forwarding() Eric Dumazet
2013-11-02 15:46 ` Ben Hutchings
2013-11-02 18:20 ` Eric Dumazet
2013-11-02 19:58 ` [PATCH v4 " Eric Dumazet
2013-11-03 17:18 ` Christoph Paasch
2013-11-04 16:55 ` Ben Hutchings
2013-11-07 21:17 ` David Miller
2013-11-07 21:31 ` Herbert Xu
2013-11-07 21:54 ` Eric Dumazet
2013-11-08 3:59 ` Herbert Xu
2013-11-08 4:25 ` Eric Dumazet
2013-11-10 14:05 ` Herbert Xu
2013-11-11 14:36 ` Herbert Xu
2013-11-07 22:06 ` David Miller
2013-11-08 2:17 ` Herbert Xu
2013-11-08 2:42 ` Eric Dumazet
2013-11-08 2:51 ` Eric Dumazet
2013-11-08 3:23 ` Herbert Xu
2013-11-08 4:21 ` Eric Dumazet
2013-11-08 4:24 ` Herbert Xu
2013-11-08 4:40 ` Eric Dumazet
2013-11-08 4:43 ` Herbert Xu
2013-11-08 5:08 ` Eric Dumazet
2013-11-08 5:21 ` Herbert Xu
2013-11-08 5:40 ` Eric Dumazet
2013-11-11 18:58 ` Herbert Xu
2013-11-08 3:22 ` Herbert Xu
2013-11-08 4:06 ` Eric Dumazet
2013-11-08 4:10 ` Herbert Xu
2013-11-08 4:24 ` Eric Dumazet
2013-11-08 4:28 ` Herbert Xu
2013-11-21 18:29 ` David Miller
2013-11-21 18:38 ` Eric Dumazet
2013-11-03 12:28 ` [PATCH v3 " Herbert Xu
2013-11-03 16:28 ` Eric Dumazet
2013-11-03 16:31 ` Herbert Xu
2013-11-03 17:26 ` Eric Dumazet
2013-11-04 4:11 ` Herbert Xu
2013-11-04 4:23 ` Eric Dumazet
2013-11-04 4:29 ` Herbert Xu
2013-11-04 5:00 ` Eric Dumazet
2013-11-04 5:23 ` Herbert Xu
2013-11-04 6:05 ` Eric Dumazet
2013-11-04 6:22 ` Herbert Xu
2013-11-04 6:26 ` Herbert Xu
2013-11-04 7:10 ` Eric Dumazet
2013-11-04 7:21 ` Herbert Xu
2013-11-04 13:58 ` Eric Dumazet
2013-11-04 6:46 ` Eric Dumazet
2013-11-04 7:03 ` Herbert Xu
2013-11-06 1:30 ` gso: Attempt to handle mega-GRO packets Herbert Xu
2013-11-06 1:45 ` Eric Dumazet
2013-11-06 4:07 ` Herbert Xu
2013-11-06 4:23 ` Eric Dumazet
2013-11-06 4:28 ` Herbert Xu
2013-11-06 5:20 ` Eric Dumazet
2013-11-06 8:04 ` Herbert Xu
2013-11-06 8:16 ` Herbert Xu
2013-11-06 13:12 ` Herbert Xu
2013-11-06 15:01 ` Eric Dumazet
2013-11-07 0:36 ` Herbert Xu
2013-11-07 1:03 ` Eric Dumazet
2013-11-07 1:47 ` Herbert Xu
2013-11-07 2:02 ` Eric Dumazet
2013-11-07 2:08 ` Eric Dumazet
2013-11-07 2:15 ` Herbert Xu
2013-11-07 2:37 ` Eric Dumazet
2013-11-07 2:41 ` Herbert Xu
2013-11-07 5:56 ` Michael S. Tsirkin
2013-11-07 7:07 ` Eric Dumazet
2013-11-07 2:52 ` Jason Wang
2013-11-06 15:05 ` Eric Dumazet
2013-11-07 0:39 ` Herbert Xu
2013-11-06 12:39 ` Herbert Xu
2013-11-06 13:30 ` Herbert Xu
2013-11-06 14:39 ` Herbert Xu
2013-11-06 15:06 ` Eric Dumazet
2013-11-06 17:25 ` Joe Perches
2013-11-06 19:47 ` Eric Dumazet
2013-11-07 0:15 ` Eric Dumazet
2013-11-07 0:47 ` Herbert Xu
2013-11-07 0:56 ` Eric Dumazet
2013-11-07 1:00 ` Herbert Xu
2013-11-07 1:08 ` Eric Dumazet
2013-11-07 1:13 ` Hannes Frederic Sowa
2013-11-07 1:21 ` Eric Dumazet
2013-11-07 1:34 ` Eric Dumazet
2013-11-07 2:03 ` Hannes Frederic Sowa
2013-11-07 3:05 ` Eric Dumazet
2013-11-07 6:59 ` Eric Dumazet
2013-11-07 0:43 ` Herbert Xu
2013-11-07 6:22 ` Herbert Xu
2013-11-07 7:03 ` [1/3] gso: Add to segs at end of loop in skb_segment Herbert Xu
2013-11-07 7:06 ` [2/3] gso: Handle new frag_list of frags GRO packets Herbert Xu
2013-11-07 7:08 ` [3/3] gso: Handle malicious GRO packets without crashing Herbert Xu
2013-11-07 18:18 ` Ben Hutchings
2013-11-07 19:13 ` Sergei Shtylyov
2013-11-11 18:55 ` Herbert Xu
2013-11-07 18:16 ` [2/3] gso: Handle new frag_list of frags GRO packets Ben Hutchings
2013-11-11 18:54 ` Herbert Xu
2013-11-11 18:52 ` Herbert Xu
2013-11-12 10:12 ` David Laight
2013-11-13 1:13 ` gso: " Eric Dumazet
2013-11-13 1:29 ` Herbert Xu
2013-11-13 2:14 ` Eric Dumazet
2013-11-13 2:17 ` Eric Dumazet
2013-11-13 2:22 ` Herbert Xu
2013-11-13 2:25 ` Herbert Xu
2013-11-13 2:45 ` Eric Dumazet
2013-11-13 14:26 ` Herbert Xu
2013-11-13 15:06 ` Eric Dumazet
2013-11-14 8:11 ` Herbert Xu
2013-11-15 4:37 ` Eric Dumazet
2013-11-13 2:31 ` Eric Dumazet
2013-11-07 7:11 ` gso: Attempt to handle mega-GRO packets Eric Dumazet
2013-11-07 7:15 ` Herbert Xu
2013-11-07 7:17 ` Herbert Xu
2013-11-07 7:31 ` Eric Dumazet
2013-11-07 7:33 ` Herbert Xu
2013-11-03 23:23 ` [PATCH v3 net-next] net: introduce dev_set_forwarding() David Miller
2013-10-30 0:53 ` [PATCH v2 net-next] net: introduce gro_frag_list_enable sysctl Eric Dumazet
2013-10-30 2:02 ` David Miller
2013-10-30 2:05 ` Herbert Xu
2013-10-30 2:13 ` Jerry Chu
2013-10-30 2:19 ` Herbert Xu
2013-10-30 2:34 ` David Miller
2013-10-30 2:33 ` David Miller
[not found] ` <44571383414236@web13j.yandex.ru>
2013-11-02 18:28 ` Eric Dumazet
2013-11-03 23:19 ` David Miller
2013-10-30 19:39 ` Ben Hutchings
2013-10-30 19:53 ` Eric Dumazet
2013-10-30 20:05 ` Ben Hutchings
2013-10-30 20:12 ` Eric Dumazet
2013-10-30 4:06 ` Eric Dumazet
2013-10-30 4:08 ` Herbert Xu
2013-10-30 4:09 ` Herbert Xu
2013-10-30 4:15 ` Jerry Chu
2013-10-30 4:16 ` Eric Dumazet
2013-10-30 4:19 ` Herbert Xu
2013-10-30 4:34 ` Eric Dumazet
2013-10-30 4:42 ` Herbert Xu
2013-10-30 17:39 ` Jerry Chu
2013-10-30 18:09 ` Vlad Yasevich
2013-10-30 19:12 ` David Miller
2013-10-30 0:03 ` Jerry Chu
2013-10-29 14:41 ` Bug in skb_segment: fskb->len != len Herbert Xu
2013-10-29 15:08 ` Eric Dumazet
2013-10-30 1:50 ` Herbert Xu
2013-10-30 4:03 ` Eric Dumazet
2013-10-30 4:06 ` Herbert Xu
2013-10-30 4:37 ` Eric Dumazet
2013-10-30 4:47 ` Herbert Xu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20131028115552.GC4408@cpaasch-mac \
--to=christoph.paasch@uclouvain.be \
--cc=eric.dumazet@gmail.com \
--cc=herbert@gondor.apana.org.au \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.