* virtio-net: kernel panic in virtio_net.c @ 2021-06-01 16:06 Corentin Noël 2021-06-01 17:07 ` Greg KH 0 siblings, 1 reply; 21+ messages in thread From: Corentin Noël @ 2021-06-01 16:06 UTC (permalink / raw) To: stable Cc: Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Xuan Zhuo I've been experiencing crashes with 5.13 that do not occur with 5.12, here is the crash trace: [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 len:3762 put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2 end:0xec0 dev:<NULL> [ 47.716267] kernel BUG at net/core/skbuff.c:110! [ 47.717197] invalid opcode: 0000 [#1] SMP PTI [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0- rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1 [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 [ 47.720656] RIP: 0010:skb_panic+0x43/0x45 [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0 [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX: 00000000ffffdfff [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: 0000000000000000 [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: 0000000000009ffb [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: ffff9e1e1e95b300 [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: 0000000000000eb2 [ 47.732541] FS: 00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000) knlGS:0000000000000000 [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: 0000000000370ee0 [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 47.738318] Call Trace: [ 47.738812] skb_put.cold+0x10/0x10 [ 47.739450] page_to_skb+0xe4/0x400 [ 47.740072] receive_buf+0x86/0x1660 [ 47.740693] ? inet_gro_receive+0x54/0x2c0 [ 47.741279] ? dev_gro_receive+0x194/0x6a0 [ 47.741846] virtnet_poll+0x2b8/0x3c0 [ 47.742357] __napi_poll+0x25/0x150 [ 47.742844] net_rx_action+0x22f/0x280 [ 47.743388] __do_softirq+0xba/0x264 [ 47.743947] irq_exit_rcu+0x90/0xb0 [ 47.744435] common_interrupt+0x40/0xa0 [ 47.744978] ? asm_common_interrupt+0x8/0x40 [ 47.745582] asm_common_interrupt+0x1e/0x40 [ 47.746182] RIP: 0033:0x7f3a7a276ed4 [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc 54 c8 c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5 d5 fa c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8 cb [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212 [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX: ffffffffffffffff [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI: 00007f3a7c0575a0 [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09: 00007f3a8c210354 [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12: 00007f3a8c210340 [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15: 00007f3a8c21033c [ 47.755354] Modules linked in: [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]--- [ 47.756606] RIP: 0010:skb_panic+0x43/0x45 [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0 [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX: 00000000ffffdfff [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: 0000000000000000 [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: 0000000000009ffb [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: ffff9e1e1e95b300 [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: 0000000000000eb2 [ 47.766261] FS: 00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000) knlGS:0000000000000000 [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: 0000000000370ee0 [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 47.771339] Kernel panic - not syncing: Fatal exception in interrupt [ 47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) I've been able to bisect the issue a little bit and the issue disappeared after reverting the 4 following commits: * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7 * af39c8f72301b268ad8b04bae646b6025918b82b * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6 * f80bd740cb7c954791279590b2e810ba6c214e52 Here is my kernel config: https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config Regards, Corentin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-01 16:06 virtio-net: kernel panic in virtio_net.c Corentin Noël @ 2021-06-01 17:07 ` Greg KH 2021-06-01 17:09 ` Corentin Noël 0 siblings, 1 reply; 21+ messages in thread From: Greg KH @ 2021-06-01 17:07 UTC (permalink / raw) To: Corentin Noël Cc: stable, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Xuan Zhuo On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote: > I've been experiencing crashes with 5.13 that do not occur with 5.12, > here is the crash trace: > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 len:3762 > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2 > end:0xec0 dev:<NULL> > [ 47.716267] kernel BUG at net/core/skbuff.c:110! > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0- > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1 > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45 > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0 > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > 0000000000009ffb > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff9e1e1e95b300 > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > 0000000000000eb2 > [ 47.732541] FS: 00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000) > knlGS:0000000000000000 > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > 0000000000370ee0 > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 47.738318] Call Trace: > [ 47.738812] skb_put.cold+0x10/0x10 > [ 47.739450] page_to_skb+0xe4/0x400 > [ 47.740072] receive_buf+0x86/0x1660 > [ 47.740693] ? inet_gro_receive+0x54/0x2c0 > [ 47.741279] ? dev_gro_receive+0x194/0x6a0 > [ 47.741846] virtnet_poll+0x2b8/0x3c0 > [ 47.742357] __napi_poll+0x25/0x150 > [ 47.742844] net_rx_action+0x22f/0x280 > [ 47.743388] __do_softirq+0xba/0x264 > [ 47.743947] irq_exit_rcu+0x90/0xb0 > [ 47.744435] common_interrupt+0x40/0xa0 > [ 47.744978] ? asm_common_interrupt+0x8/0x40 > [ 47.745582] asm_common_interrupt+0x1e/0x40 > [ 47.746182] RIP: 0033:0x7f3a7a276ed4 > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc 54 c8 > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5 d5 fa > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8 cb > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212 > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX: > ffffffffffffffff > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI: > 00007f3a7c0575a0 > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09: > 00007f3a8c210354 > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12: > 00007f3a8c210340 > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15: > 00007f3a8c21033c > [ 47.755354] Modules linked in: > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]--- > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45 > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f 4c fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 e0 > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > 0000000000009ffb > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff9e1e1e95b300 > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > 0000000000000eb2 > [ 47.766261] FS: 00007f3a82b53700(0000) GS:ffff9e1f2bd00000(0000) > knlGS:0000000000000000 > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > 0000000000370ee0 > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 47.771339] Kernel panic - not syncing: Fatal exception in interrupt > [ 47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000 > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > I've been able to bisect the issue a little bit and the issue > disappeared after reverting the 4 following commits: > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7 > * af39c8f72301b268ad8b04bae646b6025918b82b > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6 > * f80bd740cb7c954791279590b2e810ba6c214e52 > > Here is my kernel config: > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config Do you have the same problem with 5.13-rc4? thanks, greg k-h ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-01 17:07 ` Greg KH @ 2021-06-01 17:09 ` Corentin Noël 2021-06-01 17:47 ` Eric Dumazet 0 siblings, 1 reply; 21+ messages in thread From: Corentin Noël @ 2021-06-01 17:09 UTC (permalink / raw) To: Greg KH Cc: stable, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Xuan Zhuo Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit : > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote: > > I've been experiencing crashes with 5.13 that do not occur with > > 5.12, > > here is the crash trace: > > > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 > > len:3762 > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2 > > end:0xec0 dev:<NULL> > > [ 47.716267] kernel BUG at net/core/skbuff.c:110! > > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI > > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0- > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1 > > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 > > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45 > > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > 00 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f > > 4c fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 > > e0 > > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > 0000000000009ffb > > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff9e1e1e95b300 > > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > 0000000000000eb2 > > [ 47.732541] FS: 00007f3a82b53700(0000) > > GS:ffff9e1f2bd00000(0000) > > knlGS:0000000000000000 > > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > 0000000000370ee0 > > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 47.738318] Call Trace: > > [ 47.738812] skb_put.cold+0x10/0x10 > > [ 47.739450] page_to_skb+0xe4/0x400 > > [ 47.740072] receive_buf+0x86/0x1660 > > [ 47.740693] ? inet_gro_receive+0x54/0x2c0 > > [ 47.741279] ? dev_gro_receive+0x194/0x6a0 > > [ 47.741846] virtnet_poll+0x2b8/0x3c0 > > [ 47.742357] __napi_poll+0x25/0x150 > > [ 47.742844] net_rx_action+0x22f/0x280 > > [ 47.743388] __do_softirq+0xba/0x264 > > [ 47.743947] irq_exit_rcu+0x90/0xb0 > > [ 47.744435] common_interrupt+0x40/0xa0 > > [ 47.744978] ? asm_common_interrupt+0x8/0x40 > > [ 47.745582] asm_common_interrupt+0x1e/0x40 > > [ 47.746182] RIP: 0033:0x7f3a7a276ed4 > > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc > > 54 c8 > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5 > > d5 fa > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8 > > cb > > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212 > > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX: > > ffffffffffffffff > > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI: > > 00007f3a7c0575a0 > > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09: > > 00007f3a8c210354 > > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12: > > 00007f3a8c210340 > > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15: > > 00007f3a8c21033c > > [ 47.755354] Modules linked in: > > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]--- > > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45 > > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > 00 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f > > 4c fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 > > e0 > > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > 0000000000009ffb > > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff9e1e1e95b300 > > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > 0000000000000eb2 > > [ 47.766261] FS: 00007f3a82b53700(0000) > > GS:ffff9e1f2bd00000(0000) > > knlGS:0000000000000000 > > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > 0000000000370ee0 > > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 47.771339] Kernel panic - not syncing: Fatal exception in > > interrupt > > [ 47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000 > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > I've been able to bisect the issue a little bit and the issue > > disappeared after reverting the 4 following commits: > > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7 > > * af39c8f72301b268ad8b04bae646b6025918b82b > > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6 > > * f80bd740cb7c954791279590b2e810ba6c214e52 > > > > Here is my kernel config: > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config > > Do you have the same problem with 5.13-rc4? > > thanks, > > greg k-h Yes I tried with rc2, rc3 and rc4 resulting to the same panic. Thanks, Corentin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-01 17:09 ` Corentin Noël @ 2021-06-01 17:47 ` Eric Dumazet 2021-06-02 17:54 ` Corentin Noël 0 siblings, 1 reply; 21+ messages in thread From: Eric Dumazet @ 2021-06-01 17:47 UTC (permalink / raw) To: Corentin Noël Cc: Greg KH, stable, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Xuan Zhuo On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël <corentin.noel@collabora.com> wrote: > > Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit : > > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote: > > > I've been experiencing crashes with 5.13 that do not occur with > > > 5.12, > > > here is the crash trace: > > > > > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 > > > len:3762 > > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2 > > > end:0xec0 dev:<NULL> > > > [ 47.716267] kernel BUG at net/core/skbuff.c:110! > > > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI > > > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted 5.13.0- > > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1 > > > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 > > > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45 > > > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > 00 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f > > > 4c fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 > > > e0 > > > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > > 0000000000009ffb > > > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff9e1e1e95b300 > > > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > > 0000000000000eb2 > > > [ 47.732541] FS: 00007f3a82b53700(0000) > > > GS:ffff9e1f2bd00000(0000) > > > knlGS:0000000000000000 > > > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > > 0000000000370ee0 > > > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 47.738318] Call Trace: > > > [ 47.738812] skb_put.cold+0x10/0x10 > > > [ 47.739450] page_to_skb+0xe4/0x400 > > > [ 47.740072] receive_buf+0x86/0x1660 > > > [ 47.740693] ? inet_gro_receive+0x54/0x2c0 > > > [ 47.741279] ? dev_gro_receive+0x194/0x6a0 > > > [ 47.741846] virtnet_poll+0x2b8/0x3c0 > > > [ 47.742357] __napi_poll+0x25/0x150 > > > [ 47.742844] net_rx_action+0x22f/0x280 > > > [ 47.743388] __do_softirq+0xba/0x264 > > > [ 47.743947] irq_exit_rcu+0x90/0xb0 > > > [ 47.744435] common_interrupt+0x40/0xa0 > > > [ 47.744978] ? asm_common_interrupt+0x8/0x40 > > > [ 47.745582] asm_common_interrupt+0x1e/0x40 > > > [ 47.746182] RIP: 0033:0x7f3a7a276ed4 > > > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 bc > > > 54 c8 > > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 c5 > > > d5 fa > > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 55 b8 > > > cb > > > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212 > > > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX: > > > ffffffffffffffff > > > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI: > > > 00007f3a7c0575a0 > > > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09: > > > 00007f3a8c210354 > > > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12: > > > 00007f3a8c210340 > > > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15: > > > 00007f3a8c21033c > > > [ 47.755354] Modules linked in: > > > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]--- > > > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45 > > > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > 00 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 7f > > > 4c fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 c7 c6 > > > e0 > > > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > > 0000000000009ffb > > > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff9e1e1e95b300 > > > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > > 0000000000000eb2 > > > [ 47.766261] FS: 00007f3a82b53700(0000) > > > GS:ffff9e1f2bd00000(0000) > > > knlGS:0000000000000000 > > > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > > 0000000000370ee0 > > > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 47.771339] Kernel panic - not syncing: Fatal exception in > > > interrupt > > > [ 47.772814] Kernel Offset: 0x35c00000 from 0xffffffff81000000 > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > > > I've been able to bisect the issue a little bit and the issue > > > disappeared after reverting the 4 following commits: > > > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7 > > > * af39c8f72301b268ad8b04bae646b6025918b82b > > > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6 > > > * f80bd740cb7c954791279590b2e810ba6c214e52 > > > > > > Here is my kernel config: > > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config > > > > Do you have the same problem with 5.13-rc4? > > > > thanks, > > > > greg k-h > > Yes I tried with rc2, rc3 and rc4 resulting to the same panic. > > Thanks, > Could you provide a stack trace with file names and line numbers ? (ie use scripts/decode_stacktrace.sh ) Thanks. ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-01 17:47 ` Eric Dumazet @ 2021-06-02 17:54 ` Corentin Noël 0 siblings, 0 replies; 21+ messages in thread From: Corentin Noël @ 2021-06-02 17:54 UTC (permalink / raw) To: Eric Dumazet Cc: Greg KH, stable, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Xuan Zhuo Le mardi 01 juin 2021 à 19:47 +0200, Eric Dumazet a écrit : > On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël > <corentin.noel@collabora.com> wrote: > > Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit : > > > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote: > > > > I've been experiencing crashes with 5.13 that do not occur with > > > > 5.12, > > > > here is the crash trace: > > > > > > > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 > > > > len:3762 > > > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 tail:0xec2 > > > > end:0xec0 dev:<NULL> > > > > [ 47.716267] kernel BUG at net/core/skbuff.c:110! > > > > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI > > > > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted > > > > 5.13.0- > > > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1 > > > > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 > > > > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45 > > > > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 > > > > 00 > > > > 00 50 > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 > > > > 7f > > > > 4c fb > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 > > > > c7 c6 > > > > e0 > > > > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > > 00000000ffffdfff > > > > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > > 0000000000000000 > > > > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > > > 0000000000009ffb > > > > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > > ffff9e1e1e95b300 > > > > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > > > 0000000000000eb2 > > > > [ 47.732541] FS: 00007f3a82b53700(0000) > > > > GS:ffff9e1f2bd00000(0000) > > > > knlGS:0000000000000000 > > > > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: > > > > 0000000080050033 > > > > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > > > 0000000000370ee0 > > > > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > > 0000000000000000 > > > > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > > 0000000000000400 > > > > [ 47.738318] Call Trace: > > > > [ 47.738812] skb_put.cold+0x10/0x10 > > > > [ 47.739450] page_to_skb+0xe4/0x400 > > > > [ 47.740072] receive_buf+0x86/0x1660 > > > > [ 47.740693] ? inet_gro_receive+0x54/0x2c0 > > > > [ 47.741279] ? dev_gro_receive+0x194/0x6a0 > > > > [ 47.741846] virtnet_poll+0x2b8/0x3c0 > > > > [ 47.742357] __napi_poll+0x25/0x150 > > > > [ 47.742844] net_rx_action+0x22f/0x280 > > > > [ 47.743388] __do_softirq+0xba/0x264 > > > > [ 47.743947] irq_exit_rcu+0x90/0xb0 > > > > [ 47.744435] common_interrupt+0x40/0xa0 > > > > [ 47.744978] ? asm_common_interrupt+0x8/0x40 > > > > [ 47.745582] asm_common_interrupt+0x1e/0x40 > > > > [ 47.746182] RIP: 0033:0x7f3a7a276ed4 > > > > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 c5 > > > > bc > > > > 54 c8 > > > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 c0 > > > > c5 > > > > d5 fa > > > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 > > > > 55 b8 > > > > cb > > > > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212 > > > > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff RCX: > > > > ffffffffffffffff > > > > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c RDI: > > > > 00007f3a7c0575a0 > > > > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 R09: > > > > 00007f3a8c210354 > > > > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef R12: > > > > 00007f3a8c210340 > > > > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 R15: > > > > 00007f3a8c21033c > > > > [ 47.755354] Modules linked in: > > > > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]--- > > > > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45 > > > > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 > > > > 00 > > > > 00 50 > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 e8 > > > > 7f > > > > 4c fb > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 > > > > c7 c6 > > > > e0 > > > > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > > 00000000ffffdfff > > > > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > > 0000000000000000 > > > > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 R09: > > > > 0000000000009ffb > > > > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > > ffff9e1e1e95b300 > > > > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 R15: > > > > 0000000000000eb2 > > > > [ 47.766261] FS: 00007f3a82b53700(0000) > > > > GS:ffff9e1f2bd00000(0000) > > > > knlGS:0000000000000000 > > > > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: > > > > 0000000080050033 > > > > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 CR4: > > > > 0000000000370ee0 > > > > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > > 0000000000000000 > > > > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > > 0000000000000400 > > > > [ 47.771339] Kernel panic - not syncing: Fatal exception in > > > > interrupt > > > > [ 47.772814] Kernel Offset: 0x35c00000 from > > > > 0xffffffff81000000 > > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > > > > > I've been able to bisect the issue a little bit and the issue > > > > disappeared after reverting the 4 following commits: > > > > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7 > > > > * af39c8f72301b268ad8b04bae646b6025918b82b > > > > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6 > > > > * f80bd740cb7c954791279590b2e810ba6c214e52 > > > > > > > > Here is my kernel config: > > > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config > > > > > > Do you have the same problem with 5.13-rc4? > > > > > > thanks, > > > > > > greg k-h > > > > Yes I tried with rc2, rc3 and rc4 resulting to the same panic. > > > > Thanks, > > > > Could you provide a stack trace with file names and line numbers ? > > (ie use scripts/decode_stacktrace.sh ) > > Thanks. Sure, here is the decoded trace: [ 44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434 len:3762 put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2 end:0xec0 dev:<NULL> [ 44.525254] kernel BUG at net/core/skbuff.c:110! [ 44.525910] invalid opcode: 0000 [#1] SMP PTI [ 44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0- rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1 [ 44.528109] Hardware name: ChromiumOS crosvm, BIOS 0 [ 44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110) [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 4c fb ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 60 All code ======== 0: 4f 70 50 rex.WRXB jo 0x53 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax 9: 50 push %rax a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax 10: 50 push %rax 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0,%rdi 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d 2a:* 0f 0b ud2 <-- trapping instruction 2c: 48 8b 14 24 mov (%rsp),%rdx 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx 37: e8 ab ff ff ff callq 0xffffffffffffffe7 3c: 48 rex.W 3d: c7 .byte 0xc7 3e: c6 (bad) 3f: 60 (bad) Code starting with the faulting instruction =========================================== 0: 0f 0b ud2 2: 48 8b 14 24 mov (%rsp),%rdx 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx d: e8 ab ff ff ff callq 0xffffffffffffffbd 12: 48 rex.W 13: c7 .byte 0xc7 14: c6 (bad) 15: 60 (bad) [ 44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 [ 44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX: 00000000ffffdfff [ 44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: 0000000000000000 [ 44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: 0000000000009ffb [ 44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: ffff979ad2aa5600 [ 44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: 0000000000000eb2 [ 44.539300] FS: 00007fdb9cb11700(0000) GS:ffff979aebd00000(0000) knlGS:0000000000000000 [ 44.540376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: 0000000000370ee0 [ 44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 44.544063] Call Trace: [ 44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator 1) net/core/skbuff.c:5252 (discriminator 1)) [ 44.544864] page_to_skb (drivers/net/virtio_net.c:485) [ 44.545361] receive_buf (drivers/net/virtio_net.c:849 drivers/net/virtio_net.c:1131) [ 44.545870] ? netif_receive_skb_list_internal (net/core/dev.c:5714) [ 44.546628] ? dev_gro_receive (net/core/dev.c:6103) [ 44.547135] ? napi_complete_done (./include/linux/list.h:35 net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565) [ 44.547672] virtnet_poll (drivers/net/virtio_net.c:1427 drivers/net/virtio_net.c:1525) [ 44.548251] __napi_poll (net/core/dev.c:6985) [ 44.548744] net_rx_action (net/core/dev.c:7054 net/core/dev.c:7139) [ 44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19 ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 kernel/softirq.c:560) [ 44.549762] irq_exit_rcu (kernel/softirq.c:433 kernel/softirq.c:637 kernel/softirq.c:649) [ 44.551384] common_interrupt (arch/x86/kernel/irq.c:240 (discriminator 13)) [ 44.551991] ? asm_common_interrupt (./arch/x86/include/asm/idtentry.h:638) [ 44.552654] asm_common_interrupt (./arch/x86/include/asm/idtentry.h:638) [ 44.553276] RIP: 0033:0x7fdb981a82e4 [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09 c4 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5 f9 6c f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5 f9 All code ======== 0: d2 48 63 rorb %cl,0x63(%rax) 3: f6 c4 41 test $0x41,%ah 6: 7a 6f jp 0x77 8: 0c 01 or $0x1,%al a: c4 41 7a 6f 14 09 vmovdqu (%r9,%rcx,1),%xmm10 10: c4 41 7a 6f 24 11 vmovdqu (%r9,%rdx,1),%xmm12 16: c4 41 7a 6f 2c 31 vmovdqu (%r9,%rsi,1),%xmm13 1c: c4 c1 31 6a c2 vpunpckhdq %xmm10,%xmm9,%xmm0 21: c4 c1 19 6a d5 vpunpckhdq %xmm13,%xmm12,%xmm2 26: c5 f9 6c f2 vpunpcklqdq %xmm2,%xmm0,%xmm6 2a:* c5 79 6d c2 vpunpckhqdq %xmm2,%xmm0,%xmm8 <-- trapping instruction 2e: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 33: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xmm0 39: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 3e: c5 .byte 0xc5 3f: f9 stc Code starting with the faulting instruction =========================================== 0: c5 79 6d c2 vpunpckhqdq %xmm2,%xmm0,%xmm8 4: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 9: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xmm0 f: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 14: c5 .byte 0xc5 15: f9 stc [ 44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202 [ 44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX: 0000000000122d40 [ 44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI: 000055d7049b9368 [ 44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09: 00007fdb5e544040 [ 44.560042] R10: 000000000000ffff R11: 000000000000ffff R12: 0000000000000000 [ 44.560991] R13: 0000000000000000 R14: 0000000000005000 R15: 0000000000000000 [ 44.561965] Modules linked in: [ 44.562426] ---[ end trace 9a32eb9d31cb21a1 ]--- [ 44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110) [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 4c fb ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 60 All code ======== 0: 4f 70 50 rex.WRXB jo 0x53 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax 9: 50 push %rax a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax 10: 50 push %rax 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0,%rdi 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d 2a:* 0f 0b ud2 <-- trapping instruction 2c: 48 8b 14 24 mov (%rsp),%rdx 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx 37: e8 ab ff ff ff callq 0xffffffffffffffe7 3c: 48 rex.W 3d: c7 .byte 0xc7 3e: c6 (bad) 3f: 60 (bad) Code starting with the faulting instruction =========================================== 0: 0f 0b ud2 2: 48 8b 14 24 mov (%rsp),%rdx 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320,%rcx d: e8 ab ff ff ff callq 0xffffffffffffffbd 12: 48 rex.W 13: c7 .byte 0xc7 14: c6 (bad) 15: 60 (bad) [ 44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 [ 44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX: 00000000ffffdfff [ 44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: 0000000000000000 [ 44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: 0000000000009ffb [ 44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: ffff979ad2aa5600 [ 44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: 0000000000000eb2 [ 44.571483] FS: 00007fdb9cb11700(0000) GS:ffff979aebd00000(0000) knlGS:0000000000000000 [ 44.572694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: 0000000000370ee0 [ 44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 44.576618] Kernel panic - not syncing: Fatal exception in interrupt [ 44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1622599316.2056065-1-xuanzhuo@linux.alibaba.com>]
* Re: virtio-net: kernel panic in virtio_net.c [not found] <1622599316.2056065-1-xuanzhuo@linux.alibaba.com> @ 2021-06-02 17:56 ` Corentin Noël 0 siblings, 0 replies; 21+ messages in thread From: Corentin Noël @ 2021-06-02 17:56 UTC (permalink / raw) To: Xuan Zhuo, Eric Dumazet Cc: Greg KH, stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions Le mercredi 02 juin 2021 à 10:01 +0800, Xuan Zhuo a écrit : > On Tue, 1 Jun 2021 19:47:44 +0200, Eric Dumazet <edumazet@google.com> > wrote: > > On Tue, Jun 1, 2021 at 7:09 PM Corentin Noël > > <corentin.noel@collabora.com> wrote: > > > Le mardi 01 juin 2021 à 19:07 +0200, Greg KH a écrit : > > > > On Tue, Jun 01, 2021 at 06:06:50PM +0200, Corentin Noël wrote: > > > > > I've been experiencing crashes with 5.13 that do not occur > > > > > with > > > > > 5.12, > > > > > here is the crash trace: > > > > > > > > > > [ 47.713713] skbuff: skb_over_panic: text:ffffffffb73a8354 > > > > > len:3762 > > > > > put:3762 head:ffff9e1e1e48e000 data:ffff9e1e1e48e010 > > > > > tail:0xec2 > > > > > end:0xec0 dev:<NULL> > > > > > [ 47.716267] kernel BUG at net/core/skbuff.c:110! > > > > > [ 47.717197] invalid opcode: 0000 [#1] SMP PTI > > > > > [ 47.718049] CPU: 2 PID: 730 Comm: llvmpipe-0 Not tainted > > > > > 5.13.0- > > > > > rc3linux-v5.13-rc3-for-mesa-ci-87614d7f3282.tar.bz2 #1 > > > > > [ 47.719739] Hardware name: ChromiumOS crosvm, BIOS 0 > > > > > [ 47.720656] RIP: 0010:skb_panic+0x43/0x45 > > > > > [ 47.721426] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 > > > > > 00 00 > > > > > 00 50 > > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 > > > > > e8 7f > > > > > 4c fb > > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 > > > > > c7 c6 > > > > > e0 > > > > > [ 47.725944] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > > > [ 47.726735] RAX: 000000000000008b RBX: 0000000000000010 > > > > > RCX: > > > > > 00000000ffffdfff > > > > > [ 47.727820] RDX: 0000000000000000 RSI: 00000000ffffffea > > > > > RDI: > > > > > 0000000000000000 > > > > > [ 47.729096] RBP: ffffeb2700792380 R08: ffffffffb8144b08 > > > > > R09: > > > > > 0000000000009ffb > > > > > [ 47.730260] R10: 00000000ffffe000 R11: 3fffffffffffffff > > > > > R12: > > > > > ffff9e1e1e95b300 > > > > > [ 47.731411] R13: 0000000000000000 R14: ffff9e1e1e48e000 > > > > > R15: > > > > > 0000000000000eb2 > > > > > [ 47.732541] FS: 00007f3a82b53700(0000) > > > > > GS:ffff9e1f2bd00000(0000) > > > > > knlGS:0000000000000000 > > > > > [ 47.733858] CS: 0010 DS: 0000 ES: 0000 CR0: > > > > > 0000000080050033 > > > > > [ 47.734813] CR2: 00000000010d24f8 CR3: 0000000012d6e004 > > > > > CR4: > > > > > 0000000000370ee0 > > > > > [ 47.735968] DR0: 0000000000000000 DR1: 0000000000000000 > > > > > DR2: > > > > > 0000000000000000 > > > > > [ 47.737091] DR3: 0000000000000000 DR6: 00000000fffe0ff0 > > > > > DR7: > > > > > 0000000000000400 > > > > > [ 47.738318] Call Trace: > > > > > [ 47.738812] skb_put.cold+0x10/0x10 > > > > > [ 47.739450] page_to_skb+0xe4/0x400 > > > > > [ 47.740072] receive_buf+0x86/0x1660 > > > > > [ 47.740693] ? inet_gro_receive+0x54/0x2c0 > > > > > [ 47.741279] ? dev_gro_receive+0x194/0x6a0 > > > > > [ 47.741846] virtnet_poll+0x2b8/0x3c0 > > > > > [ 47.742357] __napi_poll+0x25/0x150 > > > > > [ 47.742844] net_rx_action+0x22f/0x280 > > > > > [ 47.743388] __do_softirq+0xba/0x264 > > > > > [ 47.743947] irq_exit_rcu+0x90/0xb0 > > > > > [ 47.744435] common_interrupt+0x40/0xa0 > > > > > [ 47.744978] ? asm_common_interrupt+0x8/0x40 > > > > > [ 47.745582] asm_common_interrupt+0x1e/0x40 > > > > > [ 47.746182] RIP: 0033:0x7f3a7a276ed4 > > > > > [ 47.746708] Code: a0 03 00 00 c5 fc 29 84 24 40 0f 00 00 > > > > > c5 bc > > > > > 54 c8 > > > > > c5 7c 28 84 24 80 01 00 00 c5 bc 59 e9 c5 fe 5b ed c5 fd 76 > > > > > c0 c5 > > > > > d5 fa > > > > > c0 <c5> fd db ec c5 fd 7f 84 24 20 0f 00 00 c5 fc 5b ed c4 e2 > > > > > 55 b8 > > > > > cb > > > > > [ 47.749292] RSP: 002b:00007f3a82b4dba0 EFLAGS: 00000212 > > > > > [ 47.750006] RAX: 00007f3a8c210324 RBX: ffffffffffffffff > > > > > RCX: > > > > > ffffffffffffffff > > > > > [ 47.750964] RDX: 00007f3a8c210348 RSI: 00007f3a8c21034c > > > > > RDI: > > > > > 00007f3a7c0575a0 > > > > > [ 47.752049] RBP: 00007f3a82b52ca0 R08: 00007f3a8c210350 > > > > > R09: > > > > > 00007f3a8c210354 > > > > > [ 47.753161] R10: 00007f3a8c210358 R11: 000000000000ffef > > > > > R12: > > > > > 00007f3a8c210340 > > > > > [ 47.754260] R13: 00007f3a8c210344 R14: 00007f3a7c057580 > > > > > R15: > > > > > 00007f3a8c21033c > > > > > [ 47.755354] Modules linked in: > > > > > [ 47.755871] ---[ end trace a8b692ea99c9cd9e ]--- > > > > > [ 47.756606] RIP: 0010:skb_panic+0x43/0x45 > > > > > [ 47.757297] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 > > > > > 00 00 > > > > > 00 50 > > > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 78 ae ef b7 > > > > > e8 7f > > > > > 4c fb > > > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 a0 22 d1 b7 e8 ab ff ff ff 48 > > > > > c7 c6 > > > > > e0 > > > > > [ 47.760168] RSP: 0000:ffffacec01347c20 EFLAGS: 00010246 > > > > > [ 47.760896] RAX: 000000000000008b RBX: 0000000000000010 > > > > > RCX: > > > > > 00000000ffffdfff > > > > > [ 47.761903] RDX: 0000000000000000 RSI: 00000000ffffffea > > > > > RDI: > > > > > 0000000000000000 > > > > > [ 47.762945] RBP: ffffeb2700792380 R08: ffffffffb8144b08 > > > > > R09: > > > > > 0000000000009ffb > > > > > [ 47.764059] R10: 00000000ffffe000 R11: 3fffffffffffffff > > > > > R12: > > > > > ffff9e1e1e95b300 > > > > > [ 47.765169] R13: 0000000000000000 R14: ffff9e1e1e48e000 > > > > > R15: > > > > > 0000000000000eb2 > > > > > [ 47.766261] FS: 00007f3a82b53700(0000) > > > > > GS:ffff9e1f2bd00000(0000) > > > > > knlGS:0000000000000000 > > > > > [ 47.767512] CS: 0010 DS: 0000 ES: 0000 CR0: > > > > > 0000000080050033 > > > > > [ 47.768389] CR2: 00000000010d24f8 CR3: 0000000012d6e004 > > > > > CR4: > > > > > 0000000000370ee0 > > > > > [ 47.769381] DR0: 0000000000000000 DR1: 0000000000000000 > > > > > DR2: > > > > > 0000000000000000 > > > > > [ 47.770362] DR3: 0000000000000000 DR6: 00000000fffe0ff0 > > > > > DR7: > > > > > 0000000000000400 > > > > > [ 47.771339] Kernel panic - not syncing: Fatal exception in > > > > > interrupt > > > > > [ 47.772814] Kernel Offset: 0x35c00000 from > > > > > 0xffffffff81000000 > > > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > > > > > > > I've been able to bisect the issue a little bit and the issue > > > > > disappeared after reverting the 4 following commits: > > > > > * fb32856b16ad9d5bcd75b76a274e2c515ac7b9d7 > > > > > * af39c8f72301b268ad8b04bae646b6025918b82b > > > > > * f5d7872a8b8a3176e65dc6f7f0705ce7e9a699e6 > > > > > * f80bd740cb7c954791279590b2e810ba6c214e52 > > > > > > > > > > Here is my kernel config: > > > > > https://gitlab.freedesktop.org/tintou/mesa/-/blob/e5d6c56bfae8522e924217883d2c6a6bfc1b332b/.gitlab-ci/container/x86_64.config > > Do you have XDP running? If so, you can try it > > https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=5c37711d9f27bdc83fd5980446be7f4aa2106230 I applied this patch in top of 5.13-rc4 and it resulted in the same crash > > Thanks. > > > > > Do you have the same problem with 5.13-rc4? > > > > > > > > thanks, > > > > > > > > greg k-h > > > > > > Yes I tried with rc2, rc3 and rc4 resulting to the same panic. > > > > > > Thanks, > > > > > > > Could you provide a stack trace with file names and line numbers ? > > > > (ie use scripts/decode_stacktrace.sh ) > > > > Thanks. ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1622688283.7488964-1-xuanzhuo@linux.alibaba.com>]
* Re: virtio-net: kernel panic in virtio_net.c [not found] <1622688283.7488964-1-xuanzhuo@linux.alibaba.com> @ 2021-06-03 8:57 ` Corentin Noël 2021-06-08 12:17 ` Greg KH 0 siblings, 1 reply; 21+ messages in thread From: Corentin Noël @ 2021-06-03 8:57 UTC (permalink / raw) To: Xuan Zhuo Cc: Greg KH, stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet Le jeudi 03 juin 2021 à 10:44 +0800, Xuan Zhuo a écrit : > On Wed, 02 Jun 2021 19:54:41 +0200, Corentin Noël < > corentin.noel@collabora.com> wrote: > > Sure, here is the decoded trace: > > > > [ 44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434 > > len:3762 > > put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2 > > end:0xec0 dev:<NULL> > > [ 44.525254] kernel BUG at net/core/skbuff.c:110! > > [ 44.525910] invalid opcode: 0000 [#1] SMP PTI > > [ 44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0- > > rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1 > > [ 44.528109] Hardware name: ChromiumOS crosvm, BIOS 0 > > [ 44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110) > > [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 > > 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 > > 4c fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 > > 60 > > All code > > ======== > > 0: 4f 70 50 rex.WRXB jo 0x53 > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > 9: 50 push %rax > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > 10: 50 push %rax > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0, > > %rdi > > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d > > 2a:* 0f 0b ud2 <-- > > trapping > > instruction > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > %rcx > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > 3c: 48 rex.W > > 3d: c7 .byte 0xc7 > > 3e: c6 (bad) > > 3f: 60 (bad) > > > > Code starting with the faulting instruction > > =========================================== > > 0: 0f 0b ud2 > > 2: 48 8b 14 24 mov (%rsp),%rdx > > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > %rcx > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > 12: 48 rex.W > > 13: c7 .byte 0xc7 > > 14: c6 (bad) > > 15: 60 (bad) > > [ 44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 > > [ 44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: > > 0000000000009ffb > > [ 44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff979ad2aa5600 > > [ 44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: > > 0000000000000eb2 > > [ 44.539300] FS: 00007fdb9cb11700(0000) > > GS:ffff979aebd00000(0000) > > knlGS:0000000000000000 > > [ 44.540376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: > > 0000000000370ee0 > > [ 44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 44.544063] Call Trace: > > [ 44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator > > 1) > > net/core/skbuff.c:5252 (discriminator 1)) > > [ 44.544864] page_to_skb (drivers/net/virtio_net.c:485) > > [ 44.545361] receive_buf (drivers/net/virtio_net.c:849 > > drivers/net/virtio_net.c:1131) > > [ 44.545870] ? netif_receive_skb_list_internal > > (net/core/dev.c:5714) > > [ 44.546628] ? dev_gro_receive (net/core/dev.c:6103) > > [ 44.547135] ? napi_complete_done (./include/linux/list.h:35 > > net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565) > > [ 44.547672] virtnet_poll (drivers/net/virtio_net.c:1427 > > drivers/net/virtio_net.c:1525) > > [ 44.548251] __napi_poll (net/core/dev.c:6985) > > [ 44.548744] net_rx_action (net/core/dev.c:7054 > > net/core/dev.c:7139) > > [ 44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19 > > ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 > > kernel/softirq.c:560) > > [ 44.549762] irq_exit_rcu (kernel/softirq.c:433 > > kernel/softirq.c:637 > > kernel/softirq.c:649) > > [ 44.551384] common_interrupt (arch/x86/kernel/irq.c:240 > > (discriminator 13)) > > [ 44.551991] ? asm_common_interrupt > > (./arch/x86/include/asm/idtentry.h:638) > > [ 44.552654] asm_common_interrupt > > (./arch/x86/include/asm/idtentry.h:638) > > [ 44.553276] RIP: 0033:0x7fdb981a82e4 > > [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09 > > c4 > > 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5 > > f9 6c > > f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5 > > f9 > > All code > > ======== > > 0: d2 48 63 rorb %cl,0x63(%rax) > > 3: f6 c4 41 test $0x41,%ah > > 6: 7a 6f jp 0x77 > > 8: 0c 01 or $0x1,%al > > a: c4 41 7a 6f 14 09 vmovdqu (%r9,%rcx,1),%xmm10 > > 10: c4 41 7a 6f 24 11 vmovdqu (%r9,%rdx,1),%xmm12 > > 16: c4 41 7a 6f 2c 31 vmovdqu (%r9,%rsi,1),%xmm13 > > 1c: c4 c1 31 6a c2 vpunpckhdq > > %xmm10,%xmm9,%xmm0 > > 21: c4 c1 19 6a d5 vpunpckhdq > > %xmm13,%xmm12,%xmm2 > > 26: c5 f9 6c f2 vpunpcklqdq > > %xmm2,%xmm0,%xmm6 > > 2a:* c5 79 6d c2 vpunpckhqdq > > %xmm2,%xmm0,%xmm8 > > <-- trapping instruction > > 2e: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 > > 33: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm > > m0 > > 39: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 > > 3e: c5 .byte 0xc5 > > 3f: f9 stc > > > > Code starting with the faulting instruction > > =========================================== > > 0: c5 79 6d c2 vpunpckhqdq > > %xmm2,%xmm0,%xmm8 > > 4: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 > > 9: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm > > m0 > > f: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 > > 14: c5 .byte 0xc5 > > 15: f9 stc > > [ 44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202 > > [ 44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX: > > 0000000000122d40 > > [ 44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI: > > 000055d7049b9368 > > [ 44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09: > > 00007fdb5e544040 > > [ 44.560042] R10: 000000000000ffff R11: 000000000000ffff R12: > > 0000000000000000 > > [ 44.560991] R13: 0000000000000000 R14: 0000000000005000 R15: > > 0000000000000000 > > [ 44.561965] Modules linked in: > > [ 44.562426] ---[ end trace 9a32eb9d31cb21a1 ]--- > > [ 44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110) > > [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 > > 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 > > 4c fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 > > 60 > > All code > > ======== > > 0: 4f 70 50 rex.WRXB jo 0x53 > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > 9: 50 push %rax > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > 10: 50 push %rax > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0, > > %rdi > > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d > > 2a:* 0f 0b ud2 <-- > > trapping > > instruction > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > %rcx > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > 3c: 48 rex.W > > 3d: c7 .byte 0xc7 > > 3e: c6 (bad) > > 3f: 60 (bad) > > > > Code starting with the faulting instruction > > =========================================== > > 0: 0f 0b ud2 > > 2: 48 8b 14 24 mov (%rsp),%rdx > > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > %rcx > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > 12: 48 rex.W > > 13: c7 .byte 0xc7 > > 14: c6 (bad) > > 15: 60 (bad) > > [ 44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 > > [ 44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: > > 0000000000009ffb > > [ 44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff979ad2aa5600 > > [ 44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: > > 0000000000000eb2 > > [ 44.571483] FS: 00007fdb9cb11700(0000) > > GS:ffff979aebd00000(0000) > > knlGS:0000000000000000 > > [ 44.572694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: > > 0000000000370ee0 > > [ 44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 44.576618] Kernel panic - not syncing: Fatal exception in > > interrupt > > [ 44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000 > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > Can you test this patch on the latest net branch? > > Thanks. > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > index fa407eb8b457..78a01c71a17c 100644 > --- a/drivers/net/virtio_net.c > +++ b/drivers/net/virtio_net.c > @@ -406,7 +406,7 @@ static struct sk_buff *page_to_skb(struct > virtnet_info *vi, > * add_recvbuf_mergeable() + get_mergeable_buf_len() > */ > truesize = headroom ? PAGE_SIZE : truesize; > - tailroom = truesize - len - headroom; > + tailroom = truesize - len - headroom - (hdr_padded_len - > hdr_len); > buf = p - headroom; > > len -= hdr_len; With this patch and the latest net branch I no longer get crashes. Thanks a lot for this, Corentin ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-06-03 8:57 ` Corentin Noël @ 2021-06-08 12:17 ` Greg KH [not found] ` <1623203313.4303577-1-xuanzhuo@linux.alibaba.com> 0 siblings, 1 reply; 21+ messages in thread From: Greg KH @ 2021-06-08 12:17 UTC (permalink / raw) To: Corentin Noël Cc: Xuan Zhuo, stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet On Thu, Jun 03, 2021 at 10:57:52AM +0200, Corentin Noël wrote: > Le jeudi 03 juin 2021 à 10:44 +0800, Xuan Zhuo a écrit : > > On Wed, 02 Jun 2021 19:54:41 +0200, Corentin Noël < > > corentin.noel@collabora.com> wrote: > > > Sure, here is the decoded trace: > > > > > > [ 44.523231] skbuff: skb_over_panic: text:ffffffffad1a8434 > > > len:3762 > > > put:3762 head:ffff9799e6b6b000 data:ffff9799e6b6b010 tail:0xec2 > > > end:0xec0 dev:<NULL> > > > [ 44.525254] kernel BUG at net/core/skbuff.c:110! > > > [ 44.525910] invalid opcode: 0000 [#1] SMP PTI > > > [ 44.526521] CPU: 2 PID: 245 Comm: llvmpipe-0 Not tainted 5.13.0- > > > rc4linux-v5.13-rc4-for-mesa-ci-184862285c49.tar.bz2 #1 > > > [ 44.528109] Hardware name: ChromiumOS crosvm, BIOS 0 > > > [ 44.529243] RIP: 0010:skb_panic (net/core/skbuff.c:110) > > > [ 44.530284] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 > > > 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 > > > 4c fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 > > > 60 > > > All code > > > ======== > > > 0: 4f 70 50 rex.WRXB jo 0x53 > > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > > 9: 50 push %rax > > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > > 10: 50 push %rax > > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0, > > > %rdi > > > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d > > > 2a:* 0f 0b ud2 <-- > > > trapping > > > instruction > > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > > %rcx > > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > > 3c: 48 rex.W > > > 3d: c7 .byte 0xc7 > > > 3e: c6 (bad) > > > 3f: 60 (bad) > > > > > > Code starting with the faulting instruction > > > =========================================== > > > 0: 0f 0b ud2 > > > 2: 48 8b 14 24 mov (%rsp),%rdx > > > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > > %rcx > > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > > 12: 48 rex.W > > > 13: c7 .byte 0xc7 > > > 14: c6 (bad) > > > 15: 60 (bad) > > > [ 44.533988] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 > > > [ 44.534723] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 44.535772] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 44.536693] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: > > > 0000000000009ffb > > > [ 44.537569] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff979ad2aa5600 > > > [ 44.538449] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: > > > 0000000000000eb2 > > > [ 44.539300] FS: 00007fdb9cb11700(0000) > > > GS:ffff979aebd00000(0000) > > > knlGS:0000000000000000 > > > [ 44.540376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 44.541103] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: > > > 0000000000370ee0 > > > [ 44.542057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 44.543063] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 44.544063] Call Trace: > > > [ 44.544385] skb_put.cold (net/core/skbuff.c:5254 (discriminator > > > 1) > > > net/core/skbuff.c:5252 (discriminator 1)) > > > [ 44.544864] page_to_skb (drivers/net/virtio_net.c:485) > > > [ 44.545361] receive_buf (drivers/net/virtio_net.c:849 > > > drivers/net/virtio_net.c:1131) > > > [ 44.545870] ? netif_receive_skb_list_internal > > > (net/core/dev.c:5714) > > > [ 44.546628] ? dev_gro_receive (net/core/dev.c:6103) > > > [ 44.547135] ? napi_complete_done (./include/linux/list.h:35 > > > net/core/dev.c:5867 net/core/dev.c:5862 net/core/dev.c:6565) > > > [ 44.547672] virtnet_poll (drivers/net/virtio_net.c:1427 > > > drivers/net/virtio_net.c:1525) > > > [ 44.548251] __napi_poll (net/core/dev.c:6985) > > > [ 44.548744] net_rx_action (net/core/dev.c:7054 > > > net/core/dev.c:7139) > > > [ 44.549264] __do_softirq (./arch/x86/include/asm/jump_label.h:19 > > > ./include/linux/jump_label.h:200 ./include/trace/events/irq.h:142 > > > kernel/softirq.c:560) > > > [ 44.549762] irq_exit_rcu (kernel/softirq.c:433 > > > kernel/softirq.c:637 > > > kernel/softirq.c:649) > > > [ 44.551384] common_interrupt (arch/x86/kernel/irq.c:240 > > > (discriminator 13)) > > > [ 44.551991] ? asm_common_interrupt > > > (./arch/x86/include/asm/idtentry.h:638) > > > [ 44.552654] asm_common_interrupt > > > (./arch/x86/include/asm/idtentry.h:638) > > > [ 44.553276] RIP: 0033:0x7fdb981a82e4 > > > [ 44.553809] Code: d2 48 63 f6 c4 41 7a 6f 0c 01 c4 41 7a 6f 14 09 > > > c4 > > > 41 7a 6f 24 11 c4 41 7a 6f 2c 31 c4 c1 31 6a c2 c4 c1 19 6a d5 c5 > > > f9 6c > > > f2 <c5> 79 6d c2 c5 f9 71 d6 08 c5 f9 db 44 24 20 c5 c1 71 f6 0b c5 > > > f9 > > > All code > > > ======== > > > 0: d2 48 63 rorb %cl,0x63(%rax) > > > 3: f6 c4 41 test $0x41,%ah > > > 6: 7a 6f jp 0x77 > > > 8: 0c 01 or $0x1,%al > > > a: c4 41 7a 6f 14 09 vmovdqu (%r9,%rcx,1),%xmm10 > > > 10: c4 41 7a 6f 24 11 vmovdqu (%r9,%rdx,1),%xmm12 > > > 16: c4 41 7a 6f 2c 31 vmovdqu (%r9,%rsi,1),%xmm13 > > > 1c: c4 c1 31 6a c2 vpunpckhdq > > > %xmm10,%xmm9,%xmm0 > > > 21: c4 c1 19 6a d5 vpunpckhdq > > > %xmm13,%xmm12,%xmm2 > > > 26: c5 f9 6c f2 vpunpcklqdq > > > %xmm2,%xmm0,%xmm6 > > > 2a:* c5 79 6d c2 vpunpckhqdq > > > %xmm2,%xmm0,%xmm8 > > > <-- trapping instruction > > > 2e: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 > > > 33: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm > > > m0 > > > 39: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 > > > 3e: c5 .byte 0xc5 > > > 3f: f9 stc > > > > > > Code starting with the faulting instruction > > > =========================================== > > > 0: c5 79 6d c2 vpunpckhqdq > > > %xmm2,%xmm0,%xmm8 > > > 4: c5 f9 71 d6 08 vpsrlw $0x8,%xmm6,%xmm0 > > > 9: c5 f9 db 44 24 20 vpand 0x20(%rsp),%xmm0,%xm > > > m0 > > > f: c5 c1 71 f6 0b vpsllw $0xb,%xmm6,%xmm7 > > > 14: c5 .byte 0xc5 > > > 15: f9 stc > > > [ 44.556477] RSP: 002b:00007fdb9cb10240 EFLAGS: 00000202 > > > [ 44.557224] RAX: 0000000000122d40 RBX: 00007fdb5f9e8790 RCX: > > > 0000000000122d40 > > > [ 44.558200] RDX: 0000000000122d40 RSI: 0000000000122d40 RDI: > > > 000055d7049b9368 > > > [ 44.559088] RBP: 00007fdb9cb10ba0 R08: 00007fdb981a5174 R09: > > > 00007fdb5e544040 > > > [ 44.560042] R10: 000000000000ffff R11: 000000000000ffff R12: > > > 0000000000000000 > > > [ 44.560991] R13: 0000000000000000 R14: 0000000000005000 R15: > > > 0000000000000000 > > > [ 44.561965] Modules linked in: > > > [ 44.562426] ---[ end trace 9a32eb9d31cb21a1 ]--- > > > [ 44.563091] RIP: 0010:skb_panic (net/core/skbuff.c:110) > > > [ 44.563721] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 > > > 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 f0 af cf ad e8 43 > > > 4c fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 23 b1 ad e8 ab ff ff ff 48 c7 c6 > > > 60 > > > All code > > > ======== > > > 0: 4f 70 50 rex.WRXB jo 0x53 > > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > > 9: 50 push %rax > > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > > 10: 50 push %rax > > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > > 1e: 48 c7 c7 f0 af cf ad mov $0xffffffffadcfaff0, > > > %rdi > > > 25: e8 43 4c fb ff callq 0xfffffffffffb4c6d > > > 2a:* 0f 0b ud2 <-- > > > trapping > > > instruction > > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > > 30: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > > %rcx > > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > > 3c: 48 rex.W > > > 3d: c7 .byte 0xc7 > > > 3e: c6 (bad) > > > 3f: 60 (bad) > > > > > > Code starting with the faulting instruction > > > =========================================== > > > 0: 0f 0b ud2 > > > 2: 48 8b 14 24 mov (%rsp),%rdx > > > 6: 48 c7 c1 20 23 b1 ad mov $0xffffffffadb12320, > > > %rcx > > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > > 12: 48 rex.W > > > 13: c7 .byte 0xc7 > > > 14: c6 (bad) > > > 15: 60 (bad) > > > [ 44.566252] RSP: 0000:ffffa651c134fc20 EFLAGS: 00010246 > > > [ 44.567051] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 44.567947] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 44.568839] RBP: ffffd77b009adac0 R08: ffffffffadf44b08 R09: > > > 0000000000009ffb > > > [ 44.569725] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff979ad2aa5600 > > > [ 44.570608] R13: 0000000000000000 R14: ffff9799e6b6b000 R15: > > > 0000000000000eb2 > > > [ 44.571483] FS: 00007fdb9cb11700(0000) > > > GS:ffff979aebd00000(0000) > > > knlGS:0000000000000000 > > > [ 44.572694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 44.573474] CR2: 00007f99099f4024 CR3: 0000000129558005 CR4: > > > 0000000000370ee0 > > > [ 44.574531] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 44.575597] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 44.576618] Kernel panic - not syncing: Fatal exception in > > > interrupt > > > [ 44.577996] Kernel Offset: 0x2ba00000 from 0xffffffff81000000 > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > > > > Can you test this patch on the latest net branch? > > > > Thanks. > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > > index fa407eb8b457..78a01c71a17c 100644 > > --- a/drivers/net/virtio_net.c > > +++ b/drivers/net/virtio_net.c > > @@ -406,7 +406,7 @@ static struct sk_buff *page_to_skb(struct > > virtnet_info *vi, > > * add_recvbuf_mergeable() + get_mergeable_buf_len() > > */ > > truesize = headroom ? PAGE_SIZE : truesize; > > - tailroom = truesize - len - headroom; > > + tailroom = truesize - len - headroom - (hdr_padded_len - > > hdr_len); > > buf = p - headroom; > > > > len -= hdr_len; > > With this patch and the latest net branch I no longer get crashes. Did this ever get properly submitted to the networking tree to get into 5.13-final? thanks, greg k-h ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1623203313.4303577-1-xuanzhuo@linux.alibaba.com>]
* Re: virtio-net: kernel panic in virtio_net.c [not found] ` <1623203313.4303577-1-xuanzhuo@linux.alibaba.com> @ 2021-06-09 4:50 ` Greg KH [not found] ` <1623218897.4150124-1-xuanzhuo@linux.alibaba.com> 0 siblings, 1 reply; 21+ messages in thread From: Greg KH @ 2021-06-09 4:50 UTC (permalink / raw) To: Xuan Zhuo Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Corentin Noël On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote: > > > With this patch and the latest net branch I no longer get crashes. > > > > Did this ever get properly submitted to the networking tree to get into > > 5.13-final? > > The patch has been submitted. > > [PATCH net] virtio-net: fix for skb_over_panic inside big mode Submitted where? Do you have a lore.kernel.org link somewhere? thanks, greg k-h ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1623218897.4150124-1-xuanzhuo@linux.alibaba.com>]
* Re: virtio-net: kernel panic in virtio_net.c [not found] ` <1623218897.4150124-1-xuanzhuo@linux.alibaba.com> @ 2021-06-09 6:24 ` Greg KH [not found] ` <1623225080.4793522-1-xuanzhuo@linux.alibaba.com> 0 siblings, 1 reply; 21+ messages in thread From: Greg KH @ 2021-06-09 6:24 UTC (permalink / raw) To: Xuan Zhuo Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Corentin Noël On Wed, Jun 09, 2021 at 02:08:17PM +0800, Xuan Zhuo wrote: > On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote: > > > > > With this patch and the latest net branch I no longer get crashes. > > > > > > > > Did this ever get properly submitted to the networking tree to get into > > > > 5.13-final? > > > > > > The patch has been submitted. > > > > > > [PATCH net] virtio-net: fix for skb_over_panic inside big mode > > > > Submitted where? Do you have a lore.kernel.org link somewhere? > > > https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/ So this is commit 1a8024239dac ("virtio-net: fix for skb_over_panic inside big mode") in Linus's tree, right? But why is that referencing: Fixes: fb32856b16ad ("virtio-net: page_to_skb() use build_skb when there's sufficient tailroom") when this problem was seen in stable kernels that had a different commit backported to it? Is there nothing needed to be done for the stable kernel trees? confused, greg k-h ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1623225080.4793522-1-xuanzhuo@linux.alibaba.com>]
* Re: virtio-net: kernel panic in virtio_net.c [not found] ` <1623225080.4793522-1-xuanzhuo@linux.alibaba.com> @ 2021-06-09 8:03 ` Greg KH 0 siblings, 0 replies; 21+ messages in thread From: Greg KH @ 2021-06-09 8:03 UTC (permalink / raw) To: Xuan Zhuo Cc: stable, Michael S.Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, Corentin Noël On Wed, Jun 09, 2021 at 03:51:20PM +0800, Xuan Zhuo wrote: > On Wed, 9 Jun 2021 08:24:20 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > On Wed, Jun 09, 2021 at 02:08:17PM +0800, Xuan Zhuo wrote: > > > On Wed, 9 Jun 2021 06:50:10 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > > > On Wed, Jun 09, 2021 at 09:48:33AM +0800, Xuan Zhuo wrote: > > > > > > > With this patch and the latest net branch I no longer get crashes. > > > > > > > > > > > > Did this ever get properly submitted to the networking tree to get into > > > > > > 5.13-final? > > > > > > > > > > The patch has been submitted. > > > > > > > > > > [PATCH net] virtio-net: fix for skb_over_panic inside big mode > > > > > > > > Submitted where? Do you have a lore.kernel.org link somewhere? > > > > > > > > > https://lore.kernel.org/netdev/20210603170901.66504-1-xuanzhuo@linux.alibaba.com/ > > > > So this is commit 1a8024239dac ("virtio-net: fix for skb_over_panic > > inside big mode") in Linus's tree, right? > > YES. > > > > > But why is that referencing: > > Fixes: fb32856b16ad ("virtio-net: page_to_skb() use build_skb when there's sufficient tailroom") > > This problem was indeed introduced in fb32856b16ad. > > I confirmed that this commit fb32856b16ad was first entered in 5.13-rc1, and the > previous 5.12 did not have this commit fb32856b16ad. > > I'm not sure if it helped you. Hm, then what resolves the reported problem that people were having with the 5.12.y kernel release? Is that a separate issue? thanks, greg k-h ^ permalink raw reply [flat|nested] 21+ messages in thread
* virtio-net: kernel panic in virtio_net.c
@ 2021-10-07 12:04 Corentin Noël
2021-10-07 13:10 ` Michael S. Tsirkin
[not found] ` <1633619172.5342586-1-xuanzhuo@linux.alibaba.com>
0 siblings, 2 replies; 21+ messages in thread
From: Corentin Noël @ 2021-10-07 12:04 UTC (permalink / raw)
To: stable
Cc: Michael S. Tsirkin, Jason Wang, virtualization, regressions,
Eric Dumazet, Xuan Zhuo
I've been experiencing crashes with 5.14-rc1 and above that do not
occur with 5.13,
here is the crash trace:
[ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
end:0xec0 dev:<NULL>
[ 61.369192] kernel BUG at net/core/skbuff.c:111!
[ 61.372840] invalid opcode: 0000 [#1] SMP PTI
[ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
[ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
[ 61.377222] RIP: 0010:skb_panic+0x43/0x45
[ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
[ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
[ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
0000000000370ee0
[ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 61.393635] Call Trace:
[ 61.394127] <IRQ>
[ 61.394488] skb_put.cold+0x10/0x10
[ 61.395095] page_to_skb+0xf7/0x410
[ 61.395689] receive_buf+0x81/0x1660
[ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
[ 61.397180] ? napi_gro_flush+0x97/0xe0
[ 61.397896] ? detach_buf_split+0x67/0x120
[ 61.398573] virtnet_poll+0x2cf/0x420
[ 61.399197] __napi_poll+0x25/0x150
[ 61.399764] net_rx_action+0x22f/0x280
[ 61.400394] __do_softirq+0xba/0x257
[ 61.401012] irq_exit_rcu+0x8e/0xb0
[ 61.401618] common_interrupt+0x7b/0xa0
[ 61.402270] </IRQ>
[ 61.402620] asm_common_interrupt+0x1e/0x40
[ 61.403302] RIP: 0010:default_idle+0xb/0x10
[ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
[ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
[ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
ffff8a5febd56f80
[ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
ffff8a5febd5dd00
[ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
0000000000000000
[ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
0000000000000000
[ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 61.414183] ? mwait_idle+0x70/0x70
[ 61.414805] ? mwait_idle+0x70/0x70
[ 61.415592] default_idle_call+0x2a/0xa0
[ 61.416216] do_idle+0x1e8/0x250
[ 61.416722] cpu_startup_entry+0x14/0x20
[ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb
[ 61.418144] Modules linked in:
[ 61.418622] ---[ end trace 3741c3e580a52bbd ]---
[ 61.419399] RIP: 0010:skb_panic+0x43/0x45
[ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
[ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
[ 61.431048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
0000000000370ee0
[ 61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 61.435799] Kernel panic - not syncing: Fatal exception in interrupt
[ 61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Here is my kernel config:
https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config
here is the decoded trace:
[ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762
put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2
end:0xec0 dev:<NULL>
[ 61.369192] kernel BUG at net/core/skbuff.c:111!
[ 61.372840] invalid opcode: 0000 [#1] SMP PTI
[ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0-
rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1
[ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0
[ 61.377222] RIP: skb_panic+0x43/0x45
[ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
All code
========
0: 4f 70 50 rex.WRXB jo 0x53
3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
9: 50 push %rax
a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
10: 50 push %rax
11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi
25: e8 6a 43 fb ff callq 0xfffffffffffb4394
2a:* 0f 0b ud2 <-- trapping
instruction
2c: 48 8b 14 24 mov (%rsp),%rdx
30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
37: e8 ab ff ff ff callq 0xffffffffffffffe7
3c: 48 rex.W
3d: c7 .byte 0xc7
3e: c6 (bad)
3f: 60 (bad)
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: 48 8b 14 24 mov (%rsp),%rdx
6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
d: e8 ab ff ff ff callq 0xffffffffffffffbd
12: 48 rex.W
13: c7 .byte 0xc7
14: c6 (bad)
15: 60 (bad)
[ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
[ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4:
0000000000370ee0
[ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 61.393635] Call Trace:
[ 61.394127] <IRQ>
[ 61.394488] skb_put.cold+0x10/0x10
[ 61.395095] page_to_skb+0xf7/0x410
[ 61.395689] receive_buf+0x81/0x1660
[ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0
[ 61.397180] ? napi_gro_flush+0x97/0xe0
[ 61.397896] ? detach_buf_split+0x67/0x120
[ 61.398573] virtnet_poll+0x2cf/0x420
[ 61.399197] __napi_poll+0x25/0x150
[ 61.399764] net_rx_action+0x22f/0x280
[ 61.400394] __do_softirq+0xba/0x257
[ 61.401012] irq_exit_rcu+0x8e/0xb0
[ 61.401618] common_interrupt+0x7b/0xa0
[ 61.402270] </IRQ>
[ 61.402620] asm_common_interrupt+0x1e/0x40
[ 61.403302] RIP: default_idle+0xb/0x10
[ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f
ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb
f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48
All code
========
0: 8b 04 25 00 6d 01 00 mov 0x16d00,%eax
7: f0 80 60 02 df lock andb $0xdf,0x2(%rax)
c: c3 retq
d: 0f ae f0 mfence
10: 0f ae 38 clflush (%rax)
13: 0f ae f0 mfence
16: eb b9 jmp 0xffffffffffffffd1
18: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
1f: eb 07 jmp 0x28
21: 0f 00 2d df 3e 44 00 verw 0x443edf(%rip) # 0x443f07
28: fb sti
29: f4 hlt
2a:* c3 retq <-- trapping
instruction
2b: cc int3
2c: cc int3
2d: cc int3
2e: cc int3
2f: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx
# 0x77a42f67
36: 89 d2 mov %edx,%edx
38: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax #
0x10ca20f
3f: 48 rex.W
Code starting with the faulting instruction
===========================================
0: c3 retq
1: cc int3
2: cc int3
3: cc int3
4: cc int3
5: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx
# 0x77a42f3d
c: 89 d2 mov %edx,%edx
e: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax #
0x10ca1e5
15: 48 rex.W
[ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202
[ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX:
ffff8a5febd56f80
[ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI:
ffff8a5febd5dd00
[ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09:
0000000000000000
[ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12:
0000000000000000
[ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 61.414183] ? mwait_idle+0x70/0x70
[ 61.414805] ? mwait_idle+0x70/0x70
[ 61.415592] default_idle_call+0x2a/0xa0
[ 61.416216] do_idle+0x1e8/0x250
[ 61.416722] cpu_startup_entry+0x14/0x20
[ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb
[ 61.418144] Modules linked in:
[ 61.418622] ---[ end trace 3741c3e580a52bbd ]---
[ 61.419399] RIP: skb_panic+0x43/0x45
[ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50
ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb
ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60
All code
========
0: 4f 70 50 rex.WRXB jo 0x53
3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax
9: 50 push %rax
a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax
10: 50 push %rax
11: ff b7 c8 00 00 00 pushq 0xc8(%rdi)
17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9
1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi
25: e8 6a 43 fb ff callq 0xfffffffffffb4394
2a:* 0f 0b ud2 <-- trapping
instruction
2c: 48 8b 14 24 mov (%rsp),%rdx
30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
37: e8 ab ff ff ff callq 0xffffffffffffffe7
3c: 48 rex.W
3d: c7 .byte 0xc7
3e: c6 (bad)
3f: 60 (bad)
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: 48 8b 14 24 mov (%rsp),%rdx
6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx
d: e8 ab ff ff ff callq 0xffffffffffffffbd
12: 48 rex.W
13: c7 .byte 0xc7
14: c6 (bad)
15: 60 (bad)
[ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246
[ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX:
00000000ffffdfff
[ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI:
0000000000000000
[ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09:
0000000000009ffb
[ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12:
ffff8a5ec7461200
[ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15:
0000000000000eb2
[ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000)
knlGS:0000000000000000
Regards,
Corentin
^ permalink raw reply [flat|nested] 21+ messages in thread* Re: virtio-net: kernel panic in virtio_net.c 2021-10-07 12:04 Corentin Noël @ 2021-10-07 13:10 ` Michael S. Tsirkin 2021-10-07 13:51 ` Eric Dumazet [not found] ` <1633619172.5342586-1-xuanzhuo@linux.alibaba.com> 1 sibling, 1 reply; 21+ messages in thread From: Michael S. Tsirkin @ 2021-10-07 13:10 UTC (permalink / raw) To: Corentin Noël Cc: stable, Jason Wang, virtualization, regressions, Eric Dumazet, Xuan Zhuo On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote: > I've been experiencing crashes with 5.14-rc1 and above that do not > occur with 5.13, > > here is the crash trace: > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762 > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > end:0xec0 dev:<NULL> > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > [ 61.377222] RIP: 0010:skb_panic+0x43/0x45 > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > 0000000000009ffb > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff8a5ec7461200 > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > 0000000000000eb2 > [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > knlGS:0000000000000000 > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > 0000000000370ee0 > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 61.393635] Call Trace: > [ 61.394127] <IRQ> > [ 61.394488] skb_put.cold+0x10/0x10 > [ 61.395095] page_to_skb+0xf7/0x410 > [ 61.395689] receive_buf+0x81/0x1660 > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > [ 61.397896] ? detach_buf_split+0x67/0x120 > [ 61.398573] virtnet_poll+0x2cf/0x420 > [ 61.399197] __napi_poll+0x25/0x150 > [ 61.399764] net_rx_action+0x22f/0x280 > [ 61.400394] __do_softirq+0xba/0x257 > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > [ 61.401618] common_interrupt+0x7b/0xa0 > [ 61.402270] </IRQ> > [ 61.402620] asm_common_interrupt+0x1e/0x40 > [ 61.403302] RIP: 0010:default_idle+0xb/0x10 > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48 > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > ffff8a5febd56f80 > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > ffff8a5febd5dd00 > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > 0000000000000000 > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > 0000000000000000 > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > [ 61.414183] ? mwait_idle+0x70/0x70 > [ 61.414805] ? mwait_idle+0x70/0x70 > [ 61.415592] default_idle_call+0x2a/0xa0 > [ 61.416216] do_idle+0x1e8/0x250 > [ 61.416722] cpu_startup_entry+0x14/0x20 > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > [ 61.418144] Modules linked in: > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > [ 61.419399] RIP: 0010:skb_panic+0x43/0x45 > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > 0000000000009ffb > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff8a5ec7461200 > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > 0000000000000eb2 > [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > knlGS:0000000000000000 > [ 61.431048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > 0000000000370ee0 > [ 61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 61.435799] Kernel panic - not syncing: Fatal exception in interrupt > [ 61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000 > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > Here is my kernel config: > https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config > > > here is the decoded trace: > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762 > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > end:0xec0 dev:<NULL> > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > [ 61.377222] RIP: skb_panic+0x43/0x45 > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > All code > ======== > 0: 4f 70 50 rex.WRXB jo 0x53 > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > 9: 50 push %rax > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > 10: 50 push %rax > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > 2a:* 0f 0b ud2 <-- trapping > instruction > 2c: 48 8b 14 24 mov (%rsp),%rdx > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > 3c: 48 rex.W > 3d: c7 .byte 0xc7 > 3e: c6 (bad) > 3f: 60 (bad) > > Code starting with the faulting instruction > =========================================== > 0: 0f 0b ud2 > 2: 48 8b 14 24 mov (%rsp),%rdx > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > d: e8 ab ff ff ff callq 0xffffffffffffffbd > 12: 48 rex.W > 13: c7 .byte 0xc7 > 14: c6 (bad) > 15: 60 (bad) > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > 0000000000009ffb > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff8a5ec7461200 > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > 0000000000000eb2 > [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > knlGS:0000000000000000 > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > 0000000000370ee0 > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 61.393635] Call Trace: > [ 61.394127] <IRQ> > [ 61.394488] skb_put.cold+0x10/0x10 > [ 61.395095] page_to_skb+0xf7/0x410 > [ 61.395689] receive_buf+0x81/0x1660 > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > [ 61.397896] ? detach_buf_split+0x67/0x120 > [ 61.398573] virtnet_poll+0x2cf/0x420 > [ 61.399197] __napi_poll+0x25/0x150 > [ 61.399764] net_rx_action+0x22f/0x280 > [ 61.400394] __do_softirq+0xba/0x257 > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > [ 61.401618] common_interrupt+0x7b/0xa0 > [ 61.402270] </IRQ> > [ 61.402620] asm_common_interrupt+0x1e/0x40 > [ 61.403302] RIP: default_idle+0xb/0x10 > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48 > All code > ======== > 0: 8b 04 25 00 6d 01 00 mov 0x16d00,%eax > 7: f0 80 60 02 df lock andb $0xdf,0x2(%rax) > c: c3 retq > d: 0f ae f0 mfence > 10: 0f ae 38 clflush (%rax) > 13: 0f ae f0 mfence > 16: eb b9 jmp 0xffffffffffffffd1 > 18: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) > 1f: eb 07 jmp 0x28 > 21: 0f 00 2d df 3e 44 00 verw 0x443edf(%rip) # 0x443f07 > 28: fb sti > 29: f4 hlt > 2a:* c3 retq <-- trapping > instruction > 2b: cc int3 > 2c: cc int3 > 2d: cc int3 > 2e: cc int3 > 2f: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > # 0x77a42f67 > 36: 89 d2 mov %edx,%edx > 38: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax # > 0x10ca20f > 3f: 48 rex.W > > Code starting with the faulting instruction > =========================================== > 0: c3 retq > 1: cc int3 > 2: cc int3 > 3: cc int3 > 4: cc int3 > 5: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > # 0x77a42f3d > c: 89 d2 mov %edx,%edx > e: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax # > 0x10ca1e5 > 15: 48 rex.W > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > ffff8a5febd56f80 > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > ffff8a5febd5dd00 > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > 0000000000000000 > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > 0000000000000000 > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > 0000000000000000 > [ 61.414183] ? mwait_idle+0x70/0x70 > [ 61.414805] ? mwait_idle+0x70/0x70 > [ 61.415592] default_idle_call+0x2a/0xa0 > [ 61.416216] do_idle+0x1e8/0x250 > [ 61.416722] cpu_startup_entry+0x14/0x20 > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > [ 61.418144] Modules linked in: > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > [ 61.419399] RIP: skb_panic+0x43/0x45 > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > All code > ======== > 0: 4f 70 50 rex.WRXB jo 0x53 > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > 9: 50 push %rax > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > 10: 50 push %rax > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > 2a:* 0f 0b ud2 <-- trapping > instruction > 2c: 48 8b 14 24 mov (%rsp),%rdx > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > 3c: 48 rex.W > 3d: c7 .byte 0xc7 > 3e: c6 (bad) > 3f: 60 (bad) > > Code starting with the faulting instruction > =========================================== > 0: 0f 0b ud2 > 2: 48 8b 14 24 mov (%rsp),%rdx > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > d: e8 ab ff ff ff callq 0xffffffffffffffbd > 12: 48 rex.W > 13: c7 .byte 0xc7 > 14: c6 (bad) > 15: 60 (bad) > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > 00000000ffffdfff > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > 0000000000000000 > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > 0000000000009ffb > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > ffff8a5ec7461200 > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > 0000000000000eb2 > [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > knlGS:0000000000000000 > > Regards, > Corentin Don't see anything obvious.. could be a net stack change. Any chance of a bisect? -- MST ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-07 13:10 ` Michael S. Tsirkin @ 2021-10-07 13:51 ` Eric Dumazet 2021-10-07 14:02 ` Corentin Noël 0 siblings, 1 reply; 21+ messages in thread From: Eric Dumazet @ 2021-10-07 13:51 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Corentin Noël, linux-stable, Jason Wang, virtualization, regressions, Xuan Zhuo On Thu, Oct 7, 2021 at 6:11 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote: > > I've been experiencing crashes with 5.14-rc1 and above that do not > > occur with 5.13, What about 5.14 ? 5.14-rc1 has many bugs we do not want to spend time rediscovering them... > > > > here is the crash trace: > > > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762 > > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > > end:0xec0 dev:<NULL> > > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > > [ 61.377222] RIP: 0010:skb_panic+0x43/0x45 > > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > 0000000000009ffb > > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff8a5ec7461200 > > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > 0000000000000eb2 > > [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > > knlGS:0000000000000000 > > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > > 0000000000370ee0 > > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 61.393635] Call Trace: > > [ 61.394127] <IRQ> > > [ 61.394488] skb_put.cold+0x10/0x10 > > [ 61.395095] page_to_skb+0xf7/0x410 > > [ 61.395689] receive_buf+0x81/0x1660 > > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > > [ 61.397896] ? detach_buf_split+0x67/0x120 > > [ 61.398573] virtnet_poll+0x2cf/0x420 > > [ 61.399197] __napi_poll+0x25/0x150 > > [ 61.399764] net_rx_action+0x22f/0x280 > > [ 61.400394] __do_softirq+0xba/0x257 > > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > > [ 61.401618] common_interrupt+0x7b/0xa0 > > [ 61.402270] </IRQ> > > [ 61.402620] asm_common_interrupt+0x1e/0x40 > > [ 61.403302] RIP: 0010:default_idle+0xb/0x10 > > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f > > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb > > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48 > > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > > ffff8a5febd56f80 > > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > > ffff8a5febd5dd00 > > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > > 0000000000000000 > > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > > 0000000000000000 > > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > > 0000000000000000 > > [ 61.414183] ? mwait_idle+0x70/0x70 > > [ 61.414805] ? mwait_idle+0x70/0x70 > > [ 61.415592] default_idle_call+0x2a/0xa0 > > [ 61.416216] do_idle+0x1e8/0x250 > > [ 61.416722] cpu_startup_entry+0x14/0x20 > > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > > [ 61.418144] Modules linked in: > > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > > [ 61.419399] RIP: 0010:skb_panic+0x43/0x45 > > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > 0000000000009ffb > > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff8a5ec7461200 > > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > 0000000000000eb2 > > [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > > knlGS:0000000000000000 > > [ 61.431048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > > 0000000000370ee0 > > [ 61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 61.435799] Kernel panic - not syncing: Fatal exception in interrupt > > [ 61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000 > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > Here is my kernel config: > > https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config > > > > > > here is the decoded trace: > > > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 len:3762 > > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > > end:0xec0 dev:<NULL> > > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > > [ 61.377222] RIP: skb_panic+0x43/0x45 > > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > > All code > > ======== > > 0: 4f 70 50 rex.WRXB jo 0x53 > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > 9: 50 push %rax > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > 10: 50 push %rax > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > > 2a:* 0f 0b ud2 <-- trapping > > instruction > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > 3c: 48 rex.W > > 3d: c7 .byte 0xc7 > > 3e: c6 (bad) > > 3f: 60 (bad) > > > > Code starting with the faulting instruction > > =========================================== > > 0: 0f 0b ud2 > > 2: 48 8b 14 24 mov (%rsp),%rdx > > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > 12: 48 rex.W > > 13: c7 .byte 0xc7 > > 14: c6 (bad) > > 15: 60 (bad) > > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > 0000000000009ffb > > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff8a5ec7461200 > > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > 0000000000000eb2 > > [ 61.386825] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > > knlGS:0000000000000000 > > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > > 0000000000370ee0 > > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > 0000000000000000 > > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > 0000000000000400 > > [ 61.393635] Call Trace: > > [ 61.394127] <IRQ> > > [ 61.394488] skb_put.cold+0x10/0x10 > > [ 61.395095] page_to_skb+0xf7/0x410 > > [ 61.395689] receive_buf+0x81/0x1660 > > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > > [ 61.397896] ? detach_buf_split+0x67/0x120 > > [ 61.398573] virtnet_poll+0x2cf/0x420 > > [ 61.399197] __napi_poll+0x25/0x150 > > [ 61.399764] net_rx_action+0x22f/0x280 > > [ 61.400394] __do_softirq+0xba/0x257 > > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > > [ 61.401618] common_interrupt+0x7b/0xa0 > > [ 61.402270] </IRQ> > > [ 61.402620] asm_common_interrupt+0x1e/0x40 > > [ 61.403302] RIP: default_idle+0xb/0x10 > > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae f0 0f > > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 00 fb > > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c 01 48 > > All code > > ======== > > 0: 8b 04 25 00 6d 01 00 mov 0x16d00,%eax > > 7: f0 80 60 02 df lock andb $0xdf,0x2(%rax) > > c: c3 retq > > d: 0f ae f0 mfence > > 10: 0f ae 38 clflush (%rax) > > 13: 0f ae f0 mfence > > 16: eb b9 jmp 0xffffffffffffffd1 > > 18: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) > > 1f: eb 07 jmp 0x28 > > 21: 0f 00 2d df 3e 44 00 verw 0x443edf(%rip) # 0x443f07 > > 28: fb sti > > 29: f4 hlt > > 2a:* c3 retq <-- trapping > > instruction > > 2b: cc int3 > > 2c: cc int3 > > 2d: cc int3 > > 2e: cc int3 > > 2f: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > > # 0x77a42f67 > > 36: 89 d2 mov %edx,%edx > > 38: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax # > > 0x10ca20f > > 3f: 48 rex.W > > > > Code starting with the faulting instruction > > =========================================== > > 0: c3 retq > > 1: cc int3 > > 2: cc int3 > > 3: cc int3 > > 4: cc int3 > > 5: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > > # 0x77a42f3d > > c: 89 d2 mov %edx,%edx > > e: 48 8b 05 d0 a1 0c 01 mov 0x10ca1d0(%rip),%rax # > > 0x10ca1e5 > > 15: 48 rex.W > > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > > ffff8a5febd56f80 > > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > > ffff8a5febd5dd00 > > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > > 0000000000000000 > > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > > 0000000000000000 > > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > > 0000000000000000 > > [ 61.414183] ? mwait_idle+0x70/0x70 > > [ 61.414805] ? mwait_idle+0x70/0x70 > > [ 61.415592] default_idle_call+0x2a/0xa0 > > [ 61.416216] do_idle+0x1e8/0x250 > > [ 61.416722] cpu_startup_entry+0x14/0x20 > > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > > [ 61.418144] Modules linked in: > > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > > [ 61.419399] RIP: skb_panic+0x43/0x45 > > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 00 50 > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a 43 fb > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 c6 60 > > All code > > ======== > > 0: 4f 70 50 rex.WRXB jo 0x53 > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > 9: 50 push %rax > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > 10: 50 push %rax > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > > 2a:* 0f 0b ud2 <-- trapping > > instruction > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > 3c: 48 rex.W > > 3d: c7 .byte 0xc7 > > 3e: c6 (bad) > > 3f: 60 (bad) > > > > Code starting with the faulting instruction > > =========================================== > > 0: 0f 0b ud2 > > 2: 48 8b 14 24 mov (%rsp),%rdx > > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > 12: 48 rex.W > > 13: c7 .byte 0xc7 > > 14: c6 (bad) > > 15: 60 (bad) > > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > 00000000ffffdfff > > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > 0000000000000000 > > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > 0000000000009ffb > > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > ffff8a5ec7461200 > > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > 0000000000000eb2 > > [ 61.429799] FS: 0000000000000000(0000) GS:ffff8a5febd40000(0000) > > knlGS:0000000000000000 > > > > Regards, > > Corentin > > Don't see anything obvious.. could be a net stack change. > Any chance of a bisect? > > -- > MST > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-07 13:51 ` Eric Dumazet @ 2021-10-07 14:02 ` Corentin Noël 2021-10-07 14:13 ` Greg KH 0 siblings, 1 reply; 21+ messages in thread From: Corentin Noël @ 2021-10-07 14:02 UTC (permalink / raw) To: Eric Dumazet, Michael S. Tsirkin Cc: linux-stable, Jason Wang, virtualization, regressions, Xuan Zhuo Le jeudi 07 octobre 2021 à 06:51 -0700, Eric Dumazet a écrit : > On Thu, Oct 7, 2021 at 6:11 AM Michael S. Tsirkin <mst@redhat.com> > wrote: > > On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote: > > > I've been experiencing crashes with 5.14-rc1 and above that do > > > not > > > occur with 5.13, > > What about 5.14 ? > > 5.14-rc1 has many bugs we do not want to spend time rediscovering > them... > I've tested on 5.14, 5.15-rc4 and 5.15-rc4 with latest netdev and could reproduce the crash on them all. > > > here is the crash trace: > > > > > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 > > > len:3762 > > > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > > > end:0xec0 dev:<NULL> > > > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > > > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > > > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > > > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > > > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > > > [ 61.377222] RIP: 0010:skb_panic+0x43/0x45 > > > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > 00 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a > > > 43 fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 > > > c6 60 > > > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > > 0000000000009ffb > > > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff8a5ec7461200 > > > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > > 0000000000000eb2 > > > [ 61.386825] FS: 0000000000000000(0000) > > > GS:ffff8a5febd40000(0000) > > > knlGS:0000000000000000 > > > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > > > 0000000000370ee0 > > > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 61.393635] Call Trace: > > > [ 61.394127] <IRQ> > > > [ 61.394488] skb_put.cold+0x10/0x10 > > > [ 61.395095] page_to_skb+0xf7/0x410 > > > [ 61.395689] receive_buf+0x81/0x1660 > > > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > > > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > > > [ 61.397896] ? detach_buf_split+0x67/0x120 > > > [ 61.398573] virtnet_poll+0x2cf/0x420 > > > [ 61.399197] __napi_poll+0x25/0x150 > > > [ 61.399764] net_rx_action+0x22f/0x280 > > > [ 61.400394] __do_softirq+0xba/0x257 > > > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > > > [ 61.401618] common_interrupt+0x7b/0xa0 > > > [ 61.402270] </IRQ> > > > [ 61.402620] asm_common_interrupt+0x1e/0x40 > > > [ 61.403302] RIP: 0010:default_idle+0xb/0x10 > > > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae > > > f0 0f > > > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 > > > 00 fb > > > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c > > > 01 48 > > > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > > > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > > > ffff8a5febd56f80 > > > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > > > ffff8a5febd5dd00 > > > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > > > 0000000000000000 > > > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > > > 0000000000000000 > > > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > > > 0000000000000000 > > > [ 61.414183] ? mwait_idle+0x70/0x70 > > > [ 61.414805] ? mwait_idle+0x70/0x70 > > > [ 61.415592] default_idle_call+0x2a/0xa0 > > > [ 61.416216] do_idle+0x1e8/0x250 > > > [ 61.416722] cpu_startup_entry+0x14/0x20 > > > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > > > [ 61.418144] Modules linked in: > > > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > > > [ 61.419399] RIP: 0010:skb_panic+0x43/0x45 > > > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > 00 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a > > > 43 fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 > > > c6 60 > > > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > > 0000000000009ffb > > > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff8a5ec7461200 > > > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > > 0000000000000eb2 > > > [ 61.429799] FS: 0000000000000000(0000) > > > GS:ffff8a5febd40000(0000) > > > knlGS:0000000000000000 > > > [ 61.431048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 61.431997] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > > > 0000000000370ee0 > > > [ 61.433206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 61.434502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 61.435799] Kernel panic - not syncing: Fatal exception in > > > interrupt > > > [ 61.439250] Kernel Offset: 0x6a00000 from 0xffffffff81000000 > > > (relocation range: 0xffffffff80000000-0xffffffffbfffffff) > > > > > > Here is my kernel config: > > > https://gitlab.freedesktop.org/tintou/mesa/-/raw/7cf2be0e1c53d1040ff8a973ddeeeb3d93250f8e/.gitlab-ci/container/x86_64.config > > > > > > > > > here is the decoded trace: > > > > > > [ 61.346677] skbuff: skb_over_panic: text:ffffffff881ae2c7 > > > len:3762 > > > put:3762 head:ffff8a5ec8c22000 data:ffff8a5ec8c22010 tail:0xec2 > > > end:0xec0 dev:<NULL> > > > [ 61.369192] kernel BUG at net/core/skbuff.c:111! > > > [ 61.372840] invalid opcode: 0000 [#1] SMP PTI > > > [ 61.374892] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.14.0- > > > rc1linux-v5.14-rc1-for-mesa-ci.tar.bz2 #1 > > > [ 61.376450] Hardware name: ChromiumOS crosvm, BIOS 0 > > > [ 61.377222] RIP: skb_panic+0x43/0x45 > > > [ 61.377833] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > 00 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a > > > 43 fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 > > > c6 60 > > > All code > > > ======== > > > 0: 4f 70 50 rex.WRXB jo 0x53 > > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > > 9: 50 push %rax > > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > > 10: 50 push %rax > > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > > > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > > > 2a:* 0f 0b ud2 <-- > > > trapping > > > instruction > > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > > 3c: 48 rex.W > > > 3d: c7 .byte 0xc7 > > > 3e: c6 (bad) > > > 3f: 60 (bad) > > > > > > Code starting with the faulting instruction > > > =========================================== > > > 0: 0f 0b ud2 > > > 2: 48 8b 14 24 mov (%rsp),%rdx > > > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > > 12: 48 rex.W > > > 13: c7 .byte 0xc7 > > > 14: c6 (bad) > > > 15: 60 (bad) > > > [ 61.380566] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > > [ 61.381267] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 61.382246] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 61.383376] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > > 0000000000009ffb > > > [ 61.384494] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff8a5ec7461200 > > > [ 61.385696] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > > 0000000000000eb2 > > > [ 61.386825] FS: 0000000000000000(0000) > > > GS:ffff8a5febd40000(0000) > > > knlGS:0000000000000000 > > > [ 61.388055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 61.389221] CR2: 000000000148a060 CR3: 000000011ae0e005 CR4: > > > 0000000000370ee0 > > > [ 61.390871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > > > 0000000000000000 > > > [ 61.392335] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > > > 0000000000000400 > > > [ 61.393635] Call Trace: > > > [ 61.394127] <IRQ> > > > [ 61.394488] skb_put.cold+0x10/0x10 > > > [ 61.395095] page_to_skb+0xf7/0x410 > > > [ 61.395689] receive_buf+0x81/0x1660 > > > [ 61.396228] ? netif_receive_skb_list_internal+0x1ad/0x2b0 > > > [ 61.397180] ? napi_gro_flush+0x97/0xe0 > > > [ 61.397896] ? detach_buf_split+0x67/0x120 > > > [ 61.398573] virtnet_poll+0x2cf/0x420 > > > [ 61.399197] __napi_poll+0x25/0x150 > > > [ 61.399764] net_rx_action+0x22f/0x280 > > > [ 61.400394] __do_softirq+0xba/0x257 > > > [ 61.401012] irq_exit_rcu+0x8e/0xb0 > > > [ 61.401618] common_interrupt+0x7b/0xa0 > > > [ 61.402270] </IRQ> > > > [ 61.402620] asm_common_interrupt+0x1e/0x40 > > > [ 61.403302] RIP: default_idle+0xb/0x10 > > > [ 61.404018] Code: 8b 04 25 00 6d 01 00 f0 80 60 02 df c3 0f ae > > > f0 0f > > > ae 38 0f ae f0 eb b9 0f 1f 80 00 00 00 00 eb 07 0f 00 2d df 3e 44 > > > 00 fb > > > f4 <c3> cc cc cc cc 65 8b 15 31 2f a4 77 89 d2 48 8b 05 d0 a1 0c > > > 01 48 > > > All code > > > ======== > > > 0: 8b 04 25 00 6d 01 00 mov 0x16d00,%eax > > > 7: f0 80 60 02 df lock andb $0xdf,0x2(%rax) > > > c: c3 retq > > > d: 0f ae f0 mfence > > > 10: 0f ae 38 clflush (%rax) > > > 13: 0f ae f0 mfence > > > 16: eb b9 jmp 0xffffffffffffffd1 > > > 18: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) > > > 1f: eb 07 jmp 0x28 > > > 21: 0f 00 2d df 3e 44 00 verw 0x443edf(%rip) # > > > 0x443f07 > > > 28: fb sti > > > 29: f4 hlt > > > 2a:* c3 retq <-- > > > trapping > > > instruction > > > 2b: cc int3 > > > 2c: cc int3 > > > 2d: cc int3 > > > 2e: cc int3 > > > 2f: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > > > # 0x77a42f67 > > > 36: 89 d2 mov %edx,%edx > > > 38: 48 8b 05 d0 a1 0c > > > 01 mov 0x10ca1d0(%rip),%rax # > > > 0x10ca20f > > > 3f: 48 rex.W > > > > > > Code starting with the faulting instruction > > > =========================================== > > > 0: c3 retq > > > 1: cc int3 > > > 2: cc int3 > > > 3: cc int3 > > > 4: cc int3 > > > 5: 65 8b 15 31 2f a4 77 mov %gs:0x77a42f31(%rip),%edx > > > # 0x77a42f3d > > > c: 89 d2 mov %edx,%edx > > > e: 48 8b 05 d0 a1 0c > > > 01 mov 0x10ca1d0(%rip),%rax # > > > 0x10ca1e5 > > > 15: 48 rex.W > > > [ 61.407636] RSP: 0018:ffffae258008fef8 EFLAGS: 00000202 > > > [ 61.408394] RAX: ffffffff885ce620 RBX: 0000000000000005 RCX: > > > ffff8a5febd56f80 > > > [ 61.409451] RDX: 0000000000c1ec32 RSI: 7ffffff1b7a1e726 RDI: > > > ffff8a5febd5dd00 > > > [ 61.410530] RBP: ffff8a5fc01f8000 R08: 0000000000c1ec32 R09: > > > 0000000000000000 > > > [ 61.411715] R10: 0000000000000006 R11: 0000000000000002 R12: > > > 0000000000000000 > > > [ 61.412984] R13: 0000000000000000 R14: 0000000000000000 R15: > > > 0000000000000000 > > > [ 61.414183] ? mwait_idle+0x70/0x70 > > > [ 61.414805] ? mwait_idle+0x70/0x70 > > > [ 61.415592] default_idle_call+0x2a/0xa0 > > > [ 61.416216] do_idle+0x1e8/0x250 > > > [ 61.416722] cpu_startup_entry+0x14/0x20 > > > [ 61.417347] secondary_startup_64_no_verify+0xc2/0xcb > > > [ 61.418144] Modules linked in: > > > [ 61.418622] ---[ end trace 3741c3e580a52bbd ]--- > > > [ 61.419399] RIP: skb_panic+0x43/0x45 > > > [ 61.420054] Code: 4f 70 50 8b 87 bc 00 00 00 50 8b 87 b8 00 00 > > > 00 50 > > > ff b7 c8 00 00 00 4c 8b 8f c0 00 00 00 48 c7 c7 18 f1 cf 88 e8 6a > > > 43 fb > > > ff <0f> 0b 48 8b 14 24 48 c7 c1 20 35 b1 88 e8 ab ff ff ff 48 c7 > > > c6 60 > > > All code > > > ======== > > > 0: 4f 70 50 rex.WRXB jo 0x53 > > > 3: 8b 87 bc 00 00 00 mov 0xbc(%rdi),%eax > > > 9: 50 push %rax > > > a: 8b 87 b8 00 00 00 mov 0xb8(%rdi),%eax > > > 10: 50 push %rax > > > 11: ff b7 c8 00 00 00 pushq 0xc8(%rdi) > > > 17: 4c 8b 8f c0 00 00 00 mov 0xc0(%rdi),%r9 > > > 1e: 48 c7 c7 18 f1 cf 88 mov $0xffffffff88cff118,%rdi > > > 25: e8 6a 43 fb ff callq 0xfffffffffffb4394 > > > 2a:* 0f 0b ud2 <-- > > > trapping > > > instruction > > > 2c: 48 8b 14 24 mov (%rsp),%rdx > > > 30: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > > 37: e8 ab ff ff ff callq 0xffffffffffffffe7 > > > 3c: 48 rex.W > > > 3d: c7 .byte 0xc7 > > > 3e: c6 (bad) > > > 3f: 60 (bad) > > > > > > Code starting with the faulting instruction > > > =========================================== > > > 0: 0f 0b ud2 > > > 2: 48 8b 14 24 mov (%rsp),%rdx > > > 6: 48 c7 c1 20 35 b1 88 mov $0xffffffff88b13520,%rcx > > > d: e8 ab ff ff ff callq 0xffffffffffffffbd > > > 12: 48 rex.W > > > 13: c7 .byte 0xc7 > > > 14: c6 (bad) > > > 15: 60 (bad) > > > [ 61.422606] RSP: 0018:ffffae258017cce0 EFLAGS: 00010246 > > > [ 61.423865] RAX: 000000000000008b RBX: 0000000000000010 RCX: > > > 00000000ffffdfff > > > [ 61.425031] RDX: 0000000000000000 RSI: 00000000ffffffea RDI: > > > 0000000000000000 > > > [ 61.426229] RBP: ffffde6a80230880 R08: ffffffff88f45568 R09: > > > 0000000000009ffb > > > [ 61.427439] R10: 00000000ffffe000 R11: 3fffffffffffffff R12: > > > ffff8a5ec7461200 > > > [ 61.428615] R13: ffff8a5ec8c22000 R14: 0000000000000000 R15: > > > 0000000000000eb2 > > > [ 61.429799] FS: 0000000000000000(0000) > > > GS:ffff8a5febd40000(0000) > > > knlGS:0000000000000000 > > > > > > Regards, > > > Corentin > > > > Don't see anything obvious.. could be a net stack change. > > Any chance of a bisect? > > > > -- > > MST > > ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-07 14:02 ` Corentin Noël @ 2021-10-07 14:13 ` Greg KH 0 siblings, 0 replies; 21+ messages in thread From: Greg KH @ 2021-10-07 14:13 UTC (permalink / raw) To: Corentin Noël Cc: Eric Dumazet, Michael S. Tsirkin, linux-stable, Jason Wang, virtualization, regressions, Xuan Zhuo On Thu, Oct 07, 2021 at 04:02:10PM +0200, Corentin Noël wrote: > Le jeudi 07 octobre 2021 à 06:51 -0700, Eric Dumazet a écrit : > > On Thu, Oct 7, 2021 at 6:11 AM Michael S. Tsirkin <mst@redhat.com> > > wrote: > > > On Thu, Oct 07, 2021 at 02:04:22PM +0200, Corentin Noël wrote: > > > > I've been experiencing crashes with 5.14-rc1 and above that do > > > > not > > > > occur with 5.13, > > > > What about 5.14 ? > > > > 5.14-rc1 has many bugs we do not want to spend time rediscovering > > them... > > > > I've tested on 5.14, 5.15-rc4 and 5.15-rc4 with latest netdev and could > reproduce the crash on them all. Great, any chance you can use 'git bisect' to find the offending commit? thanks, greg k-h ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1633619172.5342586-1-xuanzhuo@linux.alibaba.com>]
* Re: virtio-net: kernel panic in virtio_net.c [not found] ` <1633619172.5342586-1-xuanzhuo@linux.alibaba.com> @ 2021-10-07 15:25 ` Greg KH [not found] ` <1633623446.6192446-1-xuanzhuo@linux.alibaba.com> 0 siblings, 1 reply; 21+ messages in thread From: Greg KH @ 2021-10-07 15:25 UTC (permalink / raw) To: Xuan Zhuo Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, stable On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote: > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote: > > I've been experiencing crashes with 5.14-rc1 and above that do not > > occur with 5.13, > > I should have fixed this problem before. I don't know why, I just looked at the > latest net code, and this commit seems to be lost. > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > Can you test this patch again? That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had it in it, right? thanks, greg k-h ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1633623446.6192446-1-xuanzhuo@linux.alibaba.com>]
* Re: virtio-net: kernel panic in virtio_net.c [not found] ` <1633623446.6192446-1-xuanzhuo@linux.alibaba.com> @ 2021-10-08 8:06 ` Greg KH 2021-10-08 10:02 ` Michael S. Tsirkin [not found] ` <1633710428.4908655-1-xuanzhuo@linux.alibaba.com> 0 siblings, 2 replies; 21+ messages in thread From: Greg KH @ 2021-10-08 8:06 UTC (permalink / raw) To: Xuan Zhuo Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, stable On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote: > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote: > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote: > > > > I've been experiencing crashes with 5.14-rc1 and above that do not > > > > occur with 5.13, > > > > > > I should have fixed this problem before. I don't know why, I just looked at the > > > latest net code, and this commit seems to be lost. > > > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > > > > > Can you test this patch again? > > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had > > it in it, right? > > > > Yes, it may be lost due to conflicts during a certain merge. Really? I tried to apply that again to 5.14 and it did not work. So I do not understand what to do here, can you try to explain it better? thanks, greg k-h ^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-08 8:06 ` Greg KH @ 2021-10-08 10:02 ` Michael S. Tsirkin 2021-10-08 12:21 ` Corentin Noël [not found] ` <1633710428.4908655-1-xuanzhuo@linux.alibaba.com> 1 sibling, 1 reply; 21+ messages in thread From: Michael S. Tsirkin @ 2021-10-08 10:02 UTC (permalink / raw) To: Greg KH Cc: Xuan Zhuo, Corentin Noël, Jason Wang, virtualization, regressions, Eric Dumazet, stable On Fri, Oct 08, 2021 at 10:06:57AM +0200, Greg KH wrote: > On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote: > > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote: > > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote: > > > > > I've been experiencing crashes with 5.14-rc1 and above that do not > > > > > occur with 5.13, > > > > > > > > I should have fixed this problem before. I don't know why, I just looked at the > > > > latest net code, and this commit seems to be lost. > > > > > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > > > > > > > Can you test this patch again? > > > > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had > > > it in it, right? > > > > > > > Yes, it may be lost due to conflicts during a certain merge. > > Really? I tried to apply that again to 5.14 and it did not work. So I > do not understand what to do here, can you try to explain it better? > > thanks, > > greg k-h Hmm, something like the following perhaps then? Corentin would you like to try this? Warning: untested. diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 096c2ac6b7a6..18dd9f6d107d 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -406,12 +406,13 @@ static struct sk_buff *page_to_skb(struct virtnet_info *vi, * add_recvbuf_mergeable() + get_mergeable_buf_len() */ truesize = headroom ? PAGE_SIZE : truesize; - tailroom = truesize - len - headroom; + tailroom = truesize - headroom; buf = p - headroom; len -= hdr_len; offset += hdr_padded_len; p += hdr_padded_len; + tailroom -= hdr_padded_len + len; shinfo_size = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); ^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: virtio-net: kernel panic in virtio_net.c 2021-10-08 10:02 ` Michael S. Tsirkin @ 2021-10-08 12:21 ` Corentin Noël 0 siblings, 0 replies; 21+ messages in thread From: Corentin Noël @ 2021-10-08 12:21 UTC (permalink / raw) To: Michael S. Tsirkin, Greg KH Cc: Xuan Zhuo, Jason Wang, virtualization, regressions, Eric Dumazet, stable Le vendredi 08 octobre 2021 à 06:02 -0400, Michael S. Tsirkin a écrit : > On Fri, Oct 08, 2021 at 10:06:57AM +0200, Greg KH wrote: > > On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote: > > > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH < > > > gregkh@linuxfoundation.org> wrote: > > > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote: > > > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël < > > > > > corentin.noel@collabora.com> wrote: > > > > > > I've been experiencing crashes with 5.14-rc1 and above that > > > > > > do not > > > > > > occur with 5.13, > > > > > > > > > > I should have fixed this problem before. I don't know why, I > > > > > just looked at the > > > > > latest net code, and this commit seems to be lost. > > > > > > > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix > > > > > for skb_over_panic inside big mode > > > > > > > > > > Can you test this patch again? > > > > > > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should > > > > have had > > > > it in it, right? > > > > > > > > > > Yes, it may be lost due to conflicts during a certain merge. > > > > Really? I tried to apply that again to 5.14 and it did not > > work. So I > > do not understand what to do here, can you try to explain it > > better? > > > > thanks, > > > > greg k-h > > Hmm, something like the following perhaps then? > Corentin would you like to try this? > Warning: untested. > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > index 096c2ac6b7a6..18dd9f6d107d 100644 > --- a/drivers/net/virtio_net.c > +++ b/drivers/net/virtio_net.c > @@ -406,12 +406,13 @@ static struct sk_buff *page_to_skb(struct > virtnet_info *vi, > * add_recvbuf_mergeable() + get_mergeable_buf_len() > */ > truesize = headroom ? PAGE_SIZE : truesize; > - tailroom = truesize - len - headroom; > + tailroom = truesize - headroom; > buf = p - headroom; > > len -= hdr_len; > offset += hdr_padded_len; > p += hdr_padded_len; > + tailroom -= hdr_padded_len + len; > > shinfo_size = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); > > Thank you for the patch, I started bisecting the issue but your patch actually makes it work again. Regards, Corentin ^ permalink raw reply [flat|nested] 21+ messages in thread
[parent not found: <1633710428.4908655-1-xuanzhuo@linux.alibaba.com>]
* Re: virtio-net: kernel panic in virtio_net.c [not found] ` <1633710428.4908655-1-xuanzhuo@linux.alibaba.com> @ 2021-10-09 5:19 ` Greg KH 0 siblings, 0 replies; 21+ messages in thread From: Greg KH @ 2021-10-09 5:19 UTC (permalink / raw) To: Xuan Zhuo Cc: Corentin Noël, Michael S. Tsirkin, Jason Wang, virtualization, regressions, Eric Dumazet, stable On Sat, Oct 09, 2021 at 12:27:08AM +0800, Xuan Zhuo wrote: > On Fri, 8 Oct 2021 10:06:57 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > On Fri, Oct 08, 2021 at 12:17:26AM +0800, Xuan Zhuo wrote: > > > On Thu, 7 Oct 2021 17:25:02 +0200, Greg KH <gregkh@linuxfoundation.org> wrote: > > > > On Thu, Oct 07, 2021 at 11:06:12PM +0800, Xuan Zhuo wrote: > > > > > On Thu, 07 Oct 2021 14:04:22 +0200, Corentin Noël <corentin.noel@collabora.com> wrote: > > > > > > I've been experiencing crashes with 5.14-rc1 and above that do not > > > > > > occur with 5.13, > > > > > > > > > > I should have fixed this problem before. I don't know why, I just looked at the > > > > > latest net code, and this commit seems to be lost. > > > > > > > > > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > > > > > > > > > Can you test this patch again? > > > > > > > > That commit showed up in 5.13-rc5, so 5.14-rc1 and 5.13 should have had > > > > it in it, right? > > > > > > > > > > Yes, it may be lost due to conflicts during a certain merge. > > > > Really? I tried to apply that again to 5.14 and it did not work. So I > > do not understand what to do here, can you try to explain it better? > > I took a look, and there is actually another missing patch: > > A. 8fb7da9e990793299c89ed7a4281c235bfdd31f8 virtio_net: get build_skb() buf by data ptr > B. 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode > > A is replaced by another patch: > > commit c32325b8fdf2f979befb9fd5587918c0d5412db3 > Author: Jakub Kicinski <kuba@kernel.org> > Date: Mon Aug 2 10:57:29 2021 -0700 > > virtio-net: realign page_to_skb() after merges > > We ended up merging two versions of the same patch set: > > commit 8fb7da9e9907 ("virtio_net: get build_skb() buf by data ptr") > commit 5c37711d9f27 ("virtio-net: fix for unable to handle page fault for address") > > into net, and > > commit 7bf64460e3b2 ("virtio-net: get build_skb() buf by data ptr") > commit 6c66c147b9a4 ("virtio-net: fix for unable to handle page fault for address") > > into net-next. Redo the merge from commit 126285651b7f ("Merge > ra.kernel.org:/pub/scm/linux/kernel/git/netdev/net"), so that > the most recent code remains. > > Acked-by: Michael S. Tsirkin <mst@redhat.com> > Signed-off-by: Jakub Kicinski <kuba@kernel.org> > Acked-by: Jason Wang <jasowang@redhat.com> > Signed-off-by: David S. Miller <davem@davemloft.net> > > So after this patch, patch B can be applied normally. > > So on the latest net branch, only lost > > 1a8024239dacf53fcf39c0f07fbf2712af22864f virtio-net: fix for skb_over_panic inside big mode Again, I do not know what to do here, can you submit the needed fix to the networking developers so this gets fixed? thanks, greg k-h ^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2021-10-09 5:19 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-06-01 16:06 virtio-net: kernel panic in virtio_net.c Corentin Noël
2021-06-01 17:07 ` Greg KH
2021-06-01 17:09 ` Corentin Noël
2021-06-01 17:47 ` Eric Dumazet
2021-06-02 17:54 ` Corentin Noël
[not found] <1622599316.2056065-1-xuanzhuo@linux.alibaba.com>
2021-06-02 17:56 ` Corentin Noël
[not found] <1622688283.7488964-1-xuanzhuo@linux.alibaba.com>
2021-06-03 8:57 ` Corentin Noël
2021-06-08 12:17 ` Greg KH
[not found] ` <1623203313.4303577-1-xuanzhuo@linux.alibaba.com>
2021-06-09 4:50 ` Greg KH
[not found] ` <1623218897.4150124-1-xuanzhuo@linux.alibaba.com>
2021-06-09 6:24 ` Greg KH
[not found] ` <1623225080.4793522-1-xuanzhuo@linux.alibaba.com>
2021-06-09 8:03 ` Greg KH
-- strict thread matches above, loose matches on Subject: below --
2021-10-07 12:04 Corentin Noël
2021-10-07 13:10 ` Michael S. Tsirkin
2021-10-07 13:51 ` Eric Dumazet
2021-10-07 14:02 ` Corentin Noël
2021-10-07 14:13 ` Greg KH
[not found] ` <1633619172.5342586-1-xuanzhuo@linux.alibaba.com>
2021-10-07 15:25 ` Greg KH
[not found] ` <1633623446.6192446-1-xuanzhuo@linux.alibaba.com>
2021-10-08 8:06 ` Greg KH
2021-10-08 10:02 ` Michael S. Tsirkin
2021-10-08 12:21 ` Corentin Noël
[not found] ` <1633710428.4908655-1-xuanzhuo@linux.alibaba.com>
2021-10-09 5:19 ` Greg KH
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).