* BUG: soft lockup in sock_setsockopt @ 2019-12-18 8:43 syzbot 2019-12-18 9:08 ` Dmitry Vyukov 0 siblings, 1 reply; 4+ messages in thread From: syzbot @ 2019-12-18 8:43 UTC (permalink / raw) To: gregkh, jpoimboe, linux-kernel, namit, peterz, syzkaller-bugs, tglx Hello, syzbot found the following crash on: HEAD commit: 9065e063 Merge branch 'x86-urgent-for-linus' of git://git... git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=17185e99e00000 kernel config: https://syzkaller.appspot.com/x/.config?x=dcf10bf83926432a dashboard link: https://syzkaller.appspot.com/bug?extid=e7e13ce5d4ca294ca90a compiler: gcc (GCC) 9.0.0 20181231 (experimental) Unfortunately, I don't have any reproducer for this crash yet. IMPORTANT: if you fix the bug, please add the following tag to the commit: Reported-by: syzbot+e7e13ce5d4ca294ca90a@syzkaller.appspotmail.com watchdog: BUG: soft lockup - CPU#0 stuck for 122s! [syz-executor.3:9634] Modules linked in: irq event stamp: 35786 hardirqs last enabled at (35785): [<ffffffff81006983>] trace_hardirqs_on_thunk+0x1a/0x1c arch/x86/entry/thunk_64.S:41 hardirqs last disabled at (35786): [<ffffffff8100699f>] trace_hardirqs_off_thunk+0x1a/0x1c arch/x86/entry/thunk_64.S:42 softirqs last enabled at (5788): [<ffffffff880006cd>] __do_softirq+0x6cd/0x98c kernel/softirq.c:319 softirqs last disabled at (5707): [<ffffffff81478ceb>] invoke_softirq kernel/softirq.c:373 [inline] softirqs last disabled at (5707): [<ffffffff81478ceb>] irq_exit+0x19b/0x1e0 kernel/softirq.c:413 CPU: 0 PID: 9634 Comm: syz-executor.3 Not tainted 5.5.0-rc2-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:__read_once_size include/linux/compiler.h:199 [inline] RIP: 0010:csd_lock_wait kernel/smp.c:109 [inline] RIP: 0010:smp_call_function_single+0x188/0x480 kernel/smp.c:311 Code: 00 e8 6c 23 0b 00 48 8b 4c 24 08 48 8b 54 24 10 48 8d 74 24 40 8b 7c 24 1c e8 c4 f9 ff ff 41 89 c5 eb 07 e8 4a 23 0b 00 f3 90 <44> 8b 64 24 58 31 ff 41 83 e4 01 44 89 e6 e8 b5 24 0b 00 45 85 e4 RSP: 0018:ffffc90004d3f480 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 RAX: 0000000000040000 RBX: 1ffff920009a7e94 RCX: ffffc90010442000 RDX: 0000000000040000 RSI: ffffffff816a0856 RDI: 0000000000000005 RBP: ffffc90004d3f550 R08: ffff88809e5161c0 R09: ffffed1015d27059 R10: ffffed1015d27058 R11: ffff8880ae9382c7 R12: 0000000000000001 R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000 FS: 00007f19610d6700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fe53bcc09c0 CR3: 000000008ffc6000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: smp_call_function_many+0x7ba/0x940 kernel/smp.c:451 smp_call_function+0x42/0x90 kernel/smp.c:509 on_each_cpu+0x2f/0x1f0 kernel/smp.c:616 flush_tlb_kernel_range+0x19b/0x250 arch/x86/mm/tlb.c:839 kasan_release_vmalloc+0xb4/0xc0 mm/kasan/common.c:976 __purge_vmap_area_lazy+0xca5/0x1ef0 mm/vmalloc.c:1313 _vm_unmap_aliases mm/vmalloc.c:1730 [inline] _vm_unmap_aliases+0x396/0x480 mm/vmalloc.c:1695 vm_unmap_aliases+0x19/0x20 mm/vmalloc.c:1753 change_page_attr_set_clr+0x22e/0x840 arch/x86/mm/pageattr.c:1709 change_page_attr_clear arch/x86/mm/pageattr.c:1766 [inline] set_memory_ro+0x7b/0xa0 arch/x86/mm/pageattr.c:1899 bpf_jit_binary_lock_ro include/linux/filter.h:790 [inline] bpf_int_jit_compile+0xebd/0x12ce arch/x86/net/bpf_jit_comp.c:1659 bpf_prog_select_runtime+0x4b9/0x850 kernel/bpf/core.c:1801 bpf_migrate_filter net/core/filter.c:1275 [inline] bpf_prepare_filter net/core/filter.c:1323 [inline] bpf_prepare_filter+0x977/0xd60 net/core/filter.c:1289 __get_filter+0x212/0x2c0 net/core/filter.c:1492 sk_attach_filter+0x1e/0xa0 net/core/filter.c:1507 sock_setsockopt+0x1f44/0x22b0 net/core/sock.c:999 __sys_setsockopt+0x440/0x4c0 net/socket.c:2113 __do_sys_setsockopt net/socket.c:2133 [inline] __se_sys_setsockopt net/socket.c:2130 [inline] __x64_sys_setsockopt+0xbe/0x150 net/socket.c:2130 do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x45a919 Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007f19610d5c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000036 RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 000000000045a919 RDX: 000000000000001a RSI: 0000000000000001 RDI: 000000000000000c RBP: 000000000075c070 R08: 0000000000000010 R09: 0000000000000000 R10: 0000000020000480 R11: 0000000000000246 R12: 00007f19610d66d4 R13: 00000000004c9e34 R14: 00000000004e1f78 R15: 00000000ffffffff Sending NMI from CPU 0 to CPUs 1: NMI backtrace for cpu 1 CPU: 1 PID: 3231 Comm: kworker/1:3 Not tainted 5.5.0-rc2-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: rcu_gp process_srcu RIP: 0010:delay_tsc+0x33/0xc0 arch/x86/lib/delay.c:68 Code: bf 01 00 00 00 41 55 41 54 53 e8 58 95 8b f9 e8 63 b4 cd fb 41 89 c5 0f 01 f9 66 90 48 c1 e2 20 48 09 c2 49 89 d4 eb 16 f3 90 <bf> 01 00 00 00 e8 33 95 8b f9 e8 3e b4 cd fb 44 39 e8 75 36 0f 01 RSP: 0018:ffffc9000898fbb0 EFLAGS: 00000286 RAX: 0000000080000000 RBX: 000000c937e7f04b RCX: 0000000000000000 RDX: 0000000000000001 RSI: ffffffff8392f013 RDI: 0000000000000001 RBP: ffffc9000898fbd0 R08: ffff88809e1e2200 R09: 0000000000000040 R10: 0000000000000040 R11: ffffffff89a4f487 R12: 000000c937e7c4ab R13: 0000000000000001 R14: 0000000000002ced R15: 0000000000000047 FS: 0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fbb9d9c6000 CR3: 0000000094911000 CR4: 00000000001426e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: __delay arch/x86/lib/delay.c:161 [inline] __const_udelay+0x59/0x80 arch/x86/lib/delay.c:175 try_check_zero+0x201/0x330 kernel/rcu/srcutree.c:705 srcu_advance_state kernel/rcu/srcutree.c:1142 [inline] process_srcu+0x329/0xe10 kernel/rcu/srcutree.c:1237 process_one_work+0x9af/0x1740 kernel/workqueue.c:2264 worker_thread+0x98/0xe40 kernel/workqueue.c:2410 kthread+0x361/0x430 kernel/kthread.c:255 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 --- This bug is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at syzkaller@googlegroups.com. syzbot will keep track of this bug report. See: https://goo.gl/tpsmEJ#status for how to communicate with syzbot. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: BUG: soft lockup in sock_setsockopt 2019-12-18 8:43 BUG: soft lockup in sock_setsockopt syzbot @ 2019-12-18 9:08 ` Dmitry Vyukov 2019-12-18 13:57 ` Daniel Axtens 0 siblings, 1 reply; 4+ messages in thread From: Dmitry Vyukov @ 2019-12-18 9:08 UTC (permalink / raw) To: syzbot, kasan-dev, Daniel Axtens Cc: Greg Kroah-Hartman, Josh Poimboeuf, LKML, namit, Peter Zijlstra, syzkaller-bugs, Thomas Gleixner On Wed, Dec 18, 2019 at 9:43 AM syzbot <syzbot+e7e13ce5d4ca294ca90a@syzkaller.appspotmail.com> wrote: > > Hello, > > syzbot found the following crash on: > > HEAD commit: 9065e063 Merge branch 'x86-urgent-for-linus' of git://git... > git tree: upstream > console output: https://syzkaller.appspot.com/x/log.txt?x=17185e99e00000 > kernel config: https://syzkaller.appspot.com/x/.config?x=dcf10bf83926432a > dashboard link: https://syzkaller.appspot.com/bug?extid=e7e13ce5d4ca294ca90a > compiler: gcc (GCC) 9.0.0 20181231 (experimental) > > Unfortunately, I don't have any reproducer for this crash yet. > > IMPORTANT: if you fix the bug, please add the following tag to the commit: > Reported-by: syzbot+e7e13ce5d4ca294ca90a@syzkaller.appspotmail.com +Daniel, kasan-dev, This looks like another stall caused by KASAN+vmalloc. Now it has reached upstream and this instance uses Apparmor rather than Smack. > watchdog: BUG: soft lockup - CPU#0 stuck for 122s! [syz-executor.3:9634] > Modules linked in: > irq event stamp: 35786 > hardirqs last enabled at (35785): [<ffffffff81006983>] > trace_hardirqs_on_thunk+0x1a/0x1c arch/x86/entry/thunk_64.S:41 > hardirqs last disabled at (35786): [<ffffffff8100699f>] > trace_hardirqs_off_thunk+0x1a/0x1c arch/x86/entry/thunk_64.S:42 > softirqs last enabled at (5788): [<ffffffff880006cd>] > __do_softirq+0x6cd/0x98c kernel/softirq.c:319 > softirqs last disabled at (5707): [<ffffffff81478ceb>] invoke_softirq > kernel/softirq.c:373 [inline] > softirqs last disabled at (5707): [<ffffffff81478ceb>] irq_exit+0x19b/0x1e0 > kernel/softirq.c:413 > CPU: 0 PID: 9634 Comm: syz-executor.3 Not tainted 5.5.0-rc2-syzkaller #0 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS > Google 01/01/2011 > RIP: 0010:__read_once_size include/linux/compiler.h:199 [inline] > RIP: 0010:csd_lock_wait kernel/smp.c:109 [inline] > RIP: 0010:smp_call_function_single+0x188/0x480 kernel/smp.c:311 > Code: 00 e8 6c 23 0b 00 48 8b 4c 24 08 48 8b 54 24 10 48 8d 74 24 40 8b 7c > 24 1c e8 c4 f9 ff ff 41 89 c5 eb 07 e8 4a 23 0b 00 f3 90 <44> 8b 64 24 58 > 31 ff 41 83 e4 01 44 89 e6 e8 b5 24 0b 00 45 85 e4 > RSP: 0018:ffffc90004d3f480 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 > RAX: 0000000000040000 RBX: 1ffff920009a7e94 RCX: ffffc90010442000 > RDX: 0000000000040000 RSI: ffffffff816a0856 RDI: 0000000000000005 > RBP: ffffc90004d3f550 R08: ffff88809e5161c0 R09: ffffed1015d27059 > R10: ffffed1015d27058 R11: ffff8880ae9382c7 R12: 0000000000000001 > R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000 > FS: 00007f19610d6700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00007fe53bcc09c0 CR3: 000000008ffc6000 CR4: 00000000001426f0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > Call Trace: > smp_call_function_many+0x7ba/0x940 kernel/smp.c:451 > smp_call_function+0x42/0x90 kernel/smp.c:509 > on_each_cpu+0x2f/0x1f0 kernel/smp.c:616 > flush_tlb_kernel_range+0x19b/0x250 arch/x86/mm/tlb.c:839 > kasan_release_vmalloc+0xb4/0xc0 mm/kasan/common.c:976 > __purge_vmap_area_lazy+0xca5/0x1ef0 mm/vmalloc.c:1313 > _vm_unmap_aliases mm/vmalloc.c:1730 [inline] > _vm_unmap_aliases+0x396/0x480 mm/vmalloc.c:1695 > vm_unmap_aliases+0x19/0x20 mm/vmalloc.c:1753 > change_page_attr_set_clr+0x22e/0x840 arch/x86/mm/pageattr.c:1709 > change_page_attr_clear arch/x86/mm/pageattr.c:1766 [inline] > set_memory_ro+0x7b/0xa0 arch/x86/mm/pageattr.c:1899 > bpf_jit_binary_lock_ro include/linux/filter.h:790 [inline] > bpf_int_jit_compile+0xebd/0x12ce arch/x86/net/bpf_jit_comp.c:1659 > bpf_prog_select_runtime+0x4b9/0x850 kernel/bpf/core.c:1801 > bpf_migrate_filter net/core/filter.c:1275 [inline] > bpf_prepare_filter net/core/filter.c:1323 [inline] > bpf_prepare_filter+0x977/0xd60 net/core/filter.c:1289 > __get_filter+0x212/0x2c0 net/core/filter.c:1492 > sk_attach_filter+0x1e/0xa0 net/core/filter.c:1507 > sock_setsockopt+0x1f44/0x22b0 net/core/sock.c:999 > __sys_setsockopt+0x440/0x4c0 net/socket.c:2113 > __do_sys_setsockopt net/socket.c:2133 [inline] > __se_sys_setsockopt net/socket.c:2130 [inline] > __x64_sys_setsockopt+0xbe/0x150 net/socket.c:2130 > do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294 > entry_SYSCALL_64_after_hwframe+0x49/0xbe > RIP: 0033:0x45a919 > Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 > 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff > ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 > RSP: 002b:00007f19610d5c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000036 > RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 000000000045a919 > RDX: 000000000000001a RSI: 0000000000000001 RDI: 000000000000000c > RBP: 000000000075c070 R08: 0000000000000010 R09: 0000000000000000 > R10: 0000000020000480 R11: 0000000000000246 R12: 00007f19610d66d4 > R13: 00000000004c9e34 R14: 00000000004e1f78 R15: 00000000ffffffff > Sending NMI from CPU 0 to CPUs 1: > NMI backtrace for cpu 1 > CPU: 1 PID: 3231 Comm: kworker/1:3 Not tainted 5.5.0-rc2-syzkaller #0 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS > Google 01/01/2011 > Workqueue: rcu_gp process_srcu > RIP: 0010:delay_tsc+0x33/0xc0 arch/x86/lib/delay.c:68 > Code: bf 01 00 00 00 41 55 41 54 53 e8 58 95 8b f9 e8 63 b4 cd fb 41 89 c5 > 0f 01 f9 66 90 48 c1 e2 20 48 09 c2 49 89 d4 eb 16 f3 90 <bf> 01 00 00 00 > e8 33 95 8b f9 e8 3e b4 cd fb 44 39 e8 75 36 0f 01 > RSP: 0018:ffffc9000898fbb0 EFLAGS: 00000286 > RAX: 0000000080000000 RBX: 000000c937e7f04b RCX: 0000000000000000 > RDX: 0000000000000001 RSI: ffffffff8392f013 RDI: 0000000000000001 > RBP: ffffc9000898fbd0 R08: ffff88809e1e2200 R09: 0000000000000040 > R10: 0000000000000040 R11: ffffffff89a4f487 R12: 000000c937e7c4ab > R13: 0000000000000001 R14: 0000000000002ced R15: 0000000000000047 > FS: 0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00007fbb9d9c6000 CR3: 0000000094911000 CR4: 00000000001426e0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > Call Trace: > __delay arch/x86/lib/delay.c:161 [inline] > __const_udelay+0x59/0x80 arch/x86/lib/delay.c:175 > try_check_zero+0x201/0x330 kernel/rcu/srcutree.c:705 > srcu_advance_state kernel/rcu/srcutree.c:1142 [inline] > process_srcu+0x329/0xe10 kernel/rcu/srcutree.c:1237 > process_one_work+0x9af/0x1740 kernel/workqueue.c:2264 > worker_thread+0x98/0xe40 kernel/workqueue.c:2410 > kthread+0x361/0x430 kernel/kthread.c:255 > ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 > > > --- > This bug is generated by a bot. It may contain errors. > See https://goo.gl/tpsmEJ for more information about syzbot. > syzbot engineers can be reached at syzkaller@googlegroups.com. > > syzbot will keep track of this bug report. See: > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > > -- > You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group. > To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/00000000000021cc1a0599f66f55%40google.com. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: BUG: soft lockup in sock_setsockopt 2019-12-18 9:08 ` Dmitry Vyukov @ 2019-12-18 13:57 ` Daniel Axtens 2019-12-18 14:15 ` Dmitry Vyukov 0 siblings, 1 reply; 4+ messages in thread From: Daniel Axtens @ 2019-12-18 13:57 UTC (permalink / raw) To: Dmitry Vyukov, syzbot, kasan-dev Cc: Greg Kroah-Hartman, Josh Poimboeuf, LKML, namit, Peter Zijlstra, syzkaller-bugs, Thomas Gleixner Dmitry Vyukov <dvyukov@google.com> writes: > On Wed, Dec 18, 2019 at 9:43 AM syzbot > <syzbot+e7e13ce5d4ca294ca90a@syzkaller.appspotmail.com> wrote: >> >> Hello, >> >> syzbot found the following crash on: >> >> HEAD commit: 9065e063 Merge branch 'x86-urgent-for-linus' of git://git... >> git tree: upstream >> console output: https://syzkaller.appspot.com/x/log.txt?x=17185e99e00000 >> kernel config: https://syzkaller.appspot.com/x/.config?x=dcf10bf83926432a >> dashboard link: https://syzkaller.appspot.com/bug?extid=e7e13ce5d4ca294ca90a >> compiler: gcc (GCC) 9.0.0 20181231 (experimental) >> >> Unfortunately, I don't have any reproducer for this crash yet. >> >> IMPORTANT: if you fix the bug, please add the following tag to the commit: >> Reported-by: syzbot+e7e13ce5d4ca294ca90a@syzkaller.appspotmail.com > > +Daniel, kasan-dev, > > This looks like another stall caused by KASAN+vmalloc. Now it has > reached upstream and this instance uses Apparmor rather than Smack. weird. It looks like the hang happens while waiting for a tlb range flush - in particular CPU#0 hangs while waiting for CPU#1 to finish. I don't know what was running on CPU#1 122s ago, but it's currently SRCU. I see the SRCU code does smp_lock_irq_... but I don't know how interrupts would be disabled for that long. (Unless the IPI just gets dropped on the floor if it the CPU currently has IRQs off, but surely we'd have picked that up somewhere else first?) The whole code is deep magic to me. Am I accidentally violating any restrictions on what can be called when? Regards, Daniel > >> watchdog: BUG: soft lockup - CPU#0 stuck for 122s! [syz-executor.3:9634] >> Modules linked in: >> irq event stamp: 35786 >> hardirqs last enabled at (35785): [<ffffffff81006983>] >> trace_hardirqs_on_thunk+0x1a/0x1c arch/x86/entry/thunk_64.S:41 >> hardirqs last disabled at (35786): [<ffffffff8100699f>] >> trace_hardirqs_off_thunk+0x1a/0x1c arch/x86/entry/thunk_64.S:42 >> softirqs last enabled at (5788): [<ffffffff880006cd>] >> __do_softirq+0x6cd/0x98c kernel/softirq.c:319 >> softirqs last disabled at (5707): [<ffffffff81478ceb>] invoke_softirq >> kernel/softirq.c:373 [inline] >> softirqs last disabled at (5707): [<ffffffff81478ceb>] irq_exit+0x19b/0x1e0 >> kernel/softirq.c:413 >> CPU: 0 PID: 9634 Comm: syz-executor.3 Not tainted 5.5.0-rc2-syzkaller #0 >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS >> Google 01/01/2011 >> RIP: 0010:__read_once_size include/linux/compiler.h:199 [inline] >> RIP: 0010:csd_lock_wait kernel/smp.c:109 [inline] >> RIP: 0010:smp_call_function_single+0x188/0x480 kernel/smp.c:311 >> Code: 00 e8 6c 23 0b 00 48 8b 4c 24 08 48 8b 54 24 10 48 8d 74 24 40 8b 7c >> 24 1c e8 c4 f9 ff ff 41 89 c5 eb 07 e8 4a 23 0b 00 f3 90 <44> 8b 64 24 58 >> 31 ff 41 83 e4 01 44 89 e6 e8 b5 24 0b 00 45 85 e4 >> RSP: 0018:ffffc90004d3f480 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 >> RAX: 0000000000040000 RBX: 1ffff920009a7e94 RCX: ffffc90010442000 >> RDX: 0000000000040000 RSI: ffffffff816a0856 RDI: 0000000000000005 >> RBP: ffffc90004d3f550 R08: ffff88809e5161c0 R09: ffffed1015d27059 >> R10: ffffed1015d27058 R11: ffff8880ae9382c7 R12: 0000000000000001 >> R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000 >> FS: 00007f19610d6700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> CR2: 00007fe53bcc09c0 CR3: 000000008ffc6000 CR4: 00000000001426f0 >> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >> Call Trace: >> smp_call_function_many+0x7ba/0x940 kernel/smp.c:451 >> smp_call_function+0x42/0x90 kernel/smp.c:509 >> on_each_cpu+0x2f/0x1f0 kernel/smp.c:616 >> flush_tlb_kernel_range+0x19b/0x250 arch/x86/mm/tlb.c:839 >> kasan_release_vmalloc+0xb4/0xc0 mm/kasan/common.c:976 >> __purge_vmap_area_lazy+0xca5/0x1ef0 mm/vmalloc.c:1313 >> _vm_unmap_aliases mm/vmalloc.c:1730 [inline] >> _vm_unmap_aliases+0x396/0x480 mm/vmalloc.c:1695 >> vm_unmap_aliases+0x19/0x20 mm/vmalloc.c:1753 >> change_page_attr_set_clr+0x22e/0x840 arch/x86/mm/pageattr.c:1709 >> change_page_attr_clear arch/x86/mm/pageattr.c:1766 [inline] >> set_memory_ro+0x7b/0xa0 arch/x86/mm/pageattr.c:1899 >> bpf_jit_binary_lock_ro include/linux/filter.h:790 [inline] >> bpf_int_jit_compile+0xebd/0x12ce arch/x86/net/bpf_jit_comp.c:1659 >> bpf_prog_select_runtime+0x4b9/0x850 kernel/bpf/core.c:1801 >> bpf_migrate_filter net/core/filter.c:1275 [inline] >> bpf_prepare_filter net/core/filter.c:1323 [inline] >> bpf_prepare_filter+0x977/0xd60 net/core/filter.c:1289 >> __get_filter+0x212/0x2c0 net/core/filter.c:1492 >> sk_attach_filter+0x1e/0xa0 net/core/filter.c:1507 >> sock_setsockopt+0x1f44/0x22b0 net/core/sock.c:999 >> __sys_setsockopt+0x440/0x4c0 net/socket.c:2113 >> __do_sys_setsockopt net/socket.c:2133 [inline] >> __se_sys_setsockopt net/socket.c:2130 [inline] >> __x64_sys_setsockopt+0xbe/0x150 net/socket.c:2130 >> do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294 >> entry_SYSCALL_64_after_hwframe+0x49/0xbe >> RIP: 0033:0x45a919 >> Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 >> 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff >> ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 >> RSP: 002b:00007f19610d5c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000036 >> RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 000000000045a919 >> RDX: 000000000000001a RSI: 0000000000000001 RDI: 000000000000000c >> RBP: 000000000075c070 R08: 0000000000000010 R09: 0000000000000000 >> R10: 0000000020000480 R11: 0000000000000246 R12: 00007f19610d66d4 >> R13: 00000000004c9e34 R14: 00000000004e1f78 R15: 00000000ffffffff >> Sending NMI from CPU 0 to CPUs 1: >> NMI backtrace for cpu 1 >> CPU: 1 PID: 3231 Comm: kworker/1:3 Not tainted 5.5.0-rc2-syzkaller #0 >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS >> Google 01/01/2011 >> Workqueue: rcu_gp process_srcu >> RIP: 0010:delay_tsc+0x33/0xc0 arch/x86/lib/delay.c:68 >> Code: bf 01 00 00 00 41 55 41 54 53 e8 58 95 8b f9 e8 63 b4 cd fb 41 89 c5 >> 0f 01 f9 66 90 48 c1 e2 20 48 09 c2 49 89 d4 eb 16 f3 90 <bf> 01 00 00 00 >> e8 33 95 8b f9 e8 3e b4 cd fb 44 39 e8 75 36 0f 01 >> RSP: 0018:ffffc9000898fbb0 EFLAGS: 00000286 >> RAX: 0000000080000000 RBX: 000000c937e7f04b RCX: 0000000000000000 >> RDX: 0000000000000001 RSI: ffffffff8392f013 RDI: 0000000000000001 >> RBP: ffffc9000898fbd0 R08: ffff88809e1e2200 R09: 0000000000000040 >> R10: 0000000000000040 R11: ffffffff89a4f487 R12: 000000c937e7c4ab >> R13: 0000000000000001 R14: 0000000000002ced R15: 0000000000000047 >> FS: 0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000 >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> CR2: 00007fbb9d9c6000 CR3: 0000000094911000 CR4: 00000000001426e0 >> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 >> Call Trace: >> __delay arch/x86/lib/delay.c:161 [inline] >> __const_udelay+0x59/0x80 arch/x86/lib/delay.c:175 >> try_check_zero+0x201/0x330 kernel/rcu/srcutree.c:705 >> srcu_advance_state kernel/rcu/srcutree.c:1142 [inline] >> process_srcu+0x329/0xe10 kernel/rcu/srcutree.c:1237 >> process_one_work+0x9af/0x1740 kernel/workqueue.c:2264 >> worker_thread+0x98/0xe40 kernel/workqueue.c:2410 >> kthread+0x361/0x430 kernel/kthread.c:255 >> ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 >> >> >> --- >> This bug is generated by a bot. It may contain errors. >> See https://goo.gl/tpsmEJ for more information about syzbot. >> syzbot engineers can be reached at syzkaller@googlegroups.com. >> >> syzbot will keep track of this bug report. See: >> https://goo.gl/tpsmEJ#status for how to communicate with syzbot. >> >> -- >> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group. >> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com. >> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/00000000000021cc1a0599f66f55%40google.com. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: BUG: soft lockup in sock_setsockopt 2019-12-18 13:57 ` Daniel Axtens @ 2019-12-18 14:15 ` Dmitry Vyukov 0 siblings, 0 replies; 4+ messages in thread From: Dmitry Vyukov @ 2019-12-18 14:15 UTC (permalink / raw) To: Daniel Axtens, Paul E. McKenney Cc: syzbot, kasan-dev, Greg Kroah-Hartman, Josh Poimboeuf, LKML, namit, Peter Zijlstra, syzkaller-bugs, Thomas Gleixner On Wed, Dec 18, 2019 at 2:57 PM Daniel Axtens <dja@axtens.net> wrote: > > Dmitry Vyukov <dvyukov@google.com> writes: > > > On Wed, Dec 18, 2019 at 9:43 AM syzbot > > <syzbot+e7e13ce5d4ca294ca90a@syzkaller.appspotmail.com> wrote: > >> > >> Hello, > >> > >> syzbot found the following crash on: > >> > >> HEAD commit: 9065e063 Merge branch 'x86-urgent-for-linus' of git://git... > >> git tree: upstream > >> console output: https://syzkaller.appspot.com/x/log.txt?x=17185e99e00000 > >> kernel config: https://syzkaller.appspot.com/x/.config?x=dcf10bf83926432a > >> dashboard link: https://syzkaller.appspot.com/bug?extid=e7e13ce5d4ca294ca90a > >> compiler: gcc (GCC) 9.0.0 20181231 (experimental) > >> > >> Unfortunately, I don't have any reproducer for this crash yet. > >> > >> IMPORTANT: if you fix the bug, please add the following tag to the commit: > >> Reported-by: syzbot+e7e13ce5d4ca294ca90a@syzkaller.appspotmail.com > > > > +Daniel, kasan-dev, > > > > This looks like another stall caused by KASAN+vmalloc. Now it has > > reached upstream and this instance uses Apparmor rather than Smack. > > weird. It looks like the hang happens while waiting for a tlb range > flush - in particular CPU#0 hangs while waiting for CPU#1 to finish. I > don't know what was running on CPU#1 122s ago, but it's currently > SRCU. I see the SRCU code does smp_lock_irq_... but I don't know how > interrupts would be disabled for that long. (Unless the IPI just gets > dropped on the floor if it the CPU currently has IRQs off, but surely > we'd have picked that up somewhere else first?) > > The whole code is deep magic to me. Am I accidentally violating any > restrictions on what can be called when? Maybe bpf_int_jit_compile stack is in srcu reader critical section and try_check_zero waits forever for it to leave, while bpf_int_jit_compile tries to send IPI which does not reach try_check_zero CPU? Should this: if (--trycount + !srcu_get_delay(ssp) <= 0) return false; in try_check_zero break eventually? What is even the type of that expression? That's playing with fire in C :) Can if jump across 0 because both trycount and srcu_get_delay change at the same iteration? And then if it's unsigned type in the end, it does not become -1, but instead 0xffffffff which is not <=0?.... > >> watchdog: BUG: soft lockup - CPU#0 stuck for 122s! [syz-executor.3:9634] > >> Modules linked in: > >> irq event stamp: 35786 > >> hardirqs last enabled at (35785): [<ffffffff81006983>] > >> trace_hardirqs_on_thunk+0x1a/0x1c arch/x86/entry/thunk_64.S:41 > >> hardirqs last disabled at (35786): [<ffffffff8100699f>] > >> trace_hardirqs_off_thunk+0x1a/0x1c arch/x86/entry/thunk_64.S:42 > >> softirqs last enabled at (5788): [<ffffffff880006cd>] > >> __do_softirq+0x6cd/0x98c kernel/softirq.c:319 > >> softirqs last disabled at (5707): [<ffffffff81478ceb>] invoke_softirq > >> kernel/softirq.c:373 [inline] > >> softirqs last disabled at (5707): [<ffffffff81478ceb>] irq_exit+0x19b/0x1e0 > >> kernel/softirq.c:413 > >> CPU: 0 PID: 9634 Comm: syz-executor.3 Not tainted 5.5.0-rc2-syzkaller #0 > >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS > >> Google 01/01/2011 > >> RIP: 0010:__read_once_size include/linux/compiler.h:199 [inline] > >> RIP: 0010:csd_lock_wait kernel/smp.c:109 [inline] > >> RIP: 0010:smp_call_function_single+0x188/0x480 kernel/smp.c:311 > >> Code: 00 e8 6c 23 0b 00 48 8b 4c 24 08 48 8b 54 24 10 48 8d 74 24 40 8b 7c > >> 24 1c e8 c4 f9 ff ff 41 89 c5 eb 07 e8 4a 23 0b 00 f3 90 <44> 8b 64 24 58 > >> 31 ff 41 83 e4 01 44 89 e6 e8 b5 24 0b 00 45 85 e4 > >> RSP: 0018:ffffc90004d3f480 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 > >> RAX: 0000000000040000 RBX: 1ffff920009a7e94 RCX: ffffc90010442000 > >> RDX: 0000000000040000 RSI: ffffffff816a0856 RDI: 0000000000000005 > >> RBP: ffffc90004d3f550 R08: ffff88809e5161c0 R09: ffffed1015d27059 > >> R10: ffffed1015d27058 R11: ffff8880ae9382c7 R12: 0000000000000001 > >> R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000 > >> FS: 00007f19610d6700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 > >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > >> CR2: 00007fe53bcc09c0 CR3: 000000008ffc6000 CR4: 00000000001426f0 > >> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > >> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > >> Call Trace: > >> smp_call_function_many+0x7ba/0x940 kernel/smp.c:451 > >> smp_call_function+0x42/0x90 kernel/smp.c:509 > >> on_each_cpu+0x2f/0x1f0 kernel/smp.c:616 > >> flush_tlb_kernel_range+0x19b/0x250 arch/x86/mm/tlb.c:839 > >> kasan_release_vmalloc+0xb4/0xc0 mm/kasan/common.c:976 > >> __purge_vmap_area_lazy+0xca5/0x1ef0 mm/vmalloc.c:1313 > >> _vm_unmap_aliases mm/vmalloc.c:1730 [inline] > >> _vm_unmap_aliases+0x396/0x480 mm/vmalloc.c:1695 > >> vm_unmap_aliases+0x19/0x20 mm/vmalloc.c:1753 > >> change_page_attr_set_clr+0x22e/0x840 arch/x86/mm/pageattr.c:1709 > >> change_page_attr_clear arch/x86/mm/pageattr.c:1766 [inline] > >> set_memory_ro+0x7b/0xa0 arch/x86/mm/pageattr.c:1899 > >> bpf_jit_binary_lock_ro include/linux/filter.h:790 [inline] > >> bpf_int_jit_compile+0xebd/0x12ce arch/x86/net/bpf_jit_comp.c:1659 > >> bpf_prog_select_runtime+0x4b9/0x850 kernel/bpf/core.c:1801 > >> bpf_migrate_filter net/core/filter.c:1275 [inline] > >> bpf_prepare_filter net/core/filter.c:1323 [inline] > >> bpf_prepare_filter+0x977/0xd60 net/core/filter.c:1289 > >> __get_filter+0x212/0x2c0 net/core/filter.c:1492 > >> sk_attach_filter+0x1e/0xa0 net/core/filter.c:1507 > >> sock_setsockopt+0x1f44/0x22b0 net/core/sock.c:999 > >> __sys_setsockopt+0x440/0x4c0 net/socket.c:2113 > >> __do_sys_setsockopt net/socket.c:2133 [inline] > >> __se_sys_setsockopt net/socket.c:2130 [inline] > >> __x64_sys_setsockopt+0xbe/0x150 net/socket.c:2130 > >> do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294 > >> entry_SYSCALL_64_after_hwframe+0x49/0xbe > >> RIP: 0033:0x45a919 > >> Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 > >> 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff > >> ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 > >> RSP: 002b:00007f19610d5c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000036 > >> RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 000000000045a919 > >> RDX: 000000000000001a RSI: 0000000000000001 RDI: 000000000000000c > >> RBP: 000000000075c070 R08: 0000000000000010 R09: 0000000000000000 > >> R10: 0000000020000480 R11: 0000000000000246 R12: 00007f19610d66d4 > >> R13: 00000000004c9e34 R14: 00000000004e1f78 R15: 00000000ffffffff > >> Sending NMI from CPU 0 to CPUs 1: > >> NMI backtrace for cpu 1 > >> CPU: 1 PID: 3231 Comm: kworker/1:3 Not tainted 5.5.0-rc2-syzkaller #0 > >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS > >> Google 01/01/2011 > >> Workqueue: rcu_gp process_srcu > >> RIP: 0010:delay_tsc+0x33/0xc0 arch/x86/lib/delay.c:68 > >> Code: bf 01 00 00 00 41 55 41 54 53 e8 58 95 8b f9 e8 63 b4 cd fb 41 89 c5 > >> 0f 01 f9 66 90 48 c1 e2 20 48 09 c2 49 89 d4 eb 16 f3 90 <bf> 01 00 00 00 > >> e8 33 95 8b f9 e8 3e b4 cd fb 44 39 e8 75 36 0f 01 > >> RSP: 0018:ffffc9000898fbb0 EFLAGS: 00000286 > >> RAX: 0000000080000000 RBX: 000000c937e7f04b RCX: 0000000000000000 > >> RDX: 0000000000000001 RSI: ffffffff8392f013 RDI: 0000000000000001 > >> RBP: ffffc9000898fbd0 R08: ffff88809e1e2200 R09: 0000000000000040 > >> R10: 0000000000000040 R11: ffffffff89a4f487 R12: 000000c937e7c4ab > >> R13: 0000000000000001 R14: 0000000000002ced R15: 0000000000000047 > >> FS: 0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000 > >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > >> CR2: 00007fbb9d9c6000 CR3: 0000000094911000 CR4: 00000000001426e0 > >> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > >> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > >> Call Trace: > >> __delay arch/x86/lib/delay.c:161 [inline] > >> __const_udelay+0x59/0x80 arch/x86/lib/delay.c:175 > >> try_check_zero+0x201/0x330 kernel/rcu/srcutree.c:705 > >> srcu_advance_state kernel/rcu/srcutree.c:1142 [inline] > >> process_srcu+0x329/0xe10 kernel/rcu/srcutree.c:1237 > >> process_one_work+0x9af/0x1740 kernel/workqueue.c:2264 > >> worker_thread+0x98/0xe40 kernel/workqueue.c:2410 > >> kthread+0x361/0x430 kernel/kthread.c:255 > >> ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 > >> > >> > >> --- > >> This bug is generated by a bot. It may contain errors. > >> See https://goo.gl/tpsmEJ for more information about syzbot. > >> syzbot engineers can be reached at syzkaller@googlegroups.com. > >> > >> syzbot will keep track of this bug report. See: > >> https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > >> > >> -- > >> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group. > >> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com. > >> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/00000000000021cc1a0599f66f55%40google.com. > > -- > You received this message because you are subscribed to the Google Groups "kasan-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to kasan-dev+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/kasan-dev/87o8w5u47c.fsf%40dja-thinkpad.axtens.net. ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2019-12-18 14:16 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2019-12-18 8:43 BUG: soft lockup in sock_setsockopt syzbot 2019-12-18 9:08 ` Dmitry Vyukov 2019-12-18 13:57 ` Daniel Axtens 2019-12-18 14:15 ` Dmitry Vyukov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox