* INFO: rcu detected stall in vhost_worker @ 2019-07-26 9:38 syzbot 2019-07-26 15:26 ` syzbot 2019-07-27 8:23 ` Hillf Danton 0 siblings, 2 replies; 7+ messages in thread From: syzbot @ 2019-07-26 9:38 UTC (permalink / raw) To: jasowang, kvm, linux-kernel, mst, netdev, syzkaller-bugs, virtualization Hello, syzbot found the following crash on: HEAD commit: 13bf6d6a Add linux-next specific files for 20190725 git tree: linux-next console output: https://syzkaller.appspot.com/x/log.txt?x=141449f0600000 kernel config: https://syzkaller.appspot.com/x/.config?x=8ae987d803395886 dashboard link: https://syzkaller.appspot.com/bug?extid=36e93b425cd6eb54fcc1 compiler: gcc (GCC) 9.0.0 20181231 (experimental) syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15112f3fa00000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=131ab578600000 IMPORTANT: if you fix the bug, please add the following tag to the commit: Reported-by: syzbot+36e93b425cd6eb54fcc1@syzkaller.appspotmail.com rcu: INFO: rcu_preempt self-detected stall on CPU rcu: 0-....: (10500 ticks this GP) idle=a56/1/0x4000000000000002 softirq=12266/12266 fqs=5250 (t=10502 jiffies g=14905 q=12) NMI backtrace for cpu 0 CPU: 0 PID: 10848 Comm: vhost-10847 Not tainted 5.3.0-rc1-next-20190725 #52 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: <IRQ> __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x172/0x1f0 lib/dump_stack.c:113 nmi_cpu_backtrace.cold+0x70/0xb2 lib/nmi_backtrace.c:101 nmi_trigger_cpumask_backtrace+0x23b/0x28b lib/nmi_backtrace.c:62 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline] rcu_dump_cpu_stacks+0x183/0x1cf kernel/rcu/tree_stall.h:254 print_cpu_stall kernel/rcu/tree_stall.h:455 [inline] check_cpu_stall kernel/rcu/tree_stall.h:529 [inline] rcu_pending kernel/rcu/tree.c:2736 [inline] rcu_sched_clock_irq.cold+0x4dd/0xc13 kernel/rcu/tree.c:2183 update_process_times+0x32/0x80 kernel/time/timer.c:1639 tick_sched_handle+0xa2/0x190 kernel/time/tick-sched.c:167 tick_sched_timer+0x53/0x140 kernel/time/tick-sched.c:1296 __run_hrtimer kernel/time/hrtimer.c:1389 [inline] __hrtimer_run_queues+0x364/0xe40 kernel/time/hrtimer.c:1451 hrtimer_interrupt+0x314/0x770 kernel/time/hrtimer.c:1509 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1068 [inline] smp_apic_timer_interrupt+0x160/0x610 arch/x86/kernel/apic/apic.c:1093 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:828 </IRQ> RIP: 0010:check_memory_region_inline mm/kasan/generic.c:173 [inline] RIP: 0010:check_memory_region+0x0/0x1a0 mm/kasan/generic.c:192 Code: 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 f2 be f8 00 00 00 48 89 e5 e8 df 60 90 05 5d c3 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 <48> 85 f6 0f 84 34 01 00 00 48 b8 ff ff ff ff ff 7f ff ff 55 0f b6 RSP: 0018:ffff8880a40bf950 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 RAX: 0000000000000000 RBX: ffff8880836a8220 RCX: ffffffff81599777 RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8880836a8220 RBP: ffff8880a40bf958 R08: 1ffff110106d5044 R09: ffffed10106d5045 R10: ffffed10106d5044 R11: ffff8880836a8223 R12: 0000000000000001 R13: 0000000000000003 R14: ffffed10106d5044 R15: 0000000000000001 atomic_read include/asm-generic/atomic-instrumented.h:26 [inline] virt_spin_lock arch/x86/include/asm/qspinlock.h:83 [inline] native_queued_spin_lock_slowpath+0xb7/0x9f0 kernel/locking/qspinlock.c:325 pv_queued_spin_lock_slowpath arch/x86/include/asm/paravirt.h:642 [inline] queued_spin_lock_slowpath arch/x86/include/asm/qspinlock.h:50 [inline] queued_spin_lock include/asm-generic/qspinlock.h:81 [inline] do_raw_spin_lock+0x20e/0x2e0 kernel/locking/spinlock_debug.c:113 __raw_spin_lock include/linux/spinlock_api_smp.h:143 [inline] _raw_spin_lock+0x37/0x40 kernel/locking/spinlock.c:151 spin_lock include/linux/spinlock.h:338 [inline] vhost_setup_uaddr drivers/vhost/vhost.c:790 [inline] vhost_setup_vq_uaddr drivers/vhost/vhost.c:801 [inline] vhost_vq_map_prefetch drivers/vhost/vhost.c:1783 [inline] vq_meta_prefetch+0x2a0/0xcb0 drivers/vhost/vhost.c:1804 handle_rx+0x145/0x1890 drivers/vhost/net.c:1128 handle_rx_net+0x19/0x20 drivers/vhost/net.c:1270 vhost_worker+0x2af/0x4d0 drivers/vhost/vhost.c:473 kthread+0x361/0x430 kernel/kthread.c:255 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 --- This bug is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at syzkaller@googlegroups.com. syzbot will keep track of this bug report. See: https://goo.gl/tpsmEJ#status for how to communicate with syzbot. syzbot can test patches for this bug, for details see: https://goo.gl/tpsmEJ#testing-patches ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: INFO: rcu detected stall in vhost_worker 2019-07-26 9:38 INFO: rcu detected stall in vhost_worker syzbot @ 2019-07-26 15:26 ` syzbot 2019-07-26 21:35 ` Linus Torvalds ` (2 more replies) 2019-07-27 8:23 ` Hillf Danton 1 sibling, 3 replies; 7+ messages in thread From: syzbot @ 2019-07-26 15:26 UTC (permalink / raw) To: jasowang, kvm, linux-kbuild, linux-kernel, michal.lkml, mst, netdev, syzkaller-bugs, torvalds, virtualization, yamada.masahiro syzbot has bisected this bug to: commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 Author: Linus Torvalds <torvalds@linux-foundation.org> Date: Sun Jul 7 22:41:56 2019 +0000 Linux 5.2 bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=118810bfa00000 start commit: 13bf6d6a Add linux-next specific files for 20190725 git tree: linux-next kernel config: https://syzkaller.appspot.com/x/.config?x=8ae987d803395886 dashboard link: https://syzkaller.appspot.com/bug?extid=36e93b425cd6eb54fcc1 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15112f3fa00000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=131ab578600000 Reported-by: syzbot+36e93b425cd6eb54fcc1@syzkaller.appspotmail.com Fixes: 0ecfebd2b524 ("Linux 5.2") For information about bisection process see: https://goo.gl/tpsmEJ#bisection ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: INFO: rcu detected stall in vhost_worker 2019-07-26 15:26 ` syzbot @ 2019-07-26 21:35 ` Linus Torvalds 2019-07-26 21:35 ` Linus Torvalds 2019-07-28 8:36 ` Michael S. Tsirkin 2 siblings, 0 replies; 7+ messages in thread From: Linus Torvalds @ 2019-07-26 21:35 UTC (permalink / raw) To: syzbot Cc: michal.lkml, KVM list, Michael S. Tsirkin, Netdev, Linux Kbuild mailing list, syzkaller-bugs, Linux List Kernel Mailing, virtualization, Masahiro Yamada On Fri, Jul 26, 2019 at 8:26 AM syzbot <syzbot+36e93b425cd6eb54fcc1@syzkaller.appspotmail.com> wrote: > > syzbot has bisected this bug to: > > commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 > Author: Linus Torvalds <torvalds@linux-foundation.org> > Date: Sun Jul 7 22:41:56 2019 +0000 > > Linux 5.2 That seems very unlikely. That commit literally just changes the EXTRAVERSION part of the version string. So even if something actually depended on the version number, even that wouldn't have triggered any semantic change. Linus ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: INFO: rcu detected stall in vhost_worker 2019-07-26 15:26 ` syzbot 2019-07-26 21:35 ` Linus Torvalds @ 2019-07-26 21:35 ` Linus Torvalds 2019-07-28 8:36 ` Michael S. Tsirkin 2 siblings, 0 replies; 7+ messages in thread From: Linus Torvalds @ 2019-07-26 21:35 UTC (permalink / raw) To: syzbot Cc: Jason Wang, KVM list, Linux Kbuild mailing list, Linux List Kernel Mailing, michal.lkml, Michael S. Tsirkin, Netdev, syzkaller-bugs, virtualization, Masahiro Yamada On Fri, Jul 26, 2019 at 8:26 AM syzbot <syzbot+36e93b425cd6eb54fcc1@syzkaller.appspotmail.com> wrote: > > syzbot has bisected this bug to: > > commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 > Author: Linus Torvalds <torvalds@linux-foundation.org> > Date: Sun Jul 7 22:41:56 2019 +0000 > > Linux 5.2 That seems very unlikely. That commit literally just changes the EXTRAVERSION part of the version string. So even if something actually depended on the version number, even that wouldn't have triggered any semantic change. Linus ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: INFO: rcu detected stall in vhost_worker 2019-07-26 15:26 ` syzbot @ 2019-07-28 8:36 ` Michael S. Tsirkin 2019-07-26 21:35 ` Linus Torvalds 2019-07-28 8:36 ` Michael S. Tsirkin 2 siblings, 0 replies; 7+ messages in thread From: Michael S. Tsirkin @ 2019-07-28 8:36 UTC (permalink / raw) To: Hillf Danton Cc: syzbot, jasowang, kvm, linux-kbuild, linux-kernel, michal.lkml, netdev, syzkaller-bugs, torvalds, virtualization, yamada.masahiro On Sat, Jul 27, 2019 at 04:23:23PM +0800, Hillf Danton wrote: > > Fri, 26 Jul 2019 08:26:01 -0700 (PDT) > > syzbot has bisected this bug to: > > > > commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 > > Author: Linus Torvalds <torvalds@linux-foundation.org> > > Date: Sun Jul 7 22:41:56 2019 +0000 > > > > Linux 5.2 > > > > bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=118810bfa00000 > > start commit: 13bf6d6a Add linux-next specific files for 20190725 > > git tree: linux-next > > kernel config: https://syzkaller.appspot.com/x/.config?x=8ae987d803395886 > > dashboard link: https://syzkaller.appspot.com/bug?extid=36e93b425cd6eb54fcc1 > > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15112f3fa00000 > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=131ab578600000 > > > > Reported-by: syzbot+36e93b425cd6eb54fcc1@syzkaller.appspotmail.com > > Fixes: 0ecfebd2b524 ("Linux 5.2") > > > > For information about bisection process see: https://goo.gl/tpsmEJ#bisection > > --- a/drivers/vhost/vhost.c > +++ b/drivers/vhost/vhost.c > @@ -787,7 +787,6 @@ static void vhost_setup_uaddr(struct vho > size_t size, bool write) > { > struct vhost_uaddr *addr = &vq->uaddrs[index]; > - spin_lock(&vq->mmu_lock); > > addr->uaddr = uaddr; > addr->size = size; > @@ -797,7 +796,10 @@ static void vhost_setup_uaddr(struct vho > static void vhost_setup_vq_uaddr(struct vhost_virtqueue *vq) > { > spin_lock(&vq->mmu_lock); > - > + /* > + * deadlock if managing to take mmu_lock again while > + * setting up uaddr > + */ > vhost_setup_uaddr(vq, VHOST_ADDR_DESC, > (unsigned long)vq->desc, > vhost_get_desc_size(vq, vq->num), > -- Thanks! I reverted this whole commit. -- MST ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: INFO: rcu detected stall in vhost_worker @ 2019-07-28 8:36 ` Michael S. Tsirkin 0 siblings, 0 replies; 7+ messages in thread From: Michael S. Tsirkin @ 2019-07-28 8:36 UTC (permalink / raw) To: Hillf Danton Cc: michal.lkml, kvm, linux-kbuild, syzbot, netdev, syzkaller-bugs, linux-kernel, virtualization, yamada.masahiro, torvalds On Sat, Jul 27, 2019 at 04:23:23PM +0800, Hillf Danton wrote: > > Fri, 26 Jul 2019 08:26:01 -0700 (PDT) > > syzbot has bisected this bug to: > > > > commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 > > Author: Linus Torvalds <torvalds@linux-foundation.org> > > Date: Sun Jul 7 22:41:56 2019 +0000 > > > > Linux 5.2 > > > > bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=118810bfa00000 > > start commit: 13bf6d6a Add linux-next specific files for 20190725 > > git tree: linux-next > > kernel config: https://syzkaller.appspot.com/x/.config?x=8ae987d803395886 > > dashboard link: https://syzkaller.appspot.com/bug?extid=36e93b425cd6eb54fcc1 > > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15112f3fa00000 > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=131ab578600000 > > > > Reported-by: syzbot+36e93b425cd6eb54fcc1@syzkaller.appspotmail.com > > Fixes: 0ecfebd2b524 ("Linux 5.2") > > > > For information about bisection process see: https://goo.gl/tpsmEJ#bisection > > --- a/drivers/vhost/vhost.c > +++ b/drivers/vhost/vhost.c > @@ -787,7 +787,6 @@ static void vhost_setup_uaddr(struct vho > size_t size, bool write) > { > struct vhost_uaddr *addr = &vq->uaddrs[index]; > - spin_lock(&vq->mmu_lock); > > addr->uaddr = uaddr; > addr->size = size; > @@ -797,7 +796,10 @@ static void vhost_setup_uaddr(struct vho > static void vhost_setup_vq_uaddr(struct vhost_virtqueue *vq) > { > spin_lock(&vq->mmu_lock); > - > + /* > + * deadlock if managing to take mmu_lock again while > + * setting up uaddr > + */ > vhost_setup_uaddr(vq, VHOST_ADDR_DESC, > (unsigned long)vq->desc, > vhost_get_desc_size(vq, vq->num), > -- Thanks! I reverted this whole commit. -- MST ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: INFO: rcu detected stall in vhost_worker 2019-07-26 9:38 INFO: rcu detected stall in vhost_worker syzbot 2019-07-26 15:26 ` syzbot @ 2019-07-27 8:23 ` Hillf Danton 1 sibling, 0 replies; 7+ messages in thread From: Hillf Danton @ 2019-07-27 8:23 UTC (permalink / raw) To: syzbot Cc: michal.lkml, kvm, mst, netdev, linux-kbuild, syzkaller-bugs, linux-kernel, virtualization, yamada.masahiro, torvalds Fri, 26 Jul 2019 08:26:01 -0700 (PDT) > syzbot has bisected this bug to: > > commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 > Author: Linus Torvalds <torvalds@linux-foundation.org> > Date: Sun Jul 7 22:41:56 2019 +0000 > > Linux 5.2 > > bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=118810bfa00000 > start commit: 13bf6d6a Add linux-next specific files for 20190725 > git tree: linux-next > kernel config: https://syzkaller.appspot.com/x/.config?x=8ae987d803395886 > dashboard link: https://syzkaller.appspot.com/bug?extid=36e93b425cd6eb54fcc1 > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15112f3fa00000 > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=131ab578600000 > > Reported-by: syzbot+36e93b425cd6eb54fcc1@syzkaller.appspotmail.com > Fixes: 0ecfebd2b524 ("Linux 5.2") > > For information about bisection process see: https://goo.gl/tpsmEJ#bisection --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -787,7 +787,6 @@ static void vhost_setup_uaddr(struct vho size_t size, bool write) { struct vhost_uaddr *addr = &vq->uaddrs[index]; - spin_lock(&vq->mmu_lock); addr->uaddr = uaddr; addr->size = size; @@ -797,7 +796,10 @@ static void vhost_setup_uaddr(struct vho static void vhost_setup_vq_uaddr(struct vhost_virtqueue *vq) { spin_lock(&vq->mmu_lock); - + /* + * deadlock if managing to take mmu_lock again while + * setting up uaddr + */ vhost_setup_uaddr(vq, VHOST_ADDR_DESC, (unsigned long)vq->desc, vhost_get_desc_size(vq, vq->num), -- ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-07-28 8:36 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2019-07-26 9:38 INFO: rcu detected stall in vhost_worker syzbot 2019-07-26 15:26 ` syzbot 2019-07-26 21:35 ` Linus Torvalds 2019-07-26 21:35 ` Linus Torvalds 2019-07-28 8:36 ` Michael S. Tsirkin 2019-07-28 8:36 ` Michael S. Tsirkin 2019-07-27 8:23 ` Hillf Danton
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.