From: Pavel Begunkov <asml.silence@gmail.com>
To: syzbot <syzbot+9a512c5bdc15635eab70@syzkaller.appspotmail.com>,
axboe@kernel.dk, io-uring@vger.kernel.org,
linux-kernel@vger.kernel.org, syzkaller-bugs@googlegroups.com
Subject: Re: possible deadlock in io_link_timeout_fn
Date: Tue, 23 Feb 2021 13:50:00 +0000 [thread overview]
Message-ID: <5003d05e-4fd9-5d0b-86ed-4e81b48ccf76@gmail.com> (raw)
In-Reply-To: <00000000000097f98905bc00fd53@google.com>
On 23/02/2021 13:35, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: 31caf8b2 Merge branch 'linus' of git://git.kernel.org/pub/..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=11c64f12d00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=5a8f3a57fabb4015
> dashboard link: https://syzkaller.appspot.com/bug?extid=9a512c5bdc15635eab70
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+9a512c5bdc15635eab70@syzkaller.appspotmail.com
Ok, that IRQ recycling stuff kicks in. I guess we can take
work_clean out of spin, I'll try it out this evening.
Either, Jen's io-wq patches go 5.12 in the end, and it would probably be
solved naturally.
>
> =====================================================
> WARNING: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected
> 5.11.0-syzkaller #0 Not tainted
> -----------------------------------------------------
> syz-executor.0/12185 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
> ffff888013db4820 (&fs->lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
> ffff888013db4820 (&fs->lock){+.+.}-{2:2}, at: io_req_clean_work fs/io_uring.c:1405 [inline]
> ffff888013db4820 (&fs->lock){+.+.}-{2:2}, at: io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
>
> and this task is already holding:
> ffff88806bea6718 (&ctx->completion_lock){-...}-{2:2}, at: io_req_complete_post+0x4e/0x920 fs/io_uring.c:1923
> which would create a new lock dependency:
> (&ctx->completion_lock){-...}-{2:2} -> (&fs->lock){+.+.}-{2:2}
>
> but this new dependency connects a HARDIRQ-irq-safe lock:
> (&ctx->completion_lock){-...}-{2:2}
>
> ... which became HARDIRQ-irq-safe at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
> io_link_timeout_fn+0xbf/0x720 fs/io_uring.c:6495
> __run_hrtimer kernel/time/hrtimer.c:1519 [inline]
> __hrtimer_run_queues+0x609/0xe40 kernel/time/hrtimer.c:1583
> hrtimer_interrupt+0x334/0x940 kernel/time/hrtimer.c:1645
> local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
> __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
> asm_call_irq_on_stack+0xf/0x20
> __run_sysvec_on_irqstack arch/x86/include/asm/irq_stack.h:37 [inline]
> run_sysvec_on_irqstack_cond arch/x86/include/asm/irq_stack.h:89 [inline]
> sysvec_apic_timer_interrupt+0xbd/0x100 arch/x86/kernel/apic/apic.c:1100
> asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:635
> __raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline]
> _raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:199
> spin_unlock_irq include/linux/spinlock.h:404 [inline]
> io_queue_linked_timeout+0x194/0x1f0 fs/io_uring.c:6541
> __io_queue_sqe+0x32f/0xdb0 fs/io_uring.c:6607
> __io_req_task_submit+0x18e/0x240 fs/io_uring.c:2344
> __tctx_task_work fs/io_uring.c:2204 [inline]
> tctx_task_work+0x12b/0x330 fs/io_uring.c:2230
> task_work_run+0xdd/0x1a0 kernel/task_work.c:140
> tracehook_notify_signal include/linux/tracehook.h:212 [inline]
> handle_signal_work kernel/entry/common.c:145 [inline]
> exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
> exit_to_user_mode_prepare+0x221/0x250 kernel/entry/common.c:208
> __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
> syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
> entry_SYSCALL_64_after_hwframe+0x44/0xae
>
> to a HARDIRQ-irq-unsafe lock:
> (&fs->lock){+.+.}-{2:2}
>
> ... which became HARDIRQ-irq-unsafe at:
> ...
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
> init_chdir+0x106/0x14e fs/init.c:54
> devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
> devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
> kthread+0x3b1/0x4a0 kernel/kthread.c:292
> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
>
> other info that might help us debug this:
>
> Possible interrupt unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(&fs->lock);
> local_irq_disable();
> lock(&ctx->completion_lock);
> lock(&fs->lock);
> <Interrupt>
> lock(&ctx->completion_lock);
>
> *** DEADLOCK ***
>
> 1 lock held by syz-executor.0/12185:
> #0: ffff88806bea6718 (&ctx->completion_lock){-...}-{2:2}, at: io_req_complete_post+0x4e/0x920 fs/io_uring.c:1923
>
> the dependencies between HARDIRQ-irq-safe lock and the holding lock:
> -> (&ctx->completion_lock){-...}-{2:2} {
> IN-HARDIRQ-W at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
> io_link_timeout_fn+0xbf/0x720 fs/io_uring.c:6495
> __run_hrtimer kernel/time/hrtimer.c:1519 [inline]
> __hrtimer_run_queues+0x609/0xe40 kernel/time/hrtimer.c:1583
> hrtimer_interrupt+0x334/0x940 kernel/time/hrtimer.c:1645
> local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
> __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
> asm_call_irq_on_stack+0xf/0x20
> __run_sysvec_on_irqstack arch/x86/include/asm/irq_stack.h:37 [inline]
> run_sysvec_on_irqstack_cond arch/x86/include/asm/irq_stack.h:89 [inline]
> sysvec_apic_timer_interrupt+0xbd/0x100 arch/x86/kernel/apic/apic.c:1100
> asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:635
> __raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline]
> _raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:199
> spin_unlock_irq include/linux/spinlock.h:404 [inline]
> io_queue_linked_timeout+0x194/0x1f0 fs/io_uring.c:6541
> __io_queue_sqe+0x32f/0xdb0 fs/io_uring.c:6607
> __io_req_task_submit+0x18e/0x240 fs/io_uring.c:2344
> __tctx_task_work fs/io_uring.c:2204 [inline]
> tctx_task_work+0x12b/0x330 fs/io_uring.c:2230
> task_work_run+0xdd/0x1a0 kernel/task_work.c:140
> tracehook_notify_signal include/linux/tracehook.h:212 [inline]
> handle_signal_work kernel/entry/common.c:145 [inline]
> exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
> exit_to_user_mode_prepare+0x221/0x250 kernel/entry/common.c:208
> __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
> syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> INITIAL USE at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
> io_req_complete_post+0x4e/0x920 fs/io_uring.c:1923
> __io_req_complete fs/io_uring.c:1963 [inline]
> io_req_complete fs/io_uring.c:1968 [inline]
> io_queue_sqe+0xa3b/0xfa0 fs/io_uring.c:6620
> io_submit_sqe fs/io_uring.c:6707 [inline]
> io_submit_sqes+0x15f5/0x2b60 fs/io_uring.c:6939
> __do_sys_io_uring_enter+0x1154/0x1f50 fs/io_uring.c:9454
> do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> }
> ... key at: [<ffffffff8fe69a80>] __key.9+0x0/0x40
> ... acquired at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> io_req_clean_work fs/io_uring.c:1405 [inline]
> io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
> io_req_complete_post+0xf6/0x920 fs/io_uring.c:1933
> __io_req_complete fs/io_uring.c:1963 [inline]
> io_req_complete fs/io_uring.c:1968 [inline]
> io_cancel_defer_files fs/io_uring.c:8945 [inline]
> io_uring_cancel_task_requests+0x67e/0xea0 fs/io_uring.c:9052
> __io_uring_files_cancel+0x110/0x1b0 fs/io_uring.c:9140
> io_uring_files_cancel include/linux/io_uring.h:65 [inline]
> do_exit+0x2fe/0x2ae0 kernel/exit.c:780
> do_group_exit+0x125/0x310 kernel/exit.c:922
> get_signal+0x42c/0x2100 kernel/signal.c:2773
> arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
> handle_signal_work kernel/entry/common.c:147 [inline]
> exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
> exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:208
> __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
> syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
> entry_SYSCALL_64_after_hwframe+0x44/0xae
>
>
> the dependencies between the lock to be acquired
> and HARDIRQ-irq-unsafe lock:
> -> (&fs->lock){+.+.}-{2:2} {
> HARDIRQ-ON-W at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
> init_chdir+0x106/0x14e fs/init.c:54
> devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
> devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
> kthread+0x3b1/0x4a0 kernel/kthread.c:292
> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
> SOFTIRQ-ON-W at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
> init_chdir+0x106/0x14e fs/init.c:54
> devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
> devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
> kthread+0x3b1/0x4a0 kernel/kthread.c:292
> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
> INITIAL USE at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
> init_chdir+0x106/0x14e fs/init.c:54
> devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
> devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
> kthread+0x3b1/0x4a0 kernel/kthread.c:292
> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
> }
> ... key at: [<ffffffff8fe68260>] __key.1+0x0/0x40
> ... acquired at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> io_req_clean_work fs/io_uring.c:1405 [inline]
> io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
> io_req_complete_post+0xf6/0x920 fs/io_uring.c:1933
> __io_req_complete fs/io_uring.c:1963 [inline]
> io_req_complete fs/io_uring.c:1968 [inline]
> io_cancel_defer_files fs/io_uring.c:8945 [inline]
> io_uring_cancel_task_requests+0x67e/0xea0 fs/io_uring.c:9052
> __io_uring_files_cancel+0x110/0x1b0 fs/io_uring.c:9140
> io_uring_files_cancel include/linux/io_uring.h:65 [inline]
> do_exit+0x2fe/0x2ae0 kernel/exit.c:780
> do_group_exit+0x125/0x310 kernel/exit.c:922
> get_signal+0x42c/0x2100 kernel/signal.c:2773
> arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
> handle_signal_work kernel/entry/common.c:147 [inline]
> exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
> exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:208
> __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
> syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
> entry_SYSCALL_64_after_hwframe+0x44/0xae
>
>
> stack backtrace:
> CPU: 3 PID: 12185 Comm: syz-executor.0 Not tainted 5.11.0-syzkaller #0
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014
> Call Trace:
> __dump_stack lib/dump_stack.c:79 [inline]
> dump_stack+0xfa/0x151 lib/dump_stack.c:120
> print_bad_irq_dependency kernel/locking/lockdep.c:2460 [inline]
> check_irq_usage.cold+0x50d/0x744 kernel/locking/lockdep.c:2689
> check_prev_add kernel/locking/lockdep.c:2940 [inline]
> check_prevs_add kernel/locking/lockdep.c:3059 [inline]
> validate_chain kernel/locking/lockdep.c:3674 [inline]
> __lock_acquire+0x2b2c/0x54c0 kernel/locking/lockdep.c:4900
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> io_req_clean_work fs/io_uring.c:1405 [inline]
> io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
> io_req_complete_post+0xf6/0x920 fs/io_uring.c:1933
> __io_req_complete fs/io_uring.c:1963 [inline]
> io_req_complete fs/io_uring.c:1968 [inline]
> io_cancel_defer_files fs/io_uring.c:8945 [inline]
> io_uring_cancel_task_requests+0x67e/0xea0 fs/io_uring.c:9052
> __io_uring_files_cancel+0x110/0x1b0 fs/io_uring.c:9140
> io_uring_files_cancel include/linux/io_uring.h:65 [inline]
> do_exit+0x2fe/0x2ae0 kernel/exit.c:780
> do_group_exit+0x125/0x310 kernel/exit.c:922
> get_signal+0x42c/0x2100 kernel/signal.c:2773
> arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
> handle_signal_work kernel/entry/common.c:147 [inline]
> exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
> exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:208
> __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
> syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> RIP: 0033:0x465ef9
> Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
> RSP: 002b:00007ffb56aa0218 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
> RAX: 0000000000000000 RBX: 000000000056bf68 RCX: 0000000000465ef9
> RDX: 0000000000000000 RSI: 0000000000000080 RDI: 000000000056bf68
> RBP: 000000000056bf60 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf6c
> R13: 00007fff198147ff R14: 00007ffb56aa0300 R15: 0000000000022000
>
>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@googlegroups.com.
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>
--
Pavel Begunkov
prev parent reply other threads:[~2021-02-23 13:54 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-23 13:35 possible deadlock in io_link_timeout_fn syzbot
2021-02-23 13:50 ` Pavel Begunkov [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5003d05e-4fd9-5d0b-86ed-4e81b48ccf76@gmail.com \
--to=asml.silence@gmail.com \
--cc=axboe@kernel.dk \
--cc=io-uring@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=syzbot+9a512c5bdc15635eab70@syzkaller.appspotmail.com \
--cc=syzkaller-bugs@googlegroups.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.