[syzbot] [block?] possible deadlock in blk_mq_alloc_request

[syzbot] [block?] possible deadlock in blk_mq_alloc_request

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    c88416ba074a Add linux-next specific files for 20241101
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=17e59aa7980000
kernel config:  https://syzkaller.appspot.com/x/.config?x=704b6be2ac2f205f
dashboard link: https://syzkaller.appspot.com/bug?extid=ca7d7c797fee31d2b474
compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=1250b630580000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/760a8c88d0c3/disk-c88416ba.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/46e4b0a851a2/vmlinux-c88416ba.xz
kernel image: https://storage.googleapis.com/syzbot-assets/428e2c784b75/bzImage-c88416ba.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+ca7d7c797fee31d2b474@syzkaller.appspotmail.com

============================================
WARNING: possible recursive locking detected
6.12.0-rc5-next-20241101-syzkaller #0 Not tainted
--------------------------------------------
udevd/6086 is trying to acquire lock:
ffff8880288261c0 (&q->q_usage_counter(queue)#67){++++}-{0:0}, at: blk_mq_alloc_request+0x26b/0xab0 block/blk-mq.c:626

but task is already holding lock:
ffff8880288261c0 (&q->q_usage_counter(queue)#67){++++}-{0:0}, at: blk_freeze_queue block/blk-mq.c:177 [inline]
ffff8880288261c0 (&q->q_usage_counter(queue)#67){++++}-{0:0}, at: blk_mq_freeze_queue+0x15/0x20 block/blk-mq.c:187

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&q->q_usage_counter(queue)#67);
  lock(&q->q_usage_counter(queue)#67);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

3 locks held by udevd/6086:
 #0: ffff888034a534c8 (&disk->open_mutex){+.+.}-{4:4}, at: bdev_open+0xf0/0xc50 block/bdev.c:904
 #1: ffff888028826188 (&q->q_usage_counter(io)#81){+.+.}-{0:0}, at: blk_freeze_queue block/blk-mq.c:177 [inline]
 #1: ffff888028826188 (&q->q_usage_counter(io)#81){+.+.}-{0:0}, at: blk_mq_freeze_queue+0x15/0x20 block/blk-mq.c:187
 #2: ffff8880288261c0 (&q->q_usage_counter(queue)#67){++++}-{0:0}, at: blk_freeze_queue block/blk-mq.c:177 [inline]
 #2: ffff8880288261c0 (&q->q_usage_counter(queue)#67){++++}-{0:0}, at: blk_mq_freeze_queue+0x15/0x20 block/blk-mq.c:187

stack backtrace:
CPU: 0 UID: 0 PID: 6086 Comm: udevd Not tainted 6.12.0-rc5-next-20241101-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120
 print_deadlock_bug+0x483/0x620 kernel/locking/lockdep.c:3037
 check_deadlock kernel/locking/lockdep.c:3089 [inline]
 validate_chain+0x15e2/0x5920 kernel/locking/lockdep.c:3891
 __lock_acquire+0x1397/0x2100 kernel/locking/lockdep.c:5226
 lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5849
 blk_queue_enter+0xe1/0x600 block/blk-core.c:328
 blk_mq_alloc_request+0x26b/0xab0 block/blk-mq.c:626
 scsi_alloc_request drivers/scsi/scsi_lib.c:1222 [inline]
 scsi_execute_cmd+0x177/0x1090 drivers/scsi/scsi_lib.c:304
 ioctl_internal_command+0x16a/0x7f0 drivers/scsi/scsi_ioctl.c:79
 scsi_set_medium_removal+0x15a/0x340 drivers/scsi/scsi_ioctl.c:139
 sd_open+0x43c/0x610 drivers/scsi/sd.c:1564
 blkdev_get_whole+0x8e/0x450 block/bdev.c:689
 bdev_open+0x2d4/0xc50 block/bdev.c:916
 blkdev_open+0x389/0x4f0 block/fops.c:627
 do_dentry_open+0xbe1/0x1b70 fs/open.c:962
 vfs_open+0x3e/0x330 fs/open.c:1092
 do_open fs/namei.c:3774 [inline]
 path_openat+0x2c84/0x3590 fs/namei.c:3933
 do_filp_open+0x235/0x490 fs/namei.c:3960
 do_sys_openat2+0x13e/0x1d0 fs/open.c:1419
 do_sys_open fs/open.c:1434 [inline]
 __do_sys_openat fs/open.c:1450 [inline]
 __se_sys_openat fs/open.c:1445 [inline]
 __x64_sys_openat+0x247/0x2a0 fs/open.c:1445
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f21713169a4
Code: 24 20 48 8d 44 24 30 48 89 44 24 28 64 8b 04 25 18 00 00 00 85 c0 75 2c 44 89 e2 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 76 60 48 8b 15 55 a4 0d 00 f7 d8 64 89 02 48 83
RSP: 002b:00007ffec20c7d50 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 00005559ea473800 RCX: 00007f21713169a4
RDX: 00000000000a0800 RSI: 00005559ea474730 RDI: 00000000ffffff9c
RBP: 00005559ea474730 R08: 0000000000000006 R09: 7fffffffffffffff
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000000a0800
R13: 00005559ea468070 R14: 0000000000000001 R15: 00005559ea457910
 </TASK>
udevd[6086]: inotify_add_watch(7, /dev/sde, 10) failed: No such file or directory
sd 6:0:0:1: [sde] Test Unit Ready failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
udevd[6086]: inotify_add_watch(7, /dev/sde, 10) failed: No such file or directory
udevd[6086]: inotify_add_watch(7, /dev/sdd, 10) failed: No such file or directory


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Re: [syzbot] [block?] possible deadlock in blk_mq_alloc_request

[Ming Lei]

On Tue, Nov 05, 2024 at 06:40:22PM -0800, syzbot wrote:
> Hello,
> 
> syzbot found the following issue on:
> 
> HEAD commit:    c88416ba074a Add linux-next specific files for 20241101
> git tree:       linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=17e59aa7980000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=704b6be2ac2f205f
> dashboard link: https://syzkaller.appspot.com/bug?extid=ca7d7c797fee31d2b474
> compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=1250b630580000
> 
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/760a8c88d0c3/disk-c88416ba.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/46e4b0a851a2/vmlinux-c88416ba.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/428e2c784b75/bzImage-c88416ba.xz
> 
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+ca7d7c797fee31d2b474@syzkaller.appspotmail.com
> 
> ============================================
> WARNING: possible recursive locking detected
> 6.12.0-rc5-next-20241101-syzkaller #0 Not tainted
> --------------------------------------------
> udevd/6086 is trying to acquire lock:
> ffff8880288261c0 (&q->q_usage_counter(queue)#67){++++}-{0:0}, at: blk_mq_alloc_request+0x26b/0xab0 block/blk-mq.c:626
> 
> but task is already holding lock:
> ffff8880288261c0 (&q->q_usage_counter(queue)#67){++++}-{0:0}, at: blk_freeze_queue block/blk-mq.c:177 [inline]
> ffff8880288261c0 (&q->q_usage_counter(queue)#67){++++}-{0:0}, at: blk_mq_freeze_queue+0x15/0x20 block/blk-mq.c:187
> 
> other info that might help us debug this:
>  Possible unsafe locking scenario:
> 
>        CPU0
>        ----
>   lock(&q->q_usage_counter(queue)#67);
>   lock(&q->q_usage_counter(queue)#67);
> 
>  *** DEADLOCK ***
> 
>  May be due to missing lock nesting notation
> 
> 3 locks held by udevd/6086:
>  #0: ffff888034a534c8 (&disk->open_mutex){+.+.}-{4:4}, at: bdev_open+0xf0/0xc50 block/bdev.c:904
>  #1: ffff888028826188 (&q->q_usage_counter(io)#81){+.+.}-{0:0}, at: blk_freeze_queue block/blk-mq.c:177 [inline]
>  #1: ffff888028826188 (&q->q_usage_counter(io)#81){+.+.}-{0:0}, at: blk_mq_freeze_queue+0x15/0x20 block/blk-mq.c:187
>  #2: ffff8880288261c0 (&q->q_usage_counter(queue)#67){++++}-{0:0}, at: blk_freeze_queue block/blk-mq.c:177 [inline]
>  #2: ffff8880288261c0 (&q->q_usage_counter(queue)#67){++++}-{0:0}, at: blk_mq_freeze_queue+0x15/0x20 block/blk-mq.c:187

Not get idea how blk_mq_freeze_queue is called in this context.

Is the blk_mq_unfreeze_queue() in sd_revalidate_disk() not released?

Anyway, please test the not-merged fixes.

#syz test: https://github.com/ming1/linux.git for-next


Thanks, 
Ming

Re: [syzbot] [block?] possible deadlock in blk_mq_alloc_request

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-by: syzbot+ca7d7c797fee31d2b474@syzkaller.appspotmail.com
Tested-by: syzbot+ca7d7c797fee31d2b474@syzkaller.appspotmail.com

Tested on:

commit:         72697401 block: don't verify IO lock for freeze/unfree..
git tree:       https://github.com/ming1/linux.git for-next
console output: https://syzkaller.appspot.com/x/log.txt?x=172b9d5f980000
kernel config:  https://syzkaller.appspot.com/x/.config?x=dfea72efa3e2aef2
dashboard link: https://syzkaller.appspot.com/bug?extid=ca7d7c797fee31d2b474
compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40

Note: no patches were applied.
Note: testing is done by a robot and is best-effort only.

Re: [syzbot] [usb?] [block?] possible deadlock in blk_mq_alloc_request

[syzbot]

syzbot has bisected this issue to:

commit f1be1788a32e8fa63416ad4518bbd1a85a825c9d
Author: Ming Lei <ming.lei@redhat.com>
Date:   Fri Oct 25 00:37:20 2024 +0000

    block: model freeze & enter queue as lock for supporting lockdep

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=15fd34c4580000
start commit:   c88416ba074a Add linux-next specific files for 20241101
git tree:       linux-next
final oops:     https://syzkaller.appspot.com/x/report.txt?x=17fd34c4580000
console output: https://syzkaller.appspot.com/x/log.txt?x=13fd34c4580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=704b6be2ac2f205f
dashboard link: https://syzkaller.appspot.com/bug?extid=ca7d7c797fee31d2b474
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=1250b630580000

Reported-by: syzbot+ca7d7c797fee31d2b474@syzkaller.appspotmail.com
Fixes: f1be1788a32e ("block: model freeze & enter queue as lock for supporting lockdep")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection