Re: [syzbot] [block?] possible deadlock in blk_mq_submit_bio

syzbot
2024-12-10T11:12:03Z

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
possible deadlock in __submit_bio

======================================================
WARNING: possible circular locking dependency detected
6.13.0-rc1-syzkaller-00011-gc018ec9dd144 #0 Not tainted
------------------------------------------------------
syz.0.15/7623 is trying to acquire lock:
ffff0000ca7b1de8 (&q->q_usage_counter(io)#17){++++}-{0:0}, at: __submit_bio+0x1a0/0x4f8 block/blk-core.c:629

but task is already holding lock:
ffff0000d771a0b0 (&tree->tree_lock){+.+.}-{4:4}, at: hfsplus_find_init+0x144/0x1bc fs/hfsplus/bfind.c:28

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #2 (&tree->tree_lock){+.+.}-{4:4}:
__mutex_lock_common+0x218/0x28f4 kernel/locking/mutex.c:585
__mutex_lock kernel/locking/mutex.c:735 [inline]
mutex_lock_nested+0x2c/0x38 kernel/locking/mutex.c:787
hfsplus_find_init+0x144/0x1bc fs/hfsplus/bfind.c:28
hfsplus_cat_write_inode+0x1a4/0xd48 fs/hfsplus/inode.c:589
hfsplus_write_inode+0x15c/0x4dc fs/hfsplus/super.c:161
write_inode fs/fs-writeback.c:1525 [inline]
__writeback_single_inode+0x5a0/0x15a4 fs/fs-writeback.c:1745
writeback_single_inode+0x18c/0x554 fs/fs-writeback.c:1801
sync_inode_metadata+0xc4/0x12c fs/fs-writeback.c:2871
hfsplus_file_fsync+0xe4/0x4c8 fs/hfsplus/inode.c:316
vfs_fsync_range fs/sync.c:187 [inline]
vfs_fsync+0x154/0x18c fs/sync.c:201
__loop_update_dio+0x248/0x420 drivers/block/loop.c:204
loop_set_status+0x538/0x7f4 drivers/block/loop.c:1289
lo_ioctl+0xf10/0x1c48
blkdev_ioctl+0x3a8/0xa8c block/ioctl.c:693
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:906 [inline]
__se_sys_ioctl fs/ioctl.c:892 [inline]
__arm64_sys_ioctl+0x14c/0x1cc fs/ioctl.c:892
__invoke_syscall arch/arm64/kernel/syscall.c:35 [inline]
invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:49
el0_svc_common+0x130/0x23c arch/arm64/kernel/syscall.c:132
do_el0_svc+0x48/0x58 arch/arm64/kernel/syscall.c:151
el0_svc+0x54/0x168 arch/arm64/kernel/entry-common.c:744
el0t_64_sync_handler+0x84/0x108 arch/arm64/kernel/entry-common.c:762
el0t_64_sync+0x198/0x19c arch/arm64/kernel/entry.S:600

-> #1 (&sb->s_type->i_mutex_key#20){+.+.}-{4:4}:
down_write+0x50/0xc0 kernel/locking/rwsem.c:1577
inode_lock include/linux/fs.h:818 [inline]
hfsplus_file_fsync+0xd8/0x4c8 fs/hfsplus/inode.c:311
vfs_fsync_range fs/sync.c:187 [inline]
vfs_fsync+0x154/0x18c fs/sync.c:201
__loop_update_dio+0x248/0x420 drivers/block/loop.c:204
loop_set_status+0x538/0x7f4 drivers/block/loop.c:1289
lo_ioctl+0xf10/0x1c48
blkdev_ioctl+0x3a8/0xa8c block/ioctl.c:693
vfs_ioctl fs/ioctl.c:51 [inline]
__do_sys_ioctl fs/ioctl.c:906 [inline]
__se_sys_ioctl fs/ioctl.c:892 [inline]
__arm64_sys_ioctl+0x14c/0x1cc fs/ioctl.c:892
__invoke_syscall arch/arm64/kernel/syscall.c:35 [inline]
invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:49
el0_svc_common+0x130/0x23c arch/arm64/kernel/syscall.c:132
do_el0_svc+0x48/0x58 arch/arm64/kernel/syscall.c:151
el0_svc+0x54/0x168 arch/arm64/kernel/entry-common.c:744
el0t_64_sync_handler+0x84/0x108 arch/arm64/kernel/entry-common.c:762
el0t_64_sync+0x198/0x19c arch/arm64/kernel/entry.S:600

-> #0 (&q->q_usage_counter(io)#17){++++}-{0:0}:
check_prev_add kernel/locking/lockdep.c:3161 [inline]
check_prevs_add kernel/locking/lockdep.c:3280 [inline]
validate_chain kernel/locking/lockdep.c:3904 [inline]
__lock_acquire+0x34f0/0x7904 kernel/locking/lockdep.c:5226
lock_acquire+0x23c/0x724 kernel/locking/lockdep.c:5849
bio_queue_enter block/blk.h:75 [inline]
blk_mq_submit_bio+0x1254/0x2070 block/blk-mq.c:3093
__submit_bio+0x1a0/0x4f8 block/blk-core.c:629
__submit_bio_noacct_mq block/blk-core.c:710 [inline]
submit_bio_noacct_nocheck+0x3bc/0xcbc block/blk-core.c:739
submit_bio_noacct+0xc6c/0x166c block/blk-core.c:868
submit_bio+0x374/0x564 block/blk-core.c:910
submit_bh_wbc+0x3f8/0x4c8 fs/buffer.c:2814
submit_bh fs/buffer.c:2819 [inline]
block_read_full_folio+0x7d0/0x950 fs/buffer.c:2446
hfsplus_read_folio+0x28/0x38 fs/hfsplus/inode.c:28
filemap_read_folio+0x108/0x318 mm/filemap.c:2366
do_read_cache_folio+0x368/0x5c0 mm/filemap.c:3826
do_read_cache_page mm/filemap.c:3892 [inline]
read_cache_page+0x6c/0x15c mm/filemap.c:3901
read_mapping_page include/linux/pagemap.h:1005 [inline]
__hfs_bnode_create+0x3dc/0x6d4 fs/hfsplus/bnode.c:440
hfsplus_bnode_find+0x200/0xe60 fs/hfsplus/bnode.c:486
hfsplus_brec_find+0x134/0x4a0 fs/hfsplus/bfind.c:172
hfsplus_brec_read+0x38/0x128 fs/hfsplus/bfind.c:211
hfsplus_find_cat+0x140/0x4a0 fs/hfsplus/catalog.c:202
hfsplus_iget+0x34c/0x584 fs/hfsplus/super.c:83
hfsplus_fill_super+0xa5c/0x16f8 fs/hfsplus/super.c:504
get_tree_bdev_flags+0x38c/0x494 fs/super.c:1636
get_tree_bdev+0x2c/0x3c fs/super.c:1659
hfsplus_get_tree+0x28/0x38 fs/hfsplus/super.c:640
vfs_get_tree+0x90/0x28c fs/super.c:1814
do_new_mount+0x278/0x900 fs/namespace.c:3507
path_mount+0x590/0xe04 fs/namespace.c:3834
do_mount fs/namespace.c:3847 [inline]
__do_sys_mount fs/namespace.c:4057 [inline]
__se_sys_mount fs/namespace.c:4034 [inline]
__arm64_sys_mount+0x4d4/0x5ac fs/namespace.c:4034
__invoke_syscall arch/arm64/kernel/syscall.c:35 [inline]
invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:49
el0_svc_common+0x130/0x23c arch/arm64/kernel/syscall.c:132
do_el0_svc+0x48/0x58 arch/arm64/kernel/syscall.c:151
el0_svc+0x54/0x168 arch/arm64/kernel/entry-common.c:744
el0t_64_sync_handler+0x84/0x108 arch/arm64/kernel/entry-common.c:762
el0t_64_sync+0x198/0x19c arch/arm64/kernel/entry.S:600

other info that might help us debug this:

Chain exists of:
&q->q_usage_counter(io)#17 --> &sb->s_type->i_mutex_key#20 --> &tree->tree_lock

Possible unsafe locking scenario:

CPU0 CPU1
---- ----
lock(&tree->tree_lock);
lock(&sb->s_type->i_mutex_key#20);
lock(&tree->tree_lock);
rlock(&q->q_usage_counter(io)#17);

*** DEADLOCK ***

2 locks held by syz.0.15/7623:
#0: ffff0000cb7e60e0 (&type->s_umount_key#51/1){+.+.}-{4:4}, at: alloc_super+0x1b0/0x834 fs/super.c:344
#1: ffff0000d771a0b0 (&tree->tree_lock){+.+.}-{4:4}, at: hfsplus_find_init+0x144/0x1bc fs/hfsplus/bfind.c:28

stack backtrace:
CPU: 1 UID: 0 PID: 7623 Comm: syz.0.15 Not tainted 6.13.0-rc1-syzkaller-00011-gc018ec9dd144 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Call trace:
show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:484 (C)
__dump_stack lib/dump_stack.c:94 [inline]
dump_stack_lvl+0xe4/0x150 lib/dump_stack.c:120
dump_stack+0x1c/0x28 lib/dump_stack.c:129
print_circular_bug+0x154/0x1c0 kernel/locking/lockdep.c:2074
check_noncircular+0x310/0x404 kernel/locking/lockdep.c:2206
check_prev_add kernel/locking/lockdep.c:3161 [inline]
check_prevs_add kernel/locking/lockdep.c:3280 [inline]
validate_chain kernel/locking/lockdep.c:3904 [inline]
__lock_acquire+0x34f0/0x7904 kernel/locking/lockdep.c:5226
lock_acquire+0x23c/0x724 kernel/locking/lockdep.c:5849
bio_queue_enter block/blk.h:75 [inline]
blk_mq_submit_bio+0x1254/0x2070 block/blk-mq.c:3093
__submit_bio+0x1a0/0x4f8 block/blk-core.c:629
__submit_bio_noacct_mq block/blk-core.c:710 [inline]
submit_bio_noacct_nocheck+0x3bc/0xcbc block/blk-core.c:739
submit_bio_noacct+0xc6c/0x166c block/blk-core.c:868
submit_bio+0x374/0x564 block/blk-core.c:910
submit_bh_wbc+0x3f8/0x4c8 fs/buffer.c:2814
submit_bh fs/buffer.c:2819 [inline]
block_read_full_folio+0x7d0/0x950 fs/buffer.c:2446
hfsplus_read_folio+0x28/0x38 fs/hfsplus/inode.c:28
filemap_read_folio+0x108/0x318 mm/filemap.c:2366
do_read_cache_folio+0x368/0x5c0 mm/filemap.c:3826
do_read_cache_page mm/filemap.c:3892 [inline]
read_cache_page+0x6c/0x15c mm/filemap.c:3901
read_mapping_page include/linux/pagemap.h:1005 [inline]
__hfs_bnode_create+0x3dc/0x6d4 fs/hfsplus/bnode.c:440
hfsplus_bnode_find+0x200/0xe60 fs/hfsplus/bnode.c:486
hfsplus_brec_find+0x134/0x4a0 fs/hfsplus/bfind.c:172
hfsplus_brec_read+0x38/0x128 fs/hfsplus/bfind.c:211
hfsplus_find_cat+0x140/0x4a0 fs/hfsplus/catalog.c:202
hfsplus_iget+0x34c/0x584 fs/hfsplus/super.c:83
hfsplus_fill_super+0xa5c/0x16f8 fs/hfsplus/super.c:504
get_tree_bdev_flags+0x38c/0x494 fs/super.c:1636
get_tree_bdev+0x2c/0x3c fs/super.c:1659
hfsplus_get_tree+0x28/0x38 fs/hfsplus/super.c:640
vfs_get_tree+0x90/0x28c fs/super.c:1814
do_new_mount+0x278/0x900 fs/namespace.c:3507
path_mount+0x590/0xe04 fs/namespace.c:3834
do_mount fs/namespace.c:3847 [inline]
__do_sys_mount fs/namespace.c:4057 [inline]
__se_sys_mount fs/namespace.c:4034 [inline]
__arm64_sys_mount+0x4d4/0x5ac fs/namespace.c:4034
__invoke_syscall arch/arm64/kernel/syscall.c:35 [inline]
invoke_syscall+0x98/0x2b8 arch/arm64/kernel/syscall.c:49
el0_svc_common+0x130/0x23c arch/arm64/kernel/syscall.c:132
do_el0_svc+0x48/0x58 arch/arm64/kernel/syscall.c:151
el0_svc+0x54/0x168 arch/arm64/kernel/entry-common.c:744
el0t_64_sync_handler+0x84/0x108 arch/arm64/kernel/entry-common.c:762
el0t_64_sync+0x198/0x19c arch/arm64/kernel/entry.S:600
loop0: detected capacity change from 1024 to 3
Dev loop0: unable to read RDB block 3
loop0: unable to read partition table
loop0: partition table beyond EOD, truncated
loop_reread_partitions: partition scan of loop0 (�Rt�\v*�3\f!6{\x06bO�0�\x7f�\x17.�Qʝ�\x03� H�"Uqd\�'�Lz�8�\b���w1�A\bH��\x10�\x19��) failed (rc=-5)

Tested on:

commit: c018ec9d block: rnull: Initialize the module in place
git tree: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git for-6.14/block
console output: https://syzkaller.appspot.com/x/log.txt?x=124c68f8580000
kernel config: https://syzkaller.appspot.com/x/.config?x=bd60186d08e947a5
dashboard link: https://syzkaller.appspot.com/bug?extid=5218c85078236fc46227
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
userspace arch: arm64

Note: no patches were applied.