* [PATCH fs/hfs] hfs: fix deadlock in hfs_extend_file()
2024-04-18 14:41 [syzbot] [hfs?] possible deadlock in hfs_extend_file (3) syzbot
@ 2024-04-23 9:31 ` Jeongjun Park
2024-06-23 9:36 ` [syzbot] [hfs?] possible deadlock in hfs_extend_file (3) syzbot
1 sibling, 0 replies; 3+ messages in thread
From: Jeongjun Park @ 2024-04-23 9:31 UTC (permalink / raw)
To: syzbot+2a62f58f1a4951a549bb
Cc: jlayton, jack, linux-fsdevel, linux-kernel, syzkaller-bugs,
Jeongjun Park
======================================================
WARNING: possible circular locking dependency detected
6.9.0-rc4-syzkaller-00274-g3b68086599f8 #0 Not tainted
------------------------------------------------------
kworker/u8:6/1059 is trying to acquire lock:
ffff88805bd4a7f8 (&HFS_I(tree->inode)->extents_lock){+.+.}-{3:3}, at: hfs_extend_file+0xff/0x1450 fs/hfs/extent.c:397
but task is already holding lock:
ffff88806a48c0b0 (&tree->tree_lock#2/1){+.+.}-{3:3}, at: hfs_find_init+0x16e/0x1f0
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&tree->tree_lock#2/1){+.+.}-{3:3}:
lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754
__mutex_lock_common kernel/locking/mutex.c:608 [inline]
__mutex_lock+0x136/0xd70 kernel/locking/mutex.c:752
hfs_find_init+0x16e/0x1f0
hfs_ext_read_extent fs/hfs/extent.c:200 [inline]
hfs_extend_file+0x31b/0x1450 fs/hfs/extent.c:401
hfs_bmap_reserve+0xd9/0x400 fs/hfs/btree.c:234
hfs_cat_create+0x1e0/0x970 fs/hfs/catalog.c:104
hfs_create+0x66/0xe0 fs/hfs/dir.c:202
lookup_open fs/namei.c:3497 [inline]
open_last_lookups fs/namei.c:3566 [inline]
path_openat+0x1425/0x3240 fs/namei.c:3796
do_filp_open+0x235/0x490 fs/namei.c:3826
do_sys_openat2+0x13e/0x1d0 fs/open.c:1406
do_sys_open fs/open.c:1421 [inline]
__do_sys_openat fs/open.c:1437 [inline]
__se_sys_openat fs/open.c:1432 [inline]
__x64_sys_openat+0x247/0x2a0 fs/open.c:1432
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xf5/0x240 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x77/0x7f
-> #0 (&HFS_I(tree->inode)->extents_lock){+.+.}-{3:3}:
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18cb/0x58e0 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754
__mutex_lock_common kernel/locking/mutex.c:608 [inline]
__mutex_lock+0x136/0xd70 kernel/locking/mutex.c:752
hfs_extend_file+0xff/0x1450 fs/hfs/extent.c:397
hfs_bmap_reserve+0xd9/0x400 fs/hfs/btree.c:234
__hfs_ext_write_extent+0x22e/0x4f0 fs/hfs/extent.c:121
hfs_ext_write_extent+0x154/0x1d0 fs/hfs/extent.c:144
hfs_write_inode+0xbc/0xec0 fs/hfs/inode.c:427
write_inode fs/fs-writeback.c:1498 [inline]
__writeback_single_inode+0x6b9/0x10b0 fs/fs-writeback.c:1715
writeback_sb_inodes+0x905/0x1260 fs/fs-writeback.c:1941
wb_writeback+0x457/0xce0 fs/fs-writeback.c:2117
wb_do_writeback fs/fs-writeback.c:2264 [inline]
wb_workfn+0x410/0x1090 fs/fs-writeback.c:2304
process_one_work kernel/workqueue.c:3254 [inline]
process_scheduled_works+0xa10/0x17c0 kernel/workqueue.c:3335
worker_thread+0x86d/0xd70 kernel/workqueue.c:3416
kthread+0x2f0/0x390 kernel/kthread.c:388
ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&tree->tree_lock#2/1);
lock(&HFS_I(tree->inode)->extents_lock);
lock(&tree->tree_lock#2/1);
lock(&HFS_I(tree->inode)->extents_lock);
*** DEADLOCK ***
3 locks held by kworker/u8:6/1059:
#0: ffff88801be87148 ((wq_completion)writeback){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3229 [inline]
#0: ffff88801be87148 ((wq_completion)writeback){+.+.}-{0:0}, at: process_scheduled_works+0x8e0/0x17c0 kernel/workqueue.c:3335
#1: ffffc90004197d00 ((work_completion)(&(&wb->dwork)->work)){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3230 [inline]
#1: ffffc90004197d00 ((work_completion)(&(&wb->dwork)->work)){+.+.}-{0:0}, at: process_scheduled_works+0x91b/0x17c0 kernel/workqueue.c:3335
#2: ffff88806a48c0b0 (&tree->tree_lock#2/1){+.+.}-{3:3}, at: hfs_find_init+0x16e/0x1f0
======================================================
When a file expansion operation occurs in the hfs file system,
unnecessary locking occurs due to recursion. This situation does
not appear to be easy to reproduce, but it is very strange logic
and must be fixed.
Whether this recursion is intended behavior or not, I think it is a
good idea to prevent deadlock by placing mutex_lock() in a higher
function than hfs_extend_file().
Reported-by: syzbot+2a62f58f1a4951a549bb@syzkaller.appspotmail.com
Fixes: 39f8d472f280 ("hfs: convert extents_lock in a mutex")
Fixes: 1267a07be5eb ("hfs: fix return value of hfs_get_block()")
Fixes: 54640c7502e5 ("hfs: prevent btree data loss on ENOSPC")
Signed-off-by: Jeongjun Park <aha310510@gmail.com>
---
fs/hfs/catalog.c | 4 ++++
fs/hfs/extent.c | 26 +++++++++++++++-----------
2 files changed, 19 insertions(+), 11 deletions(-)
diff --git a/fs/hfs/catalog.c b/fs/hfs/catalog.c
index d63880e7d9d6..cece4333f7a7 100644
--- a/fs/hfs/catalog.c
+++ b/fs/hfs/catalog.c
@@ -101,7 +101,9 @@ int hfs_cat_create(u32 cnid, struct inode *dir, const struct qstr *str, struct i
* Fail early and avoid ENOSPC during the btree operations. We may
* have to split the root node at most once.
*/
+ mutex_lock(&HFS_I(fd.tree->inode)->extents_lock);
err = hfs_bmap_reserve(fd.tree, 2 * fd.tree->depth);
+ mutex_unlock(&HFS_I(fd.tree->inode)->extents_lock);
if (err)
goto err2;
@@ -307,7 +309,9 @@ int hfs_cat_move(u32 cnid, struct inode *src_dir, const struct qstr *src_name,
* Fail early and avoid ENOSPC during the btree operations. We may
* have to split the root node at most once.
*/
+ mutex_lock(&HFS_I(src_fd.tree->inode)->extents_lock);
err = hfs_bmap_reserve(src_fd.tree, 2 * src_fd.tree->depth);
+ mutex_unlock(&HFS_I(src_fd.tree->inode)->extents_lock);
if (err)
goto out;
diff --git a/fs/hfs/extent.c b/fs/hfs/extent.c
index 6d1878b99b30..25de1d48b667 100644
--- a/fs/hfs/extent.c
+++ b/fs/hfs/extent.c
@@ -338,38 +338,41 @@ int hfs_get_block(struct inode *inode, sector_t block,
{
struct super_block *sb;
u16 dblock, ablock;
- int res;
+ int res = 0;
sb = inode->i_sb;
/* Convert inode block to disk allocation block */
ablock = (u32)block / HFS_SB(sb)->fs_div;
+ mutex_lock(&HFS_I(inode)->extents_lock);
if (block >= HFS_I(inode)->fs_blocks) {
if (!create)
- return 0;
- if (block > HFS_I(inode)->fs_blocks)
- return -EIO;
+ goto out;
+ if (block > HFS_I(inode)->fs_blocks){
+ res = -EIO;
+ goto out;
+ }
if (ablock >= HFS_I(inode)->alloc_blocks) {
res = hfs_extend_file(inode);
if (res)
- return res;
+ goto out;
}
} else
create = 0;
if (ablock < HFS_I(inode)->first_blocks) {
dblock = hfs_ext_find_block(HFS_I(inode)->first_extents, ablock);
+ mutex_unlock(&HFS_I(inode)->extents_lock);
goto done;
}
- mutex_lock(&HFS_I(inode)->extents_lock);
res = hfs_ext_read_extent(inode, ablock);
if (!res)
dblock = hfs_ext_find_block(HFS_I(inode)->cached_extents,
ablock - HFS_I(inode)->cached_start);
else {
- mutex_unlock(&HFS_I(inode)->extents_lock);
- return -EIO;
+ res = -EIO;
+ goto out;
}
mutex_unlock(&HFS_I(inode)->extents_lock);
@@ -385,7 +388,10 @@ int hfs_get_block(struct inode *inode, sector_t block,
inode_add_bytes(inode, sb->s_blocksize);
mark_inode_dirty(inode);
}
- return 0;
+ return res;
+out:
+ mutex_unlock(&HFS_I(inode)->extents_lock);
+ return res;
}
int hfs_extend_file(struct inode *inode)
@@ -394,7 +400,6 @@ int hfs_extend_file(struct inode *inode)
u32 start, len, goal;
int res;
- mutex_lock(&HFS_I(inode)->extents_lock);
if (HFS_I(inode)->alloc_blocks == HFS_I(inode)->first_blocks)
goal = hfs_ext_lastblock(HFS_I(inode)->first_extents);
else {
@@ -444,7 +449,6 @@ int hfs_extend_file(struct inode *inode)
goto insert_extent;
}
out:
- mutex_unlock(&HFS_I(inode)->extents_lock);
if (!res) {
HFS_I(inode)->alloc_blocks += len;
mark_inode_dirty(inode);
--
2.34.1
^ permalink raw reply related [flat|nested] 3+ messages in thread* Re: [syzbot] [hfs?] possible deadlock in hfs_extend_file (3)
2024-04-18 14:41 [syzbot] [hfs?] possible deadlock in hfs_extend_file (3) syzbot
2024-04-23 9:31 ` [PATCH fs/hfs] hfs: fix deadlock in hfs_extend_file() Jeongjun Park
@ 2024-06-23 9:36 ` syzbot
1 sibling, 0 replies; 3+ messages in thread
From: syzbot @ 2024-06-23 9:36 UTC (permalink / raw)
To: aha310510, jack, jlayton, linux-fsdevel, linux-kernel,
syzkaller-bugs
syzbot has found a reproducer for the following issue on:
HEAD commit: 563a50672d8a Merge tag 'xfs-6.10-fixes-4' of git://git.ker..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=11ca148e980000
kernel config: https://syzkaller.appspot.com/x/.config?x=12f98862a3c0c799
dashboard link: https://syzkaller.appspot.com/bug?extid=2a62f58f1a4951a549bb
compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1287d83e980000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=12cb8151980000
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/8f92c1547793/disk-563a5067.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/21bb27a22e67/vmlinux-563a5067.xz
kernel image: https://storage.googleapis.com/syzbot-assets/168847d060c0/bzImage-563a5067.xz
mounted in repro #1: https://storage.googleapis.com/syzbot-assets/048c1910668d/mount_0.gz
mounted in repro #2: https://storage.googleapis.com/syzbot-assets/097a7c874267/mount_1.gz
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+2a62f58f1a4951a549bb@syzkaller.appspotmail.com
loop0: detected capacity change from 0 to 64
======================================================
WARNING: possible circular locking dependency detected
6.10.0-rc4-syzkaller-00283-g563a50672d8a #0 Not tainted
------------------------------------------------------
syz-executor283/5110 is trying to acquire lock:
ffff88805ee675f8 (&HFS_I(tree->inode)->extents_lock){+.+.}-{3:3}, at: hfs_extend_file+0xff/0x1450 fs/hfs/extent.c:397
but task is already holding lock:
ffff88802992a0b0 (&tree->tree_lock/1){+.+.}-{3:3}, at: hfs_find_init+0x16e/0x1f0
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (&tree->tree_lock/1){+.+.}-{3:3}:
lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754
__mutex_lock_common kernel/locking/mutex.c:608 [inline]
__mutex_lock+0x136/0xd70 kernel/locking/mutex.c:752
hfs_find_init+0x16e/0x1f0
hfs_ext_read_extent fs/hfs/extent.c:200 [inline]
hfs_extend_file+0x31b/0x1450 fs/hfs/extent.c:401
hfs_bmap_reserve+0xd9/0x400 fs/hfs/btree.c:234
hfs_cat_create+0x1e0/0x970 fs/hfs/catalog.c:104
hfs_create+0x66/0xe0 fs/hfs/dir.c:202
lookup_open fs/namei.c:3505 [inline]
open_last_lookups fs/namei.c:3574 [inline]
path_openat+0x1425/0x3280 fs/namei.c:3804
do_filp_open+0x235/0x490 fs/namei.c:3834
do_sys_openat2+0x13e/0x1d0 fs/open.c:1405
do_sys_open fs/open.c:1420 [inline]
__do_sys_openat fs/open.c:1436 [inline]
__se_sys_openat fs/open.c:1431 [inline]
__x64_sys_openat+0x247/0x2a0 fs/open.c:1431
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x77/0x7f
-> #0 (&HFS_I(tree->inode)->extents_lock){+.+.}-{3:3}:
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18e0/0x5900 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754
__mutex_lock_common kernel/locking/mutex.c:608 [inline]
__mutex_lock+0x136/0xd70 kernel/locking/mutex.c:752
hfs_extend_file+0xff/0x1450 fs/hfs/extent.c:397
hfs_bmap_reserve+0xd9/0x400 fs/hfs/btree.c:234
__hfs_ext_write_extent+0x22e/0x4f0 fs/hfs/extent.c:121
__hfs_ext_cache_extent+0x6a/0x990 fs/hfs/extent.c:174
hfs_ext_read_extent fs/hfs/extent.c:202 [inline]
hfs_extend_file+0x344/0x1450 fs/hfs/extent.c:401
hfs_get_block+0x3e4/0xb60 fs/hfs/extent.c:353
__block_write_begin_int+0x50c/0x1a70 fs/buffer.c:2128
__block_write_begin fs/buffer.c:2177 [inline]
block_write_begin+0x9b/0x1e0 fs/buffer.c:2236
cont_write_begin+0x645/0x890 fs/buffer.c:2590
hfs_write_begin+0x8a/0xd0 fs/hfs/inode.c:53
generic_perform_write+0x322/0x640 mm/filemap.c:4015
generic_file_write_iter+0xaf/0x310 mm/filemap.c:4136
new_sync_write fs/read_write.c:497 [inline]
vfs_write+0xa72/0xc90 fs/read_write.c:590
ksys_write+0x1a0/0x2c0 fs/read_write.c:643
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x77/0x7f
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&tree->tree_lock/1);
lock(&HFS_I(tree->inode)->extents_lock);
lock(&tree->tree_lock/1);
lock(&HFS_I(tree->inode)->extents_lock);
*** DEADLOCK ***
4 locks held by syz-executor283/5110:
#0: ffff888029a14420 (sb_writers#9){.+.+}-{0:0}, at: file_start_write include/linux/fs.h:2854 [inline]
#0: ffff888029a14420 (sb_writers#9){.+.+}-{0:0}, at: vfs_write+0x227/0xc90 fs/read_write.c:586
#1: ffff8880784d0fa8 (&sb->s_type->i_mutex_key#14){+.+.}-{3:3}, at: inode_lock include/linux/fs.h:791 [inline]
#1: ffff8880784d0fa8 (&sb->s_type->i_mutex_key#14){+.+.}-{3:3}, at: generic_file_write_iter+0x83/0x310 mm/filemap.c:4133
#2: ffff8880784d0df8 (&HFS_I(inode)->extents_lock){+.+.}-{3:3}, at: hfs_extend_file+0xff/0x1450 fs/hfs/extent.c:397
#3: ffff88802992a0b0 (&tree->tree_lock/1){+.+.}-{3:3}, at: hfs_find_init+0x16e/0x1f0
stack backtrace:
CPU: 1 PID: 5110 Comm: syz-executor283 Not tainted 6.10.0-rc4-syzkaller-00283-g563a50672d8a #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/07/2024
Call Trace:
<TASK>
__dump_stack lib/dump_stack.c:88 [inline]
dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187
check_prev_add kernel/locking/lockdep.c:3134 [inline]
check_prevs_add kernel/locking/lockdep.c:3253 [inline]
validate_chain+0x18e0/0x5900 kernel/locking/lockdep.c:3869
__lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137
lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754
__mutex_lock_common kernel/locking/mutex.c:608 [inline]
__mutex_lock+0x136/0xd70 kernel/locking/mutex.c:752
hfs_extend_file+0xff/0x1450 fs/hfs/extent.c:397
hfs_bmap_reserve+0xd9/0x400 fs/hfs/btree.c:234
__hfs_ext_write_extent+0x22e/0x4f0 fs/hfs/extent.c:121
__hfs_ext_cache_extent+0x6a/0x990 fs/hfs/extent.c:174
hfs_ext_read_extent fs/hfs/extent.c:202 [inline]
hfs_extend_file+0x344/0x1450 fs/hfs/extent.c:401
hfs_get_block+0x3e4/0xb60 fs/hfs/extent.c:353
__block_write_begin_int+0x50c/0x1a70 fs/buffer.c:2128
__block_write_begin fs/buffer.c:2177 [inline]
block_write_begin+0x9b/0x1e0 fs/buffer.c:2236
cont_write_begin+0x645/0x890 fs/buffer.c:2590
hfs_write_begin+0x8a/0xd0 fs/hfs/inode.c:53
generic_perform_write+0x322/0x640 mm/filemap.c:4015
generic_file_write_iter+0xaf/0x310 mm/filemap.c:4136
new_sync_write fs/read_write.c:497 [inline]
vfs_write+0xa72/0xc90 fs/read_write.c:590
ksys_write+0x1a0/0x2c0 fs/read_write.c:643
do_syscall_x64 arch/x86/entry/common.c:52 [inline]
do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fe0c6fea2e9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 21 18 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffefba1b208 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fe0c6fea2e9
RDX: 000000000000fea7 RSI: 0000000020000000 RDI: 0000000000000004
RBP: 0000000000000000 R08: 00000000200005c0 R09: 00007ffefba1b240
R10: 0000000000000280 R11: 0000000000000246 R12: 00007ffefba1b22c
R13: 000000000000000b R14: 431bde82d7b634db R15: 00007ffefba1b260
</TASK>
---
If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.
^ permalink raw reply [flat|nested] 3+ messages in thread