[syzbot] [bcachefs?] WARNING in bch2_fs_read_write_early

[syzbot] [bcachefs?] WARNING in bch2_fs_read_write_early

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    2ccbdf43d5e7 Merge tag 'for-linus' of git://git.kernel.org..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=1689a256980000
kernel config:  https://syzkaller.appspot.com/x/.config?x=b8786f381e62940f
dashboard link: https://syzkaller.appspot.com/bug?extid=4366624c0b5aac4906cf
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=110cda0e980000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=164f91fe980000

Downloadable assets:
disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/7bc7510fe41f/non_bootable_disk-2ccbdf43.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/c40c1cd990d2/vmlinux-2ccbdf43.xz
kernel image: https://storage.googleapis.com/syzbot-assets/a2a94050804e/bzImage-2ccbdf43.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/a79f654b4644/mount_0.gz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+4366624c0b5aac4906cf@syzkaller.appspotmail.com

------------[ cut here ]------------
WARNING: CPU: 0 PID: 10 at fs/bcachefs/super.c:525 bch2_fs_read_write_early+0x8a/0xa0 fs/bcachefs/super.c:525
Modules linked in:
CPU: 0 PID: 10 Comm: kworker/0:1 Not tainted 6.10.0-rc3-syzkaller-00044-g2ccbdf43d5e7 #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
Workqueue: bcachefs_write_ref bch2_delete_dead_snapshots_work
RIP: 0010:bch2_fs_read_write_early+0x8a/0xa0 fs/bcachefs/super.c:525
Code: ac b8 76 fd 48 8d bb 78 02 00 00 be ff ff ff ff e8 0b cd d0 06 31 ff 89 c5 89 c6 e8 90 b3 76 fd 85 ed 75 c7 e8 87 b8 76 fd 90 <0f> 0b 90 eb bc 48 c7 c7 58 a2 e4 8f e8 c5 8b d2 fd eb 9b 0f 1f 00
RSP: 0018:ffffc900000d7910 EFLAGS: 00010293
RAX: 0000000000000000 RBX: ffff888041080000 RCX: ffffffff84172cc0
RDX: ffff888017e90000 RSI: ffffffff84172cc9 RDI: 0000000000000005
RBP: 0000000000000000 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000000 R11: 1ffffffff272c7ee R12: ffff8880410801a8
R13: ffff888041080ae0 R14: 0000000000000001 R15: ffff88801f971800
FS:  0000000000000000(0000) GS:ffff88806b000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555581085788 CR3: 0000000025a9e000 CR4: 0000000000350ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 bch2_delete_dead_snapshots+0xfdc/0x4e50 fs/bcachefs/snapshot.c:1569
 bch2_delete_dead_snapshots_work+0x20/0x160 fs/bcachefs/snapshot.c:1690
 process_one_work+0x9fb/0x1b60 kernel/workqueue.c:3231
 process_scheduled_works kernel/workqueue.c:3312 [inline]
 worker_thread+0x6c8/0xf70 kernel/workqueue.c:3393
 kthread+0x2c1/0x3a0 kernel/kthread.c:389
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Re: [syzbot] [bcachefs?] WARNING in bch2_fs_read_write_early

[Edward Adam Davis]

please test no write lock in bch2_fs_read_write_early

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 2ccbdf43d5e7

diff --git a/fs/bcachefs/snapshot.c b/fs/bcachefs/snapshot.c
index 51918acfd726..b27a4327274d 100644
--- a/fs/bcachefs/snapshot.c
+++ b/fs/bcachefs/snapshot.c
@@ -1566,7 +1566,9 @@ int bch2_delete_dead_snapshots(struct bch_fs *c)
 		return 0;
 
 	if (!test_bit(BCH_FS_started, &c->flags)) {
+		down_write(&c->state_lock);
 		ret = bch2_fs_read_write_early(c);
+		up_write(&c->state_lock);
 		bch_err_msg(c, ret, "deleting dead snapshots: error going rw");
 		if (ret)
 			return ret;

Re: [syzbot] [bcachefs?] WARNING in bch2_fs_read_write_early

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
turning off the locking correctness validator.
CPU: 2 PID: 5280 Comm: kworker/2:3 Not tainted 6.10.0-rc3-syzkaller-00044-g2ccbdf43d5e7-dirty #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
Workqueue: bcachefs_write_ref bch2_delete_dead_snapshots_work
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x116/0x1f0 lib/dump_stack.c:114
 add_chain_cache kernel/locking/lockdep.c:3735 [inline]
 lookup_chain_cache_add kernel/locking/lockdep.c:3816 [inline]
 validate_chain kernel/locking/lockdep.c:3837 [inline]
 __lock_acquire+0x2ea6/0x3b30 kernel/locking/lockdep.c:5137
 lock_acquire kernel/locking/lockdep.c:5754 [inline]
 lock_acquire+0x1b1/0x560 kernel/locking/lockdep.c:5719
 do_write_seqcount_begin_nested include/linux/seqlock.h:469 [inline]
 do_write_seqcount_begin include/linux/seqlock.h:495 [inline]
 psi_account_irqtime+0x2a3/0x510 kernel/sched/psi.c:1016
 update_rq_clock_task kernel/sched/core.c:726 [inline]
 update_rq_clock kernel/sched/core.c:770 [inline]
 update_rq_clock+0x250/0xb30 kernel/sched/core.c:751
 ttwu_queue kernel/sched/core.c:4056 [inline]
 try_to_wake_up+0x56a/0x13e0 kernel/sched/core.c:4378
 kick_pool+0x2a0/0x7a0 kernel/workqueue.c:1279
 __queue_work+0x94d/0x1020 kernel/workqueue.c:2360
 queue_work_on+0x11a/0x140 kernel/workqueue.c:2410
 queue_work include/linux/workqueue.h:621 [inline]
 __bch2_btree_node_write+0x1fc1/0x2d60 fs/bcachefs/btree_io.c:2232
 bch2_btree_node_write+0x127/0x2f0 fs/bcachefs/btree_io.c:2307
 btree_split+0x1087/0x3010 fs/bcachefs/btree_update_interior.c:1706
 bch2_btree_split_leaf+0x108/0x770 fs/bcachefs/btree_update_interior.c:1857
 bch2_trans_commit_error+0x327/0xd00 fs/bcachefs/btree_trans_commit.c:918
 __bch2_trans_commit+0x4eb1/0x7ad0 fs/bcachefs/btree_trans_commit.c:1138
 bch2_trans_commit fs/bcachefs/btree_update.h:170 [inline]
 bch2_delete_dead_snapshots+0x1b22/0x4e70 fs/bcachefs/snapshot.c:1619
 bch2_delete_dead_snapshots_work+0x20/0x160 fs/bcachefs/snapshot.c:1692
 process_one_work+0x9fb/0x1b60 kernel/workqueue.c:3231
 process_scheduled_works kernel/workqueue.c:3312 [inline]
 worker_thread+0x6c8/0xf70 kernel/workqueue.c:3393
 kthread+0x2c1/0x3a0 kernel/kthread.c:389
 ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>
bcachefs (loop0): bch2_delete_dead_snapshots(): error deleting keys from dying snapshots erofs_trans_commit
bcachefs (loop0): bch2_delete_dead_snapshots(): error erofs_trans_commit
bcachefs (loop0): bch2_delete_dead_snapshots(): error deleting keys from dying snapshots erofs_trans_commit
bcachefs (loop0): bch2_delete_dead_snapshots(): error erofs_trans_commit
bcachefs (loop0): bch2_delete_dead_snapshots(): error deleting snapshot 4294967295 ENOENT_bkey_type_mismatch
bcachefs (loop0): bch2_delete_dead_snapshots(): error ENOENT_bkey_type_mismatch
bcachefs (loop0): bch2_delete_dead_snapshots(): error deleting snapshot 4294967295 ENOENT_bkey_type_mismatch
bcachefs (loop0): bch2_delete_dead_snapshots(): error ENOENT_bkey_type_mismatch


Tested on:

commit:         2ccbdf43 Merge tag 'for-linus' of git://git.kernel.org..
git tree:       https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
console output: https://syzkaller.appspot.com/x/log.txt?x=16e190b6980000
kernel config:  https://syzkaller.appspot.com/x/.config?x=b8786f381e62940f
dashboard link: https://syzkaller.appspot.com/bug?extid=4366624c0b5aac4906cf
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
patch:          https://syzkaller.appspot.com/x/patch.diff?x=122fbb36980000

Re: [syzbot] [bcachefs?] WARNING in bch2_fs_read_write_early

[Edward Adam Davis]

please test no write lock in bch2_fs_read_write_early

#syz test https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 2ccbdf43d5e7

diff --git a/fs/bcachefs/snapshot.c b/fs/bcachefs/snapshot.c
index 51918acfd726..b27a4327274d 100644
--- a/fs/bcachefs/snapshot.c
+++ b/fs/bcachefs/snapshot.c
@@ -1566,7 +1566,9 @@ int bch2_delete_dead_snapshots(struct bch_fs *c)
 		return 0;
 
 	if (!test_bit(BCH_FS_started, &c->flags)) {
+		down_write(&c->state_lock);
 		ret = bch2_fs_read_write_early(c);
+		up_write(&c->state_lock);
 		bch_err_msg(c, ret, "deleting dead snapshots: error going rw");
 		if (ret)
 			return ret;
diff --git a/kernel/locking/lockdep_internals.h b/kernel/locking/lockdep_internals.h
index bbe9000260d0..cbccc0bbee76 100644
--- a/kernel/locking/lockdep_internals.h
+++ b/kernel/locking/lockdep_internals.h
@@ -99,9 +99,9 @@ static const unsigned long LOCKF_USED_IN_IRQ_READ =
 #define MAX_STACK_TRACE_ENTRIES	262144UL
 #define STACK_TRACE_HASH_SIZE	8192
 #else
-#define MAX_LOCKDEP_ENTRIES	(1UL << CONFIG_LOCKDEP_BITS)
+#define MAX_LOCKDEP_ENTRIES	(1UL << 19)
 
-#define MAX_LOCKDEP_CHAINS_BITS	CONFIG_LOCKDEP_CHAINS_BITS
+#define MAX_LOCKDEP_CHAINS_BITS	20
 
 /*
  * Stack-trace: tightly packed array of stack backtrace

Re: [syzbot] [bcachefs?] WARNING in bch2_fs_read_write_early

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+4366624c0b5aac4906cf@syzkaller.appspotmail.com

Tested on:

commit:         2ccbdf43 Merge tag 'for-linus' of git://git.kernel.org..
git tree:       https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
console output: https://syzkaller.appspot.com/x/log.txt?x=10f1ceda980000
kernel config:  https://syzkaller.appspot.com/x/.config?x=b8786f381e62940f
dashboard link: https://syzkaller.appspot.com/bug?extid=4366624c0b5aac4906cf
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
patch:          https://syzkaller.appspot.com/x/patch.diff?x=16c8cbfe980000

Note: testing is done by a robot and is best-effort only.

[PATCH] bcachefs: bch2_fs_read_write_early needs to hold write lock

[Edward Adam Davis]

bch2_fs_read_write_early() needs to hold state_lock to pretect and sync data.

Reported-by: syzbot+4366624c0b5aac4906cf@syzkaller.appspotmail.com
Signed-off-by: Edward Adam Davis <eadavis@qq.com>
---
 fs/bcachefs/snapshot.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/fs/bcachefs/snapshot.c b/fs/bcachefs/snapshot.c
index 51918acfd726..b27a4327274d 100644
--- a/fs/bcachefs/snapshot.c
+++ b/fs/bcachefs/snapshot.c
@@ -1566,7 +1566,9 @@ int bch2_delete_dead_snapshots(struct bch_fs *c)
 		return 0;
 
 	if (!test_bit(BCH_FS_started, &c->flags)) {
+		down_write(&c->state_lock);
 		ret = bch2_fs_read_write_early(c);
+		up_write(&c->state_lock);
 		bch_err_msg(c, ret, "deleting dead snapshots: error going rw");
 		if (ret)
 			return ret;
-- 
2.43.0

Re: [PATCH] bcachefs: bch2_fs_read_write_early needs to hold write lock

[Kent Overstreet]

On Sat, Jun 15, 2024 at 07:44:04PM +0800, Edward Adam Davis wrote:
> bch2_fs_read_write_early() needs to hold state_lock to pretect and sync data.
> 
> Reported-by: syzbot+4366624c0b5aac4906cf@syzkaller.appspotmail.com
> Signed-off-by: Edward Adam Davis <eadavis@qq.com>

this is incorrect - delete_dead_snapshots() may be called synchronously
or asynchronously, and if it's called asynchronously we do hold
state_lock, so this will deadlock

> ---
>  fs/bcachefs/snapshot.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/fs/bcachefs/snapshot.c b/fs/bcachefs/snapshot.c
> index 51918acfd726..b27a4327274d 100644
> --- a/fs/bcachefs/snapshot.c
> +++ b/fs/bcachefs/snapshot.c
> @@ -1566,7 +1566,9 @@ int bch2_delete_dead_snapshots(struct bch_fs *c)
>  		return 0;
>  
>  	if (!test_bit(BCH_FS_started, &c->flags)) {
> +		down_write(&c->state_lock);
>  		ret = bch2_fs_read_write_early(c);
> +		up_write(&c->state_lock);
>  		bch_err_msg(c, ret, "deleting dead snapshots: error going rw");
>  		if (ret)
>  			return ret;
> -- 
> 2.43.0
> 

Re: [syzbot] [bcachefs?] WARNING in bch2_fs_read_write_early

[Kent Overstreet]

On Fri, Jun 14, 2024 at 05:16:20AM -0700, syzbot wrote:
> Hello,
> 
> syzbot found the following issue on:
> 
> HEAD commit:    2ccbdf43d5e7 Merge tag 'for-linus' of git://git.kernel.org..
> git tree:       upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=1689a256980000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=b8786f381e62940f
> dashboard link: https://syzkaller.appspot.com/bug?extid=4366624c0b5aac4906cf
> compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=110cda0e980000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=164f91fe980000
> 
> Downloadable assets:
> disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/7bc7510fe41f/non_bootable_disk-2ccbdf43.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/c40c1cd990d2/vmlinux-2ccbdf43.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/a2a94050804e/bzImage-2ccbdf43.xz
> mounted in repro: https://storage.googleapis.com/syzbot-assets/a79f654b4644/mount_0.gz
> 
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+4366624c0b5aac4906cf@syzkaller.appspotmail.com
> 
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 10 at fs/bcachefs/super.c:525 bch2_fs_read_write_early+0x8a/0xa0 fs/bcachefs/super.c:525
> Modules linked in:
> CPU: 0 PID: 10 Comm: kworker/0:1 Not tainted 6.10.0-rc3-syzkaller-00044-g2ccbdf43d5e7 #0
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> Workqueue: bcachefs_write_ref bch2_delete_dead_snapshots_work
> RIP: 0010:bch2_fs_read_write_early+0x8a/0xa0 fs/bcachefs/super.c:525
> Code: ac b8 76 fd 48 8d bb 78 02 00 00 be ff ff ff ff e8 0b cd d0 06 31 ff 89 c5 89 c6 e8 90 b3 76 fd 85 ed 75 c7 e8 87 b8 76 fd 90 <0f> 0b 90 eb bc 48 c7 c7 58 a2 e4 8f e8 c5 8b d2 fd eb 9b 0f 1f 00
> RSP: 0018:ffffc900000d7910 EFLAGS: 00010293
> RAX: 0000000000000000 RBX: ffff888041080000 RCX: ffffffff84172cc0
> RDX: ffff888017e90000 RSI: ffffffff84172cc9 RDI: 0000000000000005
> RBP: 0000000000000000 R08: 0000000000000005 R09: 0000000000000000
> R10: 0000000000000000 R11: 1ffffffff272c7ee R12: ffff8880410801a8
> R13: ffff888041080ae0 R14: 0000000000000001 R15: ffff88801f971800
> FS:  0000000000000000(0000) GS:ffff88806b000000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000555581085788 CR3: 0000000025a9e000 CR4: 0000000000350ef0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>  <TASK>
>  bch2_delete_dead_snapshots+0xfdc/0x4e50 fs/bcachefs/snapshot.c:1569
>  bch2_delete_dead_snapshots_work+0x20/0x160 fs/bcachefs/snapshot.c:1690
>  process_one_work+0x9fb/0x1b60 kernel/workqueue.c:3231
>  process_scheduled_works kernel/workqueue.c:3312 [inline]
>  worker_thread+0x6c8/0xf70 kernel/workqueue.c:3393
>  kthread+0x2c1/0x3a0 kernel/kthread.c:389
>  ret_from_fork+0x45/0x80 arch/x86/kernel/process.c:147
>  ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
>  </TASK>
> 
> 
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@googlegroups.com.
> 
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> 
> If the report is already addressed, let syzbot know by replying with:
#syz fix: bcachefs: delete_dead_snapshots() doesn't need to go RW