Bug: INFO_ task hung in lock_two_nondirectories

Bug: INFO_ task hung in lock_two_nondirectories

[Kun Hu]

Hello,

When using our customized fuzzer tool to fuzz the latest Linux kernel, the following crash (43s)
was triggered.

HEAD commit: 9d89551994a430b50c4fffcb1e617a057fa76e20
git tree: upstream
Console output: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/log0
Kernel config: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/config.txt
C reproducer: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/12generated_program.c
Syzlang reproducer: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/12_repro.txt

We first found the issue without a stable C and Syzlang reproducers, but later I tried multiple rounds of replication and got the C and Syzlang reproducers.

I suspect the issue may stem from resource contention or synchronization delays, as indicated by functions like lock_two_nondirectories and vfs_rename. There could also be potential deadlocks or inconsistencies in file system state management (e.g., sb_writers or inode locks) within the bcachefs subsystem. Additionally, interactions between lock_rename and concurrent rename operations might be contributing factors.

Could you kindly help review these areas to narrow down the root cause? Your expertise would be greatly appreciated.

If you fix this issue, please add the following tag to the commit:
Reported-by: Kun Hu <huk23@m.fudan.edu.cn>, Jiaji Qin <jjtan24@m.fudan.edu.cn>

INFO: task syz.0.12:1823 blocked for more than 143 seconds.
      Tainted: G        W          6.13.0-rc6 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz.0.12        state:D stack:25728 pid:1823  tgid:1714  ppid:442    flags:0x00000004
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5369 [inline]
 __schedule+0xe60/0x4120 kernel/sched/core.c:6756
 __schedule_loop kernel/sched/core.c:6833 [inline]
 schedule+0xd4/0x210 kernel/sched/core.c:6848
 schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:6905
 rwsem_down_write_slowpath+0x551/0x1660 kernel/locking/rwsem.c:1176
 __down_write_common kernel/locking/rwsem.c:1304 [inline]
 __down_write kernel/locking/rwsem.c:1313 [inline]
 down_write_nested+0x1db/0x210 kernel/locking/rwsem.c:1694
 inode_lock_nested include/linux/fs.h:853 [inline]
 lock_two_nondirectories+0x107/0x210 fs/inode.c:1283
 vfs_rename+0x14c7/0x2a20 fs/namei.c:5038
 do_renameat2+0xb81/0xc90 fs/namei.c:5224
 __do_sys_renameat2 fs/namei.c:5258 [inline]
 __se_sys_renameat2 fs/namei.c:5255 [inline]
 __x64_sys_renameat2+0xe4/0x120 fs/namei.c:5255
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xc3/0x1d0 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fca967c071d
RSP: 002b:00007fca953f2ba8 EFLAGS: 00000246 ORIG_RAX: 000000000000013c
RAX: ffffffffffffffda RBX: 00007fca96983058 RCX: 00007fca967c071d
RDX: ffffffffffffff9c RSI: 0000000020000580 RDI: ffffffffffffff9c
RBP: 00007fca96835425 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000200005c0 R11: 0000000000000246 R12: 0000000000000000
R13: 00007fca96983064 R14: 00007fca969830f0 R15: 00007fca953f2d40
 </TASK>

Showing all locks held in the system:
1 lock held by khungtaskd/45:
 #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
 #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
 #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x75/0x340 kernel/locking/lockdep.c:6744
1 lock held by in:imklog/329:
5 locks held by syz.0.12/1715:
 #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: do_open fs/namei.c:3821 [inline]
 #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: path_openat+0x1577/0x2970 fs/namei.c:3987
 #1: ff110000415c1780 (&sb->s_type->i_mutex_key#21){++++}-{4:4}, at: inode_lock include/linux/fs.h:818 [inline]
 #1: ff110000415c1780 (&sb->s_type->i_mutex_key#21){++++}-{4:4}, at: do_truncate+0x131/0x200 fs/open.c:63
 #2: ff11000047780a38 (&c->snapshot_create_lock){.+.+}-{4:4}, at: bch2_truncate+0x145/0x260 fs/bcachefs/io_misc.c:292
 #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_lock_acquire include/linux/srcu.h:158 [inline]
 #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_read_lock include/linux/srcu.h:249 [inline]
 #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: __bch2_trans_get+0x625/0xf60 fs/bcachefs/btree_iter.c:3228
 #4: ff110000477a66d0 (&c->gc_lock){.+.+}-{4:4}, at: bch2_btree_update_start+0xb18/0x2010 fs/bcachefs/btree_update_interior.c:1197
3 locks held by syz.0.12/1823:
 #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: do_renameat2+0x3ea/0xc90 fs/namei.c:5154
 #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: inode_lock_nested include/linux/fs.h:853 [inline]
 #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: lock_rename fs/namei.c:3215 [inline]
 #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: lock_rename+0xa5/0xc0 fs/namei.c:3212
 #2: ff110000415c1780 (&sb->s_type->i_mutex_key#21/4){+.+.}-{4:4}, at: inode_lock_nested include/linux/fs.h:853 [inline]
 #2: ff110000415c1780 (&sb->s_type->i_mutex_key#21/4){+.+.}-{4:4}, at: lock_two_nondirectories+0x107/0x210 fs/inode.c:1283
4 locks held by bch-reclaim/loo/1811:
 #0: ff110000477cb0a8 (&j->reclaim_lock){+.+.}-{4:4}, at: bch2_journal_reclaim_thread+0x101/0x560 fs/bcachefs/journal_reclaim.c:739
 #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_lock_acquire include/linux/srcu.h:158 [inline]
 #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_read_lock include/linux/srcu.h:249 [inline]
 #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: __bch2_trans_get+0x625/0xf60 fs/bcachefs/btree_iter.c:3228
 #2: ff11000047784740 (&wb->flushing.lock){+.+.}-{4:4}, at: btree_write_buffer_flush_seq+0x69f/0xa40 fs/bcachefs/btree_write_buffer.c:516
 #3: ff110000477a66d0 (&c->gc_lock){.+.+}-{4:4}, at: bch2_btree_update_start+0xb18/0x2010 fs/bcachefs/btree_update_interior.c:1197
1 lock held by syz-executor/5484:
2 locks held by syz.1.1333/8755:
1 lock held by syz.7.1104/8876:
 #0: ffffffff9fb27ef8 (rcu_state.exp_mutex){+.+.}-{4:4}, at: exp_funnel_lock+0x29f/0x3d0 kernel/rcu/tree_exp.h:297
2 locks held by syz.8.1367/8889:

=============================================

---------------
thanks,
Kun Hu

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Jan Kara]

Hello!

First I'd note that the list of recipients of this report seems somewhat
arbitrary. linux-kernel & linux-fsdevel makes sense but I'm not sure how
you have arrived at the list of other persons you have CCed :). I have to
say syzbot folks have done a pretty good job at implementing mechanisms how
to determine recipients of the reports (as well as managing existing
reports over the web / email). Maybe you could take some inspiration there
or just contribute your syzkaller modifications to syzbot folks so that
your reports can benefit from all the other infrastructure they have?

Anyway, in this particular case, based on locks reported by lockdep, the
problem seems to be most likely somewhere in bcachefs. Added relevant CCs.

								Honza

On Sun 12-01-25 18:00:24, Kun Hu wrote:
> When using our customized fuzzer tool to fuzz the latest Linux kernel, the following crash (43s)
> was triggered.
> 
> HEAD commit: 9d89551994a430b50c4fffcb1e617a057fa76e20
> git tree: upstream
> Console output: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/log0
> Kernel config: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/config.txt
> C reproducer: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/12generated_program.c
> Syzlang reproducer: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/12_repro.txt
> 
> We first found the issue without a stable C and Syzlang reproducers, but later I tried multiple rounds of replication and got the C and Syzlang reproducers.
> 
> I suspect the issue may stem from resource contention or synchronization delays, as indicated by functions like lock_two_nondirectories and vfs_rename. There could also be potential deadlocks or inconsistencies in file system state management (e.g., sb_writers or inode locks) within the bcachefs subsystem. Additionally, interactions between lock_rename and concurrent rename operations might be contributing factors.
> 
> Could you kindly help review these areas to narrow down the root cause? Your expertise would be greatly appreciated.
> 
> If you fix this issue, please add the following tag to the commit:
> Reported-by: Kun Hu <huk23@m.fudan.edu.cn>, Jiaji Qin <jjtan24@m.fudan.edu.cn>
> 
> INFO: task syz.0.12:1823 blocked for more than 143 seconds.
>       Tainted: G        W          6.13.0-rc6 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> task:syz.0.12        state:D stack:25728 pid:1823  tgid:1714  ppid:442    flags:0x00000004
> Call Trace:
>  <TASK>
>  context_switch kernel/sched/core.c:5369 [inline]
>  __schedule+0xe60/0x4120 kernel/sched/core.c:6756
>  __schedule_loop kernel/sched/core.c:6833 [inline]
>  schedule+0xd4/0x210 kernel/sched/core.c:6848
>  schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:6905
>  rwsem_down_write_slowpath+0x551/0x1660 kernel/locking/rwsem.c:1176
>  __down_write_common kernel/locking/rwsem.c:1304 [inline]
>  __down_write kernel/locking/rwsem.c:1313 [inline]
>  down_write_nested+0x1db/0x210 kernel/locking/rwsem.c:1694
>  inode_lock_nested include/linux/fs.h:853 [inline]
>  lock_two_nondirectories+0x107/0x210 fs/inode.c:1283
>  vfs_rename+0x14c7/0x2a20 fs/namei.c:5038
>  do_renameat2+0xb81/0xc90 fs/namei.c:5224
>  __do_sys_renameat2 fs/namei.c:5258 [inline]
>  __se_sys_renameat2 fs/namei.c:5255 [inline]
>  __x64_sys_renameat2+0xe4/0x120 fs/namei.c:5255
>  do_syscall_x64 arch/x86/entry/common.c:52 [inline]
>  do_syscall_64+0xc3/0x1d0 arch/x86/entry/common.c:83
>  entry_SYSCALL_64_after_hwframe+0x77/0x7f
> RIP: 0033:0x7fca967c071d
> RSP: 002b:00007fca953f2ba8 EFLAGS: 00000246 ORIG_RAX: 000000000000013c
> RAX: ffffffffffffffda RBX: 00007fca96983058 RCX: 00007fca967c071d
> RDX: ffffffffffffff9c RSI: 0000000020000580 RDI: ffffffffffffff9c
> RBP: 00007fca96835425 R08: 0000000000000000 R09: 0000000000000000
> R10: 00000000200005c0 R11: 0000000000000246 R12: 0000000000000000
> R13: 00007fca96983064 R14: 00007fca969830f0 R15: 00007fca953f2d40
>  </TASK>
> 
> Showing all locks held in the system:
> 1 lock held by khungtaskd/45:
>  #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
>  #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
>  #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x75/0x340 kernel/locking/lockdep.c:6744
> 1 lock held by in:imklog/329:
> 5 locks held by syz.0.12/1715:
>  #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: do_open fs/namei.c:3821 [inline]
>  #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: path_openat+0x1577/0x2970 fs/namei.c:3987
>  #1: ff110000415c1780 (&sb->s_type->i_mutex_key#21){++++}-{4:4}, at: inode_lock include/linux/fs.h:818 [inline]
>  #1: ff110000415c1780 (&sb->s_type->i_mutex_key#21){++++}-{4:4}, at: do_truncate+0x131/0x200 fs/open.c:63
>  #2: ff11000047780a38 (&c->snapshot_create_lock){.+.+}-{4:4}, at: bch2_truncate+0x145/0x260 fs/bcachefs/io_misc.c:292
>  #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_lock_acquire include/linux/srcu.h:158 [inline]
>  #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_read_lock include/linux/srcu.h:249 [inline]
>  #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: __bch2_trans_get+0x625/0xf60 fs/bcachefs/btree_iter.c:3228
>  #4: ff110000477a66d0 (&c->gc_lock){.+.+}-{4:4}, at: bch2_btree_update_start+0xb18/0x2010 fs/bcachefs/btree_update_interior.c:1197
> 3 locks held by syz.0.12/1823:
>  #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: do_renameat2+0x3ea/0xc90 fs/namei.c:5154
>  #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: inode_lock_nested include/linux/fs.h:853 [inline]
>  #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: lock_rename fs/namei.c:3215 [inline]
>  #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: lock_rename+0xa5/0xc0 fs/namei.c:3212
>  #2: ff110000415c1780 (&sb->s_type->i_mutex_key#21/4){+.+.}-{4:4}, at: inode_lock_nested include/linux/fs.h:853 [inline]
>  #2: ff110000415c1780 (&sb->s_type->i_mutex_key#21/4){+.+.}-{4:4}, at: lock_two_nondirectories+0x107/0x210 fs/inode.c:1283
> 4 locks held by bch-reclaim/loo/1811:
>  #0: ff110000477cb0a8 (&j->reclaim_lock){+.+.}-{4:4}, at: bch2_journal_reclaim_thread+0x101/0x560 fs/bcachefs/journal_reclaim.c:739
>  #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_lock_acquire include/linux/srcu.h:158 [inline]
>  #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_read_lock include/linux/srcu.h:249 [inline]
>  #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: __bch2_trans_get+0x625/0xf60 fs/bcachefs/btree_iter.c:3228
>  #2: ff11000047784740 (&wb->flushing.lock){+.+.}-{4:4}, at: btree_write_buffer_flush_seq+0x69f/0xa40 fs/bcachefs/btree_write_buffer.c:516
>  #3: ff110000477a66d0 (&c->gc_lock){.+.+}-{4:4}, at: bch2_btree_update_start+0xb18/0x2010 fs/bcachefs/btree_update_interior.c:1197
> 1 lock held by syz-executor/5484:
> 2 locks held by syz.1.1333/8755:
> 1 lock held by syz.7.1104/8876:
>  #0: ffffffff9fb27ef8 (rcu_state.exp_mutex){+.+.}-{4:4}, at: exp_funnel_lock+0x29f/0x3d0 kernel/rcu/tree_exp.h:297
> 2 locks held by syz.8.1367/8889:
> 
> =============================================
> 
> ---------------
> thanks,
> Kun Hu
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Kent Overstreet]

On Mon, Jan 13, 2025 at 11:28:43AM +0100, Jan Kara wrote:
> Hello!
> 
> First I'd note that the list of recipients of this report seems somewhat
> arbitrary. linux-kernel & linux-fsdevel makes sense but I'm not sure how
> you have arrived at the list of other persons you have CCed :). I have to
> say syzbot folks have done a pretty good job at implementing mechanisms how
> to determine recipients of the reports (as well as managing existing
> reports over the web / email). Maybe you could take some inspiration there
> or just contribute your syzkaller modifications to syzbot folks so that
> your reports can benefit from all the other infrastructure they have?

Is there some political reason that collaboration isn't happening? I've
found the syzbot people to be great to work with.

I'll give some analysis on this bug, but in general I won't in the
future until you guys start collaborating (or at least tell us what the
blocker is) - I don't want to be duplicating work I've already done for
syzbot.

> Anyway, in this particular case, based on locks reported by lockdep, the
> problem seems to be most likely somewhere in bcachefs. Added relevant CCs.
> 
> 								Honza
> 
> On Sun 12-01-25 18:00:24, Kun Hu wrote:
> > When using our customized fuzzer tool to fuzz the latest Linux kernel, the following crash (43s)
> > was triggered.
> > 
> > HEAD commit: 9d89551994a430b50c4fffcb1e617a057fa76e20
> > git tree: upstream
> > Console output: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/log0
> > Kernel config: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/config.txt
> > C reproducer: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/12generated_program.c
> > Syzlang reproducer: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/12_repro.txt
> > 
> > We first found the issue without a stable C and Syzlang reproducers, but later I tried multiple rounds of replication and got the C and Syzlang reproducers.
> > 
> > I suspect the issue may stem from resource contention or synchronization delays, as indicated by functions like lock_two_nondirectories and vfs_rename. There could also be potential deadlocks or inconsistencies in file system state management (e.g., sb_writers or inode locks) within the bcachefs subsystem. Additionally, interactions between lock_rename and concurrent rename operations might be contributing factors.
> > 
> > Could you kindly help review these areas to narrow down the root cause? Your expertise would be greatly appreciated.

We need to know what the other threads are doing. Since lockdep didn't
detect an actual deadlock, it seems most likely that the blocked thread
is blocked because the thread holding the inode lock is spinning and
livelocked.

That's not in the dump, so to debug this we'd need to reproduce this in
a local VM and poke around with e.g. top/perf and see what's going on.

I've a tool to reproduce syzbot bugs locally in a single command [1], so
that would be my starting point - except it doesn't work with your
forked version. Doh.

Another thing we could do is write some additional code for the hung
task detector that, when lockdep is enabled, uses it to figure out which
task it's blocked on and additionally print a backtrace for that.

[1]: https://evilpiepirate.org/git/ktest.git/tree/tests/syzbot-repro.ktest

> > 
> > If you fix this issue, please add the following tag to the commit:
> > Reported-by: Kun Hu <huk23@m.fudan.edu.cn>, Jiaji Qin <jjtan24@m.fudan.edu.cn>
> > 
> > INFO: task syz.0.12:1823 blocked for more than 143 seconds.
> >       Tainted: G        W          6.13.0-rc6 #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > task:syz.0.12        state:D stack:25728 pid:1823  tgid:1714  ppid:442    flags:0x00000004
> > Call Trace:
> >  <TASK>
> >  context_switch kernel/sched/core.c:5369 [inline]
> >  __schedule+0xe60/0x4120 kernel/sched/core.c:6756
> >  __schedule_loop kernel/sched/core.c:6833 [inline]
> >  schedule+0xd4/0x210 kernel/sched/core.c:6848
> >  schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:6905
> >  rwsem_down_write_slowpath+0x551/0x1660 kernel/locking/rwsem.c:1176
> >  __down_write_common kernel/locking/rwsem.c:1304 [inline]
> >  __down_write kernel/locking/rwsem.c:1313 [inline]
> >  down_write_nested+0x1db/0x210 kernel/locking/rwsem.c:1694
> >  inode_lock_nested include/linux/fs.h:853 [inline]
> >  lock_two_nondirectories+0x107/0x210 fs/inode.c:1283
> >  vfs_rename+0x14c7/0x2a20 fs/namei.c:5038
> >  do_renameat2+0xb81/0xc90 fs/namei.c:5224
> >  __do_sys_renameat2 fs/namei.c:5258 [inline]
> >  __se_sys_renameat2 fs/namei.c:5255 [inline]
> >  __x64_sys_renameat2+0xe4/0x120 fs/namei.c:5255
> >  do_syscall_x64 arch/x86/entry/common.c:52 [inline]
> >  do_syscall_64+0xc3/0x1d0 arch/x86/entry/common.c:83
> >  entry_SYSCALL_64_after_hwframe+0x77/0x7f
> > RIP: 0033:0x7fca967c071d
> > RSP: 002b:00007fca953f2ba8 EFLAGS: 00000246 ORIG_RAX: 000000000000013c
> > RAX: ffffffffffffffda RBX: 00007fca96983058 RCX: 00007fca967c071d
> > RDX: ffffffffffffff9c RSI: 0000000020000580 RDI: ffffffffffffff9c
> > RBP: 00007fca96835425 R08: 0000000000000000 R09: 0000000000000000
> > R10: 00000000200005c0 R11: 0000000000000246 R12: 0000000000000000
> > R13: 00007fca96983064 R14: 00007fca969830f0 R15: 00007fca953f2d40
> >  </TASK>
> > 
> > Showing all locks held in the system:
> > 1 lock held by khungtaskd/45:
> >  #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
> >  #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
> >  #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x75/0x340 kernel/locking/lockdep.c:6744
> > 1 lock held by in:imklog/329:
> > 5 locks held by syz.0.12/1715:
> >  #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: do_open fs/namei.c:3821 [inline]
> >  #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: path_openat+0x1577/0x2970 fs/namei.c:3987
> >  #1: ff110000415c1780 (&sb->s_type->i_mutex_key#21){++++}-{4:4}, at: inode_lock include/linux/fs.h:818 [inline]
> >  #1: ff110000415c1780 (&sb->s_type->i_mutex_key#21){++++}-{4:4}, at: do_truncate+0x131/0x200 fs/open.c:63
> >  #2: ff11000047780a38 (&c->snapshot_create_lock){.+.+}-{4:4}, at: bch2_truncate+0x145/0x260 fs/bcachefs/io_misc.c:292
> >  #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_lock_acquire include/linux/srcu.h:158 [inline]
> >  #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_read_lock include/linux/srcu.h:249 [inline]
> >  #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: __bch2_trans_get+0x625/0xf60 fs/bcachefs/btree_iter.c:3228
> >  #4: ff110000477a66d0 (&c->gc_lock){.+.+}-{4:4}, at: bch2_btree_update_start+0xb18/0x2010 fs/bcachefs/btree_update_interior.c:1197
> > 3 locks held by syz.0.12/1823:
> >  #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: do_renameat2+0x3ea/0xc90 fs/namei.c:5154
> >  #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: inode_lock_nested include/linux/fs.h:853 [inline]
> >  #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: lock_rename fs/namei.c:3215 [inline]
> >  #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: lock_rename+0xa5/0xc0 fs/namei.c:3212
> >  #2: ff110000415c1780 (&sb->s_type->i_mutex_key#21/4){+.+.}-{4:4}, at: inode_lock_nested include/linux/fs.h:853 [inline]
> >  #2: ff110000415c1780 (&sb->s_type->i_mutex_key#21/4){+.+.}-{4:4}, at: lock_two_nondirectories+0x107/0x210 fs/inode.c:1283
> > 4 locks held by bch-reclaim/loo/1811:
> >  #0: ff110000477cb0a8 (&j->reclaim_lock){+.+.}-{4:4}, at: bch2_journal_reclaim_thread+0x101/0x560 fs/bcachefs/journal_reclaim.c:739
> >  #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_lock_acquire include/linux/srcu.h:158 [inline]
> >  #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_read_lock include/linux/srcu.h:249 [inline]
> >  #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: __bch2_trans_get+0x625/0xf60 fs/bcachefs/btree_iter.c:3228
> >  #2: ff11000047784740 (&wb->flushing.lock){+.+.}-{4:4}, at: btree_write_buffer_flush_seq+0x69f/0xa40 fs/bcachefs/btree_write_buffer.c:516
> >  #3: ff110000477a66d0 (&c->gc_lock){.+.+}-{4:4}, at: bch2_btree_update_start+0xb18/0x2010 fs/bcachefs/btree_update_interior.c:1197
> > 1 lock held by syz-executor/5484:
> > 2 locks held by syz.1.1333/8755:
> > 1 lock held by syz.7.1104/8876:
> >  #0: ffffffff9fb27ef8 (rcu_state.exp_mutex){+.+.}-{4:4}, at: exp_funnel_lock+0x29f/0x3d0 kernel/rcu/tree_exp.h:297
> > 2 locks held by syz.8.1367/8889:
> > 
> > =============================================
> > 
> > ---------------
> > thanks,
> > Kun Hu
> -- 
> Jan Kara <jack@suse.com>
> SUSE Labs, CR

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Dmitry Vyukov]

On Mon, 13 Jan 2025 at 21:00, Kent Overstreet <kent.overstreet@linux.dev> wrote:
>
> On Mon, Jan 13, 2025 at 11:28:43AM +0100, Jan Kara wrote:
> > Hello!
> >
> > First I'd note that the list of recipients of this report seems somewhat
> > arbitrary. linux-kernel & linux-fsdevel makes sense but I'm not sure how
> > you have arrived at the list of other persons you have CCed :). I have to
> > say syzbot folks have done a pretty good job at implementing mechanisms how
> > to determine recipients of the reports (as well as managing existing
> > reports over the web / email). Maybe you could take some inspiration there
> > or just contribute your syzkaller modifications to syzbot folks so that
> > your reports can benefit from all the other infrastructure they have?
>
> Is there some political reason that collaboration isn't happening? I've
> found the syzbot people to be great to work with.
>
> I'll give some analysis on this bug, but in general I won't in the
> future until you guys start collaborating (or at least tell us what the
> blocker is) - I don't want to be duplicating work I've already done for
> syzbot.

I suspect the bulk of the reports are coming from academia
researchers. In lots of academia papers based on syzkaller I see "we
also reported X bugs to the upstream kernel". Somehow there seems to
be a preference to keep things secret before publication, so upstream
syzbot integration is problematic. Though it is well possible to
publish papers based on OSS work, these usually tend to be higher
quality and have better evaluation.

I also don't fully understand the value of "we also reported X bugs to
the upstream kernel" for research papers. There is little correlation
with the quality/novelty of research.


> > Anyway, in this particular case, based on locks reported by lockdep, the
> > problem seems to be most likely somewhere in bcachefs. Added relevant CCs.
> >
> >                                                               Honza
> >
> > On Sun 12-01-25 18:00:24, Kun Hu wrote:
> > > When using our customized fuzzer tool to fuzz the latest Linux kernel, the following crash (43s)
> > > was triggered.
> > >
> > > HEAD commit: 9d89551994a430b50c4fffcb1e617a057fa76e20
> > > git tree: upstream
> > > Console output: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/log0
> > > Kernel config: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/config.txt
> > > C reproducer: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/12generated_program.c
> > > Syzlang reproducer: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/12_repro.txt
> > >
> > > We first found the issue without a stable C and Syzlang reproducers, but later I tried multiple rounds of replication and got the C and Syzlang reproducers.
> > >
> > > I suspect the issue may stem from resource contention or synchronization delays, as indicated by functions like lock_two_nondirectories and vfs_rename. There could also be potential deadlocks or inconsistencies in file system state management (e.g., sb_writers or inode locks) within the bcachefs subsystem. Additionally, interactions between lock_rename and concurrent rename operations might be contributing factors.
> > >
> > > Could you kindly help review these areas to narrow down the root cause? Your expertise would be greatly appreciated.
>
> We need to know what the other threads are doing. Since lockdep didn't
> detect an actual deadlock, it seems most likely that the blocked thread
> is blocked because the thread holding the inode lock is spinning and
> livelocked.
>
> That's not in the dump, so to debug this we'd need to reproduce this in
> a local VM and poke around with e.g. top/perf and see what's going on.
>
> I've a tool to reproduce syzbot bugs locally in a single command [1], so
> that would be my starting point - except it doesn't work with your
> forked version. Doh.
>
> Another thing we could do is write some additional code for the hung
> task detector that, when lockdep is enabled, uses it to figure out which
> task it's blocked on and additionally print a backtrace for that.
>
> [1]: https://evilpiepirate.org/git/ktest.git/tree/tests/syzbot-repro.ktest
>
> > >
> > > If you fix this issue, please add the following tag to the commit:
> > > Reported-by: Kun Hu <huk23@m.fudan.edu.cn>, Jiaji Qin <jjtan24@m.fudan.edu.cn>
> > >
> > > INFO: task syz.0.12:1823 blocked for more than 143 seconds.
> > >       Tainted: G        W          6.13.0-rc6 #1
> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > task:syz.0.12        state:D stack:25728 pid:1823  tgid:1714  ppid:442    flags:0x00000004
> > > Call Trace:
> > >  <TASK>
> > >  context_switch kernel/sched/core.c:5369 [inline]
> > >  __schedule+0xe60/0x4120 kernel/sched/core.c:6756
> > >  __schedule_loop kernel/sched/core.c:6833 [inline]
> > >  schedule+0xd4/0x210 kernel/sched/core.c:6848
> > >  schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:6905
> > >  rwsem_down_write_slowpath+0x551/0x1660 kernel/locking/rwsem.c:1176
> > >  __down_write_common kernel/locking/rwsem.c:1304 [inline]
> > >  __down_write kernel/locking/rwsem.c:1313 [inline]
> > >  down_write_nested+0x1db/0x210 kernel/locking/rwsem.c:1694
> > >  inode_lock_nested include/linux/fs.h:853 [inline]
> > >  lock_two_nondirectories+0x107/0x210 fs/inode.c:1283
> > >  vfs_rename+0x14c7/0x2a20 fs/namei.c:5038
> > >  do_renameat2+0xb81/0xc90 fs/namei.c:5224
> > >  __do_sys_renameat2 fs/namei.c:5258 [inline]
> > >  __se_sys_renameat2 fs/namei.c:5255 [inline]
> > >  __x64_sys_renameat2+0xe4/0x120 fs/namei.c:5255
> > >  do_syscall_x64 arch/x86/entry/common.c:52 [inline]
> > >  do_syscall_64+0xc3/0x1d0 arch/x86/entry/common.c:83
> > >  entry_SYSCALL_64_after_hwframe+0x77/0x7f
> > > RIP: 0033:0x7fca967c071d
> > > RSP: 002b:00007fca953f2ba8 EFLAGS: 00000246 ORIG_RAX: 000000000000013c
> > > RAX: ffffffffffffffda RBX: 00007fca96983058 RCX: 00007fca967c071d
> > > RDX: ffffffffffffff9c RSI: 0000000020000580 RDI: ffffffffffffff9c
> > > RBP: 00007fca96835425 R08: 0000000000000000 R09: 0000000000000000
> > > R10: 00000000200005c0 R11: 0000000000000246 R12: 0000000000000000
> > > R13: 00007fca96983064 R14: 00007fca969830f0 R15: 00007fca953f2d40
> > >  </TASK>
> > >
> > > Showing all locks held in the system:
> > > 1 lock held by khungtaskd/45:
> > >  #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
> > >  #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
> > >  #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x75/0x340 kernel/locking/lockdep.c:6744
> > > 1 lock held by in:imklog/329:
> > > 5 locks held by syz.0.12/1715:
> > >  #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: do_open fs/namei.c:3821 [inline]
> > >  #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: path_openat+0x1577/0x2970 fs/namei.c:3987
> > >  #1: ff110000415c1780 (&sb->s_type->i_mutex_key#21){++++}-{4:4}, at: inode_lock include/linux/fs.h:818 [inline]
> > >  #1: ff110000415c1780 (&sb->s_type->i_mutex_key#21){++++}-{4:4}, at: do_truncate+0x131/0x200 fs/open.c:63
> > >  #2: ff11000047780a38 (&c->snapshot_create_lock){.+.+}-{4:4}, at: bch2_truncate+0x145/0x260 fs/bcachefs/io_misc.c:292
> > >  #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_lock_acquire include/linux/srcu.h:158 [inline]
> > >  #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_read_lock include/linux/srcu.h:249 [inline]
> > >  #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: __bch2_trans_get+0x625/0xf60 fs/bcachefs/btree_iter.c:3228
> > >  #4: ff110000477a66d0 (&c->gc_lock){.+.+}-{4:4}, at: bch2_btree_update_start+0xb18/0x2010 fs/bcachefs/btree_update_interior.c:1197
> > > 3 locks held by syz.0.12/1823:
> > >  #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: do_renameat2+0x3ea/0xc90 fs/namei.c:5154
> > >  #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: inode_lock_nested include/linux/fs.h:853 [inline]
> > >  #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: lock_rename fs/namei.c:3215 [inline]
> > >  #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: lock_rename+0xa5/0xc0 fs/namei.c:3212
> > >  #2: ff110000415c1780 (&sb->s_type->i_mutex_key#21/4){+.+.}-{4:4}, at: inode_lock_nested include/linux/fs.h:853 [inline]
> > >  #2: ff110000415c1780 (&sb->s_type->i_mutex_key#21/4){+.+.}-{4:4}, at: lock_two_nondirectories+0x107/0x210 fs/inode.c:1283
> > > 4 locks held by bch-reclaim/loo/1811:
> > >  #0: ff110000477cb0a8 (&j->reclaim_lock){+.+.}-{4:4}, at: bch2_journal_reclaim_thread+0x101/0x560 fs/bcachefs/journal_reclaim.c:739
> > >  #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_lock_acquire include/linux/srcu.h:158 [inline]
> > >  #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_read_lock include/linux/srcu.h:249 [inline]
> > >  #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: __bch2_trans_get+0x625/0xf60 fs/bcachefs/btree_iter.c:3228
> > >  #2: ff11000047784740 (&wb->flushing.lock){+.+.}-{4:4}, at: btree_write_buffer_flush_seq+0x69f/0xa40 fs/bcachefs/btree_write_buffer.c:516
> > >  #3: ff110000477a66d0 (&c->gc_lock){.+.+}-{4:4}, at: bch2_btree_update_start+0xb18/0x2010 fs/bcachefs/btree_update_interior.c:1197
> > > 1 lock held by syz-executor/5484:
> > > 2 locks held by syz.1.1333/8755:
> > > 1 lock held by syz.7.1104/8876:
> > >  #0: ffffffff9fb27ef8 (rcu_state.exp_mutex){+.+.}-{4:4}, at: exp_funnel_lock+0x29f/0x3d0 kernel/rcu/tree_exp.h:297
> > > 2 locks held by syz.8.1367/8889:
> > >
> > > =============================================
> > >
> > > ---------------
> > > thanks,
> > > Kun Hu
> > --
> > Jan Kara <jack@suse.com>
> > SUSE Labs, CR

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Kun Hu]

> 
> I suspect the bulk of the reports are coming from academia
> researchers. In lots of academia papers based on syzkaller I see "we
> also reported X bugs to the upstream kernel". Somehow there seems to
> be a preference to keep things secret before publication, so upstream
> syzbot integration is problematic. Though it is well possible to
> publish papers based on OSS work, these usually tend to be higher
> quality and have better evaluation.
> 
> I also don't fully understand the value of "we also reported X bugs to
> the upstream kernel" for research papers. There is little correlation
> with the quality/novelty of research.

It's nice to have a statement from a report. Because academics may not be familiar with the process of reporting, and based on some of the wrong experiences with past Mailing lists, they may continue to use it and make this redundant process reproduce over and over again. I personally support this.😂

-kun

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Theodore Ts'o]

On Tue, Jan 14, 2025 at 10:07:03AM +0100, Dmitry Vyukov wrote:
> I suspect the bulk of the reports are coming from academia
> researchers. In lots of academia papers based on syzkaller I see "we
> also reported X bugs to the upstream kernel". Somehow there seems to
> be a preference to keep things secret before publication, so upstream
> syzbot integration is problematic. Though it is well possible to
> publish papers based on OSS work, these usually tend to be higher
> quality and have better evaluation.
> 
> I also don't fully understand the value of "we also reported X bugs to
> the upstream kernel" for research papers. There is little correlation
> with the quality/novelty of research.

Oh, that's easy.  Statements make it more likely that program
committee members will more likely accept the paper because it's "real
world impact".

And if you're an academic, it's publish or perish, because due to the
gamification of tenure track committees.  Apparently in some countries
the pressure is so huge that academics have started submit fake/sham
papers:

   The startling rise in the publication of sham science papers has
   its roots in China, where young doctors and scientists seeking
   promotion were required to have published scientific papers. Shadow
   organisations – known as “paper mills” – began to supply fabricated
   work for publication in journals there.

   The practice has since spread to India, Iran, Russia, former Soviet
   Union states and eastern Europe, with paper mills supplying
   fabricated studies to more and more journals as increasing numbers
   of young scientists try to boost their careers by claiming false
   research experience. In some cases, journal editors have been
   bribed to accept articles, while paper mills have managed to
   establish their own agents as guest editors who then allow reams of
   falsified work to be published.

   https://www.theguardian.com/science/2024/feb/03/the-situation-has-become-appalling-fake-scientific-papers-push-research-credibility-to-crisis-point

At least in this case it appears to be real syzkaller reports,
although if they were submitting sham papers to sham journals, they at
least wouldn't be wasting upstream kernel developers' time.  :-)

It would be *nice* if researchers at *least* checked to see if their
reports had already been discovered using an unmodified Syzkaller (for
example, by checking the upstream Syzbot web pages).  After all, if
the unmodified/upstream Syzkaller can find the problem, in addition to
wasting our time even more, it's *clearly* not a new/novel result.

	    	    	     	    		- Ted

P.S.  If you want to push back on this nonsense, Usenix program
committee chairs are very much looking for open source professionals
to participate on the program committees for Usenix ATC (Annual
Technical Conference) and FAST (File System and Storage Technologies)
conference.  It's a huge amount of work; easily 40-60 hours of work over
3-4 months.  But it does get you to see what academics are up to, and
it's a way to help point out bogus research, and to push back on
research groups using ancient kernels (typically whatever kernel was
current when the lead professor was a graduate student)....

If you're interested, I can put you in touch with some of those
Program Committee chairs when they are asking me to serve --- there's
more than enough opportunity to go around.  :-)

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Dave Chinner]

On Tue, Jan 14, 2025 at 08:57:51AM -0500, Theodore Ts'o wrote:
> P.S.  If you want to push back on this nonsense, Usenix program
> committee chairs are very much looking for open source professionals
> to participate on the program committees for Usenix ATC (Annual
> Technical Conference) and FAST (File System and Storage Technologies)
> conference.

The problem is that the Usenix/FAST paper committees will not reach
out to OSS subject matter experts to review papers that they have
been asked to review for the conference.

Let me give you a recent example of a clear failure of the FAST
paper committee w.r.t. plagarism.

The core of this paper from FAST 2022:

https://www.usenix.org/conference/fast22/presentation/kim-dohyun

"ScaleXFS: Getting scalability of XFS back on the ring"

is based on the per-CPU CIL logging work I prototyped and posted an
RFC for early in 2021:

https://lore.kernel.org/linux-xfs/20200512092811.1846252-1-david@fromorbit.com/

The main core of the improvements described in the ScaleXFS paper
are the exact per-cpu CIL algorithm in that was contained in the
above RFC patchset.

That algorithm had serious problems that meant it was unworkable in
practice - these didn't show up until journal recovery was tested
and it resulted in random filesystem corruptions. I didn't
understand the root cause of the problem until months later.

These problems were all based on failures to correctly order the
per-CPU log items in the journal due to the per-CPU CIL being
inherently racy.  The algorithm I proposed 6 months later (and
eventually got merged in July 2022) had significant changes to the
way the per-CPU CIL ordered operations to address these problems.

IOWs, object ordering on the CIL is the single most important
critical correctness citeria for the entire journalling algorithm
and hence a fundamental algorithmic constraint for the per-CPU CIL
implementation.

However, the ScaleXFS paper does not make any mention of this
fundamental algorithmic constraint - I did not publish anything
about this constraint until the November 2022 patch set....

There were more clear tell-tales in the paper that indicate
that the "research" was based on that early per-CPU CIL RFC I
posted, but I won't go into details.

I brought this to the FAST committee almost immediately after I was
able to review the paper (a couple of days after the FAST conference
itself). I provided them with all the links to public postings of
the algorithm, detailed analysis of the paper and publicly posted
code, etc. In response, they basically did nothing and brushed my
concerns off. It would take weeks to get any response from the paper
committee, and the overall response really felt like the Usenix
people simply didn't care at all about what was obviously plagarised
work.

IOWs, the Usenix/FAST peer review process for OSS related papers is
broken, and they don't seem to care when experts from the OSS
community actually bring clear cases of academic malpractice to
them...

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Darrick J. Wong]

On Wed, Jan 15, 2025 at 08:21:04AM +1100, Dave Chinner wrote:
> On Tue, Jan 14, 2025 at 08:57:51AM -0500, Theodore Ts'o wrote:
> > P.S.  If you want to push back on this nonsense, Usenix program
> > committee chairs are very much looking for open source professionals
> > to participate on the program committees for Usenix ATC (Annual
> > Technical Conference) and FAST (File System and Storage Technologies)
> > conference.
> 
> The problem is that the Usenix/FAST paper committees will not reach
> out to OSS subject matter experts to review papers that they have
> been asked to review for the conference.
> 
> Let me give you a recent example of a clear failure of the FAST
> paper committee w.r.t. plagarism.
> 
> The core of this paper from FAST 2022:
> 
> https://www.usenix.org/conference/fast22/presentation/kim-dohyun
> 
> "ScaleXFS: Getting scalability of XFS back on the ring"
> 
> is based on the per-CPU CIL logging work I prototyped and posted an
> RFC for early in 2021:
> 
> https://lore.kernel.org/linux-xfs/20200512092811.1846252-1-david@fromorbit.com/
> 
> The main core of the improvements described in the ScaleXFS paper
> are the exact per-cpu CIL algorithm in that was contained in the
> above RFC patchset.
> 
> That algorithm had serious problems that meant it was unworkable in
> practice - these didn't show up until journal recovery was tested
> and it resulted in random filesystem corruptions. I didn't
> understand the root cause of the problem until months later.
> 
> These problems were all based on failures to correctly order the
> per-CPU log items in the journal due to the per-CPU CIL being
> inherently racy.  The algorithm I proposed 6 months later (and
> eventually got merged in July 2022) had significant changes to the
> way the per-CPU CIL ordered operations to address these problems.
> 
> IOWs, object ordering on the CIL is the single most important
> critical correctness citeria for the entire journalling algorithm
> and hence a fundamental algorithmic constraint for the per-CPU CIL
> implementation.
> 
> However, the ScaleXFS paper does not make any mention of this
> fundamental algorithmic constraint - I did not publish anything
> about this constraint until the November 2022 patch set....
> 
> There were more clear tell-tales in the paper that indicate
> that the "research" was based on that early per-CPU CIL RFC I
> posted, but I won't go into details.
> 
> I brought this to the FAST committee almost immediately after I was
> able to review the paper (a couple of days after the FAST conference
> itself). I provided them with all the links to public postings of
> the algorithm, detailed analysis of the paper and publicly posted
> code, etc. In response, they basically did nothing and brushed my
> concerns off. It would take weeks to get any response from the paper
> committee, and the overall response really felt like the Usenix
> people simply didn't care at all about what was obviously plagarised
> work.
> 
> IOWs, the Usenix/FAST peer review process for OSS related papers is
> broken, and they don't seem to care when experts from the OSS
> community actually bring clear cases of academic malpractice to
> them...

Heh, I had some pretty strong suspicions of that when I saw the paper.

--D

> -Dave.
> -- 
> Dave Chinner
> david@fromorbit.com
> 

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Kent Overstreet]

On Wed, Jan 15, 2025 at 08:21:04AM +1100, Dave Chinner wrote:
> On Tue, Jan 14, 2025 at 08:57:51AM -0500, Theodore Ts'o wrote:
> > P.S.  If you want to push back on this nonsense, Usenix program
> > committee chairs are very much looking for open source professionals
> > to participate on the program committees for Usenix ATC (Annual
> > Technical Conference) and FAST (File System and Storage Technologies)
> > conference.
> 
> The problem is that the Usenix/FAST paper committees will not reach
> out to OSS subject matter experts to review papers that they have
> been asked to review for the conference.
> 
> Let me give you a recent example of a clear failure of the FAST
> paper committee w.r.t. plagarism.
> 
> The core of this paper from FAST 2022:
> 
> https://www.usenix.org/conference/fast22/presentation/kim-dohyun
> 
> "ScaleXFS: Getting scalability of XFS back on the ring"
> 
> is based on the per-CPU CIL logging work I prototyped and posted an
> RFC for early in 2021:
> 
> https://lore.kernel.org/linux-xfs/20200512092811.1846252-1-david@fromorbit.com/
> 
> The main core of the improvements described in the ScaleXFS paper
> are the exact per-cpu CIL algorithm in that was contained in the
> above RFC patchset.
> 
> That algorithm had serious problems that meant it was unworkable in
> practice - these didn't show up until journal recovery was tested
> and it resulted in random filesystem corruptions. I didn't
> understand the root cause of the problem until months later.
> 
> These problems were all based on failures to correctly order the
> per-CPU log items in the journal due to the per-CPU CIL being
> inherently racy.  The algorithm I proposed 6 months later (and
> eventually got merged in July 2022) had significant changes to the
> way the per-CPU CIL ordered operations to address these problems.
> 
> IOWs, object ordering on the CIL is the single most important
> critical correctness citeria for the entire journalling algorithm
> and hence a fundamental algorithmic constraint for the per-CPU CIL
> implementation.
> 
> However, the ScaleXFS paper does not make any mention of this
> fundamental algorithmic constraint - I did not publish anything
> about this constraint until the November 2022 patch set....
> 
> There were more clear tell-tales in the paper that indicate
> that the "research" was based on that early per-CPU CIL RFC I
> posted, but I won't go into details.
> 
> I brought this to the FAST committee almost immediately after I was
> able to review the paper (a couple of days after the FAST conference
> itself). I provided them with all the links to public postings of
> the algorithm, detailed analysis of the paper and publicly posted
> code, etc. In response, they basically did nothing and brushed my
> concerns off. It would take weeks to get any response from the paper
> committee, and the overall response really felt like the Usenix
> people simply didn't care at all about what was obviously plagarised
> work.
> 
> IOWs, the Usenix/FAST peer review process for OSS related papers is
> broken, and they don't seem to care when experts from the OSS
> community actually bring clear cases of academic malpractice to
> them...

Yeah, that does look like misconduct, of the type that merits a
boycott...

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Jan Kara]

On Tue 14-01-25 10:07:03, Dmitry Vyukov wrote:
> I also don't fully understand the value of "we also reported X bugs to
> the upstream kernel" for research papers. There is little correlation
> with the quality/novelty of research.

Since I was working in academia in the (distant) pass, let me share my
(slightly educated) guess: In the paper you're supposed to show practical
applicability and relevance of the improvement you propose. It doesn't have
to be really useful but it has to sound useful enough to convince paper
reviewer. I suppose in the fuzzer area this "practical applicability" part
boils down how many bugs were reported...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: Bug: INFO_ task hung in lock_two_nondirectories

[James Bottomley]

On Tue, 2025-01-14 at 14:58 +0100, Jan Kara wrote:
> On Tue 14-01-25 10:07:03, Dmitry Vyukov wrote:
> > I also don't fully understand the value of "we also reported X bugs
> > to the upstream kernel" for research papers. There is little
> > correlation with the quality/novelty of research.
> 
> Since I was working in academia in the (distant) pass, let me share
> my (slightly educated) guess: In the paper you're supposed to show
> practical applicability and relevance of the improvement you propose.
> It doesn't have to be really useful but it has to sound useful enough
> to convince paper reviewer. I suppose in the fuzzer area this
> "practical applicability" part boils down how many bugs were
> reported...

It's not just that, as a recent reviewer for several Academic
Conferences, you always ask about the upstream status.  Chances are if
someone worked on open source but didn't send anything upstream that
was because there wasn't enough value to send.  However, when stuff
does go to upstream lists, you can at least look at what upstream made
of it as part of the review (the guilty confession would be this can be
done quite easily and does break supposedly blind reviews, but it is
very valuable).  Conferences now have open source badges and artifacts
to encourage this behaviour.  I'm afraid this now means that if you're
aiming at a Conference and you didn't send anything upstream you're
quite likely to get a rejection on that fact alone.

Regards,

James

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Kent Overstreet]

On Tue, Jan 14, 2025 at 05:14:39PM +0800, Kun Hu wrote:
> 
> 
> > 2025年1月14日 17:07，Dmitry Vyukov <dvyukov@google.com> 写道：
> > 
> > On Mon, 13 Jan 2025 at 21:00, Kent Overstreet <kent.overstreet@linux.dev <mailto:kent.overstreet@linux.dev>> wrote:
> >> 
> >> On Mon, Jan 13, 2025 at 11:28:43AM +0100, Jan Kara wrote:
> >>> Hello!
> >>> 
> >>> First I'd note that the list of recipients of this report seems somewhat
> >>> arbitrary. linux-kernel & linux-fsdevel makes sense but I'm not sure how
> >>> you have arrived at the list of other persons you have CCed :). I have to
> >>> say syzbot folks have done a pretty good job at implementing mechanisms how
> >>> to determine recipients of the reports (as well as managing existing
> >>> reports over the web / email). Maybe you could take some inspiration there
> >>> or just contribute your syzkaller modifications to syzbot folks so that
> >>> your reports can benefit from all the other infrastructure they have?
> >> 
> >> Is there some political reason that collaboration isn't happening? I've
> >> found the syzbot people to be great to work with.
> >> 
> >> I'll give some analysis on this bug, but in general I won't in the
> >> future until you guys start collaborating (or at least tell us what the
> >> blocker is) - I don't want to be duplicating work I've already done for
> >> syzbot.
> > 
> > I suspect the bulk of the reports are coming from academia
> > researchers. In lots of academia papers based on syzkaller I see "we
> > also reported X bugs to the upstream kernel". Somehow there seems to
> > be a preference to keep things secret before publication, so upstream
> > syzbot integration is problematic. Though it is well possible to
> > publish papers based on OSS work, these usually tend to be higher
> > quality and have better evaluation.
> > 
> > I also don't fully understand the value of "we also reported X bugs to
> > the upstream kernel" for research papers. There is little correlation
> > with the quality/novelty of research.
> > 
> > 
> >>> Anyway, in this particular case, based on locks reported by lockdep, the
> >>> problem seems to be most likely somewhere in bcachefs. Added relevant CCs.
> >>> 
> >>>                                                              Honza
> >>> 
> >>> On Sun 12-01-25 18:00:24, Kun Hu wrote:
> >>>> When using our customized fuzzer tool to fuzz the latest Linux kernel, the following crash (43s)
> >>>> was triggered.
> >>>> 
> >>>> HEAD commit: 9d89551994a430b50c4fffcb1e617a057fa76e20
> >>>> git tree: upstream
> >>>> Console output: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/log0
> >>>> Kernel config: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/config.txt
> >>>> C reproducer: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/12generated_program.c
> >>>> Syzlang reproducer: https://github.com/pghk13/Kernel-Bug/blob/main/0110_6.13rc6/43-INFO_%20task%20hung%20in%20lock_two_nondirectories/12_repro.txt
> >>>> 
> >>>> We first found the issue without a stable C and Syzlang reproducers, but later I tried multiple rounds of replication and got the C and Syzlang reproducers.
> >>>> 
> >>>> I suspect the issue may stem from resource contention or synchronization delays, as indicated by functions like lock_two_nondirectories and vfs_rename. There could also be potential deadlocks or inconsistencies in file system state management (e.g., sb_writers or inode locks) within the bcachefs subsystem. Additionally, interactions between lock_rename and concurrent rename operations might be contributing factors.
> >>>> 
> >>>> Could you kindly help review these areas to narrow down the root cause? Your expertise would be greatly appreciated.
> >> 
> >> We need to know what the other threads are doing. Since lockdep didn't
> >> detect an actual deadlock, it seems most likely that the blocked thread
> >> is blocked because the thread holding the inode lock is spinning and
> >> livelocked.
> >> 
> >> That's not in the dump, so to debug this we'd need to reproduce this in
> >> a local VM and poke around with e.g. top/perf and see what's going on.
> >> 
> >> I've a tool to reproduce syzbot bugs locally in a single command [1], so
> >> that would be my starting point - except it doesn't work with your
> >> forked version. Doh.
> >> 
> >> Another thing we could do is write some additional code for the hung
> >> task detector that, when lockdep is enabled, uses it to figure out which
> >> task it's blocked on and additionally print a backtrace for that.
> >> 
> >> [1]: https://evilpiepirate.org/git/ktest.git/tree/tests/syzbot-repro.ktest
> >> 
> >>>> 
> >>>> If you fix this issue, please add the following tag to the commit:
> >>>> Reported-by: Kun Hu <huk23@m.fudan.edu.cn>, Jiaji Qin <jjtan24@m.fudan.edu.cn>
> >>>> 
> >>>> INFO: task syz.0.12:1823 blocked for more than 143 seconds.
> >>>>      Tainted: G        W          6.13.0-rc6 #1
> >>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> >>>> task:syz.0.12        state:D stack:25728 pid:1823  tgid:1714  ppid:442    flags:0x00000004
> >>>> Call Trace:
> >>>> <TASK>
> >>>> context_switch kernel/sched/core.c:5369 [inline]
> >>>> __schedule+0xe60/0x4120 kernel/sched/core.c:6756
> >>>> __schedule_loop kernel/sched/core.c:6833 [inline]
> >>>> schedule+0xd4/0x210 kernel/sched/core.c:6848
> >>>> schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:6905
> >>>> rwsem_down_write_slowpath+0x551/0x1660 kernel/locking/rwsem.c:1176
> >>>> __down_write_common kernel/locking/rwsem.c:1304 [inline]
> >>>> __down_write kernel/locking/rwsem.c:1313 [inline]
> >>>> down_write_nested+0x1db/0x210 kernel/locking/rwsem.c:1694
> >>>> inode_lock_nested include/linux/fs.h:853 [inline]
> >>>> lock_two_nondirectories+0x107/0x210 fs/inode.c:1283
> >>>> vfs_rename+0x14c7/0x2a20 fs/namei.c:5038
> >>>> do_renameat2+0xb81/0xc90 fs/namei.c:5224
> >>>> __do_sys_renameat2 fs/namei.c:5258 [inline]
> >>>> __se_sys_renameat2 fs/namei.c:5255 [inline]
> >>>> __x64_sys_renameat2+0xe4/0x120 fs/namei.c:5255
> >>>> do_syscall_x64 arch/x86/entry/common.c:52 [inline]
> >>>> do_syscall_64+0xc3/0x1d0 arch/x86/entry/common.c:83
> >>>> entry_SYSCALL_64_after_hwframe+0x77/0x7f
> >>>> RIP: 0033:0x7fca967c071d
> >>>> RSP: 002b:00007fca953f2ba8 EFLAGS: 00000246 ORIG_RAX: 000000000000013c
> >>>> RAX: ffffffffffffffda RBX: 00007fca96983058 RCX: 00007fca967c071d
> >>>> RDX: ffffffffffffff9c RSI: 0000000020000580 RDI: ffffffffffffff9c
> >>>> RBP: 00007fca96835425 R08: 0000000000000000 R09: 0000000000000000
> >>>> R10: 00000000200005c0 R11: 0000000000000246 R12: 0000000000000000
> >>>> R13: 00007fca96983064 R14: 00007fca969830f0 R15: 00007fca953f2d40
> >>>> </TASK>
> >>>> 
> >>>> Showing all locks held in the system:
> >>>> 1 lock held by khungtaskd/45:
> >>>> #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
> >>>> #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
> >>>> #0: ffffffff9fb1aea0 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x75/0x340 kernel/locking/lockdep.c:6744
> >>>> 1 lock held by in:imklog/329:
> >>>> 5 locks held by syz.0.12/1715:
> >>>> #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: do_open fs/namei.c:3821 [inline]
> >>>> #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: path_openat+0x1577/0x2970 fs/namei.c:3987
> >>>> #1: ff110000415c1780 (&sb->s_type->i_mutex_key#21){++++}-{4:4}, at: inode_lock include/linux/fs.h:818 [inline]
> >>>> #1: ff110000415c1780 (&sb->s_type->i_mutex_key#21){++++}-{4:4}, at: do_truncate+0x131/0x200 fs/open.c:63
> >>>> #2: ff11000047780a38 (&c->snapshot_create_lock){.+.+}-{4:4}, at: bch2_truncate+0x145/0x260 fs/bcachefs/io_misc.c:292
> >>>> #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_lock_acquire include/linux/srcu.h:158 [inline]
> >>>> #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_read_lock include/linux/srcu.h:249 [inline]
> >>>> #3: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: __bch2_trans_get+0x625/0xf60 fs/bcachefs/btree_iter.c:3228
> >>>> #4: ff110000477a66d0 (&c->gc_lock){.+.+}-{4:4}, at: bch2_btree_update_start+0xb18/0x2010 fs/bcachefs/btree_update_interior.c:1197
> >>>> 3 locks held by syz.0.12/1823:
> >>>> #0: ff1100003ff8c420 (sb_writers#20){.+.+}-{0:0}, at: do_renameat2+0x3ea/0xc90 fs/namei.c:5154
> >>>> #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: inode_lock_nested include/linux/fs.h:853 [inline]
> >>>> #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: lock_rename fs/namei.c:3215 [inline]
> >>>> #1: ff110000415c0148 (&sb->s_type->i_mutex_key#21/1){+.+.}-{4:4}, at: lock_rename+0xa5/0xc0 fs/namei.c:3212
> >>>> #2: ff110000415c1780 (&sb->s_type->i_mutex_key#21/4){+.+.}-{4:4}, at: inode_lock_nested include/linux/fs.h:853 [inline]
> >>>> #2: ff110000415c1780 (&sb->s_type->i_mutex_key#21/4){+.+.}-{4:4}, at: lock_two_nondirectories+0x107/0x210 fs/inode.c:1283
> >>>> 4 locks held by bch-reclaim/loo/1811:
> >>>> #0: ff110000477cb0a8 (&j->reclaim_lock){+.+.}-{4:4}, at: bch2_journal_reclaim_thread+0x101/0x560 fs/bcachefs/journal_reclaim.c:739
> >>>> #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_lock_acquire include/linux/srcu.h:158 [inline]
> >>>> #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: srcu_read_lock include/linux/srcu.h:249 [inline]
> >>>> #1: ff11000047784398 (&c->btree_trans_barrier){.+.+}-{0:0}, at: __bch2_trans_get+0x625/0xf60 fs/bcachefs/btree_iter.c:3228
> >>>> #2: ff11000047784740 (&wb->flushing.lock){+.+.}-{4:4}, at: btree_write_buffer_flush_seq+0x69f/0xa40 fs/bcachefs/btree_write_buffer.c:516
> >>>> #3: ff110000477a66d0 (&c->gc_lock){.+.+}-{4:4}, at: bch2_btree_update_start+0xb18/0x2010 fs/bcachefs/btree_update_interior.c:1197
> >>>> 1 lock held by syz-executor/5484:
> >>>> 2 locks held by syz.1.1333/8755:
> >>>> 1 lock held by syz.7.1104/8876:
> >>>> #0: ffffffff9fb27ef8 (rcu_state.exp_mutex){+.+.}-{4:4}, at: exp_funnel_lock+0x29f/0x3d0 kernel/rcu/tree_exp.h:297
> >>>> 2 locks held by syz.8.1367/8889:
> >>>> 
> >>>> =============================================
> >>>> 
> >>>> ---------------
> >>>> thanks,
> >>>> Kun Hu
> >>> --
> >>> Jan Kara <jack@suse.com>
> >>> SUSE Labs, CR
> 
> 
> Hello D and Kent,
> 
> Thanks for the advice guys. What you guys say may be true, but I also
> want to make it clear that not working with Syzbot has not come from
> political reasons, nor for the sake of secrecy of the paper, but
> simply just dabbling in the field.

Makes sense. Sorry if I came off a bit strong, there's been a couple
syzbot copycats and I find I keep repeating myself :)

So, it sounds like you're getting nudged to work upstream, i.e. people
funding you want you to be a bit better engineers so the work you're
doing is taken up (academics tend to be lousy engineers, and vice
versa, heh).

But if you're working on fuzzing, upstream is syzbot, not the kernel -
if there's a community you should be working with, that's the one.

An individual bug report like this is pretty low value to me. I see a
ton of dups, and dups coming from yet another system is downright
painful.

The real value is all the infrastructure /around/ running tests and
finding bugs: ingesting all that data into dashboards so I can look for
patterns, and additional tooling (like the ktest/syzbot integration, as
well as other things) for getting the most out of every bug report
possible.

If you're working on fuzzing, you don't want to be taking all that on
solo. That's the power of working with a community :) And more than
that, we do _very_ much need more community minded people getting
involved with testing in general, not just fuzzing.

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Kun Hu]

> 
> Makes sense. Sorry if I came off a bit strong, there's been a couple
> syzbot copycats and I find I keep repeating myself :)
> 
> So, it sounds like you're getting nudged to work upstream, i.e. people
> funding you want you to be a bit better engineers so the work you're
> doing is taken up (academics tend to be lousy engineers, and vice
> versa, heh).
> 
> But if you're working on fuzzing, upstream is syzbot, not the kernel -
> if there's a community you should be working with, that's the one.
> 
> An individual bug report like this is pretty low value to me. I see a
> ton of dups, and dups coming from yet another system is downright
> painful.
> 
> The real value is all the infrastructure /around/ running tests and
> finding bugs: ingesting all that data into dashboards so I can look for
> patterns, and additional tooling (like the ktest/syzbot integration, as
> well as other things) for getting the most out of every bug report
> possible.
> 
> If you're working on fuzzing, you don't want to be taking all that on
> solo. That's the power of working with a community :) And more than
> that, we do _very_ much need more community minded people getting
> involved with testing in general, not just fuzzing.
> 


Hi Kent,

Thank you for your insights.

I have a couple of questions I would like to ask for your advice on:

From a testing perspective, we have modified syzkaller and discovered some issues. It’s true that researchers working on fuzzing often lack a deep understanding of the kernel, making it difficult to precisely determine the scope of reported problems. Meanwhile, syzbot provides a description of the reporting process (please refer to the link below) and encourages researchers to report bugs to maintainers. However, there seems to be a significant gap here—researchers may end up reporting bugs to the wrong maintainers or submitting reports that lack value. This seems like a serious issue. Could the kernel community consider establishing a standardized process to reduce wasted time on all sides and prevent researchers from inflating bug counts just to validate their papers?

Link: https://github.com/google/syzkaller/blob/master/docs/linux/reporting_kernel_bugs.md

It is often suggested that researchers collaborate with syzbot. What should such collaboration look like in terms of form and content? From a maintainer's perspective, how would you prefer to see researchers who report bugs work with syzbot? Since I’m new to this field, I’m not very familiar with the process and would greatly appreciate your guidance.

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Kent Overstreet]

On Thu, Jan 16, 2025 at 05:37:24PM +0800, Kun Hu wrote:
> 
> > 
> > Makes sense. Sorry if I came off a bit strong, there's been a couple
> > syzbot copycats and I find I keep repeating myself :)
> > 
> > So, it sounds like you're getting nudged to work upstream, i.e. people
> > funding you want you to be a bit better engineers so the work you're
> > doing is taken up (academics tend to be lousy engineers, and vice
> > versa, heh).
> > 
> > But if you're working on fuzzing, upstream is syzbot, not the kernel -
> > if there's a community you should be working with, that's the one.
> > 
> > An individual bug report like this is pretty low value to me. I see a
> > ton of dups, and dups coming from yet another system is downright
> > painful.
> > 
> > The real value is all the infrastructure /around/ running tests and
> > finding bugs: ingesting all that data into dashboards so I can look for
> > patterns, and additional tooling (like the ktest/syzbot integration, as
> > well as other things) for getting the most out of every bug report
> > possible.
> > 
> > If you're working on fuzzing, you don't want to be taking all that on
> > solo. That's the power of working with a community :) And more than
> > that, we do _very_ much need more community minded people getting
> > involved with testing in general, not just fuzzing.
> > 
> 
> 
> Hi Kent,
> 
> Thank you for your insights.
> 
> I have a couple of questions I would like to ask for your advice on:
> 
> From a testing perspective, we have modified syzkaller and discovered
> some issues. It’s true that researchers working on fuzzing often lack
> a deep understanding of the kernel, making it difficult to precisely
> determine the scope of reported problems. Meanwhile, syzbot provides a
> description of the reporting process (please refer to the link below)
> and encourages researchers to report bugs to maintainers. However,
> there seems to be a significant gap here—researchers may end up
> reporting bugs to the wrong maintainers or submitting reports that
> lack value. This seems like a serious issue. Could the kernel
> community consider establishing a standardized process to reduce
> wasted time on all sides and prevent researchers from inflating bug
> counts just to validate their papers?

Again: the standard mantra is "work wth upstream". If you forked syzbot
and you're working on fuzzing, syzbot is your upstream, not the kernel.

If you work with the syzbot people on getting your fuzz testing
improvements merged, the process of reporting the bugs fuzzing finds to
us kernel folks won't be on your plate anymore, and you get to focus on
the stuff you actually want to be doing. :)

> Link: https://github.com/google/syzkaller/blob/master/docs/linux/reporting_kernel_bugs.md
> 
> It is often suggested that researchers collaborate with syzbot. What
> should such collaboration look like in terms of form and content? From
> a maintainer's perspective, how would you prefer to see researchers
> who report bugs work with syzbot? Since I’m new to this field, I’m not
> very familiar with the process and would greatly appreciate your
> guidance.

Talk to the syzbot folks, they're a friendly bunch :)

The main thing to keep in mind is that it's never _just_ about getting
your code merged, you need to spend some time learning the lay of the
land so you can understand where your work fits in, and what people need
and are interested in - you want to make sure that you're not
duplicating work, and you want your code to be as maintainable and
broadly useful to everone else as it can be.

Have you shared with them what your research interests are?

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Kun Hu]

> 
> Again: the standard mantra is "work wth upstream". If you forked syzbot
> and you're working on fuzzing, syzbot is your upstream, not the kernel.
> 
> If you work with the syzbot people on getting your fuzz testing
> improvements merged, the process of reporting the bugs fuzzing finds to
> us kernel folks won't be on your plate anymore, and you get to focus on
> the stuff you actually want to be doing. :)
> 
>> Link: https://github.com/google/syzkaller/blob/master/docs/linux/reporting_kernel_bugs.md
>> 
>> It is often suggested that researchers collaborate with syzbot. What
>> should such collaboration look like in terms of form and content? From
>> a maintainer's perspective, how would you prefer to see researchers
>> who report bugs work with syzbot? Since I’m new to this field, I’m not
>> very familiar with the process and would greatly appreciate your
>> guidance.
> 
> Talk to the syzbot folks, they're a friendly bunch :)
> 
> The main thing to keep in mind is that it's never _just_ about getting
> your code merged, you need to spend some time learning the lay of the
> land so you can understand where your work fits in, and what people need
> and are interested in - you want to make sure that you're not
> duplicating work, and you want your code to be as maintainable and
> broadly useful to everone else as it can be.
> 
> Have you shared with them what your research interests are?


Sorry for late. Thank you very much for your advice, we’ll try to work with the syzbot community. 

But an interesting interaction relationship is that for researchers from academia to prove the advanced technology of their fuzzer, they seem to need to use their personal finding of real-world bugs as an important experimental metric. I think that's why you get reports that are modeled after syzbot (the official description of syzkaller describes the process for independent reports). If the quality of the individual reports is low, it does affect the judgment of the maintainer, but also it is also a waste of everyone's time.

This seems to be a problem that exists in the linux, syzbot community and academia….

———————————
Best,
Kun

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Kent Overstreet]

On Fri, Jan 24, 2025 at 08:22:57PM +0800, Kun Hu wrote:
> 
> > 
> > Again: the standard mantra is "work wth upstream". If you forked syzbot
> > and you're working on fuzzing, syzbot is your upstream, not the kernel.
> > 
> > If you work with the syzbot people on getting your fuzz testing
> > improvements merged, the process of reporting the bugs fuzzing finds to
> > us kernel folks won't be on your plate anymore, and you get to focus on
> > the stuff you actually want to be doing. :)
> > 
> >> Link: https://github.com/google/syzkaller/blob/master/docs/linux/reporting_kernel_bugs.md
> >> 
> >> It is often suggested that researchers collaborate with syzbot. What
> >> should such collaboration look like in terms of form and content? From
> >> a maintainer's perspective, how would you prefer to see researchers
> >> who report bugs work with syzbot? Since I’m new to this field, I’m not
> >> very familiar with the process and would greatly appreciate your
> >> guidance.
> > 
> > Talk to the syzbot folks, they're a friendly bunch :)
> > 
> > The main thing to keep in mind is that it's never _just_ about getting
> > your code merged, you need to spend some time learning the lay of the
> > land so you can understand where your work fits in, and what people need
> > and are interested in - you want to make sure that you're not
> > duplicating work, and you want your code to be as maintainable and
> > broadly useful to everone else as it can be.
> > 
> > Have you shared with them what your research interests are?
> 
> 
> Sorry for late. Thank you very much for your advice, we’ll try to work with the syzbot community. 
> 
> But an interesting interaction relationship is that for researchers
> from academia to prove the advanced technology of their fuzzer, they
> seem to need to use their personal finding of real-world bugs as an
> important experimental metric. I think that's why you get reports that
> are modeled after syzbot (the official description of syzkaller
> describes the process for independent reports). If the quality of the
> individual reports is low, it does affect the judgment of the
> maintainer, but also it is also a waste of everyone's time.

Ah, metrics and TPS reports. Symptoms of MBA culture, "we just want to
manage by watching the numbers and making sure they go up - deep
expertise in the thing we're managing? what's that?"

Push back on that mindset wherever you can and your life will be less
nonsensical and frustrating :)

Maybe see if you can find some other way of showing the value of your
work? Things like patches accepted into upstream syzbot, public
feedback from the syzbot maintainers, or just describing your work
yourself to your higher ups instead of relying on metrics - or see if
you can help them come up with some better metrics.

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Theodore Ts'o]

On Fri, Jan 24, 2025 at 08:22:57PM +0800, Kun Hu wrote:
> 
> But an interesting interaction relationship is that for researchers
> from academia to prove the advanced technology of their fuzzer, they
> seem to need to use their personal finding of real-world bugs as an
> important experimental metric. I think that's why you get reports
> that are modeled after syzbot (the official description of syzkaller
> describes the process for independent reports). If the quality of
> the individual reports is low, it does affect the judgment of the
> maintainer, but also it is also a waste of everyone's time.

If you're going to do this, I would suggest that you make sure that
you're actually finding new bugs.  More often than not, after wasting
a huge amount of time of upstream developers because the researchers
don't set up the syzbot dashboard and e-mail service to test to see if
a patch fixes a bug (often which we can't reproduce on our own because
it's highly environment sensitive), we discover that it's already been
reported on the upstream syzbot.

Which makes the academic syzbot a *double* waste of time, and this
trains upstream developers to simply ignore reports from these
research forks of syzbot unless it comes with a reproducer, or maybe
(this hasn't happened yet) if the researchers actually set up the web
dashboard and e-mail responder.

I also blame the peer reviewers for the journals, for not asking the
question, "why haven't you shown that the 'real world' bugs that your
forked syzbot has found are ones that the original syzcaller hasn't
found yet?"  And for not demanding of the academics, "if you want
*real* impact, get your changes merged upstream with the upstream
syzcaller, so that it will continue to find and fix vulnerabilities
instead of ceasing the moment we accept your paper."

Cheers,

						- Ted

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Kent Overstreet]

On Fri, Jan 24, 2025 at 01:28:17PM -0500, Theodore Ts'o wrote:
> On Fri, Jan 24, 2025 at 08:22:57PM +0800, Kun Hu wrote:
> > 
> > But an interesting interaction relationship is that for researchers
> > from academia to prove the advanced technology of their fuzzer, they
> > seem to need to use their personal finding of real-world bugs as an
> > important experimental metric. I think that's why you get reports
> > that are modeled after syzbot (the official description of syzkaller
> > describes the process for independent reports). If the quality of
> > the individual reports is low, it does affect the judgment of the
> > maintainer, but also it is also a waste of everyone's time.
> 
> If you're going to do this, I would suggest that you make sure that
> you're actually finding new bugs.  More often than not, after wasting
> a huge amount of time of upstream developers because the researchers
> don't set up the syzbot dashboard and e-mail service to test to see if
> a patch fixes a bug (often which we can't reproduce on our own because
> it's highly environment sensitive), we discover that it's already been
> reported on the upstream syzbot.
> 
> Which makes the academic syzbot a *double* waste of time, and this
> trains upstream developers to simply ignore reports from these
> research forks of syzbot unless it comes with a reproducer, or maybe
> (this hasn't happened yet) if the researchers actually set up the web
> dashboard and e-mail responder.
> 
> I also blame the peer reviewers for the journals, for not asking the
> question, "why haven't you shown that the 'real world' bugs that your
> forked syzbot has found are ones that the original syzcaller hasn't
> found yet?"  And for not demanding of the academics, "if you want
> *real* impact, get your changes merged upstream with the upstream
> syzcaller, so that it will continue to find and fix vulnerabilities
> instead of ceasing the moment we accept your paper."

Yeah, I think these are good reasons why "number of bugs reported" is
just a poor metric.

Patches accepted upstream by syzbot would be much better, that indicates
that the actual practicioners in your field have found your work
valuable and checked that you aren't duplicating work that's already
been done.

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Kun Hu]

> 
> Is there some political reason that collaboration isn't happening? I've
> found the syzbot people to be great to work with.
> 
> I'll give some analysis on this bug, but in general I won't in the
> future until you guys start collaborating (or at least tell us what the
> blocker is) - I don't want to be duplicating work I've already done for
> syzbot.

Thanks, no political reasons, simply just researching this area and not familiar enough with it, I'll keep an eye out for this later.


> 
> We need to know what the other threads are doing. Since lockdep didn't
> detect an actual deadlock, it seems most likely that the blocked thread
> is blocked because the thread holding the inode lock is spinning and
> livelocked.
> 
> That's not in the dump, so to debug this we'd need to reproduce this in
> a local VM and poke around with e.g. top/perf and see what's going on.
> 
> I've a tool to reproduce syzbot bugs locally in a single command [1], so
> that would be my starting point - except it doesn't work with your
> forked version. Doh.
> 
> Another thing we could do is write some additional code for the hung
> task detector that, when lockdep is enabled, uses it to figure out which
> task it's blocked on and additionally print a backtrace for that.
> 
> [1]: https://evilpiepirate.org/git/ktest.git/tree/tests/syzbot-repro.ktest
> 
>> 

Ok, I'll try it with this tool. For now, we are experimenting based on the early November 2024 version (df3dc63), let's see if we can debug this tool of yours.

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Kun Hu]

> 
> 
> Hello!
> 
> First I'd note that the list of recipients of this report seems somewhat
> arbitrary. linux-kernel & linux-fsdevel makes sense but I'm not sure how
> you have arrived at the list of other persons you have CCed :). I have to
> say syzbot folks have done a pretty good job at implementing mechanisms how
> to determine recipients of the reports (as well as managing existing
> reports over the web / email). Maybe you could take some inspiration there
> or just contribute your syzkaller modifications to syzbot folks so that
> your reports can benefit from all the other infrastructure they have?
> 
> Anyway, in this particular case, based on locks reported by lockdep, the
> problem seems to be most likely somewhere in bcachefs. Added relevant CCs.


Hi Jan,

Thank you very much for your feedback. The current method of determining recipients is indeed less structured than syzbot`s mechanism, and we have only improved the method of generating the seed, and have not yet entered very much into understanding his infrastructure. We will definitely consider taking our approach from `syzbot` and not have this problem next time.

Thanks for pointing out that it may have originated from `bcachefs`.

————
Kun Hu

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Christian Brauner]

On Sun, Jan 12, 2025 at 06:00:24PM +0800, Kun Hu wrote:
> Hello,
> 
> When using our customized fuzzer tool to fuzz the latest Linux kernel, the following crash (43s)
> was triggered.

I think we need to come to an agreement at LSFMM or somewhere else that
we will by default ingore but reports from non-syzbot fuzzers. Because
we're all wasting time on them.

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Matthew Wilcox]

On Mon, Jan 13, 2025 at 03:38:57PM +0100, Christian Brauner wrote:
> On Sun, Jan 12, 2025 at 06:00:24PM +0800, Kun Hu wrote:
> > Hello,
> > 
> > When using our customized fuzzer tool to fuzz the latest Linux kernel, the following crash (43s)
> > was triggered.
> 
> I think we need to come to an agreement at LSFMM or somewhere else that
> we will by default ingore but reports from non-syzbot fuzzers. Because
> we're all wasting time on them.

I think it needs to be broader than that to also include "AI generated
bug reports" (while not excluding AI-translated bug reports); see

https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/

so really, any "automated bug report" system is out of bounds unless
previously arranged with the developers who it's supposed to be helping.
We need to write that down somewhere in Documentation/process/ so we
can point misguided people at it.

We should also talk about how some parts of the kernel are basically
unmaintained and unused, and that automated testing should be focused
on parts of the kernel that are actually used.  A report about being
able to crash a stock configuration of ext4 is more useful than being
able to crash an unusual configuration of ufs.

Distinguishing between warnings, BUG()s and actual crashes would also
be a useful thing to put in this document.

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Darrick J. Wong]

On Mon, Jan 13, 2025 at 04:19:01PM +0000, Matthew Wilcox wrote:
> On Mon, Jan 13, 2025 at 03:38:57PM +0100, Christian Brauner wrote:
> > On Sun, Jan 12, 2025 at 06:00:24PM +0800, Kun Hu wrote:
> > > Hello,
> > > 
> > > When using our customized fuzzer tool to fuzz the latest Linux kernel, the following crash (43s)
> > > was triggered.
> > 
> > I think we need to come to an agreement at LSFMM or somewhere else that
> > we will by default ingore but reports from non-syzbot fuzzers. Because
> > we're all wasting time on them.

No need to wait until LSFMM, I already agree with the premise of
deprioritizing/ignoring piles of reports that come in all at once with
very little analysis, an IOCCC-esque reproducer, and no effort on the
part of the reporter to do *anything* about the bug.

While the Google syzbot dashboard has improved remarkably since 2018,
particularly in the past couple of years, thanks to the people who did
that!  It's nice that I can fire off patches at the bot and it'll test
them.  That said, I don't perceive Google management to be funding much
of anyone to solve the problems that their fuzzer uncovers.

This is to say nothing of the people who are coyly running their own
instances of syzbot sans dashboard and expecting me to download random
crap from Google Drive.  Hell no, I don't do that kind of thing in 2025.

> I think it needs to be broader than that to also include "AI generated
> bug reports" (while not excluding AI-translated bug reports); see
> 
> https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/
> 
> so really, any "automated bug report" system is out of bounds unless
> previously arranged with the developers who it's supposed to be helping.

Agree.  That's been my stance since syzbot first emerged in 2017-18.

> We need to write that down somewhere in Documentation/process/ so we
> can point misguided people at it.
> 
> We should also talk about how some parts of the kernel are basically
> unmaintained and unused, and that automated testing should be focused
> on parts of the kernel that are actually used.  A report about being
> able to crash a stock configuration of ext4 is more useful than being
> able to crash an unusual configuration of ufs.

Or maybe we should try to make fuse + iouring fast enough that we can
kick all these old legacy drivers out to userspace. ;)

> Distinguishing between warnings, BUG()s and actual crashes would also
> be a useful thing to put in this document.

Yes.  And also state that panic_on_warn=1 is a signal that you wanted
fail(over) fast mode.

--D

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Dmitry Vyukov]

On Mon, 13 Jan 2025 at 19:08, Darrick J. Wong <djwong@kernel.org> wrote:
>
> On Mon, Jan 13, 2025 at 04:19:01PM +0000, Matthew Wilcox wrote:
> > On Mon, Jan 13, 2025 at 03:38:57PM +0100, Christian Brauner wrote:
> > > On Sun, Jan 12, 2025 at 06:00:24PM +0800, Kun Hu wrote:
> > > > Hello,
> > > >
> > > > When using our customized fuzzer tool to fuzz the latest Linux kernel, the following crash (43s)
> > > > was triggered.
> > >
> > > I think we need to come to an agreement at LSFMM or somewhere else that
> > > we will by default ingore but reports from non-syzbot fuzzers. Because
> > > we're all wasting time on them.
>
> No need to wait until LSFMM, I already agree with the premise of
> deprioritizing/ignoring piles of reports that come in all at once with
> very little analysis, an IOCCC-esque reproducer, and no effort on the
> part of the reporter to do *anything* about the bug.

+1

It would be good to publish it somewhere on kernel.org (Documentation/
or people.kernel.org).
We could include the link to our guidelines for external reporters
(where we ask to not test old kernels, reporting dups, not including
essential info, and other silly things):
https://github.com/google/syzkaller/blob/master/docs/linux/reporting_kernel_bugs.mdThere
is unfortunately no way to make people read all docs on the internet
beforehand, but at least it's handy to have a link to an existing doc
to give to people.

> While the Google syzbot dashboard has improved remarkably since 2018,
> particularly in the past couple of years, thanks to the people who did
> that!  It's nice that I can fire off patches at the bot and it'll test
> them.  That said, I don't perceive Google management to be funding much
> of anyone to solve the problems that their fuzzer uncovers.
>
> This is to say nothing of the people who are coyly running their own
> instances of syzbot sans dashboard and expecting me to download random
> crap from Google Drive.  Hell no, I don't do that kind of thing in 2025.
>
> > I think it needs to be broader than that to also include "AI generated
> > bug reports" (while not excluding AI-translated bug reports); see
> >
> > https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/
> >
> > so really, any "automated bug report" system is out of bounds unless
> > previously arranged with the developers who it's supposed to be helping.
>
> Agree.  That's been my stance since syzbot first emerged in 2017-18.
>
> > We need to write that down somewhere in Documentation/process/ so we
> > can point misguided people at it.
> >
> > We should also talk about how some parts of the kernel are basically
> > unmaintained and unused, and that automated testing should be focused
> > on parts of the kernel that are actually used.  A report about being
> > able to crash a stock configuration of ext4 is more useful than being
> > able to crash an unusual configuration of ufs.
>
> Or maybe we should try to make fuse + iouring fast enough that we can
> kick all these old legacy drivers out to userspace. ;)
>
> > Distinguishing between warnings, BUG()s and actual crashes would also
> > be a useful thing to put in this document.
>
> Yes.  And also state that panic_on_warn=1 is a signal that you wanted
> fail(over) fast mode.
>
> --D

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Darrick J. Wong]

On Tue, Jan 14, 2025 at 09:59:08AM +0100, Dmitry Vyukov wrote:
> On Mon, 13 Jan 2025 at 19:08, Darrick J. Wong <djwong@kernel.org> wrote:
> >
> > On Mon, Jan 13, 2025 at 04:19:01PM +0000, Matthew Wilcox wrote:
> > > On Mon, Jan 13, 2025 at 03:38:57PM +0100, Christian Brauner wrote:
> > > > On Sun, Jan 12, 2025 at 06:00:24PM +0800, Kun Hu wrote:
> > > > > Hello,
> > > > >
> > > > > When using our customized fuzzer tool to fuzz the latest Linux kernel, the following crash (43s)
> > > > > was triggered.
> > > >
> > > > I think we need to come to an agreement at LSFMM or somewhere else that
> > > > we will by default ingore but reports from non-syzbot fuzzers. Because
> > > > we're all wasting time on them.
> >
> > No need to wait until LSFMM, I already agree with the premise of
> > deprioritizing/ignoring piles of reports that come in all at once with
> > very little analysis, an IOCCC-esque reproducer, and no effort on the
> > part of the reporter to do *anything* about the bug.
> 
> +1
> 
> It would be good to publish it somewhere on kernel.org (Documentation/
> or people.kernel.org).
> We could include the link to our guidelines for external reporters
> (where we ask to not test old kernels, reporting dups, not including
> essential info, and other silly things):
> https://github.com/google/syzkaller/blob/master/docs/linux/reporting_kernel_bugs.mdThere

/me reads, wonders if "Do NOT mimic syzbot reports" is behind why
everyone claims they have a "custom fuzzer" and proceed to leak the
letters s, y, and z in the report. :P

> is unfortunately no way to make people read all docs on the internet
> beforehand, but at least it's handy to have a link to an existing doc
> to give to people.

<nod> Good idea, we probably ought to have a kernel document setting out
our general policies about automated bug finders and linking to the
known good ones.

BTW, if one got handed a syzbot report, is there an easy way to ask your
dashboard if it already knows about that report?

--D

> > While the Google syzbot dashboard has improved remarkably since 2018,
> > particularly in the past couple of years, thanks to the people who did
> > that!  It's nice that I can fire off patches at the bot and it'll test
> > them.  That said, I don't perceive Google management to be funding much
> > of anyone to solve the problems that their fuzzer uncovers.
> >
> > This is to say nothing of the people who are coyly running their own
> > instances of syzbot sans dashboard and expecting me to download random
> > crap from Google Drive.  Hell no, I don't do that kind of thing in 2025.
> >
> > > I think it needs to be broader than that to also include "AI generated
> > > bug reports" (while not excluding AI-translated bug reports); see
> > >
> > > https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/
> > >
> > > so really, any "automated bug report" system is out of bounds unless
> > > previously arranged with the developers who it's supposed to be helping.
> >
> > Agree.  That's been my stance since syzbot first emerged in 2017-18.
> >
> > > We need to write that down somewhere in Documentation/process/ so we
> > > can point misguided people at it.
> > >
> > > We should also talk about how some parts of the kernel are basically
> > > unmaintained and unused, and that automated testing should be focused
> > > on parts of the kernel that are actually used.  A report about being
> > > able to crash a stock configuration of ext4 is more useful than being
> > > able to crash an unusual configuration of ufs.
> >
> > Or maybe we should try to make fuse + iouring fast enough that we can
> > kick all these old legacy drivers out to userspace. ;)
> >
> > > Distinguishing between warnings, BUG()s and actual crashes would also
> > > be a useful thing to put in this document.
> >
> > Yes.  And also state that panic_on_warn=1 is a signal that you wanted
> > fail(over) fast mode.
> >
> > --D

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Dmitry Vyukov]

On Mon, 13 Jan 2025 at 19:08, Darrick J. Wong <djwong@kernel.org> wrote:
>
> On Mon, Jan 13, 2025 at 04:19:01PM +0000, Matthew Wilcox wrote:
> > On Mon, Jan 13, 2025 at 03:38:57PM +0100, Christian Brauner wrote:
> > > On Sun, Jan 12, 2025 at 06:00:24PM +0800, Kun Hu wrote:
> > > > Hello,
> > > >
> > > > When using our customized fuzzer tool to fuzz the latest Linux kernel, the following crash (43s)
> > > > was triggered.
> > >
> > > I think we need to come to an agreement at LSFMM or somewhere else that
> > > we will by default ingore but reports from non-syzbot fuzzers. Because
> > > we're all wasting time on them.
>
> No need to wait until LSFMM, I already agree with the premise of
> deprioritizing/ignoring piles of reports that come in all at once with
> very little analysis, an IOCCC-esque reproducer, and no effort on the
> part of the reporter to do *anything* about the bug.
>
> While the Google syzbot dashboard has improved remarkably since 2018,
> particularly in the past couple of years, thanks to the people who did
> that!

And, thanks, Darrick!
Most credit goes to Aleksandr Nogikh, who worked on improvements in
the past years.
We don't always have cycles to implement everything immediately, but
we are listening.

>  It's nice that I can fire off patches at the bot and it'll test
> them.  That said, I don't perceive Google management to be funding much
> of anyone to solve the problems that their fuzzer uncovers.
>
> This is to say nothing of the people who are coyly running their own
> instances of syzbot sans dashboard and expecting me to download random
> crap from Google Drive.  Hell no, I don't do that kind of thing in 2025.
>
> > I think it needs to be broader than that to also include "AI generated
> > bug reports" (while not excluding AI-translated bug reports); see
> >
> > https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/
> >
> > so really, any "automated bug report" system is out of bounds unless
> > previously arranged with the developers who it's supposed to be helping.
>
> Agree.  That's been my stance since syzbot first emerged in 2017-18.
>
> > We need to write that down somewhere in Documentation/process/ so we
> > can point misguided people at it.
> >
> > We should also talk about how some parts of the kernel are basically
> > unmaintained and unused, and that automated testing should be focused
> > on parts of the kernel that are actually used.  A report about being
> > able to crash a stock configuration of ext4 is more useful than being
> > able to crash an unusual configuration of ufs.
>
> Or maybe we should try to make fuse + iouring fast enough that we can
> kick all these old legacy drivers out to userspace. ;)
>
> > Distinguishing between warnings, BUG()s and actual crashes would also
> > be a useful thing to put in this document.
>
> Yes.  And also state that panic_on_warn=1 is a signal that you wanted
> fail(over) fast mode.
>
> --D

Re: Bug: INFO_ task hung in lock_two_nondirectories

[Darrick J. Wong]

On Tue, Jan 14, 2025 at 10:15:28AM +0100, Dmitry Vyukov wrote:
> On Mon, 13 Jan 2025 at 19:08, Darrick J. Wong <djwong@kernel.org> wrote:
> >
> > On Mon, Jan 13, 2025 at 04:19:01PM +0000, Matthew Wilcox wrote:
> > > On Mon, Jan 13, 2025 at 03:38:57PM +0100, Christian Brauner wrote:
> > > > On Sun, Jan 12, 2025 at 06:00:24PM +0800, Kun Hu wrote:
> > > > > Hello,
> > > > >
> > > > > When using our customized fuzzer tool to fuzz the latest Linux kernel, the following crash (43s)
> > > > > was triggered.
> > > >
> > > > I think we need to come to an agreement at LSFMM or somewhere else that
> > > > we will by default ingore but reports from non-syzbot fuzzers. Because
> > > > we're all wasting time on them.
> >
> > No need to wait until LSFMM, I already agree with the premise of
> > deprioritizing/ignoring piles of reports that come in all at once with
> > very little analysis, an IOCCC-esque reproducer, and no effort on the
> > part of the reporter to do *anything* about the bug.
> >
> > While the Google syzbot dashboard has improved remarkably since 2018,
> > particularly in the past couple of years, thanks to the people who did
> > that!
> 
> And, thanks, Darrick!
> Most credit goes to Aleksandr Nogikh, who worked on improvements in
> the past years.
> We don't always have cycles to implement everything immediately, but
> we are listening.

You're welcome, and to both of you, thank you for all the improvements
over the last 8-9 years. :)

--D

> >  It's nice that I can fire off patches at the bot and it'll test
> > them.  That said, I don't perceive Google management to be funding much
> > of anyone to solve the problems that their fuzzer uncovers.
> >
> > This is to say nothing of the people who are coyly running their own
> > instances of syzbot sans dashboard and expecting me to download random
> > crap from Google Drive.  Hell no, I don't do that kind of thing in 2025.
> >
> > > I think it needs to be broader than that to also include "AI generated
> > > bug reports" (while not excluding AI-translated bug reports); see
> > >
> > > https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/
> > >
> > > so really, any "automated bug report" system is out of bounds unless
> > > previously arranged with the developers who it's supposed to be helping.
> >
> > Agree.  That's been my stance since syzbot first emerged in 2017-18.
> >
> > > We need to write that down somewhere in Documentation/process/ so we
> > > can point misguided people at it.
> > >
> > > We should also talk about how some parts of the kernel are basically
> > > unmaintained and unused, and that automated testing should be focused
> > > on parts of the kernel that are actually used.  A report about being
> > > able to crash a stock configuration of ext4 is more useful than being
> > > able to crash an unusual configuration of ufs.
> >
> > Or maybe we should try to make fuse + iouring fast enough that we can
> > kick all these old legacy drivers out to userspace. ;)
> >
> > > Distinguishing between warnings, BUG()s and actual crashes would also
> > > be a useful thing to put in this document.
> >
> > Yes.  And also state that panic_on_warn=1 is a signal that you wanted
> > fail(over) fast mode.
> >
> > --D