[syzbot] [btrfs?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (6)

[syzbot] [btrfs?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (6)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    5c43d43bad35 Merge branches 'for-next/acpi', 'for-next/mis..
git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
console output: https://syzkaller.appspot.com/x/log.txt?x=13471a05980000
kernel config:  https://syzkaller.appspot.com/x/.config?x=c91f83ae59feaa1f
dashboard link: https://syzkaller.appspot.com/bug?extid=dfb6eff2a68b42d557d3
compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
userspace arch: arm64
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=10efded5980000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=12e94093980000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/cc2dd4be620e/disk-5c43d43b.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/81d40d99ddbf/vmlinux-5c43d43b.xz
kernel image: https://storage.googleapis.com/syzbot-assets/bc6aed0f2bc5/Image-5c43d43b.gz.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/d55321fffedc/mount_0.gz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+dfb6eff2a68b42d557d3@syzkaller.appspotmail.com

BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
turning off the locking correctness validator.
CPU: 0 UID: 0 PID: 2037 Comm: kworker/u8:8 Not tainted 6.11.0-rc3-syzkaller-g5c43d43bad35 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/27/2024
Workqueue: btrfs-endio-write btrfs_work_helper
Call trace:
 dump_backtrace+0x1b8/0x1e4 arch/arm64/kernel/stacktrace.c:317
 show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:324
 __dump_stack lib/dump_stack.c:93 [inline]
 dump_stack_lvl+0xe4/0x150 lib/dump_stack.c:119
 dump_stack+0x1c/0x28 lib/dump_stack.c:128
 lookup_chain_cache_add kernel/locking/lockdep.c:3815 [inline]
 validate_chain kernel/locking/lockdep.c:3836 [inline]
 __lock_acquire+0x1fa0/0x779c kernel/locking/lockdep.c:5142
 lock_acquire+0x240/0x728 kernel/locking/lockdep.c:5759
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x48/0x60 kernel/locking/spinlock.c:154
 spin_lock include/linux/spinlock.h:351 [inline]
 free_extent_buffer_stale+0x2c/0x1ec fs/btrfs/extent_io.c:3333
 btrfs_force_cow_block+0xf2c/0x1c9c fs/btrfs/ctree.c:659
 btrfs_cow_block+0x318/0xa28 fs/btrfs/ctree.c:754
 btrfs_search_slot+0xba0/0x2a08
 btrfs_lookup_file_extent+0x124/0x1bc fs/btrfs/file-item.c:267
 btrfs_drop_extents+0x370/0x2ad8 fs/btrfs/file.c:251
 insert_reserved_file_extent+0x2b4/0xa6c fs/btrfs/inode.c:2911
 insert_ordered_extent_file_extent+0x348/0x508 fs/btrfs/inode.c:3016
 btrfs_finish_one_ordered+0x6a0/0x129c fs/btrfs/inode.c:3124
 btrfs_finish_ordered_io+0x120/0x134 fs/btrfs/inode.c:3266
 finish_ordered_fn+0x20/0x30 fs/btrfs/ordered-data.c:331
 btrfs_work_helper+0x340/0xd28 fs/btrfs/async-thread.c:314
 process_one_work+0x79c/0x15b8 kernel/workqueue.c:3231
 process_scheduled_works kernel/workqueue.c:3312 [inline]
 worker_thread+0x938/0xebc kernel/workqueue.c:3390
 kthread+0x288/0x310 kernel/kthread.c:389
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Re: [syzbot] [btrfs?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (6)

[Josef Bacik]

On Wed, Aug 21, 2024 at 12:45:25PM -0700, syzbot wrote:
> Hello,
> 
> syzbot found the following issue on:
> 
> HEAD commit:    5c43d43bad35 Merge branches 'for-next/acpi', 'for-next/mis..
> git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
> console output: https://syzkaller.appspot.com/x/log.txt?x=13471a05980000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=c91f83ae59feaa1f
> dashboard link: https://syzkaller.appspot.com/bug?extid=dfb6eff2a68b42d557d3
> compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> userspace arch: arm64
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=10efded5980000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=12e94093980000
> 
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/cc2dd4be620e/disk-5c43d43b.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/81d40d99ddbf/vmlinux-5c43d43b.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/bc6aed0f2bc5/Image-5c43d43b.gz.xz
> mounted in repro: https://storage.googleapis.com/syzbot-assets/d55321fffedc/mount_0.gz
> 
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+dfb6eff2a68b42d557d3@syzkaller.appspotmail.com
> 
> BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

Can we disable syzbot issues for this specific error?  Btrfs uses lockdep
annotations for our tree locks, so we _easily_ cross this threshold on the
default configuration.  Our CI config requires the following settings to get
lockdep to work longer than two or three tests

CONFIG_LOCKDEP_BITS=20
CONFIG_LOCKDEP_CHAINS_BITS=20
CONFIG_LOCKDEP_STACK_TRACE_BITS=19
CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14
CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12

but there's no way to require that in our config (nor do I think we should
really be able to tbqh).  It makes more sense for syzbot to just ignore this
particular error as it's not actually a bug.  Thanks,

Josef

Re: [syzbot] [btrfs?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (6)

[Dmitry Vyukov]

On Thu, 22 Aug 2024 at 10:57, Josef Bacik <josef@toxicpanda.com> wrote:
>
> On Wed, Aug 21, 2024 at 12:45:25PM -0700, syzbot wrote:
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit:    5c43d43bad35 Merge branches 'for-next/acpi', 'for-next/mis..
> > git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
> > console output: https://syzkaller.appspot.com/x/log.txt?x=13471a05980000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=c91f83ae59feaa1f
> > dashboard link: https://syzkaller.appspot.com/bug?extid=dfb6eff2a68b42d557d3
> > compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> > userspace arch: arm64
> > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=10efded5980000
> > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=12e94093980000
> >
> > Downloadable assets:
> > disk image: https://storage.googleapis.com/syzbot-assets/cc2dd4be620e/disk-5c43d43b.raw.xz
> > vmlinux: https://storage.googleapis.com/syzbot-assets/81d40d99ddbf/vmlinux-5c43d43b.xz
> > kernel image: https://storage.googleapis.com/syzbot-assets/bc6aed0f2bc5/Image-5c43d43b.gz.xz
> > mounted in repro: https://storage.googleapis.com/syzbot-assets/d55321fffedc/mount_0.gz
> >
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: syzbot+dfb6eff2a68b42d557d3@syzkaller.appspotmail.com
> >
> > BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
>
> Can we disable syzbot issues for this specific error?  Btrfs uses lockdep
> annotations for our tree locks, so we _easily_ cross this threshold on the
> default configuration.  Our CI config requires the following settings to get
> lockdep to work longer than two or three tests
>
> CONFIG_LOCKDEP_BITS=20
> CONFIG_LOCKDEP_CHAINS_BITS=20
> CONFIG_LOCKDEP_STACK_TRACE_BITS=19
> CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14
> CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12
>
> but there's no way to require that in our config (nor do I think we should
> really be able to tbqh).  It makes more sense for syzbot to just ignore this
> particular error as it's not actually a bug.  Thanks,

Hi Josef,

We could bump these values, the last 3 are already this or higher on syzbot.
Do you know if increasing CONFIG_LOCKDEP_BITS and
CONFIG_LOCKDEP_CHAINS_BITS significantly increases memory usage?

Ignoring random bugs on unknown heuristics is really not scalable.
Consider: there are hundreds of kernel subsystems, if each of them
declares a random subset of bugs as not bugs. What's the maintenance
story here? And it's not syzbot specific, any automated and manual
testing will have the same problem.
The only scalable way to mark false reports is to not produce them.

Re: [syzbot] [btrfs?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (6)

[David Sterba]

On Thu, Aug 22, 2024 at 02:05:01PM +0200, Dmitry Vyukov wrote:
> > > BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> >
> > Can we disable syzbot issues for this specific error?  Btrfs uses lockdep
> > annotations for our tree locks, so we _easily_ cross this threshold on the
> > default configuration.  Our CI config requires the following settings to get
> > lockdep to work longer than two or three tests
> >
> > CONFIG_LOCKDEP_BITS=20
> > CONFIG_LOCKDEP_CHAINS_BITS=20
> > CONFIG_LOCKDEP_STACK_TRACE_BITS=19
> > CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14
> > CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12
> >
> > but there's no way to require that in our config (nor do I think we should
> > really be able to tbqh).  It makes more sense for syzbot to just ignore this
> > particular error as it's not actually a bug.  Thanks,
> 
> Hi Josef,
> 
> We could bump these values, the last 3 are already this or higher on syzbot.
> Do you know if increasing CONFIG_LOCKDEP_BITS and
> CONFIG_LOCKDEP_CHAINS_BITS significantly increases memory usage?
> 
> Ignoring random bugs on unknown heuristics is really not scalable.

This is not a random bug. The warning has been reported many times, it
does not point to a specific problem in code that uses lockdep but
rather some defficiency in the lockdep mechanism itself.

> Consider: there are hundreds of kernel subsystems, if each of them
> declares a random subset of bugs as not bugs.

"If each of them", no this won't happen. Or, if you add this one and
reject the others you'll still make people happy.

> What's the maintenance
> story here? And it's not syzbot specific, any automated and manual
> testing will have the same problem.

Yes this does not avoid reports but at least it won't be a syzbot report
that somebody thinks is worth time. Everybody else will be told "ignore"
or poitned to documentation or the report ignored completely
(https://btrfs.readthedocs.io/en/latest/dev/Development-notes.html#bug-max-lockdep-chain-hlocks-too-low).

> The only scalable way to mark false reports is to not produce them.

In an ideal case yes. So far we have only the workaround with increasing
the config value (which makes sense on a distro config), otherwise I
remembet locking guys to suggest some fix but I can't find it now in the
numerous reports.

Re: [syzbot] [btrfs?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (6)

[Dmitry Vyukov]

On Sat, 24 Aug 2024 at 21:38, David Sterba <dsterba@suse.cz> wrote:
>
> On Thu, Aug 22, 2024 at 02:05:01PM +0200, Dmitry Vyukov wrote:
> > > > BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > >
> > > Can we disable syzbot issues for this specific error?  Btrfs uses lockdep
> > > annotations for our tree locks, so we _easily_ cross this threshold on the
> > > default configuration.  Our CI config requires the following settings to get
> > > lockdep to work longer than two or three tests
> > >
> > > CONFIG_LOCKDEP_BITS=20
> > > CONFIG_LOCKDEP_CHAINS_BITS=20
> > > CONFIG_LOCKDEP_STACK_TRACE_BITS=19
> > > CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14
> > > CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12
> > >
> > > but there's no way to require that in our config (nor do I think we should
> > > really be able to tbqh).  It makes more sense for syzbot to just ignore this
> > > particular error as it's not actually a bug.  Thanks,
> >
> > Hi Josef,
> >
> > We could bump these values, the last 3 are already this or higher on syzbot.
> > Do you know if increasing CONFIG_LOCKDEP_BITS and
> > CONFIG_LOCKDEP_CHAINS_BITS significantly increases memory usage?
> >
> > Ignoring random bugs on unknown heuristics is really not scalable.
>
> This is not a random bug. The warning has been reported many times, it
> does not point to a specific problem in code that uses lockdep but
> rather some defficiency in the lockdep mechanism itself.

By "random" I meant that the predicate is some custom English
sentence, rather than something that can be expressed in the code. So
on the global kernel scale it's hard/impossible to filter out such
reports.

Additional complication here is that the predicate involves knowing
that exactly system calls triggered this warning, since the warning is
generic. We don't generally know what exact syscall sequence triggered
a report. So it would only be possible to ignore "BUG:
MAX_LOCKDEP_CHAIN_HLOCKS too low" globally, which is not good.

> > Consider: there are hundreds of kernel subsystems, if each of them
> > declares a random subset of bugs as not bugs.
>
> "If each of them", no this won't happen. Or, if you add this one and
> reject the others you'll still make people happy.
>
> > What's the maintenance
> > story here? And it's not syzbot specific, any automated and manual
> > testing will have the same problem.
>
> Yes this does not avoid reports but at least it won't be a syzbot report
> that somebody thinks is worth time. Everybody else will be told "ignore"
> or poitned to documentation or the report ignored completely
> (https://btrfs.readthedocs.io/en/latest/dev/Development-notes.html#bug-max-lockdep-chain-hlocks-too-low).
>
> > The only scalable way to mark false reports is to not produce them.
>
> In an ideal case yes. So far we have only the workaround with increasing
> the config value (which makes sense on a distro config), otherwise I
> remembet locking guys to suggest some fix but I can't find it now in the
> numerous reports.

I've bumped LOCKDEP parameters in syzbot configs:
https://github.com/google/syzkaller/commit/f4865e39dd0bcae7e5f3f5d59807d6ac9a8a99ba

So this can be closed:

#syz invalid

Re: [syzbot] [btrfs?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (6)

[David Sterba]

On Fri, Aug 30, 2024 at 05:02:09PM +0200, Dmitry Vyukov wrote:
> On Sat, 24 Aug 2024 at 21:38, David Sterba <dsterba@suse.cz> wrote:
> >
> > On Thu, Aug 22, 2024 at 02:05:01PM +0200, Dmitry Vyukov wrote:
> > > > > BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > > >
> > > > Can we disable syzbot issues for this specific error?  Btrfs uses lockdep
> > > > annotations for our tree locks, so we _easily_ cross this threshold on the
> > > > default configuration.  Our CI config requires the following settings to get
> > > > lockdep to work longer than two or three tests
> > > >
> > > > CONFIG_LOCKDEP_BITS=20
> > > > CONFIG_LOCKDEP_CHAINS_BITS=20
> > > > CONFIG_LOCKDEP_STACK_TRACE_BITS=19
> > > > CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14
> > > > CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12
> > > >
> > > > but there's no way to require that in our config (nor do I think we should
> > > > really be able to tbqh).  It makes more sense for syzbot to just ignore this
> > > > particular error as it's not actually a bug.  Thanks,
> > >
> > > Hi Josef,
> > >
> > > We could bump these values, the last 3 are already this or higher on syzbot.
> > > Do you know if increasing CONFIG_LOCKDEP_BITS and
> > > CONFIG_LOCKDEP_CHAINS_BITS significantly increases memory usage?
> > >
> > > Ignoring random bugs on unknown heuristics is really not scalable.
> >
> > This is not a random bug. The warning has been reported many times, it
> > does not point to a specific problem in code that uses lockdep but
> > rather some defficiency in the lockdep mechanism itself.
> 
> By "random" I meant that the predicate is some custom English
> sentence, rather than something that can be expressed in the code. So
> on the global kernel scale it's hard/impossible to filter out such
> reports.
> 
> Additional complication here is that the predicate involves knowing
> that exactly system calls triggered this warning, since the warning is
> generic. We don't generally know what exact syscall sequence triggered
> a report. So it would only be possible to ignore "BUG:
> MAX_LOCKDEP_CHAIN_HLOCKS too low" globally, which is not good.
> 
> > > Consider: there are hundreds of kernel subsystems, if each of them
> > > declares a random subset of bugs as not bugs.
> >
> > "If each of them", no this won't happen. Or, if you add this one and
> > reject the others you'll still make people happy.
> >
> > > What's the maintenance
> > > story here? And it's not syzbot specific, any automated and manual
> > > testing will have the same problem.
> >
> > Yes this does not avoid reports but at least it won't be a syzbot report
> > that somebody thinks is worth time. Everybody else will be told "ignore"
> > or poitned to documentation or the report ignored completely
> > (https://btrfs.readthedocs.io/en/latest/dev/Development-notes.html#bug-max-lockdep-chain-hlocks-too-low).
> >
> > > The only scalable way to mark false reports is to not produce them.
> >
> > In an ideal case yes. So far we have only the workaround with increasing
> > the config value (which makes sense on a distro config), otherwise I
> > remembet locking guys to suggest some fix but I can't find it now in the
> > numerous reports.
> 
> I've bumped LOCKDEP parameters in syzbot configs:
> https://github.com/google/syzkaller/commit/f4865e39dd0bcae7e5f3f5d59807d6ac9a8a99ba

LOCKDEP_CHAINS_BITS=20 will improve the situation, thanks.