Re: [syzbot] [btrfs?] [netfilter?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (2)

Re: [syzbot] [btrfs?] [netfilter?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (2)

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit:    e40939bbfc68 Merge branch 'for-next/core' into for-kernelci
git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
console output: https://syzkaller.appspot.com/x/log.txt?x=15d92aaaa80000
kernel config:  https://syzkaller.appspot.com/x/.config?x=c4a2640e4213bc2f
dashboard link: https://syzkaller.appspot.com/bug?extid=9bbbacfbf1e04d5221f7
compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
userspace arch: arm64
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=149b2d66a80000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=1214348aa80000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/9d87aa312c0e/disk-e40939bb.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/22a11d32a8b2/vmlinux-e40939bb.xz
kernel image: https://storage.googleapis.com/syzbot-assets/0978b5788b52/Image-e40939bb.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+9bbbacfbf1e04d5221f7@syzkaller.appspotmail.com

team3253: Mode changed to "activebackup"
BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
turning off the locking correctness validator.
CPU: 1 PID: 9973 Comm: syz-executor246 Not tainted 6.4.0-rc7-syzkaller-ge40939bbfc68 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/03/2023
Call trace:
 dump_backtrace+0x1b8/0x1e4 arch/arm64/kernel/stacktrace.c:233
 show_stack+0x2c/0x44 arch/arm64/kernel/stacktrace.c:240
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xd0/0x124 lib/dump_stack.c:106
 dump_stack+0x1c/0x28 lib/dump_stack.c:113
 lookup_chain_cache_add kernel/locking/lockdep.c:3794 [inline]
 validate_chain kernel/locking/lockdep.c:3815 [inline]
 __lock_acquire+0x1c44/0x7604 kernel/locking/lockdep.c:5088
 lock_acquire+0x23c/0x71c kernel/locking/lockdep.c:5705
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x48/0x60 kernel/locking/spinlock.c:154
 spin_lock include/linux/spinlock.h:350 [inline]
 pl011_console_write+0x180/0x708 drivers/tty/serial/amba-pl011.c:2333
 console_emit_next_record kernel/printk/printk.c:2877 [inline]
 console_flush_all+0x5c0/0xb54 kernel/printk/printk.c:2933
 console_unlock+0x148/0x274 kernel/printk/printk.c:3007
 vprintk_emit+0x14c/0x2e4 kernel/printk/printk.c:2307
 vprintk_default+0xa0/0xe4 kernel/printk/printk.c:2318
 vprintk+0x218/0x2f0 kernel/printk/printk_safe.c:50
 _printk+0xdc/0x128 kernel/printk/printk.c:2328
 __netdev_printk+0x1f8/0x39c net/core/dev.c:11273
 netdev_info+0x104/0x150 net/core/dev.c:11320
 team_change_mode drivers/net/team/team.c:619 [inline]
 team_mode_option_set+0x350/0x390 drivers/net/team/team.c:1388
 team_option_set drivers/net/team/team.c:374 [inline]
 team_nl_cmd_options_set+0x7e0/0xdec drivers/net/team/team.c:2663
 genl_family_rcv_msg_doit net/netlink/genetlink.c:968 [inline]
 genl_family_rcv_msg net/netlink/genetlink.c:1048 [inline]
 genl_rcv_msg+0x938/0xc1c net/netlink/genetlink.c:1065
 netlink_rcv_skb+0x214/0x3c4 net/netlink/af_netlink.c:2546
 genl_rcv+0x38/0x50 net/netlink/genetlink.c:1076
 netlink_unicast_kernel net/netlink/af_netlink.c:1339 [inline]
 netlink_unicast+0x660/0x8d4 net/netlink/af_netlink.c:1365
 netlink_sendmsg+0x834/0xb18 net/netlink/af_netlink.c:1913
 sock_sendmsg_nosec net/socket.c:724 [inline]
 sock_sendmsg net/socket.c:747 [inline]
 ____sys_sendmsg+0x568/0x81c net/socket.c:2503
 ___sys_sendmsg net/socket.c:2557 [inline]
 __sys_sendmsg+0x26c/0x33c net/socket.c:2586
 __do_sys_sendmsg net/socket.c:2595 [inline]
 __se_sys_sendmsg net/socket.c:2593 [inline]
 __arm64_sys_sendmsg+0x80/0x94 net/socket.c:2593
 __invoke_syscall arch/arm64/kernel/syscall.c:38 [inline]
 invoke_syscall+0x98/0x2c0 arch/arm64/kernel/syscall.c:52
 el0_svc_common+0x138/0x244 arch/arm64/kernel/syscall.c:142
 do_el0_svc+0x64/0x198 arch/arm64/kernel/syscall.c:191
 el0_svc+0x4c/0x160 arch/arm64/kernel/entry-common.c:647
 el0t_64_sync_handler+0x84/0xfc arch/arm64/kernel/entry-common.c:665
 el0t_64_sync+0x190/0x194 arch/arm64/kernel/entry.S:591


---
If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

Re: [syzbot] [btrfs?] [netfilter?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (2)

[David Sterba]

On Wed, Jul 19, 2023 at 02:32:51AM -0700, syzbot wrote:
> syzbot has found a reproducer for the following issue on:
> 
> HEAD commit:    e40939bbfc68 Merge branch 'for-next/core' into for-kernelci
> git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
> console output: https://syzkaller.appspot.com/x/log.txt?x=15d92aaaa80000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=c4a2640e4213bc2f
> dashboard link: https://syzkaller.appspot.com/bug?extid=9bbbacfbf1e04d5221f7
> compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> userspace arch: arm64
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=149b2d66a80000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=1214348aa80000
> 
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/9d87aa312c0e/disk-e40939bb.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/22a11d32a8b2/vmlinux-e40939bb.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/0978b5788b52/Image-e40939bb.gz.xz

#syz unset btrfs

The MAX_LOCKDEP_CHAIN_HLOCKS bugs/warnings can be worked around by
configuration, otherwise are considered invalid. This report has also
'netfilter' label so I'm not closing it right away.

Re: [syzbot] [btrfs?] [netfilter?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (2)

[syzbot]

> On Wed, Jul 19, 2023 at 02:32:51AM -0700, syzbot wrote:
>> syzbot has found a reproducer for the following issue on:
>> 
>> HEAD commit:    e40939bbfc68 Merge branch 'for-next/core' into for-kernelci
>> git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
>> console output: https://syzkaller.appspot.com/x/log.txt?x=15d92aaaa80000
>> kernel config:  https://syzkaller.appspot.com/x/.config?x=c4a2640e4213bc2f
>> dashboard link: https://syzkaller.appspot.com/bug?extid=9bbbacfbf1e04d5221f7
>> compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
>> userspace arch: arm64
>> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=149b2d66a80000
>> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=1214348aa80000
>> 
>> Downloadable assets:
>> disk image: https://storage.googleapis.com/syzbot-assets/9d87aa312c0e/disk-e40939bb.raw.xz
>> vmlinux: https://storage.googleapis.com/syzbot-assets/22a11d32a8b2/vmlinux-e40939bb.xz
>> kernel image: https://storage.googleapis.com/syzbot-assets/0978b5788b52/Image-e40939bb.gz.xz
>
> #syz unset btrfs

The following labels did not exist: btrfs

>
> The MAX_LOCKDEP_CHAIN_HLOCKS bugs/warnings can be worked around by
> configuration, otherwise are considered invalid. This report has also
> 'netfilter' label so I'm not closing it right away.

Re: [syzbot] [btrfs?] [netfilter?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (2)

[Aleksandr Nogikh]

On Wed, Jul 19, 2023 at 7:11 PM syzbot
<syzbot+9bbbacfbf1e04d5221f7@syzkaller.appspotmail.com> wrote:
>
> > On Wed, Jul 19, 2023 at 02:32:51AM -0700, syzbot wrote:
> >> syzbot has found a reproducer for the following issue on:
> >>
> >> HEAD commit:    e40939bbfc68 Merge branch 'for-next/core' into for-kernelci
> >> git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
> >> console output: https://syzkaller.appspot.com/x/log.txt?x=15d92aaaa80000
> >> kernel config:  https://syzkaller.appspot.com/x/.config?x=c4a2640e4213bc2f
> >> dashboard link: https://syzkaller.appspot.com/bug?extid=9bbbacfbf1e04d5221f7
> >> compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> >> userspace arch: arm64
> >> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=149b2d66a80000
> >> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=1214348aa80000
> >>
> >> Downloadable assets:
> >> disk image: https://storage.googleapis.com/syzbot-assets/9d87aa312c0e/disk-e40939bb.raw.xz
> >> vmlinux: https://storage.googleapis.com/syzbot-assets/22a11d32a8b2/vmlinux-e40939bb.xz
> >> kernel image: https://storage.googleapis.com/syzbot-assets/0978b5788b52/Image-e40939bb.gz.xz
> >
> > #syz unset btrfs
>
> The following labels did not exist: btrfs

#syz set subsystems: netfilter

>
> >
> > The MAX_LOCKDEP_CHAIN_HLOCKS bugs/warnings can be worked around by
> > configuration, otherwise are considered invalid. This report has also
> > 'netfilter' label so I'm not closing it right away.
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/00000000000042a3ac0600da1f69%40google.com.

Re: [syzbot] [btrfs?] [netfilter?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (2)

[Florian Westphal]

Aleksandr Nogikh <nogikh@google.com> wrote:
> On Wed, Jul 19, 2023 at 7:11 PM syzbot
> <syzbot+9bbbacfbf1e04d5221f7@syzkaller.appspotmail.com> wrote:
> >
> > > On Wed, Jul 19, 2023 at 02:32:51AM -0700, syzbot wrote:
> > >> syzbot has found a reproducer for the following issue on:
> > >>
> > >> HEAD commit:    e40939bbfc68 Merge branch 'for-next/core' into for-kernelci
> > >> git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
> > >> console output: https://syzkaller.appspot.com/x/log.txt?x=15d92aaaa80000
> > >> kernel config:  https://syzkaller.appspot.com/x/.config?x=c4a2640e4213bc2f
> > >> dashboard link: https://syzkaller.appspot.com/bug?extid=9bbbacfbf1e04d5221f7
> > >> compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> > >> userspace arch: arm64
> > >> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=149b2d66a80000
> > >> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=1214348aa80000
> > >>
> > >> Downloadable assets:
> > >> disk image: https://storage.googleapis.com/syzbot-assets/9d87aa312c0e/disk-e40939bb.raw.xz
> > >> vmlinux: https://storage.googleapis.com/syzbot-assets/22a11d32a8b2/vmlinux-e40939bb.xz
> > >> kernel image: https://storage.googleapis.com/syzbot-assets/0978b5788b52/Image-e40939bb.gz.xz
> > >
> > > #syz unset btrfs
> >
> > The following labels did not exist: btrfs
> 
> #syz set subsystems: netfilter

I don't see any netfilter involvement here.

The repro just creates a massive amount of team devices.

At the time it hits the LOCKDEP limits on my test vm it has
created ~2k team devices, system load is at +14 because udev
is also busy spawing hotplug scripts for the new devices.

After reboot and suspending the running reproducer after about 1500
devices (before hitting lockdep limits), followed by 'ip link del' for
the team devices gets the lockdep entries down to ~8k (from 40k),
which is in the range that it has on this VM after a fresh boot.

So as far as I can see this workload is just pushing lockdep
past what it can handle with the configured settings and is
not triggering any actual bug.

Re: [syzbot] [btrfs?] [netfilter?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (2)

[Jakub Kicinski]

On Thu, 20 Jul 2023 01:12:07 +0200 Florian Westphal wrote:
> I don't see any netfilter involvement here.
> 
> The repro just creates a massive amount of team devices.
> 
> At the time it hits the LOCKDEP limits on my test vm it has
> created ~2k team devices, system load is at +14 because udev
> is also busy spawing hotplug scripts for the new devices.
> 
> After reboot and suspending the running reproducer after about 1500
> devices (before hitting lockdep limits), followed by 'ip link del' for
> the team devices gets the lockdep entries down to ~8k (from 40k),
> which is in the range that it has on this VM after a fresh boot.
> 
> So as far as I can see this workload is just pushing lockdep
> past what it can handle with the configured settings and is
> not triggering any actual bug.

The lockdep splat because of netdevice stacking is one of our top
reports from syzbot. Is anyone else feeling like we should add 
an artificial but very high limit on netdev stacking? :(

Re: [syzbot] [btrfs?] [netfilter?] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! (2)

[Paolo Abeni]

On Wed, 2023-07-19 at 20:30 -0700, Jakub Kicinski wrote:
> On Thu, 20 Jul 2023 01:12:07 +0200 Florian Westphal wrote:
> > I don't see any netfilter involvement here.
> > 
> > The repro just creates a massive amount of team devices.
> > 
> > At the time it hits the LOCKDEP limits on my test vm it has
> > created ~2k team devices, system load is at +14 because udev
> > is also busy spawing hotplug scripts for the new devices.
> > 
> > After reboot and suspending the running reproducer after about 1500
> > devices (before hitting lockdep limits), followed by 'ip link del' for
> > the team devices gets the lockdep entries down to ~8k (from 40k),
> > which is in the range that it has on this VM after a fresh boot.
> > 
> > So as far as I can see this workload is just pushing lockdep
> > past what it can handle with the configured settings and is
> > not triggering any actual bug.
> 
> The lockdep splat because of netdevice stacking is one of our top
> reports from syzbot. Is anyone else feeling like we should add 
> an artificial but very high limit on netdev stacking? :(

We already have a similar limit for xmit: XMIT_RECURSION_LIMIT. I guess
stacking more then such devices will be quite useless/non functional.
We could use such value to limit the device stacking, too.

Cheers,

Paolo