[syzbot] [wireguard?] INFO: rcu detected stall in wg_ratelimiter_gc_entries (2)

[syzbot] [wireguard?] INFO: rcu detected stall in wg_ratelimiter_gc_entries (2)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    ace0ab3a4b54 Revert "vlan: Fix VLAN 0 memory leak"
git tree:       net
console output: https://syzkaller.appspot.com/x/log.txt?x=16153769a80000
kernel config:  https://syzkaller.appspot.com/x/.config?x=3e670757e16affb
dashboard link: https://syzkaller.appspot.com/bug?extid=c1cc0083f159b67cb192
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=1227599ba80000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=17414927a80000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/e03bf2f0ff9c/disk-ace0ab3a.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/ad6e79c01723/vmlinux-ace0ab3a.xz
kernel image: https://storage.googleapis.com/syzbot-assets/617319e5afb7/bzImage-ace0ab3a.xz

The issue was bisected to:

commit c2368b19807affd7621f7c4638cd2e17fec13021
Author: Jiri Pirko <jiri@nvidia.com>
Date:   Fri Jul 29 07:10:35 2022 +0000

    net: devlink: introduce "unregistering" mark and use it during devlinks iteration

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=17901617a80000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=14501617a80000
console output: https://syzkaller.appspot.com/x/log.txt?x=10501617a80000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+c1cc0083f159b67cb192@syzkaller.appspotmail.com
Fixes: c2368b19807a ("net: devlink: introduce "unregistering" mark and use it during devlinks iteration")

rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 	1-....: (10499 ticks this GP) idle=2d5c/1/0x4000000000000000 softirq=8994/8995 fqs=4737
rcu: 	         hardirqs   softirqs   csw/system
rcu: 	 number:        0          0            0
rcu: 	cputime:    32198      20291           25   ==> 52490(ms)
rcu: 	(t=10500 jiffies g=7889 q=546 ncpus=2)
CPU: 1 PID: 5075 Comm: kworker/1:6 Not tainted 6.5.0-rc5-syzkaller-00194-gace0ab3a4b54 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023
Workqueue: events_power_efficient wg_ratelimiter_gc_entries
RIP: 0010:taprio_next_tc_txq net/sched/sch_taprio.c:771 [inline]
RIP: 0010:taprio_dequeue_tc_priority+0x2fb/0x4b0 net/sched/sch_taprio.c:801
Code: 01 00 00 48 be 00 00 00 00 00 fc ff df 48 8b 4c 24 28 48 89 c8 48 c1 e8 03 0f b6 14 30 48 89 c8 83 e0 07 83 c0 01 38 d0 7c 08 <84> d2 0f 85 da 00 00 00 48 8b 04 24 45 0f b7 75 fe 0f b6 00 38 44
RSP: 0000:ffffc900001e0d60 EFLAGS: 00000202
RAX: 0000000000000007 RBX: ffff88806f6f6394 RCX: ffff88807b860b5e
RDX: 0000000000000000 RSI: dffffc0000000000 RDI: dffffc0000000000
RBP: 000000000000000b R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000000 R11: 000000000000004e R12: 0000000000000008
R13: ffff88807b860b60 R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5ace99f5c0 CR3: 000000006f75b000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 taprio_dequeue+0x12e/0x5f0 net/sched/sch_taprio.c:868
 dequeue_skb net/sched/sch_generic.c:292 [inline]
 qdisc_restart net/sched/sch_generic.c:397 [inline]
 __qdisc_run+0x1c4/0x19d0 net/sched/sch_generic.c:415
 qdisc_run include/net/pkt_sched.h:125 [inline]
 qdisc_run include/net/pkt_sched.h:122 [inline]
 net_tx_action+0x71e/0xc80 net/core/dev.c:5049
 __do_softirq+0x218/0x965 kernel/softirq.c:553
 invoke_softirq kernel/softirq.c:427 [inline]
 __irq_exit_rcu kernel/softirq.c:632 [inline]
 irq_exit_rcu+0xb7/0x120 kernel/softirq.c:644
 sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1109
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:645
RIP: 0010:lock_acquire+0x1ef/0x510 kernel/locking/lockdep.c:5729
Code: c1 05 d5 6e 9b 7e 83 f8 01 0f 85 b0 02 00 00 9c 58 f6 c4 02 0f 85 9b 02 00 00 48 85 ed 74 01 fb 48 b8 00 00 00 00 00 fc ff df <48> 01 c3 48 c7 03 00 00 00 00 48 c7 43 08 00 00 00 00 48 8b 84 24
RSP: 0000:ffffc90003e1fb98 EFLAGS: 00000206
RAX: dffffc0000000000 RBX: 1ffff920007c3f75 RCX: 0000000000000001
RDX: 1ffff11003f03c80 RSI: ffffffff8a6c83a0 RDI: ffffffff8ac811a0
RBP: 0000000000000200 R08: 0000000000000000 R09: fffffbfff2309dea
R10: ffffffff9184ef57 R11: 0000000000000000 R12: 0000000000000001
R13: 0000000000000000 R14: ffffffff8d89afb8 R15: 0000000000000000
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154
 spin_lock include/linux/spinlock.h:351 [inline]
 wg_ratelimiter_gc_entries+0xc6/0x520 drivers/net/wireguard/ratelimiter.c:63
 process_one_work+0xaa2/0x16f0 kernel/workqueue.c:2600
 worker_thread+0x687/0x1110 kernel/workqueue.c:2751
 kthread+0x33a/0x430 kernel/kthread.c:389
 ret_from_fork+0x2c/0x70 arch/x86/kernel/process.c:145
 ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:304
 </TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
For information about bisection process see: https://goo.gl/tpsmEJ#bisection

If the bug is already fixed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite bug's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the bug is a duplicate of another bug, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Re: [syzbot] [wireguard?] INFO: rcu detected stall in wg_ratelimiter_gc_entries (2)

[syzbot]

Hello,

syzbot has tested the proposed patch but the reproducer is still triggering an issue:
INFO: rcu detected stall in process_one_work

rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 0-...D } 2685 jiffies s: 52849 root: 0x1/.
rcu: blocking rcu_node structures (internal RCU debug):
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 12 Comm: kworker/u4:1 Not tainted 6.5.0-rc5-syzkaller-00194-gace0ab3a4b54-dirty #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/26/2023
Workqueue: bat_events batadv_bla_periodic_work
RIP: 0010:__sanitizer_cov_trace_const_cmp4+0x0/0x20 kernel/kcov.c:303
Code: d6 fe ff ff 66 0f 1f 44 00 00 f3 0f 1e fa 48 8b 0c 24 0f b7 d6 0f b7 f7 bf 03 00 00 00 e9 b8 fe ff ff 0f 1f 84 00 00 00 00 00 <f3> 0f 1e fa 48 8b 0c 24 89 f2 89 fe bf 05 00 00 00 e9 9a fe ff ff
RSP: 0018:ffffc90000007e38 EFLAGS: 00000046
RAX: 0000000000000001 RBX: ffff8880b982b980 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: 0000000000000001 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000001 R11: 000000000000004e R12: ffff8880b982b880
R13: 177c2886f5028478 R14: ffff8880719ee340 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa162667440 CR3: 000000000c776000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 __run_hrtimer kernel/time/hrtimer.c:1654 [inline]
 __hrtimer_run_queues+0x964/0xc10 kernel/time/hrtimer.c:1752
 hrtimer_interrupt+0x31b/0x800 kernel/time/hrtimer.c:1814
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1098 [inline]
 __sysvec_apic_timer_interrupt+0x14a/0x430 arch/x86/kernel/apic/apic.c:1115
 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1109
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:645
RIP: 0010:queue_delayed_work_on+0x9a/0x130 kernel/workqueue.c:1908
Code: ff 48 89 ee e8 c7 26 31 00 48 85 ed 75 42 e8 2d 2b 31 00 9c 5b 81 e3 00 02 00 00 31 ff 48 89 de e8 ab 26 31 00 48 85 db 75 71 <e8> 11 2b 31 00 44 89 e8 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f
RSP: 0018:ffffc90000117cb0 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff88801664d940 RSI: ffffffff8154ddd6 RDI: 0000000000000007
RBP: 0000000000000200 R08: 0000000000000007 R09: 0000000000000000
R10: 0000000000000000 R11: 000000000000004e R12: ffff88801465ad00
R13: 0000000000000001 R14: ffff888028eae000 R15: 00000000000003e8
 process_one_work+0xaa2/0x16f0 kernel/workqueue.c:2600
 worker_thread+0x687/0x1110 kernel/workqueue.c:2751
 kthread+0x33a/0x430 kernel/kthread.c:389
 ret_from_fork+0x2c/0x70 arch/x86/kernel/process.c:145
 ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:304
 </TASK>


Tested on:

commit:         ace0ab3a Revert "vlan: Fix VLAN 0 memory leak"
git tree:       https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git
console output: https://syzkaller.appspot.com/x/log.txt?x=160073c3a80000
kernel config:  https://syzkaller.appspot.com/x/.config?x=3e670757e16affb
dashboard link: https://syzkaller.appspot.com/bug?extid=c1cc0083f159b67cb192
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
patch:          https://syzkaller.appspot.com/x/patch.diff?x=13c69eefa80000

Re: [syzbot] [wireguard?] INFO: rcu detected stall in wg_ratelimiter_gc_entries (2)

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+c1cc0083f159b67cb192@syzkaller.appspotmail.com

Tested on:

commit:         ace0ab3a Revert "vlan: Fix VLAN 0 memory leak"
git tree:       https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git
console output: https://syzkaller.appspot.com/x/log.txt?x=1742e46ba80000
kernel config:  https://syzkaller.appspot.com/x/.config?x=3e670757e16affb
dashboard link: https://syzkaller.appspot.com/bug?extid=c1cc0083f159b67cb192
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
patch:          https://syzkaller.appspot.com/x/patch.diff?x=113c5e6ba80000

Note: testing is done by a robot and is best-effort only.

Re: [syzbot] [wireguard?] INFO: rcu detected stall in wg_ratelimiter_gc_entries (2)

[syzbot]

syzbot suspects this issue was fixed by commit:

commit da71714e359b64bd7aab3bd56ec53f307f058133
Author: Jamal Hadi Salim <jhs@mojatatu.com>
Date:   Tue Aug 22 10:12:31 2023 +0000

    net/sched: fix a qdisc modification with ambiguous command request

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=129c464e680000
start commit:   8a519a572598 net: veth: Page pool creation error handling ..
git tree:       net
kernel config:  https://syzkaller.appspot.com/x/.config?x=3e670757e16affb
dashboard link: https://syzkaller.appspot.com/bug?extid=c1cc0083f159b67cb192
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=129f8553a80000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=1205baada80000

If the result looks correct, please mark the issue as fixed by replying with:

#syz fix: net/sched: fix a qdisc modification with ambiguous command request

For information about bisection process see: https://goo.gl/tpsmEJ#bisection