[syzbot] [rcu?] WARNING in srcu_invoke_callbacks (2)

[syzbot] [rcu?] WARNING in srcu_invoke_callbacks (2)

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    a64dcfb451e2 Linux 6.14-rc2
git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
console output: https://syzkaller.appspot.com/x/log.txt?x=12398f18580000
kernel config:  https://syzkaller.appspot.com/x/.config?x=c9c47badcd079906
dashboard link: https://syzkaller.appspot.com/bug?extid=ae5b16688c0c675b1a1f
compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
userspace arch: arm64

Unfortunately, I don't have any reproducer for this issue yet.

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/c0a862fcec77/disk-a64dcfb4.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/f03793fc001b/vmlinux-a64dcfb4.xz
kernel image: https://storage.googleapis.com/syzbot-assets/ae71c33eae14/Image-a64dcfb4.gz.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+ae5b16688c0c675b1a1f@syzkaller.appspotmail.com

------------[ cut here ]------------
WARNING: CPU: 0 PID: 6430 at kernel/rcu/srcutree.c:1798 srcu_invoke_callbacks+0x368/0x3d8
Modules linked in:
CPU: 0 UID: 0 PID: 6430 Comm: kworker/0:3 Not tainted 6.14.0-rc2-syzkaller-ga64dcfb451e2 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 12/27/2024
Workqueue: rcu_gp srcu_invoke_callbacks
pstate: 80400005 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : srcu_invoke_callbacks+0x368/0x3d8
lr : srcu_invoke_callbacks+0x1ec/0x3d8 kernel/rcu/srcutree.c:1791
sp : ffff8000a3497960
x29: ffff8000a3497a20 x28: ffff700014692f34 x27: ffff80009c6f6ce8
x26: 0000000000000000 x25: ffff80008b87d7e0 x24: dfff800000000000
x23: fffffdffbf6e3d08 x22: fffffdffbf6e3c80 x21: ffff8000a34979c0
x20: fffffdffbf6e3c40 x19: fffffdffbf6e3d60 x18: 1ffff0001472cf04
x17: ffff80008fbad000 x16: ffff8000804683c4 x15: 0000000000000001
x14: 1ffff000138ded9f x13: ffff8000a3498000 x12: 0000000000000003
x11: 0000000000000000 x10: 0000000000ff0100 x9 : 0000000000000000
x8 : 0000000000000001 x7 : ffff80008044b5b8 x6 : 0000000000000000
x5 : 0000000000000001 x4 : 0000000000000001 x3 : 0000000000000000
x2 : 0000000000000006 x1 : 0000000000000080 x0 : 0000000000000000
Call trace:
 srcu_invoke_callbacks+0x368/0x3d8 (P)
 process_one_work+0x810/0x1638 kernel/workqueue.c:3236
 process_scheduled_works kernel/workqueue.c:3317 [inline]
 worker_thread+0x97c/0xeec kernel/workqueue.c:3398
 kthread+0x65c/0x7b0 kernel/kthread.c:464
 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:862
irq event stamp: 206698
hardirqs last  enabled at (206697): [<ffff800080310c38>] __local_bh_enable_ip+0x224/0x44c kernel/softirq.c:394
hardirqs last disabled at (206698): [<ffff80008b7b3b54>] el1_dbg+0x24/0x80 arch/arm64/kernel/entry-common.c:488
softirqs last  enabled at (206696): [<ffff80008050b40c>] local_bh_enable+0xc/0x2c include/linux/bottom_half.h:32
softirqs last disabled at (206692): [<ffff80008050b3e0>] local_bh_disable+0xc/0x2c include/linux/bottom_half.h:19
---[ end trace 0000000000000000 ]---


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Re: [syzbot] [rcu?] WARNING in srcu_invoke_callbacks (2)

[Joel Fernandes]

On Tue, Feb 18, 2025 at 09:47:27AM -0800, syzbot wrote:
> Hello,
> 
> syzbot found the following issue on:
> 
> HEAD commit:    a64dcfb451e2 Linux 6.14-rc2
> git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
> console output: https://syzkaller.appspot.com/x/log.txt?x=12398f18580000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=c9c47badcd079906
> dashboard link: https://syzkaller.appspot.com/bug?extid=ae5b16688c0c675b1a1f
> compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> userspace arch: arm64
> 
> Unfortunately, I don't have any reproducer for this issue yet.
> 
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/c0a862fcec77/disk-a64dcfb4.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/f03793fc001b/vmlinux-a64dcfb4.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/ae71c33eae14/Image-a64dcfb4.gz.xz
> 
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+ae5b16688c0c675b1a1f@syzkaller.appspotmail.com
> 
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 6430 at kernel/rcu/srcutree.c:1798 srcu_invoke_callbacks+0x368/0x3d8

This is weird, happens to be WARN_ON_ONCE(ready_cbs.len); in
srcu_invoke_callbacks().

Perhaps, stack corruption or the SRCU cblist's ->seglen got corrupted?

thanks,

 - Joel

Re: [syzbot] [rcu?] WARNING in srcu_invoke_callbacks (2)

[Paul E. McKenney]

On Wed, Feb 19, 2025 at 08:25:38AM -0500, Joel Fernandes wrote:
> On Tue, Feb 18, 2025 at 09:47:27AM -0800, syzbot wrote:
> > Hello,
> > 
> > syzbot found the following issue on:
> > 
> > HEAD commit:    a64dcfb451e2 Linux 6.14-rc2
> > git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
> > console output: https://syzkaller.appspot.com/x/log.txt?x=12398f18580000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=c9c47badcd079906
> > dashboard link: https://syzkaller.appspot.com/bug?extid=ae5b16688c0c675b1a1f
> > compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> > userspace arch: arm64
> > 
> > Unfortunately, I don't have any reproducer for this issue yet.
> > 
> > Downloadable assets:
> > disk image: https://storage.googleapis.com/syzbot-assets/c0a862fcec77/disk-a64dcfb4.raw.xz
> > vmlinux: https://storage.googleapis.com/syzbot-assets/f03793fc001b/vmlinux-a64dcfb4.xz
> > kernel image: https://storage.googleapis.com/syzbot-assets/ae71c33eae14/Image-a64dcfb4.gz.xz
> > 
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: syzbot+ae5b16688c0c675b1a1f@syzkaller.appspotmail.com
> > 
> > ------------[ cut here ]------------
> > WARNING: CPU: 0 PID: 6430 at kernel/rcu/srcutree.c:1798 srcu_invoke_callbacks+0x368/0x3d8
> 
> This is weird, happens to be WARN_ON_ONCE(ready_cbs.len); in
> srcu_invoke_callbacks().

Thank you for tracking that down!

> Perhaps, stack corruption or the SRCU cblist's ->seglen got corrupted?

That has been the usual cause.  And a double call_srcu() is a common
cause of ->cblist corruption.  We have CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
set up for SRCU as well as for RCU, it appears, so perhaps retry with
that should this ever become reproducible?

							Thanx, Paul

Re: [syzbot] [rcu?] WARNING in srcu_invoke_callbacks (2)

[Joel Fernandes]

On 2/19/2025 8:58 AM, Paul E. McKenney wrote:
> On Wed, Feb 19, 2025 at 08:25:38AM -0500, Joel Fernandes wrote:
>> On Tue, Feb 18, 2025 at 09:47:27AM -0800, syzbot wrote:
>>> Hello,
>>>
>>> syzbot found the following issue on:
>>>
>>> HEAD commit:    a64dcfb451e2 Linux 6.14-rc2
>>> git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
>>> console output: https://syzkaller.appspot.com/x/log.txt?x=12398f18580000
>>> kernel config:  https://syzkaller.appspot.com/x/.config?x=c9c47badcd079906
>>> dashboard link: https://syzkaller.appspot.com/bug?extid=ae5b16688c0c675b1a1f
>>> compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
>>> userspace arch: arm64
>>>
>>> Unfortunately, I don't have any reproducer for this issue yet.
>>>
>>> Downloadable assets:
>>> disk image: https://storage.googleapis.com/syzbot-assets/c0a862fcec77/disk-a64dcfb4.raw.xz
>>> vmlinux: https://storage.googleapis.com/syzbot-assets/f03793fc001b/vmlinux-a64dcfb4.xz
>>> kernel image: https://storage.googleapis.com/syzbot-assets/ae71c33eae14/Image-a64dcfb4.gz.xz
>>>
>>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
>>> Reported-by: syzbot+ae5b16688c0c675b1a1f@syzkaller.appspotmail.com
>>>
>>> ------------[ cut here ]------------
>>> WARNING: CPU: 0 PID: 6430 at kernel/rcu/srcutree.c:1798 srcu_invoke_callbacks+0x368/0x3d8
>>
>> This is weird, happens to be WARN_ON_ONCE(ready_cbs.len); in
>> srcu_invoke_callbacks().
> 
> Thank you for tracking that down!

Sure!

>> Perhaps, stack corruption or the SRCU cblist's ->seglen got corrupted?
> 
> That has been the usual cause.  And a double call_srcu() is a common
> cause of ->cblist corruption.  We have CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
> set up for SRCU as well as for RCU, it appears, so perhaps retry with
> that should this ever become reproducible?
Makes sense, will keep an eye out for it!

thanks,

 - Joel

Re: [syzbot] [rcu?] WARNING in srcu_invoke_callbacks (2)

[Youngmin Nam]

[-- Attachment #1: Type: text/plain, Size: 2811 bytes --]

On Wed, Feb 19, 2025 at 10:09:06AM -0500, Joel Fernandes wrote:
> 
> 
> On 2/19/2025 8:58 AM, Paul E. McKenney wrote:
> > On Wed, Feb 19, 2025 at 08:25:38AM -0500, Joel Fernandes wrote:
> >> On Tue, Feb 18, 2025 at 09:47:27AM -0800, syzbot wrote:
> >>> Hello,
> >>>
> >>> syzbot found the following issue on:
> >>>
> >>> HEAD commit:    a64dcfb451e2 Linux 6.14-rc2
> >>> git tree:       git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
> >>> console output: https://syzkaller.appspot.com/x/log.txt?x=12398f18580000
> >>> kernel config:  https://syzkaller.appspot.com/x/.config?x=c9c47badcd079906
> >>> dashboard link: https://syzkaller.appspot.com/bug?extid=ae5b16688c0c675b1a1f
> >>> compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> >>> userspace arch: arm64
> >>>
> >>> Unfortunately, I don't have any reproducer for this issue yet.
> >>>
> >>> Downloadable assets:
> >>> disk image: https://storage.googleapis.com/syzbot-assets/c0a862fcec77/disk-a64dcfb4.raw.xz
> >>> vmlinux: https://storage.googleapis.com/syzbot-assets/f03793fc001b/vmlinux-a64dcfb4.xz
> >>> kernel image: https://storage.googleapis.com/syzbot-assets/ae71c33eae14/Image-a64dcfb4.gz.xz
> >>>
> >>> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> >>> Reported-by: syzbot+ae5b16688c0c675b1a1f@syzkaller.appspotmail.com
> >>>
> >>> ------------[ cut here ]------------
> >>> WARNING: CPU: 0 PID: 6430 at kernel/rcu/srcutree.c:1798 srcu_invoke_callbacks+0x368/0x3d8
> >>
> >> This is weird, happens to be WARN_ON_ONCE(ready_cbs.len); in
> >> srcu_invoke_callbacks().
> > 
> > Thank you for tracking that down!
> 
> Sure!
> 
> >> Perhaps, stack corruption or the SRCU cblist's ->seglen got corrupted?
> > 
> > That has been the usual cause.  And a double call_srcu() is a common
> > cause of ->cblist corruption.  We have CONFIG_DEBUG_OBJECTS_RCU_HEAD=y
> > set up for SRCU as well as for RCU, it appears, so perhaps retry with
> > that should this ever become reproducible?
> Makes sense, will keep an eye out for it!
> 
> thanks,
> 
>  - Joel
> 
Hello all,

syzbot reports that this issue is still occurring on recent kernels,
for example 6.16 (see https://syzkaller.appspot.com/bug?extid=ae5b16688c0c675b1a1f).

And the "CONFIG_DEBUG_OBJECTS_RCU_HEAD=y" has been set as well.
(see https://syzkaller.appspot.com/text?tag=KernelConfig&x=dcbf08d90aa89bca)

But there were no logs related with double call_srcu().
(see https://syzkaller.appspot.com/text?tag=CrashLog&x=13e40434580000)

Unfortunately we don’t yet have a simple reproducer to reliably trigger the warning.

Could you please advise whether a patch for this has already landed in mainline,
or suggest any additional methods we could use to further validate and test it?

Thanks.
Youngmin

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]