Re: [syzbot] [kernel?] INFO: rcu detected stall in __hrtimer_run_queues (3)

[Thomas Gleixner <tglx@kernel.org>]

On Fri, Sep 04 2026 at 15:51, Aleksandr Nogikh wrote:
> On Fri, Sep 4, 2026 at 7:00 AM 'Thomas Gleixner' via syzkaller-bugs
> <syzkaller-bugs@googlegroups.com> wrote:
>> On Mon, Aug 31 2026 at 08:55, syzbot wrote:
>>
>> CC+: ARM64 folks. It's their magic git tree ....
>>
>> Maybe some day syzbot people get their act together and actually figure
>> out who should be CC'ed. Hope dies last...
>
> In this particular case, the fact that the reproducer ran against the
> arm64 tree does not mean the bug is arm64-specific. As shown on the
> dashboard (https://syzkaller.appspot.com/bug?extid=6d991f16f4bcc8eeea24),
> most crashes are coming from the x86_64 instances.

Honestly I have no time to click through dashboards to figure this
out. If you already have that information then please include it. With
the gazillion of trees which have fresh patches in them it's not
necessarily a wrong assumption that it might be tree specific. It's the
same problem as bug reports against some random out of date kernel
versions.

> Judging by what reproducers do, it seems to be related to
> net/sched/act_gate.c's gate_timer_func(), for which a patch has
> recently been posted by Edward Adam Davis (Cc'd):
> https://lore.kernel.org/all/tencent_19FD76F366C43D4737EE9682371A4BF87409@qq.com/

They look similar, but the reproducers are completely different and the
act_gate backtrace shows the actual problematic code. It might be the
same bug. Edward should be able to tell.

> Syzbot attributes reports to maintainers based on the crash stack
> trace rather than the git tree URL. Because CPU lockups and stalls in
> a looping timer callback were caught inside kernel/time/hrtimer.c,
> get_maintainer.pl pointed it at the timer subsystem.

I get CC'ed on boatloads of reports which happen to show a back trace
somewhere in the code I maintain. In the vast majority of cases the
[hr]timer, interrupt ... core code is just the messenger. The hrtimer
interrupt does not magically rearm timers rapidly or leaves NULL
pointers around.

Don't get me wrong. syszbot is extremly useful, but the amount of work
it creates for those on the receiving end is not really managable and
it's not surprising to me that a lot of reports just fall through the
cracks in general. I try to avoid that, but at times I have to just deal
with more important problems and by the time I come back to it there is
a pile of new reports to wade through.

So having better information in the report would make this definitely
more managable for me and probably for others too:

 - a quick summary that this happens on upstream too and is not
   restricted to the tree which is in the report

 - a TLDR summary of the reproducer operations so it can be seen quickly
   which subsystems might be involved without decoding the reproducer
   file manually.

Also for these kind of issues the robot might grow some rules how to
provide better debug information. Let's look at this particular report:

  __run_hrtimer kernel/time/hrtimer.c:2028 [inline]
  __hrtimer_run_queues+0x22c/0xbe0 kernel/time/hrtimer.c:2096
  hrtimer_run_softirq+0x15c/0x21c kernel/time/hrtimer.c:2113
  handle_softirqs+0x2ec/0xd98 kernel/softirq.c:622
  __do_softirq+0x14/0x20 kernel/softirq.c:656
  ____do_softirq+0x14/0x20 arch/arm64/kernel/irq.c:78
  call_on_irq_stack+0x30/0x48 arch/arm64/kernel/entry.S:885
  do_softirq_own_stack+0x20/0x2c arch/arm64/kernel/irq.c:83
  invoke_softirq kernel/softirq.c:503 [inline]

So it deduces that the lockup is in __hrtimer_run_queues(). So far so
good.

So if I'd had have the time to download and run the reproducer I would
have enabled the hrtimer trace points with a relatively small trace
buffer, added 'ftrace_dump_on_oops' to the command line and let the
kernel dump it when it crashes. In case of a self rearming timer which
keeps it in that loop, that surely would dump a series of
start/expire/start/ events with the same timer and the expire_entry
tracepoint would have the offending function in it.

Just a thought, but maybe there is something which can be done.

Let me stare at __hrtimer_run_queues() whether we can add some
lightweight mechanism to prevent such issues.

Thanks,

        tglx