[PATCH v1] timers: Feed soft lockup watchdog in expire_timers()

[PATCH v1] timers: Feed soft lockup watchdog in expire_timers()

[tanze]

expire_timers() iterates over all expired timers and invokes their
callbacks one by one in softirq context.  Each individual callback may
complete quickly, but when a very large number of timers expire at the
same tick, the aggregate time spent in this loop can exceed the soft
lockup watchdog threshold, causing a false positive:

  watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [syz.6.2424:14364]
  Call Trace:
   <IRQ>
   _raw_spin_unlock_irq+0x30/0x50
   __run_timers+0x75b/0xb20
   run_timer_softirq+0x22/0x40
   handle_softirqs+0x20b/0x7f0
   __irq_exit_rcu+0x184/0x1d0
   sysvec_apic_timer_interrupt+0x72/0x90
   </IRQ>

   <TASK>
   generic_exec_single+0xa0/0x530
   smp_call_function_single+0x328/0x410
   task_function_call+0xee/0x180
   perf_install_in_context+0x2df/0x580
   __do_sys_perf_event_open+0x1602/0x26e0
   do_syscall_64+0x73/0x2c0
   </TASK>

This is reproducible by issuing many perf_event_open() syscalls with
PERF_COUNT_SW_CPU_CLOCK and a very small sample_period, which creates
a large number of software timers that expire simultaneously.  The CPU
is making forward progress -- executing timer callbacks one after
another -- but the watchdog is never reset during the batch.  Found by
running syzkaller on a local QEMU instance.

Add touch_softlockup_watchdog() after each call_timer_fn() invocation
in expire_timers().  This resets the watchdog timestamp after every
successfully completed callback.  If a single callback itself is stuck,
the touch_softlockup_watchdog() after it will never be reached, so the
watchdog correctly fires for genuine lockups.

Signed-off-by: tanze <tanze@kylinos.cn>
---
 kernel/time/timer.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index 655a8c6cd84d..7f3369ad8a5c 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -42,6 +42,7 @@
 #include <linux/sched/debug.h>
 #include <linux/slab.h>
 #include <linux/compat.h>
+#include <linux/nmi.h>
 #include <linux/random.h>
 #include <linux/sysctl.h>
 
@@ -1792,11 +1793,13 @@ static void expire_timers(struct timer_base *base, struct hlist_head *head)
 		if (timer->flags & TIMER_IRQSAFE) {
 			raw_spin_unlock(&base->lock);
 			call_timer_fn(timer, fn, baseclk);
+			touch_softlockup_watchdog();
 			raw_spin_lock(&base->lock);
 			base->running_timer = NULL;
 		} else {
 			raw_spin_unlock_irq(&base->lock);
 			call_timer_fn(timer, fn, baseclk);
+			touch_softlockup_watchdog();
 			raw_spin_lock_irq(&base->lock);
 			base->running_timer = NULL;
 			timer_sync_wait_running(base);
-- 
2.43.0

Re: [PATCH v1] timers: Feed soft lockup watchdog in expire_timers()

[Thomas Gleixner]

On Fri, Jul 03 2026 at 16:36, tanze@kylinos.cn wrote:

Thanks for your patch.

First of all, you have to provide your real name for signing off on a
patch. Your nickname does not cut it. See Documentation/process.

> expire_timers() iterates over all expired timers and invokes their
> callbacks one by one in softirq context.  Each individual callback may
> complete quickly, but when a very large number of timers expire at the
> same tick, the aggregate time spent in this loop can exceed the soft
> lockup watchdog threshold, causing a false positive:

No. It's not a false positive.

> This is reproducible by issuing many perf_event_open() syscalls with
> PERF_COUNT_SW_CPU_CLOCK and a very small sample_period, which creates
> a large number of software timers that expire simultaneously.  The CPU
> is making forward progress -- executing timer callbacks one after
> another -- but the watchdog is never reset during the batch.  Found by
> running syzkaller on a local QEMU instance.

The watchdog validates that the CPU is making forward progress for
tasks. Spending 26 seconds in soft interrupt handling does therefore not
qualify.

> Add touch_softlockup_watchdog() after each call_timer_fn() invocation
> in expire_timers().  This resets the watchdog timestamp after every
> successfully completed callback.

Which papers over the underlying problem.

> @@ -1792,11 +1793,13 @@ static void expire_timers(struct timer_base *base, struct hlist_head *head)
>  		if (timer->flags & TIMER_IRQSAFE) {
>  			raw_spin_unlock(&base->lock);
>  			call_timer_fn(timer, fn, baseclk);
> +			touch_softlockup_watchdog();
>  			raw_spin_lock(&base->lock);
>  			base->running_timer = NULL;
>  		} else {
>  			raw_spin_unlock_irq(&base->lock);
>  			call_timer_fn(timer, fn, baseclk);
> +			touch_softlockup_watchdog();
>  			raw_spin_lock_irq(&base->lock);
>  			base->running_timer = NULL;
>  			timer_sync_wait_running(base);

PERF_COUNT_SW_CPU_CLOCK does not even use timer list timers, so how is
expire_timers() even involved?

Thanks,

        tglx