[PATCH] irq: add support for warning on long-running IRQ handlers

[PATCH] irq: add support for warning on long-running IRQ handlers

[Wladislav Wiebe]

Introduce a new option CONFIG_IRQ_LATENCY_WARN that enables warnings when
IRQ handlers take an unusually long time to execute.

When triggered, the warning includes the CPU, IRQ number, handler address,
name, and execution duration, for example:

  [CPU0] latency on IRQ[787:bad_irq_handler+0x1/0x34 [bad_irq]], took: 5 jiffies (~50 ms)

To keep runtime overhead minimal, this implementation uses a jiffies-based
timing mechanism. While coarse, it is sufficient to detect problematic IRQs.

A warning is triggered only when IRQs are disabled on one CPU long enough to
stall jiffies updates and exceed MAX_STALLED_JIFFIES.
Optionally, the reporting threshold can be adjusted by adding
extra jiffies via CONFIG_IRQ_LATENCY_WARN_THRESHOLD.

This approach avoids relying on high-resolution timers and aims for negligible
impact during normal operation.

Signed-off-by: Wladislav Wiebe <wladislav.wiebe@nokia.com>
---
 include/linux/interrupt.h | 25 +++++++++++++++++++++++++
 include/linux/tick.h      |  2 ++
 kernel/irq/handle.c       |  2 ++
 kernel/time/tick-sched.c  |  2 --
 lib/Kconfig.debug         | 29 +++++++++++++++++++++++++++++
 5 files changed, 58 insertions(+), 2 deletions(-)

diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
index 51b6484c0493..d3c5f920c4ea 100644
--- a/include/linux/interrupt.h
+++ b/include/linux/interrupt.h
@@ -15,6 +15,7 @@
 #include <linux/cpumask_types.h>
 #include <linux/workqueue.h>
 #include <linux/jump_label.h>
+#include <linux/tick.h>
 
 #include <linux/atomic.h>
 #include <asm/ptrace.h>
@@ -881,4 +882,28 @@ extern int arch_early_irq_init(void);
 
 #define __softirq_entry  __section(".softirqentry.text")
 
+#ifdef CONFIG_IRQ_LATENCY_WARN
+static inline void warn_on_irq_latency(struct irqaction *action, unsigned int irq,
+				       unsigned long jiffies_start)
+{
+	unsigned long delta = jiffies - jiffies_start;
+
+	/*
+	 * Warn about long IRQ handler latency only if jiffies are reliable.
+	 * The reporting condition hits only when there are at least two CPUs
+	 * with active ticks.
+	 * Jiffies updates are stalled on this CPU until MAX_STALLED_JIFFIES
+	 * reaches and a force update happens on another CPU with active ticks.
+	 */
+	if (unlikely(delta >= (MAX_STALLED_JIFFIES + CONFIG_IRQ_LATENCY_WARN_THRESHOLD))) {
+		pr_warn_ratelimited("[CPU%d] latency on IRQ[%u:%pS], took: %lu jiffies (~%u ms)\n",
+				    smp_processor_id(), irq, action->handler,
+				    delta, jiffies_to_msecs(delta));
+	}
+}
+#else
+static inline void warn_on_irq_latency(struct irqaction *action, unsigned int irq,
+				       unsigned long jiffies_start) { }
+#endif
+
 #endif
diff --git a/include/linux/tick.h b/include/linux/tick.h
index ac76ae9fa36d..543bd96b0653 100644
--- a/include/linux/tick.h
+++ b/include/linux/tick.h
@@ -14,6 +14,8 @@
 #include <linux/rcupdate.h>
 #include <linux/static_key.h>
 
+#define MAX_STALLED_JIFFIES 5
+
 #ifdef CONFIG_GENERIC_CLOCKEVENTS
 extern void __init tick_init(void);
 /* Should be core only, but ARM BL switcher requires it */
diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
index 9489f93b3db3..273aebd71d8d 100644
--- a/kernel/irq/handle.c
+++ b/kernel/irq/handle.c
@@ -145,6 +145,7 @@ irqreturn_t __handle_irq_event_percpu(struct irq_desc *desc)
 	record_irq_time(desc);
 
 	for_each_action_of_desc(desc, action) {
+		unsigned long __maybe_unused jiffies_start = jiffies;
 		irqreturn_t res;
 
 		/*
@@ -156,6 +157,7 @@ irqreturn_t __handle_irq_event_percpu(struct irq_desc *desc)
 
 		trace_irq_handler_entry(irq, action);
 		res = action->handler(irq, action->dev_id);
+		warn_on_irq_latency(action, irq, jiffies_start);
 		trace_irq_handler_exit(irq, action, res);
 
 		if (WARN_ONCE(!irqs_disabled(),"irq %u handler %pS enabled interrupts\n",
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index c527b421c865..5daee2bb3a18 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -201,8 +201,6 @@ static inline void tick_sched_flag_clear(struct tick_sched *ts,
 	ts->flags &= ~flag;
 }
 
-#define MAX_STALLED_JIFFIES 5
-
 static void tick_sched_do_timer(struct tick_sched *ts, ktime_t now)
 {
 	int tick_cpu, cpu = smp_processor_id();
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index ebe33181b6e6..88566adf4381 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -1856,6 +1856,35 @@ config LATENCYTOP
 	  Enable this option if you want to use the LatencyTOP tool
 	  to find out which userspace is blocking on what kernel operations.
 
+config IRQ_LATENCY_WARN
+	bool "Warn on IRQ latency"
+	depends on NR_CPUS >= 2
+	default n
+	help
+	  Enable this option to receive warnings when IRQ handlers take too long.
+
+	  To keep overhead very low, this implementation uses jiffies-based
+	  timing - which is coarse, but sufficient to detect problematic IRQs.
+
+	  The minimal possible threshold is hardcoded by MAX_STALLED_JIFFIES
+	  as the reporting condition hits only when there are at least two CPUs
+	  with active ticks. The reporting threshold can be extended by adding
+	  additional jiffies to CONFIG_IRQ_LATENCY_WARN_THRESHOLD.
+
+	  The warning includes the affected CPU, IRQ number, handler address,
+	  name, and the duration it took to execute.
+
+	  Say Y if you want to identify problematic IRQs in the system.
+
+config IRQ_LATENCY_WARN_THRESHOLD
+	int "IRQ latency warning threshold in jiffies"
+	depends on IRQ_LATENCY_WARN
+	range 0 10000
+	default 0
+	help
+	  Set the latency threshold (in jiffies) for the IRQ warning messages.
+	  Consider it will be added to MAX_STALLED_JIFFIES.
+
 config DEBUG_CGROUP_REF
 	bool "Disable inlining of cgroup css reference count functions"
 	depends on DEBUG_KERNEL
-- 
2.39.3.dirty

Re: [PATCH] irq: add support for warning on long-running IRQ handlers

[Peter Zijlstra]

On Mon, Jun 30, 2025 at 02:46:44PM +0200, Wladislav Wiebe wrote:
> Introduce a new option CONFIG_IRQ_LATENCY_WARN that enables warnings when
> IRQ handlers take an unusually long time to execute.
> 
> When triggered, the warning includes the CPU, IRQ number, handler address,
> name, and execution duration, for example:
> 
>   [CPU0] latency on IRQ[787:bad_irq_handler+0x1/0x34 [bad_irq]], took: 5 jiffies (~50 ms)
> 
> To keep runtime overhead minimal, this implementation uses a jiffies-based
> timing mechanism. While coarse, it is sufficient to detect problematic IRQs.

local_clock() was found to be excessively expensive?

Re: [PATCH] irq: add support for warning on long-running IRQ handlers

[Wladislav Wiebe]

On 30/06/2025 15:25, Peter Zijlstra wrote:
> CAUTION: This is an external email. Please be very careful when clicking links or opening attachments. See the URL nok.it/ext for additional information.
>
>
>
> On Mon, Jun 30, 2025 at 02:46:44PM +0200, Wladislav Wiebe wrote:
>> Introduce a new option CONFIG_IRQ_LATENCY_WARN that enables warnings when
>> IRQ handlers take an unusually long time to execute.
>>
>> When triggered, the warning includes the CPU, IRQ number, handler address,
>> name, and execution duration, for example:
>>
>>   [CPU0] latency on IRQ[787:bad_irq_handler+0x1/0x34 [bad_irq]], took: 5 jiffies (~50 ms)
>>
>> To keep runtime overhead minimal, this implementation uses a jiffies-based
>> timing mechanism. While coarse, it is sufficient to detect problematic IRQs.
> local_clock() was found to be excessively expensive?

Perhaps not excessively expensive, but jiffies is the lowest-overhead option here, isn't it?

Re: [PATCH] irq: add support for warning on long-running IRQ handlers

[Thomas Gleixner]

On Mon, Jun 30 2025 at 14:46, Wladislav Wiebe wrote:

The subsystem prefix is 'genirq:' See

https://www.kernel.org/doc/html/latest/process/maintainer-tip.html#patch-subject

> Introduce a new option CONFIG_IRQ_LATENCY_WARN that enables warnings when
> IRQ handlers take an unusually long time to execute.
>
> When triggered, the warning includes the CPU, IRQ number, handler address,
> name, and execution duration, for example:
>
>   [CPU0] latency on IRQ[787:bad_irq_handler+0x1/0x34 [bad_irq]], took: 5 jiffies (~50 ms)
>
> To keep runtime overhead minimal, this implementation uses a jiffies-based
> timing mechanism. While coarse, it is sufficient to detect problematic IRQs.

Define sufficient. That really depends on your use case. For a real-time
system a hard interrupt handler running longer than a few microseconds
can be problematic.

So instead of adding some single purpose mechanism, can we please add
something flexible which can be used for a wide range of scenarios.

> +#ifdef CONFIG_IRQ_LATENCY_WARN
> +static inline void warn_on_irq_latency(struct irqaction *action, unsigned int irq,
> +				       unsigned long jiffies_start)

latency is the wrong term here. This is about the runtime, duration of
the handler.

> +{
> +	unsigned long delta = jiffies - jiffies_start;
> +
> +	/*
> +	 * Warn about long IRQ handler latency only if jiffies are reliable.

What means jiffies are reliable?

> +	 * The reporting condition hits only when there are at least two CPUs
> +	 * with active ticks.
> +	 * Jiffies updates are stalled on this CPU until MAX_STALLED_JIFFIES
> +	 * reaches and a force update happens on another CPU with active ticks.
> +	 */
> +	if (unlikely(delta >= (MAX_STALLED_JIFFIES + CONFIG_IRQ_LATENCY_WARN_THRESHOLD))) {
> +		pr_warn_ratelimited("[CPU%d] latency on IRQ[%u:%pS], took: %lu jiffies (~%u ms)\n",
> +				    smp_processor_id(), irq, action->handler,
> +				    delta, jiffies_to_msecs(delta));
> +	}
> +}
> +#else
> +static inline void warn_on_irq_latency(struct irqaction *action, unsigned int irq,
> +				       unsigned long jiffies_start) { }
> +#endif

I'm absolutely not fond of this #ifdeffery and yet more Kconfig
knobs. Something like this should just work:

DEFINE_STATIC_KEY_FALSE(handler_duration_check);

        if (static_branch_unlikely(&handler_duration_check))
		ts_start = local_clock();
	res = action->handler(irq, action->dev_id);
        if (static_branch_unlikely(&handler_duration_check))
		check_handler_duration(ts_start);

Then have a command-line option which allows the user to set a warning
threshold > 0 in microseconds. When the option is evaluated the
threshold is stored in a __ro_after_init variable and the static branch
is enabled.

Hmm?

Thanks,

        tglx

Re: [PATCH] irq: add support for warning on long-running IRQ handlers

[Wladislav Wiebe]

On 30/06/2025 17:42, Thomas Gleixner wrote:
> On Mon, Jun 30 2025 at 14:46, Wladislav Wiebe wrote:
>
> The subsystem prefix is 'genirq:' See
>
> https://www.kernel.org/doc/html/latest/process/maintainer-tip.html#patch-subject
>
>> Introduce a new option CONFIG_IRQ_LATENCY_WARN that enables warnings when
>> IRQ handlers take an unusually long time to execute.
>>
>> When triggered, the warning includes the CPU, IRQ number, handler address,
>> name, and execution duration, for example:
>>
>>   [CPU0] latency on IRQ[787:bad_irq_handler+0x1/0x34 [bad_irq]], took: 5 jiffies (~50 ms)
>>
>> To keep runtime overhead minimal, this implementation uses a jiffies-based
>> timing mechanism. While coarse, it is sufficient to detect problematic IRQs.
> Define sufficient. That really depends on your use case. For a real-time
> system a hard interrupt handler running longer than a few microseconds
> can be problematic.
>
> So instead of adding some single purpose mechanism, can we please add
> something flexible which can be used for a wide range of scenarios.

the initial goal was to cover regular non-RT cores, as on isolated/tickless cores
we should not have device interrupts.
However, I agree we could make this more flexible for a wider range of use cases.

>
>> +#ifdef CONFIG_IRQ_LATENCY_WARN
>> +static inline void warn_on_irq_latency(struct irqaction *action, unsigned int irq,
>> +                                    unsigned long jiffies_start)
> latency is the wrong term here. This is about the runtime, duration of
> the handler.
>
>> +{
>> +     unsigned long delta = jiffies - jiffies_start;
>> +
>> +     /*
>> +      * Warn about long IRQ handler latency only if jiffies are reliable.
> What means jiffies are reliable?

there shall be a another CPU online with active ticks enabled
which updates the jiffies. Means, on single or tickless cores, this approach will not work.

>
>> +      * The reporting condition hits only when there are at least two CPUs
>> +      * with active ticks.
>> +      * Jiffies updates are stalled on this CPU until MAX_STALLED_JIFFIES
>> +      * reaches and a force update happens on another CPU with active ticks.
>> +      */
>> +     if (unlikely(delta >= (MAX_STALLED_JIFFIES + CONFIG_IRQ_LATENCY_WARN_THRESHOLD))) {
>> +             pr_warn_ratelimited("[CPU%d] latency on IRQ[%u:%pS], took: %lu jiffies (~%u ms)\n",
>> +                                 smp_processor_id(), irq, action->handler,
>> +                                 delta, jiffies_to_msecs(delta));
>> +     }
>> +}
>> +#else
>> +static inline void warn_on_irq_latency(struct irqaction *action, unsigned int irq,
>> +                                    unsigned long jiffies_start) { }
>> +#endif
> I'm absolutely not fond of this #ifdeffery and yet more Kconfig
> knobs. Something like this should just work:
>
> DEFINE_STATIC_KEY_FALSE(handler_duration_check);
>
>         if (static_branch_unlikely(&handler_duration_check))
>                 ts_start = local_clock();
>         res = action->handler(irq, action->dev_id);
>         if (static_branch_unlikely(&handler_duration_check))
>                 check_handler_duration(ts_start);
>
> Then have a command-line option which allows the user to set a warning
> threshold > 0 in microseconds. When the option is evaluated the
> threshold is stored in a __ro_after_init variable and the static branch
> is enabled.

all right - that makes perfectly sense as well. I will refactor it and provide v2.
Thank you for the comments.

- W.W

Re: [PATCH] irq: add support for warning on long-running IRQ handlers

[Thomas Gleixner]

On Tue, Jul 01 2025 at 08:10, Wladislav Wiebe wrote:
> On 30/06/2025 17:42, Thomas Gleixner wrote:
>> Define sufficient. That really depends on your use case. For a real-time
>> system a hard interrupt handler running longer than a few microseconds
>> can be problematic.
>>
>> So instead of adding some single purpose mechanism, can we please add
>> something flexible which can be used for a wide range of scenarios.
>
> the initial goal was to cover regular non-RT cores, as on isolated/tickless cores
> we should not have device interrupts.

Who is 'we'? If you refer to your use case that might be correct, but
you cannot make assumptions about the rest of the world. Real-Time
systems are as divers in setup and configuration as anything else.

Thanks,

        tglx

Re: [PATCH] irq: add support for warning on long-running IRQ handlers

[Peter Zijlstra]

On Mon, Jun 30, 2025 at 03:59:17PM +0200, Wladislav Wiebe wrote:
> 
> On 30/06/2025 15:25, Peter Zijlstra wrote:
> > CAUTION: This is an external email. Please be very careful when clicking links or opening attachments. See the URL nok.it/ext for additional information.
> >
> >
> >
> > On Mon, Jun 30, 2025 at 02:46:44PM +0200, Wladislav Wiebe wrote:
> >> Introduce a new option CONFIG_IRQ_LATENCY_WARN that enables warnings when
> >> IRQ handlers take an unusually long time to execute.
> >>
> >> When triggered, the warning includes the CPU, IRQ number, handler address,
> >> name, and execution duration, for example:
> >>
> >>   [CPU0] latency on IRQ[787:bad_irq_handler+0x1/0x34 [bad_irq]], took: 5 jiffies (~50 ms)
> >>
> >> To keep runtime overhead minimal, this implementation uses a jiffies-based
> >> timing mechanism. While coarse, it is sufficient to detect problematic IRQs.
> > local_clock() was found to be excessively expensive?
> 
> Perhaps not excessively expensive, but jiffies is the lowest-overhead option here, isn't it?

Yeah, but since it varies in length and even the shortest (1ms) might be
too long for some, it is of very limited use.

Re: [PATCH] irq: add support for warning on long-running IRQ handlers

[Wladislav Wiebe]

On 02/07/2025 11:06, Peter Zijlstra wrote:
> On Mon, Jun 30, 2025 at 03:59:17PM +0200, Wladislav Wiebe wrote:
>> On 30/06/2025 15:25, Peter Zijlstra wrote:
>>> CAUTION: This is an external email. Please be very careful when clicking links or opening attachments. See the URL nok.it/ext for additional information.
>>>
>>>
>>>
>>> On Mon, Jun 30, 2025 at 02:46:44PM +0200, Wladislav Wiebe wrote:
>>>> Introduce a new option CONFIG_IRQ_LATENCY_WARN that enables warnings when
>>>> IRQ handlers take an unusually long time to execute.
>>>>
>>>> When triggered, the warning includes the CPU, IRQ number, handler address,
>>>> name, and execution duration, for example:
>>>>
>>>>   [CPU0] latency on IRQ[787:bad_irq_handler+0x1/0x34 [bad_irq]], took: 5 jiffies (~50 ms)
>>>>
>>>> To keep runtime overhead minimal, this implementation uses a jiffies-based
>>>> timing mechanism. While coarse, it is sufficient to detect problematic IRQs.
>>> local_clock() was found to be excessively expensive?
>> Perhaps not excessively expensive, but jiffies is the lowest-overhead option here, isn't it?
> Yeah, but since it varies in length and even the shortest (1ms) might be
> too long for some, it is of very limited use.

I plan to refactor it to cover it with local_clock() and to avoid using Kconfig knobs as based on Thomas Gleixner comments. Thanks for the feedback. - W.W.

Re: [PATCH] irq: add support for warning on long-running IRQ handlers

[Wladislav Wiebe]

On 01/07/2025 09:37, Thomas Gleixner wrote:
>
> On Tue, Jul 01 2025 at 08:10, Wladislav Wiebe wrote:
>> On 30/06/2025 17:42, Thomas Gleixner wrote:
>>> Define sufficient. That really depends on your use case. For a real-time
>>> system a hard interrupt handler running longer than a few microseconds
>>> can be problematic.
>>>
>>> So instead of adding some single purpose mechanism, can we please add
>>> something flexible which can be used for a wide range of scenarios.
>> the initial goal was to cover regular non-RT cores, as on isolated/tickless cores
>> we should not have device interrupts.
> Who is 'we'? If you refer to your use case that might be correct, but
> you cannot make assumptions about the rest of the world. Real-Time
> systems are as divers in setup and configuration as anything else.
>
> Thanks,
>
>         tglx

all right, I've updated to v2: https://lore.kernel.org/lkml/20250714084209.918-1-wladislav.wiebe@nokia.com/ Thank you for all comments! - W.W.