* [PATCH Dovetail 6.12 6.1 5.15 5.10] irq_pipeline: Fix system freeze triggered by edge IRQs
@ 2025-09-17 7:14 Florian Bezdeka
2025-09-17 8:09 ` Philippe Gerum
0 siblings, 1 reply; 2+ messages in thread
From: Florian Bezdeka @ 2025-09-17 7:14 UTC (permalink / raw)
To: xenomai; +Cc: Jan Kiszka, Philippe Gerum, Florian Bezdeka
When IRQ pipelining is enabled, it could happen that handle_edge_irq()
is interrupted while syncing the IRQ log - for the very same IRQ.
The pipeline entry code will set the IRQS_PENDING flag and bail out
forcing the sync code (lower part in handle_edge_irq()) to loop to
infinity as the (periodic) IRQ might fire again after calling
unmask_irq() and reaching the hard IRQ enablement in handle_irq_event().
There is a "race window" involved. To trigger looping to infinity the
IRQ(s) must arrive while processing the IRQ log for the same IRQ in
handle_irq_event().
Call graph, delivered by a modified NMI handler and NMI injected by
hyper-v:
[ 5.929380] Call Trace:
[ 5.929381] <NMI>
[ 5.929383] default_do_nmi+0xd9/0x110
[ 5.929387] exc_nmi+0xde/0x110
[ 5.929388] end_repeat_nmi+0xf/0x53
[ 5.929390] RIP: 0010:mask_ioapic_irq+0x86/0xf0
[ 5.929392] Code:
[ 5.929392] RSP: 0000:ffffa1c680003e48 EFLAGS: 00010086
[ 5.929393] RAX: ffffffffff5fc000 RBX: 0000000000000086 RCX: ffff8e4f01049340
[ 5.929393] RDX: 0000000000203000 RSI: 0000000000010030 RDI: 0000000000000014
[ 5.929394] RBP: ffff8e4f01049320 R08: 0000000000000012 R09: 0000000000000001
[ 5.929394] R10: 0000000000000000 R11: 770e017136e49082 R12: ffffffffac506000
[ 5.929395] R13: ffff8e4f3ec49840 R14: 0000000000000000 R15: 0000000000000000
[ 5.929402] </NMI>
[ 5.929402] <IRQ>
[ 5.929402] handle_edge_irq+0x300/0x320
[ 5.929405] generic_pipeline_irq_desc+0xc2/0x220
[ 5.929408] arch_handle_irq+0x5d/0x180
[ 5.929410] arch_pipeline_entry+0x38/0xf0
[ 5.929413] asm_common_interrupt+0x22/0x40
[ 5.929421] handle_irq_event+0xbe/0xe0
[ 5.929424] handle_edge_irq+0x1ec/0x320
[ 5.929425] arch_do_IRQ_pipelined+0xa5/0x560
[ 5.929427] </IRQ>
[ 5.929427] <TASK>
[ 5.929428] sync_current_irq_stage+0x191/0x240
[ 5.929430] __inband_irq_enable+0x48/0x60
[ 5.929431] setup_IO_APIC+0x36b/0x860
[ 5.929434] ? ioapic_read_entry+0x1a/0x80
[ 5.929436] ? clear_IO_APIC_pin+0x1d/0x240
[ 5.929437] ? clear_IO_APIC_pin+0x190/0x240
[ 5.929440] apic_intr_mode_init+0x6f/0xf0
[ 5.929441] x86_late_time_init+0x20/0x40
[ 5.929442] start_kernel+0x743/0x890
[ 5.929445] x86_64_start_reservations+0x14/0x30
[ 5.929446] x86_64_start_kernel+0xc1/0xd0
[ 5.929447] common_startup_64+0x13e/0x148
[ 5.929452] </TASK>
Enabling some debug options like CONFIG_PROVE_LOCKING widens this
window as execution time increases.
If we detect that the very same IRQ line is currently processed while we
receive another hard irq, the current event is logged into the IRQ log.
The event gets replayed within the same sync run in
sync_current_irq_stage().
Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
---
kernel/irq/chip.c | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
index 4e81a93fc7f5..ce9b09d6d07b 100644
--- a/kernel/irq/chip.c
+++ b/kernel/irq/chip.c
@@ -676,6 +676,28 @@ static inline bool should_feed_pipeline(struct irq_desc *desc, int state)
return false;
}
+#ifdef CONFIG_IRQ_PIPELINE
+static bool irq_active_on_this_cpu(struct irq_desc *desc)
+{
+ struct irq_data *data = &desc->irq_data;
+ const struct cpumask *aff;
+
+ /* desc marked as in progress, no other CPUs on !SMP */
+ if (!IS_ENABLED(CONFIG_SMP))
+ return true;
+
+ /* The below works only for single target interrupts */
+ if (!irqd_is_single_target(data) || desc->handle_irq != handle_edge_irq)
+ return false;
+
+ aff = irq_data_get_effective_affinity_mask(data);
+ if (cpumask_first(aff) != smp_processor_id())
+ return false;
+
+ return true;
+}
+#endif
+
static bool irq_may_run(struct irq_desc *desc)
{
unsigned int mask = IRQD_IRQ_INPROGRESS | IRQD_WAKEUP_ARMED;
@@ -706,6 +728,8 @@ static bool irq_may_run(struct irq_desc *desc)
WARN_ON_ONCE(irq_pipeline_debug() && !in_pipeline());
if (irqd_is_wakeup_armed(&desc->irq_data))
return true;
+ if (irq_active_on_this_cpu(desc))
+ return true;
} else if (irq_pm_check_wakeup(desc))
return false;
--
2.39.5
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [PATCH Dovetail 6.12 6.1 5.15 5.10] irq_pipeline: Fix system freeze triggered by edge IRQs
2025-09-17 7:14 [PATCH Dovetail 6.12 6.1 5.15 5.10] irq_pipeline: Fix system freeze triggered by edge IRQs Florian Bezdeka
@ 2025-09-17 8:09 ` Philippe Gerum
0 siblings, 0 replies; 2+ messages in thread
From: Philippe Gerum @ 2025-09-17 8:09 UTC (permalink / raw)
To: Florian Bezdeka; +Cc: xenomai, Jan Kiszka
Florian Bezdeka <florian.bezdeka@siemens.com> writes:
> When IRQ pipelining is enabled, it could happen that handle_edge_irq()
> is interrupted while syncing the IRQ log - for the very same IRQ.
>
> The pipeline entry code will set the IRQS_PENDING flag and bail out
> forcing the sync code (lower part in handle_edge_irq()) to loop to
> infinity as the (periodic) IRQ might fire again after calling
> unmask_irq() and reaching the hard IRQ enablement in handle_irq_event().
>
> There is a "race window" involved. To trigger looping to infinity the
> IRQ(s) must arrive while processing the IRQ log for the same IRQ in
> handle_irq_event().
>
> Call graph, delivered by a modified NMI handler and NMI injected by
> hyper-v:
> [ 5.929380] Call Trace:
> [ 5.929381] <NMI>
> [ 5.929383] default_do_nmi+0xd9/0x110
> [ 5.929387] exc_nmi+0xde/0x110
> [ 5.929388] end_repeat_nmi+0xf/0x53
> [ 5.929390] RIP: 0010:mask_ioapic_irq+0x86/0xf0
> [ 5.929392] Code:
> [ 5.929392] RSP: 0000:ffffa1c680003e48 EFLAGS: 00010086
> [ 5.929393] RAX: ffffffffff5fc000 RBX: 0000000000000086 RCX: ffff8e4f01049340
> [ 5.929393] RDX: 0000000000203000 RSI: 0000000000010030 RDI: 0000000000000014
> [ 5.929394] RBP: ffff8e4f01049320 R08: 0000000000000012 R09: 0000000000000001
> [ 5.929394] R10: 0000000000000000 R11: 770e017136e49082 R12: ffffffffac506000
> [ 5.929395] R13: ffff8e4f3ec49840 R14: 0000000000000000 R15: 0000000000000000
> [ 5.929402] </NMI>
> [ 5.929402] <IRQ>
> [ 5.929402] handle_edge_irq+0x300/0x320
> [ 5.929405] generic_pipeline_irq_desc+0xc2/0x220
> [ 5.929408] arch_handle_irq+0x5d/0x180
> [ 5.929410] arch_pipeline_entry+0x38/0xf0
> [ 5.929413] asm_common_interrupt+0x22/0x40
> [ 5.929421] handle_irq_event+0xbe/0xe0
> [ 5.929424] handle_edge_irq+0x1ec/0x320
> [ 5.929425] arch_do_IRQ_pipelined+0xa5/0x560
> [ 5.929427] </IRQ>
> [ 5.929427] <TASK>
> [ 5.929428] sync_current_irq_stage+0x191/0x240
> [ 5.929430] __inband_irq_enable+0x48/0x60
> [ 5.929431] setup_IO_APIC+0x36b/0x860
> [ 5.929434] ? ioapic_read_entry+0x1a/0x80
> [ 5.929436] ? clear_IO_APIC_pin+0x1d/0x240
> [ 5.929437] ? clear_IO_APIC_pin+0x190/0x240
> [ 5.929440] apic_intr_mode_init+0x6f/0xf0
> [ 5.929441] x86_late_time_init+0x20/0x40
> [ 5.929442] start_kernel+0x743/0x890
> [ 5.929445] x86_64_start_reservations+0x14/0x30
> [ 5.929446] x86_64_start_kernel+0xc1/0xd0
> [ 5.929447] common_startup_64+0x13e/0x148
> [ 5.929452] </TASK>
>
> Enabling some debug options like CONFIG_PROVE_LOCKING widens this
> window as execution time increases.
>
> If we detect that the very same IRQ line is currently processed while we
> receive another hard irq, the current event is logged into the IRQ log.
> The event gets replayed within the same sync run in
> sync_current_irq_stage().
>
> Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
> ---
> kernel/irq/chip.c | 24 ++++++++++++++++++++++++
> 1 file changed, 24 insertions(+)
>
> diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
> index 4e81a93fc7f5..ce9b09d6d07b 100644
> --- a/kernel/irq/chip.c
> +++ b/kernel/irq/chip.c
> @@ -676,6 +676,28 @@ static inline bool should_feed_pipeline(struct irq_desc *desc, int state)
> return false;
> }
>
> +#ifdef CONFIG_IRQ_PIPELINE
> +static bool irq_active_on_this_cpu(struct irq_desc *desc)
> +{
> + struct irq_data *data = &desc->irq_data;
> + const struct cpumask *aff;
> +
> + /* desc marked as in progress, no other CPUs on !SMP */
> + if (!IS_ENABLED(CONFIG_SMP))
> + return true;
> +
> + /* The below works only for single target interrupts */
> + if (!irqd_is_single_target(data) || desc->handle_irq != handle_edge_irq)
> + return false;
> +
> + aff = irq_data_get_effective_affinity_mask(data);
> + if (cpumask_first(aff) != smp_processor_id())
> + return false;
> +
> + return true;
> +}
> +#endif
> +
> static bool irq_may_run(struct irq_desc *desc)
> {
> unsigned int mask = IRQD_IRQ_INPROGRESS | IRQD_WAKEUP_ARMED;
> @@ -706,6 +728,8 @@ static bool irq_may_run(struct irq_desc *desc)
> WARN_ON_ONCE(irq_pipeline_debug() && !in_pipeline());
> if (irqd_is_wakeup_armed(&desc->irq_data))
> return true;
> + if (irq_active_on_this_cpu(desc))
> + return true;
> } else if (irq_pm_check_wakeup(desc))
> return false;
Merged into v6.1.y-cip, v6.12.y.
--
Philippe.
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2025-09-17 8:10 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-09-17 7:14 [PATCH Dovetail 6.12 6.1 5.15 5.10] irq_pipeline: Fix system freeze triggered by edge IRQs Florian Bezdeka
2025-09-17 8:09 ` Philippe Gerum
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox