* [PATCH Dovetail 6.16 v4] irq_pipeline: Fix system freeze triggered by edge IRQs
@ 2025-09-15 7:18 Florian Bezdeka
2025-09-17 8:09 ` Philippe Gerum
0 siblings, 1 reply; 2+ messages in thread
From: Florian Bezdeka @ 2025-09-15 7:18 UTC (permalink / raw)
To: xenomai; +Cc: Florian Bezdeka, Philippe Gerum, Jan Kiszka
When IRQ pipelining is enabled, it could happen that handle_edge_irq()
is interrupted while syncing the IRQ log - for the very same IRQ.
The pipeline entry code will set the IRQS_PENDING flag and bail out
forcing the sync code (lower part in handle_edge_irq()) to loop to
infinity as the (periodic) IRQ might fire again after calling
unmask_irq() and reaching the hard IRQ enablement in handle_irq_event().
There is a "race window" involved. To trigger looping to infinity the
IRQ(s) must arrive while processing the IRQ log for the same IRQ in
handle_irq_event().
Call graph, delivered by a modified NMI handler and NMI injected by
hyper-v:
[ 5.929380] Call Trace:
[ 5.929381] <NMI>
[ 5.929383] default_do_nmi+0xd9/0x110
[ 5.929387] exc_nmi+0xde/0x110
[ 5.929388] end_repeat_nmi+0xf/0x53
[ 5.929390] RIP: 0010:mask_ioapic_irq+0x86/0xf0
[ 5.929392] Code:
[ 5.929392] RSP: 0000:ffffa1c680003e48 EFLAGS: 00010086
[ 5.929393] RAX: ffffffffff5fc000 RBX: 0000000000000086 RCX: ffff8e4f01049340
[ 5.929393] RDX: 0000000000203000 RSI: 0000000000010030 RDI: 0000000000000014
[ 5.929394] RBP: ffff8e4f01049320 R08: 0000000000000012 R09: 0000000000000001
[ 5.929394] R10: 0000000000000000 R11: 770e017136e49082 R12: ffffffffac506000
[ 5.929395] R13: ffff8e4f3ec49840 R14: 0000000000000000 R15: 0000000000000000
[ 5.929402] </NMI>
[ 5.929402] <IRQ>
[ 5.929402] handle_edge_irq+0x300/0x320
[ 5.929405] generic_pipeline_irq_desc+0xc2/0x220
[ 5.929408] arch_handle_irq+0x5d/0x180
[ 5.929410] arch_pipeline_entry+0x38/0xf0
[ 5.929413] asm_common_interrupt+0x22/0x40
[ 5.929421] handle_irq_event+0xbe/0xe0
[ 5.929424] handle_edge_irq+0x1ec/0x320
[ 5.929425] arch_do_IRQ_pipelined+0xa5/0x560
[ 5.929427] </IRQ>
[ 5.929427] <TASK>
[ 5.929428] sync_current_irq_stage+0x191/0x240
[ 5.929430] __inband_irq_enable+0x48/0x60
[ 5.929431] setup_IO_APIC+0x36b/0x860
[ 5.929434] ? ioapic_read_entry+0x1a/0x80
[ 5.929436] ? clear_IO_APIC_pin+0x1d/0x240
[ 5.929437] ? clear_IO_APIC_pin+0x190/0x240
[ 5.929440] apic_intr_mode_init+0x6f/0xf0
[ 5.929441] x86_late_time_init+0x20/0x40
[ 5.929442] start_kernel+0x743/0x890
[ 5.929445] x86_64_start_reservations+0x14/0x30
[ 5.929446] x86_64_start_kernel+0xc1/0xd0
[ 5.929447] common_startup_64+0x13e/0x148
[ 5.929452] </TASK>
Enabling some debug options like CONFIG_PROVE_LOCKING widens this
window as execution time increases.
If we detect that the very same IRQ line is currently processed while we
receive another hard irq, the current event is logged into the IRQ log.
The event gets replayed within the same sync run in
sync_current_irq_stage().
Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
---
Hi all,
the problem could be observed and debuged quite well on hyper-v with
lockdep enabled and a slightly modified NMI handler to print out the
current call stack.
This seems to be a long standing issue for Dovetail. We had several
reports in the past that debug options (like lockdep) on hypervisors
(like VirtualBox) ended up in a boot failure.
Once we agree that the patch is correct I can send out the backport for
Dovetail 6.12 down to 5.10.
Best regards,
Florian
Cc: Philippe Gerum <rpm@xenomai.org>
Cc: Jan Kiszka <jan.kiszka@siemens.com>
---
Changes in v4:
- Remove irqd_irq_inprogress() from irq_active_on_this_cpu().
That check was always true as it is already detected by the caller.
- Add !SMP check to irq_active_on_this_cpu(), Review comment from
Philippe
- Link to v3: https://lore.kernel.org/r/20250820-wip-flo-fix-edge-irq-handling-v3-1-8cc08939e6ca@siemens.com
Changes in v3:
- Introduce irq_active_on_this_cpu()
- Link to v2: https://lore.kernel.org/r/20250820-wip-flo-fix-edge-irq-handling-v2-1-1a958d17eace@siemens.com
Changes in v2:
- v1 crashed qemu, similar to
https://lore.kernel.org/all/20250701163558.2588435-1-liangyan.peng@bytedance.com/
so v1 was analyzed further and reworked completely
- Link to v1: https://lore.kernel.org/r/20250820-wip-flo-fix-edge-irq-handling-v1-1-1a2c7805b372@siemens.com
---
kernel/irq/chip.c | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
index f5dfc378ccf2..12f520f6a197 100644
--- a/kernel/irq/chip.c
+++ b/kernel/irq/chip.c
@@ -639,6 +639,28 @@ static inline bool should_feed_pipeline(struct irq_desc *desc, int state)
return false;
}
+#ifdef CONFIG_IRQ_PIPELINE
+static bool irq_active_on_this_cpu(struct irq_desc *desc)
+{
+ struct irq_data *data = &desc->irq_data;
+ const struct cpumask *aff;
+
+ /* desc marked as in progress, no other CPUs on !SMP */
+ if (!IS_ENABLED(CONFIG_SMP))
+ return true;
+
+ /* The below works only for single target interrupts */
+ if (!irqd_is_single_target(data) || desc->handle_irq != handle_edge_irq)
+ return false;
+
+ aff = irq_data_get_effective_affinity_mask(data);
+ if (cpumask_first(aff) != smp_processor_id())
+ return false;
+
+ return true;
+}
+#endif
+
static bool irq_can_handle_pm(struct irq_desc *desc)
{
unsigned int mask = IRQD_IRQ_INPROGRESS | IRQD_WAKEUP_ARMED;
@@ -669,6 +691,8 @@ static bool irq_can_handle_pm(struct irq_desc *desc)
WARN_ON_ONCE(irq_pipeline_debug() && !in_pipeline());
if (irqd_is_wakeup_armed(&desc->irq_data))
return true;
+ if (irq_active_on_this_cpu(desc))
+ return true;
} else if (irq_pm_check_wakeup(desc))
return false;
---
base-commit: 00ee08d46dd81202d1dad7946f3bd4c1c106f3df
change-id: 20250820-wip-flo-fix-edge-irq-handling-16e48193aeac
Best regards,
--
Florian Bezdeka <florian.bezdeka@siemens.com>
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [PATCH Dovetail 6.16 v4] irq_pipeline: Fix system freeze triggered by edge IRQs
2025-09-15 7:18 [PATCH Dovetail 6.16 v4] irq_pipeline: Fix system freeze triggered by edge IRQs Florian Bezdeka
@ 2025-09-17 8:09 ` Philippe Gerum
0 siblings, 0 replies; 2+ messages in thread
From: Philippe Gerum @ 2025-09-17 8:09 UTC (permalink / raw)
To: Florian Bezdeka; +Cc: xenomai, Jan Kiszka
Florian Bezdeka <florian.bezdeka@siemens.com> writes:
> When IRQ pipelining is enabled, it could happen that handle_edge_irq()
> is interrupted while syncing the IRQ log - for the very same IRQ.
>
> The pipeline entry code will set the IRQS_PENDING flag and bail out
> forcing the sync code (lower part in handle_edge_irq()) to loop to
> infinity as the (periodic) IRQ might fire again after calling
> unmask_irq() and reaching the hard IRQ enablement in handle_irq_event().
>
> There is a "race window" involved. To trigger looping to infinity the
> IRQ(s) must arrive while processing the IRQ log for the same IRQ in
> handle_irq_event().
>
> Call graph, delivered by a modified NMI handler and NMI injected by
> hyper-v:
> [ 5.929380] Call Trace:
> [ 5.929381] <NMI>
> [ 5.929383] default_do_nmi+0xd9/0x110
> [ 5.929387] exc_nmi+0xde/0x110
> [ 5.929388] end_repeat_nmi+0xf/0x53
> [ 5.929390] RIP: 0010:mask_ioapic_irq+0x86/0xf0
> [ 5.929392] Code:
> [ 5.929392] RSP: 0000:ffffa1c680003e48 EFLAGS: 00010086
> [ 5.929393] RAX: ffffffffff5fc000 RBX: 0000000000000086 RCX: ffff8e4f01049340
> [ 5.929393] RDX: 0000000000203000 RSI: 0000000000010030 RDI: 0000000000000014
> [ 5.929394] RBP: ffff8e4f01049320 R08: 0000000000000012 R09: 0000000000000001
> [ 5.929394] R10: 0000000000000000 R11: 770e017136e49082 R12: ffffffffac506000
> [ 5.929395] R13: ffff8e4f3ec49840 R14: 0000000000000000 R15: 0000000000000000
> [ 5.929402] </NMI>
> [ 5.929402] <IRQ>
> [ 5.929402] handle_edge_irq+0x300/0x320
> [ 5.929405] generic_pipeline_irq_desc+0xc2/0x220
> [ 5.929408] arch_handle_irq+0x5d/0x180
> [ 5.929410] arch_pipeline_entry+0x38/0xf0
> [ 5.929413] asm_common_interrupt+0x22/0x40
> [ 5.929421] handle_irq_event+0xbe/0xe0
> [ 5.929424] handle_edge_irq+0x1ec/0x320
> [ 5.929425] arch_do_IRQ_pipelined+0xa5/0x560
> [ 5.929427] </IRQ>
> [ 5.929427] <TASK>
> [ 5.929428] sync_current_irq_stage+0x191/0x240
> [ 5.929430] __inband_irq_enable+0x48/0x60
> [ 5.929431] setup_IO_APIC+0x36b/0x860
> [ 5.929434] ? ioapic_read_entry+0x1a/0x80
> [ 5.929436] ? clear_IO_APIC_pin+0x1d/0x240
> [ 5.929437] ? clear_IO_APIC_pin+0x190/0x240
> [ 5.929440] apic_intr_mode_init+0x6f/0xf0
> [ 5.929441] x86_late_time_init+0x20/0x40
> [ 5.929442] start_kernel+0x743/0x890
> [ 5.929445] x86_64_start_reservations+0x14/0x30
> [ 5.929446] x86_64_start_kernel+0xc1/0xd0
> [ 5.929447] common_startup_64+0x13e/0x148
> [ 5.929452] </TASK>
>
> Enabling some debug options like CONFIG_PROVE_LOCKING widens this
> window as execution time increases.
>
> If we detect that the very same IRQ line is currently processed while we
> receive another hard irq, the current event is logged into the IRQ log.
> The event gets replayed within the same sync run in
> sync_current_irq_stage().
>
> Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
> ---
> Hi all,
>
> the problem could be observed and debuged quite well on hyper-v with
> lockdep enabled and a slightly modified NMI handler to print out the
> current call stack.
>
> This seems to be a long standing issue for Dovetail. We had several
> reports in the past that debug options (like lockdep) on hypervisors
> (like VirtualBox) ended up in a boot failure.
>
> Once we agree that the patch is correct I can send out the backport for
> Dovetail 6.12 down to 5.10.
>
> Best regards,
> Florian
>
> Cc: Philippe Gerum <rpm@xenomai.org>
> Cc: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> Changes in v4:
> - Remove irqd_irq_inprogress() from irq_active_on_this_cpu().
> That check was always true as it is already detected by the caller.
> - Add !SMP check to irq_active_on_this_cpu(), Review comment from
> Philippe
> - Link to v3: https://lore.kernel.org/r/20250820-wip-flo-fix-edge-irq-handling-v3-1-8cc08939e6ca@siemens.com
>
> Changes in v3:
> - Introduce irq_active_on_this_cpu()
> - Link to v2: https://lore.kernel.org/r/20250820-wip-flo-fix-edge-irq-handling-v2-1-1a958d17eace@siemens.com
>
> Changes in v2:
> - v1 crashed qemu, similar to
> https://lore.kernel.org/all/20250701163558.2588435-1-liangyan.peng@bytedance.com/
> so v1 was analyzed further and reworked completely
> - Link to v1: https://lore.kernel.org/r/20250820-wip-flo-fix-edge-irq-handling-v1-1-1a2c7805b372@siemens.com
> ---
> kernel/irq/chip.c | 24 ++++++++++++++++++++++++
> 1 file changed, 24 insertions(+)
>
> diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
> index f5dfc378ccf2..12f520f6a197 100644
> --- a/kernel/irq/chip.c
> +++ b/kernel/irq/chip.c
> @@ -639,6 +639,28 @@ static inline bool should_feed_pipeline(struct irq_desc *desc, int state)
> return false;
> }
>
> +#ifdef CONFIG_IRQ_PIPELINE
> +static bool irq_active_on_this_cpu(struct irq_desc *desc)
> +{
> + struct irq_data *data = &desc->irq_data;
> + const struct cpumask *aff;
> +
> + /* desc marked as in progress, no other CPUs on !SMP */
> + if (!IS_ENABLED(CONFIG_SMP))
> + return true;
> +
> + /* The below works only for single target interrupts */
> + if (!irqd_is_single_target(data) || desc->handle_irq != handle_edge_irq)
> + return false;
> +
> + aff = irq_data_get_effective_affinity_mask(data);
> + if (cpumask_first(aff) != smp_processor_id())
> + return false;
> +
> + return true;
> +}
> +#endif
> +
> static bool irq_can_handle_pm(struct irq_desc *desc)
> {
> unsigned int mask = IRQD_IRQ_INPROGRESS | IRQD_WAKEUP_ARMED;
> @@ -669,6 +691,8 @@ static bool irq_can_handle_pm(struct irq_desc *desc)
> WARN_ON_ONCE(irq_pipeline_debug() && !in_pipeline());
> if (irqd_is_wakeup_armed(&desc->irq_data))
> return true;
> + if (irq_active_on_this_cpu(desc))
> + return true;
> } else if (irq_pm_check_wakeup(desc))
> return false;
>
>
> ---
> base-commit: 00ee08d46dd81202d1dad7946f3bd4c1c106f3df
> change-id: 20250820-wip-flo-fix-edge-irq-handling-16e48193aeac
>
> Best regards,
Merged, thanks.
--
Philippe.
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2025-09-17 8:09 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-09-15 7:18 [PATCH Dovetail 6.16 v4] irq_pipeline: Fix system freeze triggered by edge IRQs Florian Bezdeka
2025-09-17 8:09 ` Philippe Gerum
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox