* [PATCH v2 00/39] arm64: Add support for FEAT_NMI
@ 2026-11-12 13:20 Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 01/39] arm64: irqflags: Introduce arm64-specific irqflags type Vladimir Murzin
` (41 more replies)
0 siblings, 42 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
FEAT_NMI provides an architected mechanism for supporting non-maskable
interrupts (NMIs) and less-masked interrupts (LMIs).
Since we already support pseudo-NMIs via priority masking, introducing
another flavour of NMI on top of the existing infrastructure could
easily become messy, making the code harder to follow and reason
about.
To avoid that, this series first makes room for the new NMI "tenant"
by restructuring the existing exception masking logic.
The main idea is to separate the logical view of exception state from
its hardware representation. To achieve this, we introduce logical
exception contexts that can be mapped onto the corresponding hardware
state. This naturally consolidates the hardware-specific handling into
a small number of places, while allowing the rest of the code to
operate purely in terms of logical exception contexts.
Since this restructuring is non-trivial and carries a risk of subtle
behavioural changes, the series adds extensive debug checks to verify
that the hardware state always matches the expected logical state.
With this restructuring in place, wiring FEAT_NMI into the new
framework becomes much more manageable. GIC support for NMI is build
on top.
This work would not have been possible without the contributions of
Ada Couprie Diaz, Mark Brown, and Lorenzo Pieralisi - please credit
them for everything that works well. Any remaining bugs or issues are
entirely my own.
Most of the testing has been done on QEMU, Fast Model and pre-silicon
environment, so the series definitely benefit from extensive testing
on real hardware (without NMI, with pseudo-NMI, and with FEAT_NMI).
Patches are organized as follow
1 - 15
Rework of existing pseudo-NMI logic to make room for FEAT_NMI
16 - 33
FEAT_NMI support including some preparation patches which have
no dependencies:
- efi/runtime-wrappers: Permit architectures to override IRQ flags checks
- arm64: smp: Abstract SGI and LPI operations
34 - 35
GICv3 support for NMI
36 - 39
GICv5 support for NMI (including some preparation patches)
Changelog
v1 -> v2
- Rebased on v7.4-rc2
- Select HAVE_PERF_EVENTS_NMI when FEAT_NMI is enabled
- Reworked APEI SEA handling to mask exceptions sequentially
- Updated GICv3 initialization to clear NMI configuration
- Added Tested-by and Reviewed-by tags where approriate
- Droped tags due to meaningful changes for:
+ "arm64: replace local_daif helpers"
+ "irqchip/gic-v3: Implement FEAT_GICv3_NMI support"
RFC v2 -> v1
- Rebased on v7.3-rc3.
- Dropped RFC tag, have not received objections, at the same time
got support from Jinjie
- Dropped prerequisite patches already present in the newer base
- Added logic to reject probing/stepping over ALLINT
- Moved addition of arm64_debug_exc_context() in separate patch
- Reworked debug logic for validating direction of exception
transition
- Split support for NMI in GICv5 in 3 separate patches
- Added review tags
RFC v1 -> RFC v2
- Rebased on v7.2-rc5.
- Improved commit descriptions.
- Split some patches into smaller ones.
- Added a graceful fallback for IPI requests when FEAT_NMI is
supported by the CPU but unsupported by the interrupt
controller.
- Split the GICv3 preparation from the FEAT_GICv3_NMI
implementation.
- Added NMI support to GICv5, primarily as an illustration of the
implementation, although it also happens to work on Fast Model.
- Added review tags only to unchanged patches. Review tags were
intentionally dropped from modified patches to reflect those
changes and encourage re-review.
RFC v1:
https://lore.kernel.org/linux-arm-kernel/20260709121333.23507-1-vladimir.murzin@arm.com/
RFC v2:
https://lore.kernel.org/linux-arm-kernel/20260727163453.7969-1-vladimir.murzin@arm.com/
v1:
https://lore.kernel.org/linux-arm-kernel/20260914102102.68597-1-vladimir.murzin@arm.com/
Thanks!
Vladimir
Ada Couprie Diaz (15):
arm64: irqflags: Introduce arm64-specific irqflags type
arm64: irqflags: save and use both DAIF and PMR
arm64: interrupts: Add common exception state helpers
arm64: entry: Introduce entry specific exception masking helpers
arm64: entry: replace DAIF helpers with entry helpers
arm64: interrupts: Introduce exception masking save/restore helpers
arm64: interrupts: introduce a helper for GIC priority initialization
arm64: replace local_daif helpers
arm64: cpuidle: use new helpers to bypass interrupt priority masking
arm64: remove daifflags.h
arm64: gicv3: remove GIC_PRIO_PSR_I_SET
arm64: ptrace: Add PSR_ALLINT_BIT
arm64: cpufeature: Detect PE support for FEAT_NMI
arm64: nmi: Manage masking for superpriority interrupts
arm64: irq: Report FEAT_NMI masking local IRQs
Lorenzo Pieralisi (2):
irqchip/gic-v3: Rename nmi to pseudo_nmi
irqchip/gic-v3: Implement FEAT_GICv3_NMI support
Mark Brown (5):
arm64: booting: Document boot requirements for FEAT_NMI
arm64: sysreg: Add definitions for immediate versions of MSR ALLINT
arm64: idreg: Add an override for FEAT_NMI
arm64: nmi: Add handling of superpriority interrupts as NMIs
arm64: nmi: Add Kconfig for NMI
Vladimir Murzin (17):
arm64: process: Lift check for exception state to generic
infrastructure
arm64: cpufeature: Remove system_has_prio_mask_debugging()
arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS
arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING
efi/runtime-wrappers: Permit architectures to override IRQ flags
checks
arm64/efi: Implement override for IRQ flags checks
arm64: probes: Reject stepping over ALLINT
arm64: suspend: Always initialise PSTATE.ALLINT
arm64/efi: Add ALLINT to IRQ flags checks
arm64: kprobes: Disable NMIs
arm64: smp: Abstract SGI and LPI operations
arm64: smp: Fall back to IRQ when IPI NMI request fails
arm64: nmi: Select HAVE_PERF_EVENTS_NMI for FEAT_NMI
arm64: smp: Add NMI support for LPI-backed IPIs
genirq: Trust local irqchip when considering IRQ for NMI
genirq: Do not call add_interrupt_randomness() in NMI context
irqchip/gic-v5: Add NMI support
Documentation/arch/arm64/booting.rst | 6 +
arch/arm64/Kconfig | 31 +-
arch/arm64/include/asm/assembler.h | 20 +-
arch/arm64/include/asm/cpucaps.h | 2 +
arch/arm64/include/asm/cpufeature.h | 14 +-
arch/arm64/include/asm/cpuidle.h | 18 +-
arch/arm64/include/asm/daifflags.h | 144 --------
arch/arm64/include/asm/efi.h | 42 ++-
arch/arm64/include/asm/entry-common.h | 8 +-
arch/arm64/include/asm/insn.h | 1 +
.../include/asm/interrupts/common_flags.h | 255 +++++++++++++
arch/arm64/include/asm/interrupts/entry.h | 102 ++++++
arch/arm64/include/asm/interrupts/masking.h | 166 +++++++++
arch/arm64/include/asm/irqflags.h | 150 ++++----
arch/arm64/include/asm/kvm_host.h | 1 -
arch/arm64/include/asm/mmu_context.h | 1 -
arch/arm64/include/asm/ptrace.h | 25 +-
arch/arm64/include/asm/sysreg.h | 2 +
arch/arm64/include/uapi/asm/ptrace.h | 1 +
arch/arm64/kernel/acpi.c | 43 +--
arch/arm64/kernel/cpufeature.c | 68 +++-
arch/arm64/kernel/debug-monitors.c | 1 -
arch/arm64/kernel/entry-common.c | 335 +++++++++++++-----
arch/arm64/kernel/entry.S | 14 +-
arch/arm64/kernel/hibernate.c | 14 +-
arch/arm64/kernel/hw_breakpoint.c | 16 +-
arch/arm64/kernel/irq.c | 45 ++-
arch/arm64/kernel/machine_kexec.c | 4 +-
arch/arm64/kernel/pi/idreg-override.c | 1 +
arch/arm64/kernel/probes/decode-insn.c | 11 +-
arch/arm64/kernel/probes/kprobes.c | 24 +-
arch/arm64/kernel/process.c | 24 +-
arch/arm64/kernel/setup.c | 4 +-
arch/arm64/kernel/signal.c | 1 -
arch/arm64/kernel/smp.c | 217 ++++++++----
arch/arm64/kernel/suspend.c | 14 +-
arch/arm64/kernel/traps.c | 1 -
arch/arm64/kvm/hyp/nvhe/switch.c | 2 +-
arch/arm64/kvm/hyp/vgic-v3-sr.c | 7 +-
arch/arm64/kvm/hyp/vhe/switch.c | 14 +-
arch/arm64/mm/fault.c | 1 -
arch/arm64/mm/mmu.c | 7 +-
arch/arm64/tools/cpucaps | 2 +
drivers/firmware/efi/runtime-wrappers.c | 32 +-
drivers/irqchip/irq-gic-v3.c | 145 +++++++-
drivers/irqchip/irq-gic-v5.c | 244 ++++++++++++-
include/linux/irq.h | 2 +-
include/linux/irqchip/arm-gic-v3-prio.h | 8 -
include/linux/irqchip/arm-gic-v3.h | 4 +
kernel/irq/handle.c | 3 +-
kernel/irq/manage.c | 5 -
51 files changed, 1696 insertions(+), 606 deletions(-)
delete mode 100644 arch/arm64/include/asm/daifflags.h
create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
create mode 100644 arch/arm64/include/asm/interrupts/entry.h
create mode 100644 arch/arm64/include/asm/interrupts/masking.h
--
2.34.1
^ permalink raw reply [flat|nested] 49+ messages in thread
* [PATCH v2 01/39] arm64: irqflags: Introduce arm64-specific irqflags type
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 02/39] arm64: irqflags: save and use both DAIF and PMR Vladimir Murzin
` (40 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
With pseudo-NMIs enabled, we have two mechanisms that control
interrupt masking in parallel :
- The DAIF flags, masking at the CPU
- The GIC PMR, masking before the CPU
However, our irqflags implementation currently assumes that only one
of the two is used at a time, so both DAIF and PMR masking use the same
`unsigned long flags` in their own way.
This is incorrect, as some parts of the kernel will mask interrupts
with DAIF directly or bypass the local_irq masking via the PMR,
and makes tracking the state and changes of both in parallel impossible.
The irqflags API expects `unsigned long`s to be passed around, but
they should not be manipulated outside of the arch-specific code.
So, we can encode the information we need however we want as long as
we return and accept `unsigned long`s.
Introduce a union type for arm64 irqflags whose first member is
a struct allowing us to track DAIF and PMR in parallel, and the second
is the `unsigned long` expected by the irqflags API.
DAIF is a two byte value, to maintain compatibility with existing defines.
PMR is a one byte value, which is the maximum amount of priority bits
allowed by the GICv3 architecture.
Update the internal irqflags functions to use this new union and convert
back and forth with the irqflags unsigned long.
There should be no functional changes.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/irqflags.h | 44 ++++++++++++++++++++++++++-----
1 file changed, 38 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index a8cb5a5c93b7..5c17e26727f9 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -9,6 +9,8 @@
#include <asm/ptrace.h>
#include <asm/sysreg.h>
+#include <linux/compiler.h>
+
/*
* Aarch64 has flags for masking: Debug, Asynchronous (serror), Interrupts and
* FIQ exceptions, in the 'daif' register. We mask and unmask them in 'daif'
@@ -20,6 +22,24 @@
* exceptions should be unmasked.
*/
+ /*
+ * Internally, we want to independently manipulate and track the different
+ * interrupt masking mechanisms.
+ * Externally, the generic irqflags API expects unsgined longs to represent
+ * the state of interrupts, which are treated as obscure arch-specific data.
+ */
+typedef union arm64_exc_hwstate {
+ struct {
+ u16 daif;
+ u8 pmr;
+ u8 __pad0;
+ u32 __pad1;
+ };
+ unsigned long flags;
+} arm64_exc_hwstate_t;
+
+static_assert(sizeof(arm64_exc_hwstate_t) == sizeof(unsigned long));
+
static __always_inline void __daif_local_irq_enable(void)
{
barrier();
@@ -79,12 +99,16 @@ static __always_inline void arch_local_irq_disable(void)
static __always_inline unsigned long __daif_local_save_flags(void)
{
- return read_sysreg(daif);
+ arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
+
+ return hwstate.flags;
}
static __always_inline unsigned long __pmr_local_save_flags(void)
{
- return read_sysreg_s(SYS_ICC_PMR_EL1);
+ arm64_exc_hwstate_t hwstate = { .pmr = read_sysreg_s(SYS_ICC_PMR_EL1) };
+
+ return hwstate.flags;
}
/*
@@ -101,12 +125,16 @@ static __always_inline unsigned long arch_local_save_flags(void)
static __always_inline bool __daif_irqs_disabled_flags(unsigned long flags)
{
- return flags & PSR_I_BIT;
+ arm64_exc_hwstate_t hwstate = { .flags = flags };
+
+ return hwstate.daif & PSR_I_BIT;
}
static __always_inline bool __pmr_irqs_disabled_flags(unsigned long flags)
{
- return flags != GIC_PRIO_IRQON;
+ arm64_exc_hwstate_t hwstate = { .flags = flags };
+
+ return hwstate.pmr != GIC_PRIO_IRQON;
}
static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
@@ -171,15 +199,19 @@ static __always_inline unsigned long arch_local_irq_save(void)
static __always_inline void __daif_local_irq_restore(unsigned long flags)
{
+ arm64_exc_hwstate_t hwstate = { .flags = flags };
+
barrier();
- write_sysreg(flags, daif);
+ write_sysreg(hwstate.daif, daif);
barrier();
}
static __always_inline void __pmr_local_irq_restore(unsigned long flags)
{
+ arm64_exc_hwstate_t hwstate = { .flags = flags };
+
barrier();
- write_sysreg_s(flags, SYS_ICC_PMR_EL1);
+ write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
pmr_sync();
barrier();
}
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 02/39] arm64: irqflags: save and use both DAIF and PMR
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 01/39] arm64: irqflags: Introduce arm64-specific irqflags type Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 03/39] arm64: interrupts: Add common exception state helpers Vladimir Murzin
` (39 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
With pseudo-NMIs enabled, both DAIF and the PMR affect interrupt masking.
Now that we have a type which can track both of them at the same time,
update our irqflags implementation to use it.
Save DAIF flags in all cases, as they can be manipulated directly by other
code, and the PMR if it is in use.
When checking if IRQs are disabled, now that we always save DAIF we
can check that the I flag is set and bypass checking the PMR if it is.
We can also properly check if PMR masks interrupts, now that we don't
need to rely on the GIC_PRIO_PSR_I_SET bit being set in the PMR to
know if DAIF is already masking interrupts. Update
`irqs_priority_unmasked()` to align with this change.
This allows us to remove the `__daif_...` and `__pmr_...` versions
of the save and check functions, as they are now unified.
We can reasonably merge the two `__{daif,pmr}_irq_restore()` functions
in the main one, as the DAIF and PMR values are properly split now.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/irqflags.h | 118 +++++++-----------------------
arch/arm64/include/asm/ptrace.h | 18 +++--
2 files changed, 38 insertions(+), 98 deletions(-)
diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index 5c17e26727f9..9f5c774cbe68 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -97,135 +97,69 @@ static __always_inline void arch_local_irq_disable(void)
}
}
-static __always_inline unsigned long __daif_local_save_flags(void)
-{
- arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
-
- return hwstate.flags;
-}
-
-static __always_inline unsigned long __pmr_local_save_flags(void)
-{
- arm64_exc_hwstate_t hwstate = { .pmr = read_sysreg_s(SYS_ICC_PMR_EL1) };
-
- return hwstate.flags;
-}
-
/*
* Save the current interrupt enable state.
*/
static __always_inline unsigned long arch_local_save_flags(void)
{
- if (system_uses_irq_prio_masking()) {
- return __pmr_local_save_flags();
- } else {
- return __daif_local_save_flags();
- }
-}
+ arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
-static __always_inline bool __daif_irqs_disabled_flags(unsigned long flags)
-{
- arm64_exc_hwstate_t hwstate = { .flags = flags };
+ if (system_uses_irq_prio_masking())
+ hwstate.pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
- return hwstate.daif & PSR_I_BIT;
+ return hwstate.flags;
}
-static __always_inline bool __pmr_irqs_disabled_flags(unsigned long flags)
+static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
{
arm64_exc_hwstate_t hwstate = { .flags = flags };
- return hwstate.pmr != GIC_PRIO_IRQON;
-}
+ if (hwstate.daif & PSR_I_BIT)
+ return true;
-static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
-{
if (system_uses_irq_prio_masking()) {
- return __pmr_irqs_disabled_flags(flags);
- } else {
- return __daif_irqs_disabled_flags(flags);
- }
-}
+ WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_IRQFLAGS) &&
+ hwstate.pmr != GIC_PRIO_IRQOFF &&
+ hwstate.pmr != GIC_PRIO_IRQON);
-static __always_inline bool __daif_irqs_disabled(void)
-{
- return __daif_irqs_disabled_flags(__daif_local_save_flags());
-}
+ return hwstate.pmr != GIC_PRIO_IRQON;
+ }
-static __always_inline bool __pmr_irqs_disabled(void)
-{
- return __pmr_irqs_disabled_flags(__pmr_local_save_flags());
+ return false;
}
static __always_inline bool arch_irqs_disabled(void)
{
- if (system_uses_irq_prio_masking()) {
- return __pmr_irqs_disabled();
- } else {
- return __daif_irqs_disabled();
- }
+ return arch_irqs_disabled_flags(arch_local_save_flags());
}
-static __always_inline unsigned long __daif_local_irq_save(void)
-{
- unsigned long flags = __daif_local_save_flags();
-
- __daif_local_irq_disable();
-
- return flags;
-}
-
-static __always_inline unsigned long __pmr_local_irq_save(void)
+static __always_inline unsigned long arch_local_irq_save(void)
{
- unsigned long flags = __pmr_local_save_flags();
+ unsigned long flags = arch_local_save_flags();
- /*
- * There are too many states with IRQs disabled, just keep the current
- * state if interrupts are already disabled/masked.
- */
- if (!__pmr_irqs_disabled_flags(flags))
+ if (system_uses_irq_prio_masking())
__pmr_local_irq_disable();
+ else
+ __daif_local_irq_disable();
return flags;
}
-static __always_inline unsigned long arch_local_irq_save(void)
-{
- if (system_uses_irq_prio_masking()) {
- return __pmr_local_irq_save();
- } else {
- return __daif_local_irq_save();
- }
-}
-
-static __always_inline void __daif_local_irq_restore(unsigned long flags)
-{
- arm64_exc_hwstate_t hwstate = { .flags = flags };
-
- barrier();
- write_sysreg(hwstate.daif, daif);
- barrier();
-}
-
-static __always_inline void __pmr_local_irq_restore(unsigned long flags)
-{
- arm64_exc_hwstate_t hwstate = { .flags = flags };
-
- barrier();
- write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
- pmr_sync();
- barrier();
-}
-
/*
* restore saved IRQ state
*/
static __always_inline void arch_local_irq_restore(unsigned long flags)
{
+ arm64_exc_hwstate_t hwstate = { .flags = flags };
+
+ barrier();
if (system_uses_irq_prio_masking()) {
- __pmr_local_irq_restore(flags);
+ write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
+ pmr_sync();
} else {
- __daif_local_irq_restore(flags);
+ write_sysreg(hwstate.daif, daif);
}
+ barrier();
}
#endif /* __ASM_IRQFLAGS_H */
diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index ad0ad30b36cd..7d7b54308b90 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -228,14 +228,20 @@ static inline void forget_syscall(struct pt_regs *regs)
#define processor_mode(regs) \
((regs)->pstate & PSR_MODE_MASK)
-#define irqs_priority_unmasked(regs) \
- (system_uses_irq_prio_masking() ? \
- (regs)->pmr == GIC_PRIO_IRQON : \
- true)
-
static __always_inline bool regs_irqs_disabled(const struct pt_regs *regs)
{
- return (regs->pstate & PSR_I_BIT) || !irqs_priority_unmasked(regs);
+ if (regs->pstate & PSR_I_BIT)
+ return true;
+
+ if (system_uses_irq_prio_masking()) {
+ WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_IRQFLAGS) &&
+ regs->pmr != GIC_PRIO_IRQOFF &&
+ regs->pmr != GIC_PRIO_IRQON);
+
+ return regs->pmr != GIC_PRIO_IRQON;
+ }
+
+ return false;
}
#define interrupts_enabled(regs) (!regs_irqs_disabled(regs))
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 03/39] arm64: interrupts: Add common exception state helpers
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 01/39] arm64: irqflags: Introduce arm64-specific irqflags type Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 02/39] arm64: irqflags: save and use both DAIF and PMR Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 04/39] arm64: process: Lift check for exception state to generic infrastructure Vladimir Murzin
` (38 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
The entry code and the rest of the kernel require different interrupt
masking APIs, but all need a common representation of the hardware
exception state.
Introduce exception contexts that map DAIF and PMR state to the
corresponding masking context. Provide helpers to translate between
exception contexts and hardware state and update the hardware state.
Updating PMR is unnecessary when a caller knows that its value has not
changed. Allow such callers to avoid the update, while providing a
force option for callers that cannot rely on the previous PMR state.
Entry specific and general purpose masking APIs will be built on these
helpers in subsequent patches.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Reviewed-by: Ashok Raj <ashok.raj@oss.qualcomm.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
.../include/asm/interrupts/common_flags.h | 163 ++++++++++++++++++
1 file changed, 163 insertions(+)
create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
new file mode 100644
index 000000000000..41364a4e9726
--- /dev/null
+++ b/arch/arm64/include/asm/interrupts/common_flags.h
@@ -0,0 +1,163 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2025 Arm Ltd.
+ */
+#ifndef __ASM_INTERRUPTS_COMMON_FLAGS_H
+#define __ASM_INTERRUPTS_COMMON_FLAGS_H
+
+#include <asm/arch_gicv3.h>
+#include <asm/bug.h>
+#include <asm/cpufeature.h>
+#include <asm/ptrace.h>
+#include <asm/sysreg.h>
+#include <asm/irqflags.h>
+
+#define DAIF_PROCCTX 0
+#define DAIF_PROCCTX_NOIRQ (PSR_I_BIT | PSR_F_BIT)
+#define DAIF_ERRCTX (PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
+#define DAIF_MASK (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
+
+/*
+ * Exception context mapping
+ *
+ * pseudo-NMI
+ *
+ * CRITICAL -> DAIF + IRQON (corresponds to the state on exception entry)
+ * ERROR -> AIF + IRQON
+ * NONMI -> IF + IRQON
+ * NOIRQ -> 0 + IRQOFF
+ * PROCESS -> 0 + IRQON
+ *
+ * Otherwise
+ *
+ * CRITICAL -> DAIF (corresponds to the state on exception entry)
+ * ERROR -> AIF
+ * NONMI -> IF
+ * NOIRQ -> IF
+ * PROCESS -> 0
+ */
+typedef enum arm64_exc_context {
+ PROCESS_CONTEXT,
+ NOIRQ_CONTEXT,
+ NONMI_CONTEXT,
+ ERROR_CONTEXT,
+ CRITICAL_CONTEXT,
+} arm64_exc_context_t;
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_process_context(void)
+{
+ if (system_uses_irq_prio_masking())
+ return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQON};
+
+ return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX};
+}
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
+{
+ if (system_uses_irq_prio_masking())
+ return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
+
+ return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
+}
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
+{
+ if (system_uses_irq_prio_masking())
+ return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
+
+ return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
+}
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
+{
+ if (system_uses_irq_prio_masking())
+ return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
+
+ return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
+}
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
+{
+ if (system_uses_irq_prio_masking())
+ return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
+
+ return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_exc_hwstate_of_context(arm64_exc_context_t context) {
+ switch (context) {
+ case PROCESS_CONTEXT:
+ return __arm64_exc_hwstate_of_process_context();
+ case NOIRQ_CONTEXT:
+ return __arm64_exc_hwstate_of_noirq_context();
+ case NONMI_CONTEXT:
+ return __arm64_exc_hwstate_of_nonmi_context();
+ case ERROR_CONTEXT:
+ return __arm64_exc_hwstate_of_error_context();
+ case CRITICAL_CONTEXT:
+ return __arm64_exc_hwstate_of_critical_context();
+ default:
+ BUG();
+ }
+}
+
+static __always_inline
+void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
+{
+ bool pseudo_nmi = system_uses_irq_prio_masking();
+
+ barrier();
+
+ if (pseudo_nmi && hwstate.pmr == GIC_PRIO_IRQOFF && force) {
+ /*
+ * There has been concern that the write to daif
+ * might be reordered before this write to PMR.
+ * From the ARM ARM DDI 0487D.a, section D1.7.1
+ * "Accessing PSTATE fields":
+ * Writes to the PSTATE fields have side-effects on
+ * various aspects of the PE operation. All of these
+ * side-effects are guaranteed:
+ * - Not to be visible to earlier instructions in
+ * the execution stream.
+ * - To be visible to later instructions in the
+ * execution stream
+ *
+ * Also, writes to PMR are self-synchronizing, so no
+ * interrupts with a lower priority than PMR is signaled
+ * to the PE after the write.
+ *
+ * So we don't need additional synchronization here.
+ */
+ write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
+ }
+
+ write_sysreg(hwstate.daif, daif);
+
+ if (pseudo_nmi && hwstate.pmr == GIC_PRIO_IRQON && force) {
+ write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
+ pmr_sync();
+ }
+
+ barrier();
+}
+
+static __always_inline
+void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate)
+{
+ __arm64_update_exc_hwstate(hwstate, true);
+}
+
+static __always_inline
+void arm64_update_exc_context(arm64_exc_context_t context)
+{
+ arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(context);
+
+ arm64_update_exc_hwstate(hwstate);
+}
+#endif /* __ASM_INTERRUPTS_COMMON_FLAGS_H */
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 04/39] arm64: process: Lift check for exception state to generic infrastructure
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (2 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 03/39] arm64: interrupts: Add common exception state helpers Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 05/39] arm64: entry: Introduce entry specific exception masking helpers Vladimir Murzin
` (37 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
debug_switch_state() explicitly checks the hardware exception state
for both the pseudo-NMI and non-NMI paths. The expected state matches
the NOIRQ_CONTEXT exception context introduced alongside the exception
masking helpers and state checkers. Such checks are handy outside of
task switch, so turn them into generic helpers available for wider
use. Make debug_switch_state() the first user of newly introduced
helpers.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
.../include/asm/interrupts/common_flags.h | 30 +++++++++++++++++++
arch/arm64/kernel/process.c | 21 ++-----------
2 files changed, 32 insertions(+), 19 deletions(-)
diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
index 41364a4e9726..8f79e8179601 100644
--- a/arch/arm64/include/asm/interrupts/common_flags.h
+++ b/arch/arm64/include/asm/interrupts/common_flags.h
@@ -107,6 +107,36 @@ arm64_exc_hwstate_t arm64_exc_hwstate_of_context(arm64_exc_context_t context) {
}
}
+static __always_inline
+void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
+{
+ arm64_exc_hwstate_t actual;
+
+ if (!IS_ENABLED(CONFIG_DEBUG_IRQFLAGS))
+ return;
+
+ actual.flags = arch_local_save_flags();
+
+ if (expected.flags == actual.flags)
+ return;
+
+ if (system_uses_irq_prio_masking()) {
+ WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
+ actual.daif, actual.pmr, expected.daif, expected.pmr);
+ } else {
+ WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
+ actual.daif, expected.daif);
+ }
+}
+
+static __always_inline
+void arm64_debug_exc_context(arm64_exc_context_t context)
+{
+ arm64_exc_hwstate_t expected = arm64_exc_hwstate_of_context(context);
+
+ arm64_debug_exc_hwstate(expected);
+}
+
static __always_inline
void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
{
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 42b15b0c069f..1013343de671 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -51,6 +51,7 @@
#include <asm/exec.h>
#include <asm/fpsimd.h>
#include <asm/gcs.h>
+#include <asm/interrupts/common_flags.h>
#include <asm/mmu_context.h>
#include <asm/mpam.h>
#include <asm/mte.h>
@@ -720,25 +721,7 @@ void update_sctlr_el1(u64 sctlr)
static inline void debug_switch_state(void)
{
- if (system_uses_irq_prio_masking()) {
- unsigned long daif_expected = 0;
- unsigned long daif_actual = read_sysreg(daif);
- unsigned long pmr_expected = GIC_PRIO_IRQOFF;
- unsigned long pmr_actual = read_sysreg_s(SYS_ICC_PMR_EL1);
-
- WARN_ONCE(daif_actual != daif_expected ||
- pmr_actual != pmr_expected,
- "Unexpected DAIF + PMR: 0x%lx + 0x%lx (expected 0x%lx + 0x%lx)\n",
- daif_actual, pmr_actual,
- daif_expected, pmr_expected);
- } else {
- unsigned long daif_expected = DAIF_PROCCTX_NOIRQ;
- unsigned long daif_actual = read_sysreg(daif);
-
- WARN_ONCE(daif_actual != daif_expected,
- "Unexpected DAIF value: 0x%lx (expected 0x%lx)\n",
- daif_actual, daif_expected);
- }
+ arm64_debug_exc_context(NOIRQ_CONTEXT);
}
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 05/39] arm64: entry: Introduce entry specific exception masking helpers
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (3 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 04/39] arm64: process: Lift check for exception state to generic infrastructure Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 06/39] arm64: entry: replace DAIF helpers with entry helpers Vladimir Murzin
` (36 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
The entry code handles interrupt masking differently from the rest of
the kernel. Exception handlers enter and exit with all exceptions
masked, but they must temporarily unmask the appropriate set of
exceptions so that the rest of the handler executes with the expected
exception state.
For EL0 handlers, this means dropping to masking context appropriate
for the work to be performed. For EL1 handlers, this means restoring
the masking context of the interrupted task. In both cases, all
exceptions must be masked again before returning from the exception
handler.
The rest of the kernel typically follows the opposite pattern: it
raises the masking context to protect a critical section and later
restores the previous context.
Given these different usage patterns, introduce a dedicated set of
exception masking helpers for the entry code. Keeping these helpers
separate from the generic interrupt masking APIs makes the intended
usage explicit and helps avoid mixing the two masking models.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
.../include/asm/interrupts/common_flags.h | 11 ++
arch/arm64/include/asm/interrupts/entry.h | 100 ++++++++++++++++++
2 files changed, 111 insertions(+)
create mode 100644 arch/arm64/include/asm/interrupts/entry.h
diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
index 8f79e8179601..b5cf3b6d0774 100644
--- a/arch/arm64/include/asm/interrupts/common_flags.h
+++ b/arch/arm64/include/asm/interrupts/common_flags.h
@@ -107,6 +107,17 @@ arm64_exc_hwstate_t arm64_exc_hwstate_of_context(arm64_exc_context_t context) {
}
}
+static __always_inline
+arm64_exc_hwstate_t arm64_exc_hwstate_of_regs(struct pt_regs *regs)
+{
+ arm64_exc_hwstate_t state = {.daif=regs->pstate & DAIF_MASK};
+
+ if (system_uses_irq_prio_masking())
+ state.pmr = regs->pmr;
+
+ return state;
+}
+
static __always_inline
void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
{
diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
new file mode 100644
index 000000000000..e34c198f4ac1
--- /dev/null
+++ b/arch/arm64/include/asm/interrupts/entry.h
@@ -0,0 +1,100 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2025 Arm Ltd.
+ */
+#ifndef __ASM_INTERRUPTS_ENTRY_H
+#define __ASM_INTERRUPTS_ENTRY_H
+
+#include <asm/arch_gicv3.h>
+#include <asm/bug.h>
+#include <asm/cpufeature.h>
+#include <asm/interrupts/common_flags.h>
+
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
+ arm64_exc_hwstate_t next)
+{
+ bool irqs_disabled = arch_irqs_disabled_flags(next.flags);
+ bool force;
+
+ arm64_debug_exc_hwstate(prev);
+
+ if (prev.flags == next.flags)
+ return next;
+
+ if (!irqs_disabled)
+ trace_hardirqs_on();
+
+ force = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
+
+ __arm64_update_exc_hwstate(next, force);
+
+ if (irqs_disabled)
+ trace_hardirqs_off();
+
+ return next;
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_inherit_exc_context(struct pt_regs *regs)
+{
+ arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
+ arm64_exc_hwstate_t next = arm64_exc_hwstate_of_regs(regs);
+
+ return __arm64_switch_exc_hwstate_to(prev, next);
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
+{
+ arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
+
+ if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
+ WARN_ON_ONCE(prev.daif < next.daif);
+
+ if (prev.daif == next.daif) {
+ /*
+ * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so larger PMR value is weaker
+ */
+ WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr > next.pmr);
+ }
+ }
+
+ return __arm64_switch_exc_hwstate_to(prev, next);
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
+{
+ arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
+
+ if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
+ WARN_ON_ONCE(prev.daif > next.daif);
+
+ if (prev.daif == next.daif) {
+ /*
+ * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so smaller PMR value is stronger
+ */
+ WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr < next.pmr);
+ }
+ }
+
+ return __arm64_switch_exc_hwstate_to(prev, next);
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_unmask_exc_context(arm64_exc_context_t context)
+{
+ arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
+
+ return arm64_drop_exc_context(prev, context);
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_mask_exc_context(arm64_exc_hwstate_t prev)
+{
+ return arm64_lift_exc_context(prev, CRITICAL_CONTEXT);
+}
+
+#endif /* __ASM_INTERRUPTS_ENTRY_H */
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 06/39] arm64: entry: replace DAIF helpers with entry helpers
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (4 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 05/39] arm64: entry: Introduce entry specific exception masking helpers Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 07/39] arm64: interrupts: Introduce exception masking save/restore helpers Vladimir Murzin
` (35 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Replace all uses of the `local_daif_...` helpers in `entry-common.c`
with the new entry-specific exception masking helpers.
Also replace `local_irq_disable()` with a switch to NOIRQ context
using the new helpers. This provides:
- consistency checks for hardware state
- a unified style for managing exception context
Now that both the irqflags API and entry code handle DAIF and PMR
correctly, remove `GIC_PRIO_PSR_I_SET` from the PMR value set by
`kernel_entry` in `entry.S` and by `init_gic_priority_masking()` in
`kernel/smp.c`.
The `local_daif_...` helpers and other specialized code still use
`GIC_PRIO_PSR_I_SET` for now; this commit does not change their
behavior.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/kernel/entry-common.c | 258 ++++++++++++++++++++-----------
arch/arm64/kernel/entry.S | 2 +-
arch/arm64/kernel/smp.c | 2 +-
3 files changed, 173 insertions(+), 89 deletions(-)
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index beb56891fc4f..ec7edb8790d5 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -18,11 +18,11 @@
#include <linux/thread_info.h>
#include <asm/cpufeature.h>
-#include <asm/daifflags.h>
#include <asm/esr.h>
#include <asm/exception.h>
#include <asm/fpsimd.h>
#include <asm/irq_regs.h>
+#include <asm/interrupts/entry.h>
#include <asm/kprobes.h>
#include <asm/mmu.h>
#include <asm/percpu.h>
@@ -91,9 +91,10 @@ static noinstr irqentry_state_t arm64_enter_from_kernel_mode(struct pt_regs *reg
* instrumentable code, or any code which may trigger an exception.
*/
static void noinstr __arm64_exit_to_kernel_mode(struct pt_regs *regs,
- irqentry_state_t state)
+ irqentry_state_t state,
+ arm64_exc_hwstate_t hwstate)
{
- local_daif_mask();
+ arm64_mask_exc_context(hwstate);
irqentry_exit_pcpu_adjust(regs);
mte_check_tfsr_exit();
irqentry_exit_to_kernel_mode_after_preempt(regs, state);
@@ -103,22 +104,24 @@ static void noinstr __arm64_exit_to_kernel_mode(struct pt_regs *regs,
* We are returning from the context which allows involuntary kernel preemption
*/
static void noinstr arm64_exit_to_kernel_mode_preempt(struct pt_regs *regs,
- irqentry_state_t state)
+ irqentry_state_t state,
+ arm64_exc_hwstate_t hwstate)
{
irqentry_exit_to_kernel_mode_preempt(regs, state);
- __arm64_exit_to_kernel_mode(regs, state);
+ __arm64_exit_to_kernel_mode(regs, state, hwstate);
}
static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
- irqentry_state_t state)
+ irqentry_state_t state,
+ arm64_exc_hwstate_t hwstate)
{
if (!regs_irqs_disabled(regs)) {
- local_irq_disable();
- arm64_exit_to_kernel_mode_preempt(regs, state);
+ hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
+ arm64_exit_to_kernel_mode_preempt(regs, state, hwstate);
return;
}
- __arm64_exit_to_kernel_mode(regs, state);
+ __arm64_exit_to_kernel_mode(regs, state, hwstate);
}
static __always_inline void arm64_syscall_enter_from_user_mode(struct pt_regs *regs)
@@ -141,11 +144,12 @@ static __always_inline void arm64_enter_from_user_mode(struct pt_regs *regs)
sme_enter_from_user_mode();
}
-static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs)
+static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs,
+ arm64_exc_hwstate_t hwstate)
{
- local_irq_disable();
+ hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
syscall_exit_to_user_mode_prepare(regs);
- local_daif_mask();
+ arm64_mask_exc_context(hwstate);
sme_exit_to_user_mode();
mte_check_tfsr_exit();
exit_to_user_mode();
@@ -156,11 +160,12 @@ static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs
* After this function returns it is not safe to call regular kernel code,
* instrumentable code, or any code which may trigger an exception.
*/
-static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
+static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs,
+ arm64_exc_hwstate_t hwstate)
{
- local_irq_disable();
+ hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
irqentry_exit_to_user_mode_prepare(regs);
- local_daif_mask();
+ arm64_mask_exc_context(hwstate);
sme_exit_to_user_mode();
mte_check_tfsr_exit();
exit_to_user_mode();
@@ -168,7 +173,10 @@ static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
asmlinkage void noinstr asm_exit_to_user_mode(struct pt_regs *regs)
{
- arm64_syscall_exit_to_user_mode(regs);
+ arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(PROCESS_CONTEXT);
+
+ arm64_syscall_exit_to_user_mode(regs, hwstate);
+ arm64_debug_exc_context(CRITICAL_CONTEXT);
}
/*
@@ -374,63 +382,69 @@ UNHANDLED(el1t, 64, error)
static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
{
unsigned long far = read_sysreg(far_el1);
+ arm64_exc_hwstate_t hwstate;
irqentry_state_t state;
state = arm64_enter_from_kernel_mode(regs);
- local_daif_inherit(regs);
+ hwstate = arm64_inherit_exc_context(regs);
do_mem_abort(far, esr, regs);
- arm64_exit_to_kernel_mode(regs, state);
+ arm64_exit_to_kernel_mode(regs, state, hwstate);
}
static void noinstr el1_pc(struct pt_regs *regs, unsigned long esr)
{
unsigned long far = read_sysreg(far_el1);
+ arm64_exc_hwstate_t hwstate;
irqentry_state_t state;
state = arm64_enter_from_kernel_mode(regs);
- local_daif_inherit(regs);
+ hwstate = arm64_inherit_exc_context(regs);
do_sp_pc_abort(far, esr, regs);
- arm64_exit_to_kernel_mode(regs, state);
+ arm64_exit_to_kernel_mode(regs, state, hwstate);
}
static void noinstr el1_undef(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
irqentry_state_t state;
state = arm64_enter_from_kernel_mode(regs);
- local_daif_inherit(regs);
+ hwstate = arm64_inherit_exc_context(regs);
do_el1_undef(regs, esr);
- arm64_exit_to_kernel_mode(regs, state);
+ arm64_exit_to_kernel_mode(regs, state, hwstate);
}
static void noinstr el1_bti(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
irqentry_state_t state;
state = arm64_enter_from_kernel_mode(regs);
- local_daif_inherit(regs);
+ hwstate = arm64_inherit_exc_context(regs);
do_el1_bti(regs, esr);
- arm64_exit_to_kernel_mode(regs, state);
+ arm64_exit_to_kernel_mode(regs, state, hwstate);
}
static void noinstr el1_gcs(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
irqentry_state_t state;
state = arm64_enter_from_kernel_mode(regs);
- local_daif_inherit(regs);
+ hwstate = arm64_inherit_exc_context(regs);
do_el1_gcs(regs, esr);
- arm64_exit_to_kernel_mode(regs, state);
+ arm64_exit_to_kernel_mode(regs, state, hwstate);
}
static void noinstr el1_mops(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
irqentry_state_t state;
state = arm64_enter_from_kernel_mode(regs);
- local_daif_inherit(regs);
+ hwstate = arm64_inherit_exc_context(regs);
do_el1_mops(regs, esr);
- arm64_exit_to_kernel_mode(regs, state);
+ arm64_exit_to_kernel_mode(regs, state, hwstate);
}
static void noinstr el1_breakpt(struct pt_regs *regs, unsigned long esr)
@@ -490,12 +504,13 @@ static void noinstr el1_brk64(struct pt_regs *regs, unsigned long esr)
static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
irqentry_state_t state;
state = arm64_enter_from_kernel_mode(regs);
- local_daif_inherit(regs);
+ hwstate = arm64_inherit_exc_context(regs);
do_el1_fpac(regs, esr);
- arm64_exit_to_kernel_mode(regs, state);
+ arm64_exit_to_kernel_mode(regs, state, hwstate);
}
asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
@@ -545,16 +560,20 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
default:
__panic_unhandled(regs, "64-bit el1h sync", esr);
}
+
+ arm64_debug_exc_context(CRITICAL_CONTEXT);
}
static __always_inline void __el1_pnmi(struct pt_regs *regs,
void (*handler)(struct pt_regs *))
{
+ arm64_exc_hwstate_t hwstate;
irqentry_state_t state;
state = irqentry_nmi_enter(regs);
+ hwstate = arm64_unmask_exc_context(NONMI_CONTEXT);
do_interrupt_handler(regs, handler);
- local_daif_mask();
+ arm64_mask_exc_context(hwstate);
irqentry_nmi_exit(regs, state);
}
@@ -565,21 +584,32 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
state = arm64_enter_from_kernel_mode(regs);
+ arm64_unmask_exc_context(NONMI_CONTEXT);
+
irq_enter_rcu();
do_interrupt_handler(regs, handler);
irq_exit_rcu();
- arm64_exit_to_kernel_mode_preempt(regs, state);
+ /*
+ * If pseudo-NMIs are enabled and the interrupted context had
+ * IRQs unmasked, the interrupt handler will have cleared DAIF
+ * and switched to PMR masking in order to handle
+ * NMIs. Otherwise it would keep IF. In both cases on return
+ * we effectivly have NOIRQ_CONTEXT - keep track of it
+ */
+ arm64_debug_exc_context(NOIRQ_CONTEXT);
+ arm64_exit_to_kernel_mode_preempt(regs, state, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
}
+
static void noinstr el1_interrupt(struct pt_regs *regs,
void (*handler)(struct pt_regs *))
{
- write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
-
if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && regs_irqs_disabled(regs))
__el1_pnmi(regs, handler);
else
__el1_irq(regs, handler);
+
+ arm64_debug_exc_context(CRITICAL_CONTEXT);
}
asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
@@ -595,28 +625,31 @@ asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
{
unsigned long esr = read_sysreg(esr_el1);
+ arm64_exc_hwstate_t hwstate;
irqentry_state_t state;
- local_daif_restore(DAIF_ERRCTX);
state = irqentry_nmi_enter(regs);
+ hwstate = arm64_unmask_exc_context(ERROR_CONTEXT);
do_serror(regs, esr);
- local_daif_mask();
+ arm64_mask_exc_context(hwstate);
irqentry_nmi_exit(regs, state);
}
static void noinstr el0_da(struct pt_regs *regs, unsigned long esr)
{
unsigned long far = read_sysreg(far_el1);
+ arm64_exc_hwstate_t hwstate;
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_mem_abort(far, esr, regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
{
unsigned long far = read_sysreg(far_el1);
+ arm64_exc_hwstate_t hwstate;
/*
* We've taken an instruction abort from userspace and not yet
@@ -627,114 +660,139 @@ static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
arm64_apply_bp_hardening();
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_mem_abort(far, esr, regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_fpsimd_acc(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_fpsimd_acc(esr, regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_sve_acc(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_sve_acc(esr, regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_sme_acc(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_sme_acc(esr, regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_fpsimd_exc(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_fpsimd_exc(esr, regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_sys(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_el0_sys(esr, regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_pc(struct pt_regs *regs, unsigned long esr)
{
unsigned long far = read_sysreg(far_el1);
+ arm64_exc_hwstate_t hwstate;
if (!is_ttbr0_addr(instruction_pointer(regs)))
arm64_apply_bp_hardening();
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_sp_pc_abort(far, esr, regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_sp(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_sp_pc_abort(regs->sp, esr, regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_undef(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_el0_undef(regs, esr);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_bti(struct pt_regs *regs)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_el0_bti(regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_el0_mops(regs, esr);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_gcs(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_el0_gcs(regs, esr);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_inv(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
bad_el0_sync(regs, 0, esr);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
if (!is_ttbr0_addr(regs->pc))
arm64_apply_bp_hardening();
@@ -742,12 +800,13 @@ static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
debug_exception_enter(regs);
do_breakpoint(esr, regs);
debug_exception_exit(regs);
- local_daif_restore(DAIF_PROCCTX);
- arm64_exit_to_user_mode(regs);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
bool step_done;
if (!is_ttbr0_addr(regs->pc))
@@ -761,50 +820,56 @@ static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
* the single-step is complete.
*/
step_done = try_step_suspended_breakpoints(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
if (!step_done)
do_el0_softstep(esr, regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_watchpt(struct pt_regs *regs, unsigned long esr)
{
/* Watchpoints are the only debug exception to write FAR_EL1 */
unsigned long far = read_sysreg(far_el1);
+ arm64_exc_hwstate_t hwstate;
arm64_enter_from_user_mode(regs);
debug_exception_enter(regs);
do_watchpoint(far, esr, regs);
debug_exception_exit(regs);
- local_daif_restore(DAIF_PROCCTX);
- arm64_exit_to_user_mode(regs);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_brk64(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_el0_brk64(esr, regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_svc(struct pt_regs *regs)
{
+ arm64_exc_hwstate_t hwstate;
arm64_syscall_enter_from_user_mode(regs);
cortex_a76_erratum_1463225_svc_handler();
fpsimd_syscall_enter();
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_el0_svc(regs);
- arm64_syscall_exit_to_user_mode(regs);
+ arm64_syscall_exit_to_user_mode(regs, hwstate);
fpsimd_syscall_exit();
}
static void noinstr el0_fpac(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_el0_fpac(regs, esr);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
@@ -873,6 +938,8 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
default:
el0_inv(regs, esr);
}
+
+ arm64_debug_exc_context(CRITICAL_CONTEXT);
}
static void noinstr el0_interrupt(struct pt_regs *regs,
@@ -880,7 +947,7 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
{
arm64_enter_from_user_mode(regs);
- write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
+ arm64_unmask_exc_context(NONMI_CONTEXT);
if (regs->pc & BIT(55))
arm64_apply_bp_hardening();
@@ -889,7 +956,14 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
do_interrupt_handler(regs, handler);
irq_exit_rcu();
- arm64_exit_to_user_mode(regs);
+ /*
+ * For the same reason as in el1_irq() we effectivly
+ * have NOIRQ_CONTEXT on return from handler - keep
+ * track of it
+ */
+ arm64_debug_exc_context(NOIRQ_CONTEXT);
+ arm64_exit_to_user_mode(regs, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
+ arm64_debug_exc_context(CRITICAL_CONTEXT);
}
static void noinstr __el0_irq_handler_common(struct pt_regs *regs)
@@ -915,15 +989,17 @@ asmlinkage void noinstr el0t_64_fiq_handler(struct pt_regs *regs)
static void noinstr __el0_error_handler_common(struct pt_regs *regs)
{
unsigned long esr = read_sysreg(esr_el1);
+ arm64_exc_hwstate_t hwstate;
irqentry_state_t state;
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_ERRCTX);
+ hwstate = arm64_unmask_exc_context(ERROR_CONTEXT);
state = irqentry_nmi_enter(regs);
do_serror(regs, esr);
irqentry_nmi_exit(regs, state);
- local_daif_restore(DAIF_PROCCTX);
- arm64_exit_to_user_mode(regs);
+ hwstate = arm64_drop_exc_context(hwstate, PROCESS_CONTEXT);
+ arm64_exit_to_user_mode(regs, hwstate);
+ arm64_debug_exc_context(CRITICAL_CONTEXT);
}
asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
@@ -934,27 +1010,33 @@ asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
#ifdef CONFIG_COMPAT
static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_el0_cp15(esr, regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_svc_compat(struct pt_regs *regs)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_syscall_enter_from_user_mode(regs);
cortex_a76_erratum_1463225_svc_handler();
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_el0_svc_compat(regs);
- arm64_syscall_exit_to_user_mode(regs);
+ arm64_syscall_exit_to_user_mode(regs, hwstate);
}
static void noinstr el0_bkpt32(struct pt_regs *regs, unsigned long esr)
{
+ arm64_exc_hwstate_t hwstate;
+
arm64_enter_from_user_mode(regs);
- local_daif_restore(DAIF_PROCCTX);
+ hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
do_bkpt32(esr, regs);
- arm64_exit_to_user_mode(regs);
+ arm64_exit_to_user_mode(regs, hwstate);
}
asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
@@ -1005,6 +1087,8 @@ asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
default:
el0_inv(regs, esr);
}
+
+ arm64_debug_exc_context(CRITICAL_CONTEXT);
}
asmlinkage void noinstr el0t_32_irq_handler(struct pt_regs *regs)
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index 8d17f61d1b5b..024b740512b0 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -328,7 +328,7 @@ alternative_else_nop_endif
mrs_s x20, SYS_ICC_PMR_EL1
str w20, [sp, #S_PMR]
- mov x20, #GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET
+ mov x20, #GIC_PRIO_IRQON
msr_s SYS_ICC_PMR_EL1, x20
.Lskip_pmr_save\@:
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 014b163ce658..0e3d0fa25dd2 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -187,7 +187,7 @@ static void init_gic_priority_masking(void)
WARN_ON(!(cpuflags & PSR_I_BIT));
WARN_ON(!(cpuflags & PSR_F_BIT));
- gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
+ gic_write_pmr(GIC_PRIO_IRQON);
}
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 07/39] arm64: interrupts: Introduce exception masking save/restore helpers
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (5 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 06/39] arm64: entry: replace DAIF helpers with entry helpers Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 08/39] arm64: interrupts: introduce a helper for GIC priority initialization Vladimir Murzin
` (34 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Outside the entry code, the local_daif_... helpers need to be replaced
so that both DAIF and PMR are handled correctly and their use can be
controlled more strongly.
Introduce local_exceptions_save_mask() and local_exceptions_restore(),
which must be used as a pair. Save both the original and requested
exception state so that inconsistent changes between the save and
restore can be detected.
There are two cases where paired save and restore operations do not
make sense:
- when initializing a CPU, and
- when preparing to turn it off.
Handle those cases with local_exceptions_cpu_init_mask() and
local_exceptions_final_mask(). When CONFIG_DEBUG_IRQFLAGS is enabled,
enforce that these helpers are not used outside their intended
contexts.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/interrupts/masking.h | 92 +++++++++++++++++++++
arch/arm64/kernel/irq.c | 31 +++++++
2 files changed, 123 insertions(+)
create mode 100644 arch/arm64/include/asm/interrupts/masking.h
diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
new file mode 100644
index 000000000000..db3439853b64
--- /dev/null
+++ b/arch/arm64/include/asm/interrupts/masking.h
@@ -0,0 +1,92 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2025 Arm Ltd.
+ */
+#ifndef __ASM_INTERRUPTS_MASKING_H
+#define __ASM_INTERRUPTS_MASKING_H
+
+#include <asm/arch_gicv3.h>
+#include <asm/bug.h>
+#include <asm/cpufeature.h>
+#include <asm/interrupts/common_flags.h>
+#include <asm/ptrace.h>
+
+typedef struct arm64_exc_hwstates {
+ arm64_exc_hwstate_t saved;
+ arm64_exc_hwstate_t expected;
+} arm64_exc_hwstates_t;
+
+static inline
+arm64_exc_hwstates_t local_exceptions_save_mask(arm64_exc_context_t new)
+{
+ arm64_exc_hwstate_t actual = {.flags = arch_local_save_flags()};
+ arm64_exc_hwstate_t state = arm64_exc_hwstate_of_context(new);
+ bool irqs_disabled = arch_irqs_disabled_flags(state.flags);
+ bool force;
+
+ /*
+ * We've just got actual HW state so we can rely on that to
+ * optimize some unnecessary updates.
+ */
+ force = system_uses_irq_prio_masking() && actual.pmr != state.pmr;
+
+ if (!irqs_disabled)
+ trace_hardirqs_on();
+
+ __arm64_update_exc_hwstate(state, force);
+
+ if (irqs_disabled)
+ trace_hardirqs_off();
+
+ return (arm64_exc_hwstates_t){.saved = actual, .expected = state};
+}
+
+static inline void local_exceptions_restore(arm64_exc_hwstates_t states)
+{
+ bool irqs_disabled = arch_irqs_disabled_flags(states.saved.flags);
+
+ arm64_debug_exc_hwstate(states.expected);
+
+ if (!irqs_disabled)
+ trace_hardirqs_on();
+
+ arm64_update_exc_hwstate(states.saved);
+
+ if (irqs_disabled)
+ trace_hardirqs_off();
+}
+
+static __always_inline
+void __local_exceptions_cpu_init_mask(arm64_exc_context_t context)
+{
+ if (context == PROCESS_CONTEXT)
+ trace_hardirqs_on();
+
+ arm64_update_exc_context(context);
+
+ if (context != PROCESS_CONTEXT)
+ trace_hardirqs_off();
+}
+
+static __always_inline void __local_exceptions_final_mask(void)
+{
+ arm64_update_exc_context(CRITICAL_CONTEXT);
+ trace_hardirqs_off();
+}
+
+#ifdef CONFIG_DEBUG_IRQFLAGS
+void local_exceptions_cpu_init_mask(arm64_exc_context_t context);
+void local_exceptions_final_mask(void);
+#else
+static inline
+void local_exceptions_cpu_init_mask(arm64_exc_context_t context)
+{
+ __local_exceptions_cpu_init_mask(context);
+}
+
+static inline void local_exceptions_final_mask(void)
+{
+ __local_exceptions_final_mask();
+}
+#endif /* CONFIG_DEBUG_IRQFLAGS */
+#endif /* __ASM_INTERRUPTS_MASKING_H */
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index 9fafd826002b..58b778083576 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -22,6 +22,7 @@
#include <linux/smp.h>
#include <linux/vmalloc.h>
#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
#include <asm/exception.h>
#include <asm/numa.h>
#include <asm/softirq_stack.h>
@@ -39,6 +40,12 @@ DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr);
DEFINE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr);
#endif
+#ifdef CONFIG_DEBUG_IRQFLAGS
+/* Make sure the CPU init/tear down masking functions are only used once. */
+static cpumask_t irqs_masks_init_done;
+static cpumask_t irqs_masks_final_done;
+#endif
+
static int __init init_irq_scs(void)
{
int cpu;
@@ -133,3 +140,27 @@ void __init init_IRQ(void)
local_daif_restore(DAIF_PROCCTX_NOIRQ);
}
}
+
+#ifdef CONFIG_DEBUG_IRQFLAGS
+void local_exceptions_cpu_init_mask(arm64_exc_context_t context)
+{
+ int cpu = smp_processor_id();
+
+ WARN_ON(cpumask_test_cpu(cpu, &irqs_masks_init_done));
+
+ __local_exceptions_cpu_init_mask(context);
+ cpumask_set_cpu(cpu, &irqs_masks_init_done);
+ cpumask_clear_cpu(cpu, &irqs_masks_final_done);
+}
+
+void local_exceptions_final_mask(void)
+{
+ int cpu = smp_processor_id();
+
+ WARN_ON(cpumask_test_cpu(cpu, &irqs_masks_final_done));
+
+ __local_exceptions_final_mask();
+ cpumask_set_cpu(cpu, &irqs_masks_final_done);
+ cpumask_clear_cpu(cpu, &irqs_masks_init_done);
+}
+#endif
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 08/39] arm64: interrupts: introduce a helper for GIC priority initialization
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (6 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 07/39] arm64: interrupts: Introduce exception masking save/restore helpers Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 09/39] arm64: replace local_daif helpers Vladimir Murzin
` (33 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Arm64's `init_IRQ()` calls `local_daif_restore()` to synchronize
interrupt masking via DAIF and switch to masking via PMR. This
depends on a very specific behaviour of `local_daif_restore()` which
will clear DAIF if masking interrupts via PMR, which will get removed
in the future.
As `setup_arch()` cleared DA only earlier, introduce a dedicated
helper that explicitly initializes the PMR to mask interrupts and
clears DAIF, switching to IRQ priority masking.
Given it is a dedicated helper, add a lockdep assertion as
`setup_arch()` should always have called `trace_hardirqs_off()` when
clearing DA, otherwise something bad happened.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/interrupts/masking.h | 19 +++++++++++++++++++
arch/arm64/kernel/irq.c | 14 +++++---------
2 files changed, 24 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
index db3439853b64..9e589bf11ec8 100644
--- a/arch/arm64/include/asm/interrupts/masking.h
+++ b/arch/arm64/include/asm/interrupts/masking.h
@@ -5,6 +5,8 @@
#ifndef __ASM_INTERRUPTS_MASKING_H
#define __ASM_INTERRUPTS_MASKING_H
+#include <linux/lockdep.h>
+
#include <asm/arch_gicv3.h>
#include <asm/bug.h>
#include <asm/cpufeature.h>
@@ -89,4 +91,21 @@ static inline void local_exceptions_final_mask(void)
__local_exceptions_final_mask();
}
#endif /* CONFIG_DEBUG_IRQFLAGS */
+
+/*
+ * During early boot, we unmask PSR.DA before the GIC has been set up.
+ * If we use IRQ priority masking, the PMR and PSR will be out of sync
+ * after the GIC is enabled : sync them up.
+ */
+static inline void local_interrupt_priority_init(void)
+{
+ if (!system_uses_irq_prio_masking())
+ return;
+
+ WARN_ON(read_sysreg(daif) & PSR_A_BIT);
+ lockdep_assert_irqs_disabled();
+
+ arm64_update_exc_context(NOIRQ_CONTEXT);
+}
+
#endif /* __ASM_INTERRUPTS_MASKING_H */
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index 58b778083576..a3d9128ba195 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -21,7 +21,6 @@
#include <linux/seq_file.h>
#include <linux/smp.h>
#include <linux/vmalloc.h>
-#include <asm/daifflags.h>
#include <asm/interrupts/masking.h>
#include <asm/exception.h>
#include <asm/numa.h>
@@ -131,14 +130,11 @@ void __init init_IRQ(void)
irqchip_init();
- if (system_uses_irq_prio_masking()) {
- /*
- * Now that we have a stack for our IRQ handler, set
- * the PMR/PSR pair to a consistent state.
- */
- WARN_ON(read_sysreg(daif) & PSR_A_BIT);
- local_daif_restore(DAIF_PROCCTX_NOIRQ);
- }
+ /*
+ * Now that we have a stack for our IRQ handler,
+ * let's mask interrupts via the PMR.
+ */
+ local_interrupt_priority_init();
}
#ifdef CONFIG_DEBUG_IRQFLAGS
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 09/39] arm64: replace local_daif helpers
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (7 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 08/39] arm64: interrupts: introduce a helper for GIC priority initialization Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 10/39] arm64: cpuidle: use new helpers to bypass interrupt priority masking Vladimir Murzin
` (32 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Replace existing `local_daif_...` usage with the new `local_exceptions_...`
helpers, adding the necessary save-state struct where required.
Rework `apei_claim_sea()`'s handling of interrupt masking to
fit with the new APIs. Currently, it saves the DAIF flags to restore
later and the `local_irq` ones to check if interrupts are disabled.
It then masks AIF to signal the SEA (forcing DAIF masking), and will
unmask to IF (re-enabling PMR masking if pseudo-NMIs are in use) if it
interrupted a task with interrupts unmaksed, restoring with the DAIF
flags from the beginning afterwards. It is not clear why masking has
to be nested, so take an opportunity to do masking sequentially:
mask/unmask AIF around the SEA signaling and mask/unmask IRQ if we
interrupted a task with interrupts unmaksed.
Remove the use of GIC_PRIO_PSR_I_SET in `kvm/hyp/nvhe/switch.c`,
as we now check both PMR and DAIF everywhere and we don't need to signal
interrupts being masked in DAIF in the PMR anymore.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
arch/arm64/kernel/acpi.c | 43 ++++++++++++++-----------------
arch/arm64/kernel/hibernate.c | 14 +++++-----
arch/arm64/kernel/hw_breakpoint.c | 16 +++++++-----
arch/arm64/kernel/machine_kexec.c | 4 +--
arch/arm64/kernel/setup.c | 4 +--
arch/arm64/kernel/smp.c | 10 +++----
arch/arm64/kernel/suspend.c | 14 +++++-----
arch/arm64/kvm/hyp/nvhe/switch.c | 2 +-
arch/arm64/kvm/hyp/vgic-v3-sr.c | 7 ++---
arch/arm64/kvm/hyp/vhe/switch.c | 14 +++++-----
arch/arm64/mm/mmu.c | 7 ++---
11 files changed, 68 insertions(+), 67 deletions(-)
diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
index 681aa2bbc399..b488e04249d0 100644
--- a/arch/arm64/kernel/acpi.c
+++ b/arch/arm64/kernel/acpi.c
@@ -33,7 +33,7 @@
#include <acpi/processor.h>
#include <asm/cputype.h>
#include <asm/cpu_ops.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
#include <asm/smp_plat.h>
int acpi_noirq = 1; /* skip ACPI IRQ initialization */
@@ -388,48 +388,43 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
*/
int apei_claim_sea(struct pt_regs *regs)
{
+ arm64_exc_hwstates_t hwstate;
+ bool irqs_disabled;
int err = -ENOENT;
- bool return_to_irqs_enabled;
- unsigned long current_flags;
if (!IS_ENABLED(CONFIG_ACPI_APEI_GHES))
return err;
- current_flags = local_daif_save_flags();
-
- /* current_flags isn't useful here as daif doesn't tell us about pNMI */
- return_to_irqs_enabled = !irqs_disabled_flags(arch_local_save_flags());
-
- if (regs)
- return_to_irqs_enabled = !regs_irqs_disabled(regs);
-
/*
* SEA can interrupt SError, mask it and describe this as an NMI so
* that APEI defers the handling.
*/
- local_daif_restore(DAIF_ERRCTX);
+ hwstate = local_exceptions_save_mask(ERROR_CONTEXT);
nmi_enter();
err = ghes_notify_sea();
nmi_exit();
+ local_exceptions_restore(hwstate);
+
+ if (err)
+ return err;
/*
* APEI NMI-like notifications are deferred to irq_work. Unless
* we interrupted irqs-masked code, we can do that now.
*/
- if (!err) {
- if (return_to_irqs_enabled) {
- local_daif_restore(DAIF_PROCCTX_NOIRQ);
- __irq_enter();
- irq_work_run();
- __irq_exit();
- } else {
- pr_warn_ratelimited("APEI work queued but not completed\n");
- err = -EINPROGRESS;
- }
+ irqs_disabled = regs ? regs_irqs_disabled(regs) : arch_irqs_disabled_flags(hwstate.saved.flags);
+
+ if (!irqs_disabled) {
+ hwstate = local_exceptions_save_mask(NOIRQ_CONTEXT);
+ __irq_enter();
+ irq_work_run();
+ __irq_exit();
+ local_exceptions_restore(hwstate);
+ } else {
+ pr_warn_ratelimited("APEI work queued but not completed\n");
+ err = -EINPROGRESS;
}
- local_daif_restore(current_flags);
-
return err;
}
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 8c39863d2d63..7712b6e87797 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -22,7 +22,7 @@
#include <asm/barrier.h>
#include <asm/cacheflush.h>
#include <asm/cputype.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
#include <asm/irqflags.h>
#include <asm/kexec.h>
#include <asm/memory.h>
@@ -346,16 +346,16 @@ static void swsusp_mte_restore_tags(void)
int swsusp_arch_suspend(void)
{
- int ret = 0;
- unsigned long flags;
struct sleep_stack_data state;
+ arm64_exc_hwstates_t hwstate;
+ int ret = 0;
if (cpus_are_stuck_in_kernel()) {
pr_err("Can't hibernate: no mechanism to offline secondary CPUs.\n");
return -EBUSY;
}
- flags = local_daif_save();
+ hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
if (__cpu_suspend_enter(&state)) {
/* make the crash dump kernel image visible/saveable */
@@ -363,7 +363,7 @@ int swsusp_arch_suspend(void)
ret = swsusp_mte_save_tags();
if (ret) {
- local_daif_restore(flags);
+ local_exceptions_restore(hwstate);
return ret;
}
@@ -407,7 +407,7 @@ int swsusp_arch_suspend(void)
spectre_v4_enable_mitigation(NULL);
}
- local_daif_restore(flags);
+ local_exceptions_restore(hwstate);
return ret;
}
@@ -491,7 +491,7 @@ int __nocfi swsusp_arch_resume(void)
* swsusp_arch_resume(), and expects to be re-entered in the
* same state : with all DAIF exceptions masked.
*/
- local_daif_save();
+ local_exceptions_save_mask(CRITICAL_CONTEXT);
hibernate_exit(virt_to_phys(tmp_pg_dir), resume_hdr.ttbr1_el1,
resume_hdr.reenter_kernel, restore_pblist,
resume_hdr.__hyp_stub_vectors, virt_to_phys(zero_page));
diff --git a/arch/arm64/kernel/hw_breakpoint.c b/arch/arm64/kernel/hw_breakpoint.c
index 32b38967433e..f20f37d9d602 100644
--- a/arch/arm64/kernel/hw_breakpoint.c
+++ b/arch/arm64/kernel/hw_breakpoint.c
@@ -20,11 +20,11 @@
#include <linux/uaccess.h>
#include <asm/current.h>
-#include <asm/daifflags.h>
#include <asm/debug-monitors.h>
#include <asm/esr.h>
#include <asm/exception.h>
#include <asm/hw_breakpoint.h>
+#include <asm/interrupts/masking.h>
#include <asm/traps.h>
#include <asm/cputype.h>
#include <asm/system_misc.h>
@@ -299,18 +299,18 @@ static int hw_breakpoint_control(struct perf_event *bp,
*/
int arch_install_hw_breakpoint(struct perf_event *bp)
{
- unsigned long daif = local_daif_save();
+ arm64_exc_hwstates_t hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
int ret = hw_breakpoint_control(bp, HW_BREAKPOINT_INSTALL);
- local_daif_restore(daif);
+ local_exceptions_restore(hwstate);
return ret;
}
void arch_uninstall_hw_breakpoint(struct perf_event *bp)
{
- unsigned long daif = local_daif_save();
+ arm64_exc_hwstates_t hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
hw_breakpoint_control(bp, HW_BREAKPOINT_UNINSTALL);
- local_daif_restore(daif);
+ local_exceptions_restore(hwstate);
}
static int get_hbp_len(u8 hbp_len)
@@ -945,7 +945,9 @@ void hw_breakpoint_thread_switch(struct task_struct *next)
*/
struct debug_info *current_debug_info, *next_debug_info;
- unsigned long daif = local_daif_save();
+ arm64_exc_hwstates_t hwstate;
+
+ hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
current_debug_info = ¤t->thread.debug;
next_debug_info = &next->thread.debug;
@@ -962,7 +964,7 @@ void hw_breakpoint_thread_switch(struct task_struct *next)
DBG_ACTIVE_EL0,
!next_debug_info->wps_disabled);
- local_daif_restore(daif);
+ local_exceptions_restore(hwstate);
}
/*
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 48f343704cb5..91f0d1e99db5 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -18,7 +18,7 @@
#include <asm/cacheflush.h>
#include <asm/cpu_ops.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
#include <asm/memory.h>
#include <asm/mmu.h>
#include <asm/mmu_context.h>
@@ -185,7 +185,7 @@ void machine_kexec(struct kimage *kimage)
pr_info("Bye!\n");
- local_daif_mask();
+ local_exceptions_final_mask();
/*
* Both restart and kernel_reloc will shutdown the MMU, disable data
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index a707deba6ae7..49914965b790 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -38,7 +38,7 @@
#include <asm/fixmap.h>
#include <asm/cpu.h>
#include <asm/cputype.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
#include <asm/elf.h>
#include <asm/cpufeature.h>
#include <asm/cpu_ops.h>
@@ -311,7 +311,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
* IRQ and FIQ will be unmasked after the root irqchip has been
* detected and initialized.
*/
- local_daif_restore(DAIF_PROCCTX_NOIRQ);
+ local_exceptions_cpu_init_mask(NOIRQ_CONTEXT);
/*
* TTBR0 is only used for the identity mapping at this stage. Make it
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 0e3d0fa25dd2..cb04f2c37d9a 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -43,7 +43,7 @@
#include <asm/cpu.h>
#include <asm/cputype.h>
#include <asm/cpu_ops.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
#include <asm/kvm_mmu.h>
#include <asm/mmu_context.h>
#include <asm/nmi.h>
@@ -267,7 +267,7 @@ asmlinkage notrace void secondary_start_kernel(void)
* as the root irqchip has already been detected and initialized we can
* unmask IRQ and FIQ at the same time.
*/
- local_daif_restore(DAIF_PROCCTX);
+ local_exceptions_cpu_init_mask(PROCESS_CONTEXT);
/*
* OK, it's off to the idle thread for us
@@ -374,7 +374,7 @@ void __noreturn cpu_die(void)
idle_task_exit();
- local_daif_mask();
+ local_exceptions_final_mask();
/* Tell cpuhp_bp_sync_dead() that this CPU is now safe to dispose of */
cpuhp_ap_report_dead();
@@ -898,14 +898,14 @@ void __noreturn arm64_nmi_cpu_stop(struct pt_regs *regs, bool die_on_crash)
bool crash = IS_ENABLED(CONFIG_KEXEC_CORE) && crash_stop;
/*
- * Use local_daif_mask() instead of local_irq_disable() to make sure
+ * Use local_exceptions_final_mask() instead of local_irq_disable() to make sure
* that pseudo-NMIs are disabled. The "stop" code starts with an IRQ
* and falls back to NMI (which might be pseudo). If the IRQ finally
* goes through right as we're timing out then the NMI could interrupt
* us. It's better to prevent the NMI and let the IRQ finish since the
* pt_regs will be better.
*/
- local_daif_mask();
+ local_exceptions_final_mask();
#ifdef CONFIG_KEXEC_CORE
if (crash && regs)
diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c
index 3691c4c32904..41cf580a7780 100644
--- a/arch/arm64/kernel/suspend.c
+++ b/arch/arm64/kernel/suspend.c
@@ -9,7 +9,7 @@
#include <asm/cacheflush.h>
#include <asm/cpufeature.h>
#include <asm/cpuidle.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
#include <asm/debug-monitors.h>
#include <asm/exec.h>
#include <asm/fpsimd.h>
@@ -69,7 +69,7 @@ void notrace __cpu_suspend_exit(void)
/*
* Restore HW breakpoint registers to sane values
* before debug exceptions are possibly reenabled
- * by cpu_suspend()s local_daif_restore() call.
+ * by cpu_suspend()s local_exceptions_restore() call.
*/
if (hw_breakpoint_restore)
hw_breakpoint_restore(cpu);
@@ -97,7 +97,7 @@ void notrace __cpu_suspend_exit(void)
int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
{
int ret = 0;
- unsigned long flags;
+ arm64_exc_hwstates_t hwstate;
struct sleep_stack_data state;
/*
@@ -119,12 +119,12 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
*
* Strictly speaking the trace_hardirqs_off() here is superfluous,
* hardirqs should be firmly off by now. This really ought to use
- * something like raw_local_daif_save().
+ * something like raw_local_exceptions_save_mask().
*
* This also unmasks interrupts in PMR in order to reliably
* resume if we're using pseudo-NMIs.
*/
- flags = local_daif_save();
+ hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
/*
* Function graph tracer state gets inconsistent when the kernel
@@ -158,11 +158,11 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
unpause_graph_tracing();
/*
- * Restore pstate flags. OS lock and mdscr have been already
+ * Restore interrupt masks. OS lock and mdscr have been already
* restored, so from this point onwards, debugging is fully
* reenabled if it was enabled when core started shutdown.
*/
- local_daif_restore(flags);
+ local_exceptions_restore(hwstate);
return ret;
}
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index 89fb9dd351ad..244260365d63 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -291,7 +291,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
* Naturally, we want to avoid this.
*/
if (system_uses_irq_prio_masking()) {
- gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
+ gic_write_pmr(GIC_PRIO_IRQON);
pmr_sync();
}
diff --git a/arch/arm64/kvm/hyp/vgic-v3-sr.c b/arch/arm64/kvm/hyp/vgic-v3-sr.c
index 2dfb5ff9db57..4e570a1ad739 100644
--- a/arch/arm64/kvm/hyp/vgic-v3-sr.c
+++ b/arch/arm64/kvm/hyp/vgic-v3-sr.c
@@ -10,6 +10,7 @@
#include <linux/irqchip/arm-gic-v3.h>
#include <linux/kvm_host.h>
+#include <asm/interrupts/masking.h>
#include <asm/kvm_emulate.h>
#include <asm/kvm_hyp.h>
#include <asm/kvm_mmu.h>
@@ -440,8 +441,8 @@ void __vgic_v3_init_lrs(void)
/* Return true if GICv3 is MMIO (GICv2) capable, false otherwise */
bool __vgic_v3_get_gic_config(void)
{
+ arm64_exc_hwstates_t hwstate;
u64 val, sre;
- unsigned long flags = 0;
sre = read_gicreg(ICC_SRE_EL1);
/*
@@ -462,7 +463,7 @@ bool __vgic_v3_get_gic_config(void)
* of the exception entry to EL2.
*/
if (has_vhe()) {
- flags = local_daif_save();
+ hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
} else {
sysreg_clear_set_hcr(0, HCR_EL2_AMO | HCR_EL2_FMO | HCR_EL2_IMO);
isb();
@@ -477,7 +478,7 @@ bool __vgic_v3_get_gic_config(void)
isb();
if (has_vhe()) {
- local_daif_restore(flags);
+ local_exceptions_restore(hwstate);
} else {
sysreg_clear_set_hcr(HCR_EL2_AMO | HCR_EL2_FMO | HCR_EL2_IMO, 0);
isb();
diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
index 3f2390a13173..31da573fb169 100644
--- a/arch/arm64/kvm/hyp/vhe/switch.c
+++ b/arch/arm64/kvm/hyp/vhe/switch.c
@@ -17,6 +17,7 @@
#include <asm/barrier.h>
#include <asm/cpufeature.h>
+#include <asm/interrupts/masking.h>
#include <asm/kprobes.h>
#include <asm/kvm_asm.h>
#include <asm/kvm_emulate.h>
@@ -666,9 +667,10 @@ NOKPROBE_SYMBOL(__kvm_vcpu_run_vhe);
int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
{
+ arm64_exc_hwstates_t hwstate;
int ret;
- local_daif_mask();
+ hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
/*
* Having IRQs masked via PMR when entering the guest means the GIC
@@ -676,18 +678,18 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
* only way to get out will be via guest exceptions.
* Naturally, we want to avoid this.
*
- * local_daif_mask() already sets GIC_PRIO_PSR_I_SET, we just need a
- * dsb to ensure the redistributor is forwards EL2 IRQs to the CPU.
+ * local_exceptions_save_mask(), via helpers, sets
+ * GIC_PRIO_IRQON along with pmr_sync(), so we can be shure
+ * that redistributor forwards EL2 IRQs to the CPU.
*/
- pmr_sync();
ret = __kvm_vcpu_run_vhe(vcpu);
/*
- * local_daif_restore() takes care to properly restore PSTATE.DAIF
+ * local_exceptions_restore() takes care to properly restore PSTATE.DAIF
* and the GIC PMR if the host is using IRQ priorities.
*/
- local_daif_restore(DAIF_PROCCTX_NOIRQ);
+ local_exceptions_restore(hwstate);
return ret;
}
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 140520b351e2..1267f49815e6 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -33,6 +33,7 @@
#include <asm/barrier.h>
#include <asm/cputype.h>
+#include <asm/interrupts/masking.h>
#include <asm/fixmap.h>
#include <asm/kasan.h>
#include <asm/kernel-pgtable.h>
@@ -2329,7 +2330,7 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
typedef void (ttbr_replace_func)(phys_addr_t);
extern ttbr_replace_func idmap_cpu_replace_ttbr1;
ttbr_replace_func *replace_phys;
- unsigned long daif;
+ arm64_exc_hwstates_t hwstate;
/* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */
phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp));
@@ -2345,9 +2346,9 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
* We really don't want to take *any* exceptions while TTBR1 is
* in the process of being replaced so mask everything.
*/
- daif = local_daif_save();
+ hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
replace_phys(ttbr1);
- local_daif_restore(daif);
+ local_exceptions_restore(hwstate);
cpu_uninstall_idmap();
}
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 10/39] arm64: cpuidle: use new helpers to bypass interrupt priority masking
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (8 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 09/39] arm64: replace local_daif helpers Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 11/39] arm64: remove daifflags.h Vladimir Murzin
` (31 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
When the CPU goes into WFI and pseudo-NMIs are in use, we need to make
sure that interrupts are not masked by the PMR otherwise they would not
reach the CPU and wake it up.
Now that we have a proper API that handles both DAIF and PMR, extend it
to provide helpers allowing to temporarily switch to masking interrupts
via DAIF only when interrupt priorities are in use.
This could allow other parts of the code to make use of it and makes it
easier to check for proper use.
Replace the custom cpuidle helper with those new helpers.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/cpuidle.h | 18 ++-----
arch/arm64/include/asm/interrupts/masking.h | 54 +++++++++++++++++++++
2 files changed, 58 insertions(+), 14 deletions(-)
diff --git a/arch/arm64/include/asm/cpuidle.h b/arch/arm64/include/asm/cpuidle.h
index 2047713e097d..b70ef5fd5e57 100644
--- a/arch/arm64/include/asm/cpuidle.h
+++ b/arch/arm64/include/asm/cpuidle.h
@@ -5,32 +5,22 @@
#include <asm/proc-fns.h>
#ifdef CONFIG_ARM64_PSEUDO_NMI
-#include <asm/arch_gicv3.h>
+#include <asm/interrupts/masking.h>
struct arm_cpuidle_irq_context {
- unsigned long pmr;
- unsigned long daif_bits;
+ arm64_exc_hwstates_t arm64_context;
};
#define arm_cpuidle_save_irq_context(__c) \
do { \
struct arm_cpuidle_irq_context *c = __c; \
- if (system_uses_irq_prio_masking()) { \
- c->daif_bits = read_sysreg(daif); \
- write_sysreg(c->daif_bits | PSR_I_BIT | PSR_F_BIT, \
- daif); \
- c->pmr = gic_read_pmr(); \
- gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); \
- } \
+ c->arm64_context = local_exceptions_force_daif_save(); \
} while (0)
#define arm_cpuidle_restore_irq_context(__c) \
do { \
struct arm_cpuidle_irq_context *c = __c; \
- if (system_uses_irq_prio_masking()) { \
- gic_write_pmr(c->pmr); \
- write_sysreg(c->daif_bits, daif); \
- } \
+ local_exceptions_force_daif_restore(c->arm64_context); \
} while (0)
#else
struct arm_cpuidle_irq_context { };
diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
index 9e589bf11ec8..8729922974b0 100644
--- a/arch/arm64/include/asm/interrupts/masking.h
+++ b/arch/arm64/include/asm/interrupts/masking.h
@@ -92,6 +92,60 @@ static inline void local_exceptions_final_mask(void)
}
#endif /* CONFIG_DEBUG_IRQFLAGS */
+/*
+ * In some cases, WFI or guest entry for example, we always want interrupts
+ * to reach the CPU even if masked. Masking via the PMR prevents them from
+ * reaching the CPU and waking it up.
+ * Force IRQ masking using DAIF by raising the priority mask
+ * and setting the IF flags.
+ *
+ * Should only be called when IRQs are already masked.
+ */
+static inline arm64_exc_hwstates_t local_exceptions_force_daif_save(void)
+{
+ arm64_exc_hwstates_t states = {};
+
+ if (!system_uses_irq_prio_masking())
+ return states;
+
+ /*
+ * Cannot use lockdep_assert here as idle entry enables hardirqs
+ * while keeping interrupts masked.
+ */
+ WARN_ON_ONCE(!irqs_disabled());
+
+ states.saved.daif = read_sysreg(daif);
+ states.saved.pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
+ states.expected.daif = states.saved.daif | DAIF_PROCCTX_NOIRQ;
+ states.expected.pmr = GIC_PRIO_IRQON;
+
+ arm64_update_exc_hwstate(states.expected);
+
+ return states;
+}
+
+/*
+ * Return to masking with the PMR, restoring previously saved DAIF and PMR.
+ *
+ * IRQs or interrupt priority masking should not have been re-enabled in between
+ * the save and restore.
+ */
+static inline
+void local_exceptions_force_daif_restore(arm64_exc_hwstates_t states)
+{
+ if (!system_uses_irq_prio_masking())
+ return;
+
+ /*
+ * Cannot use lockdep_assert here as idle entry enables hardirqs
+ * while keeping interrupts masked.
+ */
+ WARN_ON_ONCE(!irqs_disabled());
+
+ arm64_debug_exc_hwstate(states.expected);
+ arm64_update_exc_hwstate(states.saved);
+}
+
/*
* During early boot, we unmask PSR.DA before the GIC has been set up.
* If we use IRQ priority masking, the PMR and PSR will be out of sync
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 11/39] arm64: remove daifflags.h
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (9 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 10/39] arm64: cpuidle: use new helpers to bypass interrupt priority masking Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 12/39] arm64: gicv3: remove GIC_PRIO_PSR_I_SET Vladimir Murzin
` (30 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Interrupt masking is now split in different APIs and `local_daif_...`
functions are not in use anymore.
The `DAIF_...` masks defines are now in `asm/interrupts/common_flags.h`,
so we can safely remove `daifflags.h` and all `#include`s referring to it.
Only kprobes.c uses `DAIF_...` masks defines, so add the new header
there.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/daifflags.h | 144 --------------------------
arch/arm64/include/asm/entry-common.h | 1 -
arch/arm64/include/asm/kvm_host.h | 1 -
arch/arm64/include/asm/mmu_context.h | 1 -
arch/arm64/kernel/debug-monitors.c | 1 -
arch/arm64/kernel/probes/kprobes.c | 2 +-
arch/arm64/kernel/signal.c | 1 -
arch/arm64/kernel/traps.c | 1 -
arch/arm64/mm/fault.c | 1 -
9 files changed, 1 insertion(+), 152 deletions(-)
delete mode 100644 arch/arm64/include/asm/daifflags.h
diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
deleted file mode 100644
index 795b35128467..000000000000
--- a/arch/arm64/include/asm/daifflags.h
+++ /dev/null
@@ -1,144 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (C) 2017 ARM Ltd.
- */
-#ifndef __ASM_DAIFFLAGS_H
-#define __ASM_DAIFFLAGS_H
-
-#include <linux/irqflags.h>
-
-#include <asm/arch_gicv3.h>
-#include <asm/barrier.h>
-#include <asm/cpufeature.h>
-#include <asm/ptrace.h>
-
-#define DAIF_PROCCTX 0
-#define DAIF_PROCCTX_NOIRQ (PSR_I_BIT | PSR_F_BIT)
-#define DAIF_ERRCTX (PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
-#define DAIF_MASK (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
-
-
-/* mask/save/unmask/restore all exceptions, including interrupts. */
-static __always_inline void local_daif_mask(void)
-{
- WARN_ON(system_has_prio_mask_debugging() &&
- (read_sysreg_s(SYS_ICC_PMR_EL1) == (GIC_PRIO_IRQOFF |
- GIC_PRIO_PSR_I_SET)));
-
- asm volatile(
- "msr daifset, #0xf // local_daif_mask\n"
- :
- :
- : "memory");
-
- /* Don't really care for a dsb here, we don't intend to enable IRQs */
- if (system_uses_irq_prio_masking())
- gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
-
- trace_hardirqs_off();
-}
-
-static __always_inline unsigned long local_daif_save_flags(void)
-{
- unsigned long flags;
-
- flags = read_sysreg(daif);
-
- if (system_uses_irq_prio_masking()) {
- /* If IRQs are masked with PMR, reflect it in the flags */
- if (read_sysreg_s(SYS_ICC_PMR_EL1) != GIC_PRIO_IRQON)
- flags |= PSR_I_BIT | PSR_F_BIT;
- }
-
- return flags;
-}
-
-static __always_inline unsigned long local_daif_save(void)
-{
- unsigned long flags;
-
- flags = local_daif_save_flags();
-
- local_daif_mask();
-
- return flags;
-}
-
-static __always_inline void local_daif_restore(unsigned long flags)
-{
- bool irq_disabled = flags & PSR_I_BIT;
-
- WARN_ON(system_has_prio_mask_debugging() &&
- (read_sysreg(daif) & (PSR_I_BIT | PSR_F_BIT)) != (PSR_I_BIT | PSR_F_BIT));
-
- if (!irq_disabled) {
- trace_hardirqs_on();
-
- if (system_uses_irq_prio_masking()) {
- gic_write_pmr(GIC_PRIO_IRQON);
- pmr_sync();
- }
- } else if (system_uses_irq_prio_masking()) {
- u64 pmr;
-
- if (!(flags & PSR_A_BIT)) {
- /*
- * If interrupts are disabled but we can take
- * asynchronous errors, we can take NMIs
- */
- flags &= ~(PSR_I_BIT | PSR_F_BIT);
- pmr = GIC_PRIO_IRQOFF;
- } else {
- pmr = GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET;
- }
-
- /*
- * There has been concern that the write to daif
- * might be reordered before this write to PMR.
- * From the ARM ARM DDI 0487D.a, section D1.7.1
- * "Accessing PSTATE fields":
- * Writes to the PSTATE fields have side-effects on
- * various aspects of the PE operation. All of these
- * side-effects are guaranteed:
- * - Not to be visible to earlier instructions in
- * the execution stream.
- * - To be visible to later instructions in the
- * execution stream
- *
- * Also, writes to PMR are self-synchronizing, so no
- * interrupts with a lower priority than PMR is signaled
- * to the PE after the write.
- *
- * So we don't need additional synchronization here.
- */
- gic_write_pmr(pmr);
- }
-
- write_sysreg(flags, daif);
-
- if (irq_disabled)
- trace_hardirqs_off();
-}
-
-/*
- * Called by synchronous exception handlers to restore the DAIF bits that were
- * modified by taking an exception.
- */
-static __always_inline void local_daif_inherit(struct pt_regs *regs)
-{
- unsigned long flags = regs->pstate & DAIF_MASK;
-
- if (!regs_irqs_disabled(regs))
- trace_hardirqs_on();
-
- if (system_uses_irq_prio_masking())
- gic_write_pmr(regs->pmr);
-
- /*
- * We can't use local_daif_restore(regs->pstate) here as
- * system_has_prio_mask_debugging() won't restore the I bit if it can
- * use the pmr instead.
- */
- write_sysreg(flags, daif);
-}
-#endif
diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
index 1905765159aa..73d82a8d8e95 100644
--- a/arch/arm64/include/asm/entry-common.h
+++ b/arch/arm64/include/asm/entry-common.h
@@ -6,7 +6,6 @@
#include <linux/thread_info.h>
#include <asm/cpufeature.h>
-#include <asm/daifflags.h>
#include <asm/fpsimd.h>
#include <asm/mte.h>
#include <asm/stacktrace.h>
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 97332ffcb130..2ca9dd9aa03f 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -24,7 +24,6 @@
#include <asm/barrier.h>
#include <asm/cpufeature.h>
#include <asm/cputype.h>
-#include <asm/daifflags.h>
#include <asm/fpsimd.h>
#include <asm/kvm.h>
#include <asm/kvm_asm.h>
diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index 803b68758152..b3354a1350cf 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -19,7 +19,6 @@
#include <asm/cacheflush.h>
#include <asm/cpufeature.h>
-#include <asm/daifflags.h>
#include <asm/gcs.h>
#include <asm/proc-fns.h>
#include <asm/cputype.h>
diff --git a/arch/arm64/kernel/debug-monitors.c b/arch/arm64/kernel/debug-monitors.c
index e271fbac5f82..5db27c7e86bb 100644
--- a/arch/arm64/kernel/debug-monitors.c
+++ b/arch/arm64/kernel/debug-monitors.c
@@ -19,7 +19,6 @@
#include <asm/cpufeature.h>
#include <asm/cputype.h>
-#include <asm/daifflags.h>
#include <asm/debug-monitors.h>
#include <asm/exception.h>
#include <asm/kgdb.h>
diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c
index 4e0efad5caf2..68f4271ce02f 100644
--- a/arch/arm64/kernel/probes/kprobes.c
+++ b/arch/arm64/kernel/probes/kprobes.c
@@ -24,9 +24,9 @@
#include <linux/vmalloc.h>
#include <asm/cacheflush.h>
-#include <asm/daifflags.h>
#include <asm/debug-monitors.h>
#include <asm/insn.h>
+#include <asm/interrupts/common_flags.h>
#include <asm/irq.h>
#include <asm/text-patching.h>
#include <asm/ptrace.h>
diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
index 6a06a4e352b1..2e12a5622385 100644
--- a/arch/arm64/kernel/signal.c
+++ b/arch/arm64/kernel/signal.c
@@ -22,7 +22,6 @@
#include <linux/syscalls.h>
#include <linux/pkeys.h>
-#include <asm/daifflags.h>
#include <asm/debug-monitors.h>
#include <asm/elf.h>
#include <asm/exception.h>
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index d6391ab20db3..6bbfa8f543b8 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -33,7 +33,6 @@
#include <asm/atomic.h>
#include <asm/bug.h>
#include <asm/cpufeature.h>
-#include <asm/daifflags.h>
#include <asm/debug-monitors.h>
#include <asm/efi.h>
#include <asm/esr.h>
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index b22736853fcd..eb040d1e8935 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -35,7 +35,6 @@
#include <asm/cpufeature.h>
#include <asm/efi.h>
#include <asm/exception.h>
-#include <asm/daifflags.h>
#include <asm/debug-monitors.h>
#include <asm/esr.h>
#include <asm/kprobes.h>
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 12/39] arm64: gicv3: remove GIC_PRIO_PSR_I_SET
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (10 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 11/39] arm64: remove daifflags.h Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 13/39] arm64: cpufeature: Remove system_has_prio_mask_debugging() Vladimir Murzin
` (29 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
`GIC_PRIO_PSR_I_SET` was used to keep track of sections of the kernel
where interrupts always need to reach the CPU and could not be masked
via interrupt priorities, or DAIF bypassed the status of the PMR.
This flag unmasked IRQs when OR'd to GIC_PRIO_IRQON and ensured that
that IRQs were seen as disabled by the `local_irqs_...` functions,
as they would only check the PMR if pseudo-NMIs were enabled.
This is not needed anymore as our IRQ masking APIs now know how to handle
interrupt priorities properly and in parallel with DAIF : remove it.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/ptrace.h | 2 --
include/linux/irqchip/arm-gic-v3-prio.h | 8 --------
2 files changed, 10 deletions(-)
diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index 7d7b54308b90..a64eb588a781 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -24,8 +24,6 @@
#define GIC_PRIO_IRQON GICV3_PRIO_UNMASKED
#define GIC_PRIO_IRQOFF GICV3_PRIO_IRQ
-#define GIC_PRIO_PSR_I_SET GICV3_PRIO_PSR_I_SET
-
#if ARM64_S390_COMMON == 1
/* Additional SPSR bits not exposed in the UABI */
diff --git a/include/linux/irqchip/arm-gic-v3-prio.h b/include/linux/irqchip/arm-gic-v3-prio.h
index 44157c9abb78..464bf1d29453 100644
--- a/include/linux/irqchip/arm-gic-v3-prio.h
+++ b/include/linux/irqchip/arm-gic-v3-prio.h
@@ -20,17 +20,11 @@
* GICV3_PRIO_NMI is the PMR view of the priority of pseudo-NMIs. This can be
* written to the PMR to mask pseudo-NMIs.
*
- * On arm64 some code sections either automatically switch back to PSR.I or
- * explicitly require to not use priority masking. If bit GICV3_PRIO_PSR_I_SET
- * is included in the priority mask, it indicates that PSR.I should be set and
- * interrupt disabling temporarily does not rely on IRQ priorities.
*/
#define GICV3_PRIO_UNMASKED 0xe0
#define GICV3_PRIO_IRQ 0xc0
#define GICV3_PRIO_NMI 0x80
-#define GICV3_PRIO_PSR_I_SET (1 << 4)
-
#ifndef __ASSEMBLER__
#define __gicv3_prio_to_ns(p) (0xff & ((p) << 1))
@@ -45,8 +39,6 @@ static_assert(__gicv3_prio_valid_ns(GICV3_PRIO_IRQ));
static_assert(GICV3_PRIO_NMI < GICV3_PRIO_IRQ);
static_assert(GICV3_PRIO_IRQ < GICV3_PRIO_UNMASKED);
-static_assert(GICV3_PRIO_IRQ < (GICV3_PRIO_IRQ | GICV3_PRIO_PSR_I_SET));
-
#endif /* __ASSEMBLER */
#endif /* __LINUX_IRQCHIP_ARM_GIC_V3_PRIO_H */
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 13/39] arm64: cpufeature: Remove system_has_prio_mask_debugging()
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (11 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 12/39] arm64: gicv3: remove GIC_PRIO_PSR_I_SET Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 14/39] arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS Vladimir Murzin
` (28 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
Users of system_has_prio_mask_debugging() have gone, so remove it.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/cpufeature.h | 6 ------
1 file changed, 6 deletions(-)
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index e3c8a1493dc8..1c624b50b7a6 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -845,12 +845,6 @@ static inline bool system_supports_mte(void)
return alternative_has_cap_unlikely(ARM64_MTE);
}
-static inline bool system_has_prio_mask_debugging(void)
-{
- return IS_ENABLED(CONFIG_ARM64_DEBUG_PRIORITY_MASKING) &&
- system_uses_irq_prio_masking();
-}
-
static inline bool system_supports_bti(void)
{
return cpus_have_final_cap(ARM64_BTI);
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 14/39] arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (12 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 13/39] arm64: cpufeature: Remove system_has_prio_mask_debugging() Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 15/39] arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING Vladimir Murzin
` (27 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
All exception state debug logic now lives under the generic
CONFIG_DEBUG_IRQFLAGS. Move the two remaining cases from the dedicated
config option to CONFIG_DEBUG_IRQFLAGS as well, keeping all related
debug logic under a single configuration option.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/irqflags.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index 9f5c774cbe68..eedbdf7358e6 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -49,7 +49,7 @@ static __always_inline void __daif_local_irq_enable(void)
static __always_inline void __pmr_local_irq_enable(void)
{
- if (IS_ENABLED(CONFIG_ARM64_DEBUG_PRIORITY_MASKING)) {
+ if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
u32 pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
WARN_ON_ONCE(pmr != GIC_PRIO_IRQON && pmr != GIC_PRIO_IRQOFF);
}
@@ -78,7 +78,7 @@ static __always_inline void __daif_local_irq_disable(void)
static __always_inline void __pmr_local_irq_disable(void)
{
- if (IS_ENABLED(CONFIG_ARM64_DEBUG_PRIORITY_MASKING)) {
+ if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
u32 pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
WARN_ON_ONCE(pmr != GIC_PRIO_IRQON && pmr != GIC_PRIO_IRQOFF);
}
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 15/39] arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (13 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 14/39] arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 16/39] efi/runtime-wrappers: Permit architectures to override IRQ flags checks Vladimir Murzin
` (26 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
Last users of CONFIG_ARM64_DEBUG_PRIORITY_MASKING have gone, so remove
it.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/Kconfig | 11 -----------
1 file changed, 11 deletions(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 1075bc661833..4f8b0b294fe1 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2432,17 +2432,6 @@ config ARM64_PSEUDO_NMI
If unsure, say N
-if ARM64_PSEUDO_NMI
-config ARM64_DEBUG_PRIORITY_MASKING
- bool "Debug interrupt priority masking"
- help
- This adds runtime checks to functions enabling/disabling
- interrupts when using priority masking. The additional checks verify
- the validity of ICC_PMR_EL1 when calling concerned functions.
-
- If unsure, say N
-endif # ARM64_PSEUDO_NMI
-
config RELOCATABLE
bool "Build a relocatable kernel image" if EXPERT
select ARCH_HAS_RELR
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 16/39] efi/runtime-wrappers: Permit architectures to override IRQ flags checks
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (14 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 15/39] arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 17/39] arm64/efi: Implement override for " Vladimir Murzin
` (25 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
Representation of IRQ flags on arm64 has recently changed. They now
track both DAIF and PMR state while remaining compatible with the
irqflags API by encoding the state in an unsigned long.
While the generic check can still detect corruption of the IRQ flags
caused by EFI runtime services, the resulting error message is cryptic
because it does not understand the arm64-specific representation.
Permit architectures to override the IRQ flags checks and associated
error messages, while keeping the existing implementation as the
default.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
drivers/firmware/efi/runtime-wrappers.c | 32 +++++++++++++++----------
1 file changed, 20 insertions(+), 12 deletions(-)
diff --git a/drivers/firmware/efi/runtime-wrappers.c b/drivers/firmware/efi/runtime-wrappers.c
index eea4f32e6474..6be9cb844303 100644
--- a/drivers/firmware/efi/runtime-wrappers.c
+++ b/drivers/firmware/efi/runtime-wrappers.c
@@ -151,6 +151,25 @@ static bool efi_rts_abandoned;
#define arch_efi_restore_flags(state_flags) local_irq_restore(state_flags)
#endif
+#ifndef arch_efi_check_flags
+#define arch_efi_check_flags(state_flags, caller) \
+do { \
+ unsigned long cur_flags, mismatch; \
+ \
+ cur_flags = efi_call_virt_save_flags(); \
+ \
+ mismatch = state_flags ^ cur_flags; \
+ if (!WARN_ON_ONCE(mismatch & ARCH_EFI_IRQ_FLAGS_MASK)) \
+ break; \
+ \
+ add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_NOW_UNRELIABLE); \
+ pr_err_ratelimited(FW_BUG "IRQ flags corrupted (0x%08lx=>0x%08lx) by EFI call from %pS\n", \
+ state_flags, cur_flags, \
+ caller ?: __builtin_return_address(0)); \
+ arch_efi_restore_flags(state_flags); \
+} while(0)
+#endif
+
unsigned long efi_call_virt_save_flags(void)
{
unsigned long flags;
@@ -161,18 +180,7 @@ unsigned long efi_call_virt_save_flags(void)
void efi_call_virt_check_flags(unsigned long flags, const void *caller)
{
- unsigned long cur_flags, mismatch;
-
- cur_flags = efi_call_virt_save_flags();
-
- mismatch = flags ^ cur_flags;
- if (!WARN_ON_ONCE(mismatch & ARCH_EFI_IRQ_FLAGS_MASK))
- return;
-
- add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_NOW_UNRELIABLE);
- pr_err_ratelimited(FW_BUG "IRQ flags corrupted (0x%08lx=>0x%08lx) by EFI call from %pS\n",
- flags, cur_flags, caller ?: __builtin_return_address(0));
- arch_efi_restore_flags(flags);
+ arch_efi_check_flags(flags, caller);
}
/*
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 17/39] arm64/efi: Implement override for IRQ flags checks
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (15 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 16/39] efi/runtime-wrappers: Permit architectures to override IRQ flags checks Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-13 13:22 ` Mark Brown
2026-11-12 13:20 ` [PATCH v2 18/39] arm64: booting: Document boot requirements for FEAT_NMI Vladimir Murzin
` (24 subsequent siblings)
41 siblings, 1 reply; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
Take advantage of the recent changes to the way arm64 represents IRQ
flags and implement an architecture-specific override for the EFI IRQ
flags check.
IRQ flags now carry both DAIF and PMR state. EFI is not expected to
modify either of them, so check both when verifying that IRQ state is
preserved across EFI runtime calls.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/efi.h | 38 ++++++++++++++++++++++++++++++------
1 file changed, 32 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h
index e8a9783235cb..0fc5a8cd5850 100644
--- a/arch/arm64/include/asm/efi.h
+++ b/arch/arm64/include/asm/efi.h
@@ -5,6 +5,7 @@
#include <asm/boot.h>
#include <asm/cpufeature.h>
#include <asm/fpsimd.h>
+#include <asm/interrupts/common_flags.h>
#include <asm/io.h>
#include <asm/memory.h>
#include <asm/mmu_context.h>
@@ -48,18 +49,43 @@ void arch_efi_call_virt_teardown(void);
(efi_rt_stack_top != NULL && \
on_task_stack(current, READ_ONCE(efi_rt_stack_top[-1]), 1))
-#define ARCH_EFI_IRQ_FLAGS_MASK (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
-
/*
* Even when Linux uses IRQ priorities for IRQ disabling, EFI does not.
* And EFI shouldn't really play around with priority masking as it is not aware
* which priorities the OS has assigned to its interrupts.
*/
#define arch_efi_save_flags(state_flags) \
- ((void)((state_flags) = read_sysreg(daif)))
-
-#define arch_efi_restore_flags(state_flags) write_sysreg(state_flags, daif)
-
+ ((void)((state_flags) = arch_local_save_flags()))
+
+#define arch_efi_restore_flags(state_flags) \
+do { \
+ arm64_exc_hwstate_t __st = {.flags=state_flags}; \
+ \
+ arm64_update_exc_hwstate(__st); \
+} while(0)
+
+#define arch_efi_check_flags(state_flags, caller) \
+do { \
+ arm64_exc_hwstate_t __actual = {.flags = arch_local_save_flags() }; \
+ arm64_exc_hwstate_t __expected = {.flags=state_flags}; \
+ \
+ if (__expected.flags == __actual.flags) \
+ break; \
+ \
+ add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_NOW_UNRELIABLE); \
+ \
+ if (system_uses_irq_prio_masking()) { \
+ pr_err_ratelimited(FW_BUG "DAIF+PMR corrupted (0x%x + 0x%x=>0x%x + 0x%x) by EFI call from %pS\n", \
+ __expected.daif, __expected.pmr, __actual.daif, __actual.pmr, \
+ caller ?: __builtin_return_address(0)); \
+ } else { \
+ pr_err_ratelimited(FW_BUG "DAIF corrupted (0x%x=>0x%x) by EFI call from %pS\n", \
+ __expected.daif, __actual.daif, \
+ caller ?: __builtin_return_address(0)); \
+ } \
+ \
+ arch_efi_restore_flags(state_flags); \
+} while(0)
/* arch specific definitions used by the stub code */
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 18/39] arm64: booting: Document boot requirements for FEAT_NMI
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (16 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 17/39] arm64/efi: Implement override for " Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 19/39] arm64: sysreg: Add definitions for immediate versions of MSR ALLINT Vladimir Murzin
` (23 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
Mark Brown, maz, will, ruanjinjie
From: Mark Brown <broonie@kernel.org>
In order to use FEAT_NMI we must be able to use ALLINT, require that
it behave as though not trapped when it is present.
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
Documentation/arch/arm64/booting.rst | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/Documentation/arch/arm64/booting.rst b/Documentation/arch/arm64/booting.rst
index dae5ff9fae26..344a4305669a 100644
--- a/Documentation/arch/arm64/booting.rst
+++ b/Documentation/arch/arm64/booting.rst
@@ -569,6 +569,12 @@ Before jumping into the kernel, the following conditions must be met:
- HCRX_EL2.EnASR (bit 2) must be initialised to 0b1.
+ For CPUs with support for Non-maskable Interrupts (FEAT_NMI):
+
+ - If the kernel is entered at EL1 and EL2 is present:
+
+ - HCRX_EL2.TALLINT (bit 6) must be initialised to 0b0.
+
The requirements described above for CPU mode, caches, MMUs, architected
timers, coherency and system registers apply to all CPUs. All CPUs must
enter the kernel in the same exception level. Where the values documented
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 19/39] arm64: sysreg: Add definitions for immediate versions of MSR ALLINT
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (17 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 18/39] arm64: booting: Document boot requirements for FEAT_NMI Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 20/39] arm64: probes: Reject stepping over ALLINT Vladimir Murzin
` (22 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
Mark Brown, maz, will, ruanjinjie
From: Mark Brown <broonie@kernel.org>
Encodings are provided for ALLINT which allow setting of ALLINT.ALLINT
using an immediate rather than requiring that a register be loaded with
the value to write. Since these don't currently fit within the scheme we
have for sysreg generation add manual encodings like we currently do for
other similar registers such as SVCR.
Since it is required that these immediate versions be encoded with xzr
as the source register, provide asm wrappers which ensure this is the
case.
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/interrupts/common_flags.h | 16 ++++++++++++++++
arch/arm64/include/asm/sysreg.h | 2 ++
2 files changed, 18 insertions(+)
diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
index b5cf3b6d0774..a4e53c51dfe8 100644
--- a/arch/arm64/include/asm/interrupts/common_flags.h
+++ b/arch/arm64/include/asm/interrupts/common_flags.h
@@ -44,6 +44,22 @@ typedef enum arm64_exc_context {
CRITICAL_CONTEXT,
} arm64_exc_context_t;
+
+#ifdef CONFIG_ARM64_NMI
+static __always_inline void _allint_clear(void)
+{
+ asm volatile(__msr_s(SYS_ALLINT_CLR, "xzr"));
+}
+
+static __always_inline void _allint_set(void)
+{
+ asm volatile(__msr_s(SYS_ALLINT_SET, "xzr"));
+}
+#else
+static __always_inline void _allint_clear(void) {}
+static __always_inline void _allint_set(void) {}
+#endif /* CONFIG_ARM64_NMI */
+
static __always_inline
arm64_exc_hwstate_t __arm64_exc_hwstate_of_process_context(void)
{
diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index e3b60ee03023..72fa3033b3e4 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -176,6 +176,8 @@
* System registers, organised loosely by encoding but grouped together
* where the architected name contains an index. e.g. ID_MMFR<n>_EL1.
*/
+#define SYS_ALLINT_CLR sys_reg(0, 1, 4, 0, 0)
+#define SYS_ALLINT_SET sys_reg(0, 1, 4, 1, 0)
#define SYS_SVCR_SMSTOP_SM_EL0 sys_reg(0, 3, 4, 2, 3)
#define SYS_SVCR_SMSTART_SM_EL0 sys_reg(0, 3, 4, 3, 3)
#define SYS_SVCR_SMSTOP_SMZA_EL0 sys_reg(0, 3, 4, 6, 3)
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 20/39] arm64: probes: Reject stepping over ALLINT
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (18 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 19/39] arm64: sysreg: Add definitions for immediate versions of MSR ALLINT Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 21/39] arm64: ptrace: Add PSR_ALLINT_BIT Vladimir Murzin
` (21 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
ALLINT instructions are not safe for probing. Both the register and
immediate forms of MSR for ALLINT are currently rejected because they
match aarch64_insn_is_msr_{imm,reg}(). However, MRS instructions
accessing ALLINT need to be rejected explicitly.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/insn.h | 1 +
arch/arm64/kernel/probes/decode-insn.c | 11 +++++++----
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 1ce75a442638..0a81b5feb2ed 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -118,6 +118,7 @@ enum aarch64_insn_special_register {
AARCH64_INSN_SPCLREG_SP_EL0 = 0xC208,
AARCH64_INSN_SPCLREG_SPSEL = 0xC210,
AARCH64_INSN_SPCLREG_CURRENTEL = 0xC212,
+ AARCH64_INSN_SPCLREG_ALLINT = 0xC218,
AARCH64_INSN_SPCLREG_DAIF = 0xDA11,
AARCH64_INSN_SPCLREG_NZCV = 0xDA10,
AARCH64_INSN_SPCLREG_FPCR = 0xDA20,
diff --git a/arch/arm64/kernel/probes/decode-insn.c b/arch/arm64/kernel/probes/decode-insn.c
index 4137cc5ef031..a98d9d30fb4a 100644
--- a/arch/arm64/kernel/probes/decode-insn.c
+++ b/arch/arm64/kernel/probes/decode-insn.c
@@ -36,11 +36,14 @@ static bool __kprobes aarch64_insn_is_steppable(u32 insn)
/*
* The MRS instruction may not return a correct value when
* executing in the single-stepping environment. We do make one
- * exception, for reading the DAIF bits.
+ * exception, for reading the DAIF and ALLINT bits.
*/
- if (aarch64_insn_is_mrs(insn))
- return aarch64_insn_extract_system_reg(insn)
- != AARCH64_INSN_SPCLREG_DAIF;
+ if (aarch64_insn_is_mrs(insn)) {
+ u32 sysreg = aarch64_insn_extract_system_reg(insn);
+
+ return (sysreg != AARCH64_INSN_SPCLREG_DAIF) &&
+ (sysreg != AARCH64_INSN_SPCLREG_ALLINT);
+ }
/*
* The HINT instruction is steppable only if it is in whitelist
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 21/39] arm64: ptrace: Add PSR_ALLINT_BIT
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (19 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 20/39] arm64: probes: Reject stepping over ALLINT Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 22/39] arm64: idreg: Add an override for FEAT_NMI Vladimir Murzin
` (20 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
When FEAT_NMI is implemented, ALLINT is part of PSTATE and will be saved
to SPSR on exception entry.
As it can mask interrupts, depending on configuration, we will need to
check its value in interruted tasks, so add the corresponding bit
to ptrace.h.
Given its impact on interrupt and NMI masking, use it to update
`print_pstate()` so that it displays ALLINT state.
Display it after DAIF so that interrupt masking fields are together.
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/uapi/asm/ptrace.h | 1 +
arch/arm64/kernel/process.c | 3 ++-
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/uapi/asm/ptrace.h b/arch/arm64/include/uapi/asm/ptrace.h
index 0c8c6a126c3b..f43023d561c7 100644
--- a/arch/arm64/include/uapi/asm/ptrace.h
+++ b/arch/arm64/include/uapi/asm/ptrace.h
@@ -48,6 +48,7 @@
#define PSR_D_BIT 0x00000200
#define PSR_BTYPE_MASK 0x00000c00
#define PSR_SSBS_BIT 0x00001000
+#define PSR_ALLINT_BIT 0x00002000
#define PSR_PAN_BIT 0x00400000
#define PSR_UAO_BIT 0x00800000
#define PSR_DIT_BIT 0x01000000
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 1013343de671..860bfabea04a 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -183,7 +183,7 @@ static void print_pstate(struct pt_regs *regs)
const char *btype_str = btypes[(pstate & PSR_BTYPE_MASK) >>
PSR_BTYPE_SHIFT];
- printk("pstate: %08llx (%c%c%c%c %c%c%c%c %cPAN %cUAO %cTCO %cDIT %cSSBS BTYPE=%s)\n",
+ printk("pstate: %08llx (%c%c%c%c %c%c%c%c %cALLINT %cPAN %cUAO %cTCO %cDIT %cSSBS BTYPE=%s)\n",
pstate,
pstate & PSR_N_BIT ? 'N' : 'n',
pstate & PSR_Z_BIT ? 'Z' : 'z',
@@ -193,6 +193,7 @@ static void print_pstate(struct pt_regs *regs)
pstate & PSR_A_BIT ? 'A' : 'a',
pstate & PSR_I_BIT ? 'I' : 'i',
pstate & PSR_F_BIT ? 'F' : 'f',
+ pstate & PSR_ALLINT_BIT ? '+' : '-',
pstate & PSR_PAN_BIT ? '+' : '-',
pstate & PSR_UAO_BIT ? '+' : '-',
pstate & PSR_TCO_BIT ? '+' : '-',
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 22/39] arm64: idreg: Add an override for FEAT_NMI
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (20 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 21/39] arm64: ptrace: Add PSR_ALLINT_BIT Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 23/39] arm64: cpufeature: Detect PE support " Vladimir Murzin
` (19 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
Mark Brown, maz, will, ruanjinjie
From: Mark Brown <broonie@kernel.org>
Add a named override for FEAT_NMI, allowing it to be explicitly disabled
in case of problems.
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/kernel/pi/idreg-override.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c
index acaf56234973..6a995b13584c 100644
--- a/arch/arm64/kernel/pi/idreg-override.c
+++ b/arch/arm64/kernel/pi/idreg-override.c
@@ -164,6 +164,7 @@ static const struct ftr_set_desc pfr1 __prel64_initconst = {
FIELD("bt", ID_AA64PFR1_EL1_BT_SHIFT, NULL ),
FIELD("gcs", ID_AA64PFR1_EL1_GCS_SHIFT, NULL),
FIELD("mte", ID_AA64PFR1_EL1_MTE_SHIFT, NULL),
+ FIELD("nmi", ID_AA64PFR1_EL1_NMI_SHIFT, NULL),
FIELD("sme", ID_AA64PFR1_EL1_SME_SHIFT, pfr1_sme_filter),
FIELD("mpam_frac", ID_AA64PFR1_EL1_MPAM_frac_SHIFT, NULL),
{}
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 23/39] arm64: cpufeature: Detect PE support for FEAT_NMI
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (21 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 22/39] arm64: idreg: Add an override for FEAT_NMI Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 24/39] arm64: nmi: Manage masking for superpriority interrupts Vladimir Murzin
` (18 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
Mark Brown, maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Use of FEAT_NMI requires that all the PEs in the system and the GIC
have NMI support. This patch implements the PE part of that detection.
In order to avoid problematic interactions between real and pseudo NMIs
we disable the architected feature if the user has enabled pseudo NMIs
on the command line. If this is done on a system where support for the
architected feature is detected then a warning is printed during boot in
order to help users spot what is likely to be a misconfiguration.
As KVM does not care about the host kernel supporting FEAT_NMI or not
to allow guests to use it, split the CPU cap in two : ARM64_HAS_NMI is
the hardware support, ARM64_NMI is the kernel making use of it.
Co-developed-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/cpucaps.h | 2 +
arch/arm64/include/asm/cpufeature.h | 10 +++++
arch/arm64/kernel/cpufeature.c | 68 ++++++++++++++++++++++++++++-
arch/arm64/tools/cpucaps | 2 +
4 files changed, 81 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 0c3bfbe99aad..c1d07bc8d88d 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -77,6 +77,8 @@ cpucap_is_possible(const unsigned int cap)
return IS_ENABLED(CONFIG_HW_PERF_EVENTS);
case ARM64_HAS_LSUI:
return IS_ENABLED(CONFIG_ARM64_LSUI);
+ case ARM64_NMI:
+ return IS_ENABLED(CONFIG_ARM64_NMI);
}
return true;
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index 1c624b50b7a6..cc47a4c1d29f 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -840,6 +840,16 @@ static __always_inline bool system_uses_irq_prio_masking(void)
return alternative_has_cap_unlikely(ARM64_HAS_GIC_PRIO_MASKING);
}
+static __always_inline bool system_supports_nmi(void)
+{
+ return alternative_has_cap_unlikely(ARM64_HAS_NMI);
+}
+
+static __always_inline bool system_uses_nmi(void)
+{
+ return alternative_has_cap_unlikely(ARM64_NMI);
+}
+
static inline bool system_supports_mte(void)
{
return alternative_has_cap_unlikely(ARM64_MTE);
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 34c5c03c0c9b..de27b92bbc44 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -83,13 +83,14 @@
#include <asm/cpu_ops.h>
#include <asm/fpsimd.h>
#include <asm/hwcap.h>
+#include <asm/hypervisor.h>
#include <asm/insn.h>
+#include <asm/interrupts/common_flags.h>
#include <asm/kvm_host.h>
#include <asm/mmu.h>
#include <asm/mmu_context.h>
#include <asm/mpam.h>
#include <asm/mte.h>
-#include <asm/hypervisor.h>
#include <asm/processor.h>
#include <asm/smp.h>
#include <asm/sysreg.h>
@@ -317,6 +318,7 @@ static const struct arm64_ftr_bits ftr_id_aa64pfr1[] = {
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_GCS),
FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_GCS_SHIFT, 4, 0),
S_ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MTE_frac_SHIFT, 4, 0),
+ ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_NMI_SHIFT, 4, 0),
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SME),
FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_SME_SHIFT, 4, 0),
ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MPAM_frac_SHIFT, 4, 0),
@@ -2191,6 +2193,51 @@ static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry
}
#endif
+#ifdef CONFIG_ARM64_NMI
+static bool can_use_nmi(const struct arm64_cpu_capabilities *entry, int scope)
+{
+ /*
+ * ARM64_HAS_NMI has a lower index, and is a boot CPU
+ * feature, so will be detected earlier.
+ */
+ BUILD_BUG_ON(ARM64_NMI <= ARM64_HAS_NMI);
+ if (!cpus_have_cap(ARM64_HAS_NMI))
+ return false;
+
+ /*
+ * Having both real and pseudo NMIs enabled simultaneously is
+ * likely to cause confusion. Since pseudo NMIs must be
+ * enabled with an explicit command line option, if the user
+ * has set that option on a system with real NMIs for some
+ * reason assume they know what they're doing.
+ *
+ * ARM64_HAS_GIC_PRIO_MASKING has a lower index, and is a boot CPU
+ * feature, so will be detected earlier.
+ */
+ BUILD_BUG_ON(IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) &&
+ (ARM64_NMI <= ARM64_HAS_GIC_PRIO_MASKING));
+ if (cpus_have_cap(ARM64_HAS_GIC_PRIO_MASKING)) {
+ pr_info_once("Pseudo NMI enabled, not using architected NMI\n");
+ return false;
+ }
+
+ return true;
+}
+
+static void nmi_enable(const struct arm64_cpu_capabilities *__unused)
+{
+ /*
+ * Enable use of NMIs controlled by ALLINT, SPINTMASK should
+ * be clear by default but make it explicit that we are using
+ * this mode. Ensure that ALLINT is clear first in order to
+ * avoid leaving things masked.
+ */
+ _allint_clear();
+ sysreg_clear_set(sctlr_el1, SCTLR_EL1_SPINTMASK, SCTLR_EL1_NMI);
+ isb();
+}
+#endif
+
static bool can_trap_icv_dir_el1(const struct arm64_cpu_capabilities *entry,
int scope)
{
@@ -3083,6 +3130,25 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
.matches = has_cpuid_feature,
ARM64_CPUID_FIELDS(ID_AA64ISAR3_EL1, LSUI, IMP)
},
+#endif
+ {
+ .type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
+ .capability = ARM64_HAS_NMI,
+ .matches = has_cpuid_feature,
+ ARM64_CPUID_FIELDS(ID_AA64PFR1_EL1, NMI, IMP)
+ },
+#ifdef CONFIG_ARM64_NMI
+ /*
+ * Depends on ARM64_HAS_NMI
+ * Checks for conflict with pseudo-NMIs, giving them priority.
+ */
+ {
+ .desc = "Non-maskable Interrupts",
+ .type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
+ .capability = ARM64_NMI,
+ .matches = can_use_nmi,
+ .cpu_enable = nmi_enable,
+ },
#endif
{},
};
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index 20cc185649f9..cf26c450b8d9 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -52,6 +52,7 @@ HAS_LS64_V
HAS_LSUI
HAS_MOPS
HAS_NESTED_VIRT
+HAS_NMI
HAS_NV2P1
HAS_NV3
HAS_PAN
@@ -82,6 +83,7 @@ MTE
MTE_ASYMM
MTE_FAR
MTE_STORE_ONLY
+NMI
SME
SME_FA64
SME2
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 24/39] arm64: nmi: Manage masking for superpriority interrupts
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (22 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 23/39] arm64: cpufeature: Detect PE support " Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 25/39] arm64: irq: Report FEAT_NMI masking local IRQs Vladimir Murzin
` (17 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
Mark Brown, maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Extend logic to handle and debug exception context/state with knowlage
of FEAT_NMI.
Take care to order writes to ALLINT relative to DAIF, as clearing
ALLINT before DAIF could result in taking an NMI while DAIF is fully
masked, as could setting it after DAIF.
Since superpriority interrupts are not masked through DAIF like pseduo
NMIs are, we also need to modify the assembler macros for managing
DAIF to ensure that the masking is done in the assembly code.
Note that save_and_disable_irq/restore_irq and
save_and_disable_daif/restore_irq pairs are used in distinct
contextes:
- former is used in context of SW PAN to quickly disable/enable
preemption
- latter is used to completely mask all exceptions.
For that reason split save_and_disable_daif/restore_irq into more
generic exception save restore pair and plumb with FEAT_NMI logic.
Co-developed-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/assembler.h | 20 +++++++++--
.../include/asm/interrupts/common_flags.h | 35 +++++++++++++++++++
arch/arm64/include/asm/interrupts/entry.h | 6 ++--
arch/arm64/include/asm/interrupts/masking.h | 3 +-
arch/arm64/include/asm/irqflags.h | 3 +-
arch/arm64/kernel/entry.S | 12 +++----
6 files changed, 67 insertions(+), 12 deletions(-)
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 020670814bda..68f700014304 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -37,11 +37,27 @@
/*
* Save/restore interrupts.
*/
- .macro save_and_disable_daif, flags
- mrs \flags, daif
+ .macro save_and_disable_exceptions, flags, tmp
+ mrs \flags, daif // updates flags[9:6] with DAIF
+#ifdef CONFIG_ARM64_NMI
+alternative_if ARM64_NMI
+ mrs_s \tmp, SYS_ALLINT // updates tmp[13] with AllInt
+ msr_s SYS_ALLINT_SET, xzr
+ orr \flags, \flags, \tmp // now flags[13,9:6] carry pair of AllInt,DAIF
+alternative_else_nop_endif
+#endif
msr daifset, #0xf
.endm
+ .macro restore_exceptions, flags
+ msr daif, \flags // bits other than flags[9:6] are ignored
+#ifdef CONFIG_ARM64_NMI
+alternative_if ARM64_NMI
+ msr_s SYS_ALLINT, \flags // bits other than flags[13] are ignored
+alternative_else_nop_endif
+#endif
+ .endm
+
.macro save_and_disable_irq, flags
mrs \flags, daif
msr daifset, #3
diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
index a4e53c51dfe8..1055cfc8a2c4 100644
--- a/arch/arm64/include/asm/interrupts/common_flags.h
+++ b/arch/arm64/include/asm/interrupts/common_flags.h
@@ -20,6 +20,14 @@
/*
* Exception context mapping
*
+ * FEAT_NMI
+ *
+ * CRITICAL -> DAIF + AllInt (corresponds to the state on exception entry)
+ * ERROR -> AIF + AllInt
+ * NONMI -> IF + AllInt
+ * NOIRQ -> IF
+ * PROCESS -> 0
+ *
* pseudo-NMI
*
* CRITICAL -> DAIF + IRQON (corresponds to the state on exception entry)
@@ -84,6 +92,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
if (system_uses_irq_prio_masking())
return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
+ if (system_uses_nmi())
+ return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .allint=ALLINT_ALLINT};
+
return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
}
@@ -93,6 +104,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
if (system_uses_irq_prio_masking())
return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
+ if (system_uses_nmi())
+ return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .allint=ALLINT_ALLINT};
+
return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
}
@@ -102,6 +116,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
if (system_uses_irq_prio_masking())
return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
+ if (system_uses_nmi())
+ return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .allint=ALLINT_ALLINT};
+
return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
}
@@ -131,6 +148,9 @@ arm64_exc_hwstate_t arm64_exc_hwstate_of_regs(struct pt_regs *regs)
if (system_uses_irq_prio_masking())
state.pmr = regs->pmr;
+ if (system_uses_nmi())
+ state.allint = regs->pstate & PSR_ALLINT_BIT;
+
return state;
}
@@ -150,6 +170,9 @@ void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
if (system_uses_irq_prio_masking()) {
WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
actual.daif, actual.pmr, expected.daif, expected.pmr);
+ } else if (system_uses_nmi()) {
+ WARN_ONCE(1, "Unexpected DAIF+ALLINT: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
+ actual.daif, actual.allint, expected.daif, expected.allint);
} else {
WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
actual.daif, expected.daif);
@@ -168,6 +191,7 @@ static __always_inline
void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
{
bool pseudo_nmi = system_uses_irq_prio_masking();
+ bool nmi = system_uses_nmi();
barrier();
@@ -194,8 +218,19 @@ void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
}
+ /*
+ * Try to order ALLINT writes to be consistent with the DAIF state :
+ * we don't want to take an NMI with DAIF masked or when it should
+ * be masked but isn't yet.
+ */
+ if (nmi && (hwstate.allint & ALLINT_ALLINT) && force)
+ _allint_set();
+
write_sysreg(hwstate.daif, daif);
+ if (nmi && !(hwstate.allint & ALLINT_ALLINT) && force)
+ _allint_clear();
+
if (pseudo_nmi && hwstate.pmr == GIC_PRIO_IRQON && force) {
write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
pmr_sync();
diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
index e34c198f4ac1..d30a14925af8 100644
--- a/arch/arm64/include/asm/interrupts/entry.h
+++ b/arch/arm64/include/asm/interrupts/entry.h
@@ -10,7 +10,6 @@
#include <asm/cpufeature.h>
#include <asm/interrupts/common_flags.h>
-
static __always_inline
arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
arm64_exc_hwstate_t next)
@@ -26,7 +25,8 @@ arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
if (!irqs_disabled)
trace_hardirqs_on();
- force = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
+ force = (system_uses_irq_prio_masking() && prev.pmr != next.pmr) ||
+ (system_uses_nmi() && prev.allint != next.allint);
__arm64_update_exc_hwstate(next, force);
@@ -58,6 +58,7 @@ arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
* GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so larger PMR value is weaker
*/
WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr > next.pmr);
+ WARN_ON_ONCE(system_uses_nmi() && prev.allint < next.allint);
}
}
@@ -77,6 +78,7 @@ arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
* GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so smaller PMR value is stronger
*/
WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr < next.pmr);
+ WARN_ON_ONCE(system_uses_nmi() && prev.allint > next.allint);
}
}
diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
index 8729922974b0..6c5ca5befbc1 100644
--- a/arch/arm64/include/asm/interrupts/masking.h
+++ b/arch/arm64/include/asm/interrupts/masking.h
@@ -30,7 +30,8 @@ arm64_exc_hwstates_t local_exceptions_save_mask(arm64_exc_context_t new)
* We've just got actual HW state so we can rely on that to
* optimize some unnecessary updates.
*/
- force = system_uses_irq_prio_masking() && actual.pmr != state.pmr;
+ force = (system_uses_irq_prio_masking() && actual.pmr != state.pmr) ||
+ (system_uses_nmi() && actual.allint != state.allint);
if (!irqs_disabled)
trace_hardirqs_on();
diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index eedbdf7358e6..31b8f4e99532 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -31,9 +31,10 @@
typedef union arm64_exc_hwstate {
struct {
u16 daif;
+ u16 allint;
u8 pmr;
u8 __pad0;
- u32 __pad1;
+ u16 __pad1;
};
unsigned long flags;
} arm64_exc_hwstate_t;
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index 024b740512b0..405ef84d14c7 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -831,7 +831,7 @@ SYM_CODE_END(__bp_harden_el1_vectors)
*
*/
SYM_FUNC_START(cpu_switch_to)
- save_and_disable_daif x11
+ save_and_disable_exceptions x11, x12
mov x10, #THREAD_CPU_CONTEXT
add x8, x0, x10
mov x9, sp
@@ -855,7 +855,7 @@ SYM_FUNC_START(cpu_switch_to)
ptrauth_keys_install_kernel x1, x8, x9, x10
scs_save x0
scs_load_current
- restore_irq x11
+ restore_exceptions x11
ret
SYM_FUNC_END(cpu_switch_to)
NOKPROBE(cpu_switch_to)
@@ -882,7 +882,7 @@ NOKPROBE(ret_from_fork)
* Calls func(regs) using this CPU's irq stack and shadow irq stack.
*/
SYM_FUNC_START(call_on_irq_stack)
- save_and_disable_daif x9
+ save_and_disable_exceptions x9, x10
#ifdef CONFIG_SHADOW_CALL_STACK
get_current_task x16
scs_save x16
@@ -897,10 +897,10 @@ SYM_FUNC_START(call_on_irq_stack)
/* Move to the new stack and call the function there */
add sp, x16, #IRQ_STACK_SIZE
- restore_irq x9
+ restore_exceptions x9
blr x1
- save_and_disable_daif x9
+ save_and_disable_exceptions x9, x10
/*
* Restore the SP from the FP, and restore the FP and LR from the frame
* record.
@@ -908,7 +908,7 @@ SYM_FUNC_START(call_on_irq_stack)
mov sp, x29
ldp x29, x30, [sp], #16
scs_load_current
- restore_irq x9
+ restore_exceptions x9
ret
SYM_FUNC_END(call_on_irq_stack)
NOKPROBE(call_on_irq_stack)
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 25/39] arm64: irq: Report FEAT_NMI masking local IRQs
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (23 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 24/39] arm64: nmi: Manage masking for superpriority interrupts Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 26/39] arm64: nmi: Add handling of superpriority interrupts as NMIs Vladimir Murzin
` (16 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
Mark Brown, maz, will, ruanjinjie
From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
As we clear SCTLR_EL1.SPINTMASK when enabling FEAT_NMI, ALLINT masks
IRQs and FIQs regardless of superpriority.
Update irqflags.h and `regs_irqs_disabled()` to take it into account,
so we properly keep track of the local IRQ masking state.
We have documentation at the top of irqflags.h which explains the DAIF
masking. Since the additional masking with NMIs is related and also
covers the IF in DAIF, extend the comment to note what's going on
with NMIs.
Co-developed-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/irqflags.h | 17 +++++++++++++++++
arch/arm64/include/asm/ptrace.h | 3 +++
2 files changed, 20 insertions(+)
diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index 31b8f4e99532..d64d89a8f31a 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -20,6 +20,16 @@
* always masked and unmasked together, and have no side effects for other
* flags. Keeping to this order makes it easier for entry.S to know which
* exceptions should be unmasked.
+ *
+ * With the addition of the FEAT_NMI extension we gain an additional
+ * class of superpriority IRQ/FIQ which is separately masked with a
+ * choice of modes controlled by SCTLR_ELn.{SPINTMASK,NMI}.
+ * Linux sets SPINTMASK to 0 and NMI to 1 which results in ALLINT.ALLINT
+ * masking both superpriority interrupts and IRQ/FIQ regardless of the
+ * I and F settings. Since these superpriority interrupts are being
+ * used as NMIs we do not include them in the interrupt masking here,
+ * anything that requires that NMIs be masked needs to explicitly do so,
+ * but we do check for ALLINT masking IRQs/FIQs.
*/
/*
@@ -105,6 +115,9 @@ static __always_inline unsigned long arch_local_save_flags(void)
{
arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
+ if (system_uses_nmi())
+ hwstate.allint = read_sysreg_s(SYS_ALLINT);
+
if (system_uses_irq_prio_masking())
hwstate.pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
@@ -118,6 +131,10 @@ static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
if (hwstate.daif & PSR_I_BIT)
return true;
+ /* SCTLR_EL1.SPINTMASK is clear, so ALLINT masks *all* IRQs/FIQs. */
+ if (system_uses_nmi())
+ return !!(hwstate.allint & ALLINT_ALLINT);
+
if (system_uses_irq_prio_masking()) {
WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_IRQFLAGS) &&
hwstate.pmr != GIC_PRIO_IRQOFF &&
diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index a64eb588a781..2c42296b7675 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -231,6 +231,9 @@ static __always_inline bool regs_irqs_disabled(const struct pt_regs *regs)
if (regs->pstate & PSR_I_BIT)
return true;
+ if (system_uses_nmi())
+ return !!(regs->pstate & PSR_ALLINT_BIT);
+
if (system_uses_irq_prio_masking()) {
WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_IRQFLAGS) &&
regs->pmr != GIC_PRIO_IRQOFF &&
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 26/39] arm64: nmi: Add handling of superpriority interrupts as NMIs
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (24 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 25/39] arm64: irq: Report FEAT_NMI masking local IRQs Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 27/39] arm64: suspend: Always initialise PSTATE.ALLINT Vladimir Murzin
` (15 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
Mark Brown, maz, will, ruanjinjie
From: Mark Brown <broonie@kernel.org>
Our goal with superpriority interrupts is to use them as NMIs, taking
advantage of the much smaller regions where they are masked to allow
prompt handling of the most time-critical interrupts.
When an interrupt is configured with superpriority, we enter EL1 as we
do for any other interrupt. The presence of a superpriority interrupt
is indicated by a status bit in ISR_EL1. We check this bit before
unmasking interrupts in elX_interrupt(), and if a superpriority
interrupt is pending, we handle it as an NMI. Otherwise, normal
interrupts are handled as usual.
Since superpriority interrupts are always handled as NMIs, the
interrupt controller can rely on in_nmi() to distinguish them from
ordinary interrupts.
Enable IPIs to use superpriority interrupts as NMIs, matching the
existing pseudo-NMI behaviour.
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/entry-common.h | 7 +++
arch/arm64/kernel/entry-common.c | 83 +++++++++++++++++++++++----
arch/arm64/kernel/smp.c | 2 +-
3 files changed, 79 insertions(+), 13 deletions(-)
diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
index 73d82a8d8e95..078b9a6456d0 100644
--- a/arch/arm64/include/asm/entry-common.h
+++ b/arch/arm64/include/asm/entry-common.h
@@ -37,6 +37,13 @@ static inline bool arch_irqentry_exit_need_resched(void)
if (system_uses_irq_prio_masking() && read_sysreg(daif))
return false;
+ /*
+ * If AllInt is set then we must have handled an NMI, so skip
+ * preemption
+ */
+ if (system_uses_nmi() && (read_sysreg_s(SYS_ALLINT) & ALLINT_ALLINT))
+ return false;
+
/*
* Preempting a task from an IRQ means we leave copies of PSTATE
* on the stack. cpufeature's enable calls may modify PSTATE, but
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index ec7edb8790d5..1a2119f8ac11 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -33,6 +33,14 @@
#include <asm/system_misc.h>
#include <asm/vectors.h>
+static __always_inline bool is_nmi(void)
+{
+ if (!system_uses_nmi())
+ return false;
+
+ return !!(read_sysreg(isr_el1) & ISR_EL1_IS);
+}
+
/*
* Where the context being returned to had an active percpu GPR critical
* section, ensure that the offset and address GPRs are updated to match the
@@ -564,8 +572,8 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
arm64_debug_exc_context(CRITICAL_CONTEXT);
}
-static __always_inline void __el1_pnmi(struct pt_regs *regs,
- void (*handler)(struct pt_regs *))
+static __always_inline void __el1_nmi(struct pt_regs *regs,
+ void (*handler)(struct pt_regs *))
{
arm64_exc_hwstate_t hwstate;
irqentry_state_t state;
@@ -584,7 +592,23 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
state = arm64_enter_from_kernel_mode(regs);
- arm64_unmask_exc_context(NONMI_CONTEXT);
+ /*
+ * For non-NMI systems, we can switch to NOIRQ context since
+ * NMIs cannot occur.
+ *
+ * For pseudo-NMI, we must keep NONMI context until we can
+ * determine the interrupt priority, which is done by the IRQ
+ * handler.
+ *
+ * For FEAT_NMI, we already know the interrupt is an IRQ, so we
+ * can switch to NOIRQ context immediately. NMIs do not
+ * interfere because they follow a separate handling path and do
+ * not share the IRQ handling state.
+ */
+ if (system_uses_irq_prio_masking())
+ arm64_unmask_exc_context(NONMI_CONTEXT);
+ else
+ arm64_unmask_exc_context(NOIRQ_CONTEXT);
irq_enter_rcu();
do_interrupt_handler(regs, handler);
@@ -604,8 +628,9 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
static void noinstr el1_interrupt(struct pt_regs *regs,
void (*handler)(struct pt_regs *))
{
- if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && regs_irqs_disabled(regs))
- __el1_pnmi(regs, handler);
+ /* Is there a NMI to handle? */
+ if (regs_irqs_disabled(regs) || is_nmi())
+ __el1_nmi(regs, handler);
else
__el1_irq(regs, handler);
@@ -619,6 +644,8 @@ asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
{
+ WARN_ON_ONCE(system_uses_nmi() && (read_sysreg(isr_el1) & ISR_EL1_FS));
+
el1_interrupt(regs, handle_arch_fiq);
}
@@ -942,20 +969,36 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
arm64_debug_exc_context(CRITICAL_CONTEXT);
}
-static void noinstr el0_interrupt(struct pt_regs *regs,
- void (*handler)(struct pt_regs *))
+static __always_inline void __el0_nmi(struct pt_regs *regs,
+ void (*handler)(struct pt_regs *))
{
- arm64_enter_from_user_mode(regs);
+ irqentry_state_t state;
+ arm64_exc_hwstate_t hwstate;
- arm64_unmask_exc_context(NONMI_CONTEXT);
+ state = irqentry_nmi_enter(regs);
+ hwstate = arm64_unmask_exc_context(NONMI_CONTEXT);
+ do_interrupt_handler(regs, handler);
+ arm64_mask_exc_context(hwstate);
+ irqentry_nmi_exit(regs, state);
+}
- if (regs->pc & BIT(55))
- arm64_apply_bp_hardening();
+static __always_inline void __el0_irq(struct pt_regs *regs,
+ void (*handler)(struct pt_regs *))
+{
+ arm64_enter_from_user_mode(regs);
+ /*
+ * For the same reason as in el1_irq() keep NONMI
+ * context for pesudo-NMI, and switch to NOIRQ
+ * otherwise
+ */
+ if (system_uses_irq_prio_masking())
+ arm64_unmask_exc_context(NONMI_CONTEXT);
+ else
+ arm64_unmask_exc_context(NOIRQ_CONTEXT);
irq_enter_rcu();
do_interrupt_handler(regs, handler);
irq_exit_rcu();
-
/*
* For the same reason as in el1_irq() we effectivly
* have NOIRQ_CONTEXT on return from handler - keep
@@ -963,6 +1006,20 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
*/
arm64_debug_exc_context(NOIRQ_CONTEXT);
arm64_exit_to_user_mode(regs, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
+}
+
+static void noinstr el0_interrupt(struct pt_regs *regs,
+ void (*handler)(struct pt_regs *))
+{
+ if (regs->pc & BIT(55))
+ arm64_apply_bp_hardening();
+
+ /* Is there a NMI to handle? */
+ if (is_nmi())
+ __el0_nmi(regs, handler);
+ else
+ __el0_irq(regs, handler);
+
arm64_debug_exc_context(CRITICAL_CONTEXT);
}
@@ -978,6 +1035,8 @@ asmlinkage void noinstr el0t_64_irq_handler(struct pt_regs *regs)
static void noinstr __el0_fiq_handler_common(struct pt_regs *regs)
{
+ WARN_ON_ONCE(system_uses_nmi() && (read_sysreg(isr_el1) & ISR_EL1_FS));
+
el0_interrupt(regs, handle_arch_fiq);
}
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index cb04f2c37d9a..22274d775990 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -1061,7 +1061,7 @@ static void smp_cross_call(const struct cpumask *target, unsigned int ipinr)
static bool ipi_should_be_nmi(enum ipi_msg_type ipi)
{
- if (!system_uses_irq_prio_masking())
+ if (!system_uses_nmi() && !system_uses_irq_prio_masking())
return false;
switch (ipi) {
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 27/39] arm64: suspend: Always initialise PSTATE.ALLINT
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (25 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 26/39] arm64: nmi: Add handling of superpriority interrupts as NMIs Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 28/39] arm64/efi: Add ALLINT to IRQ flags checks Vladimir Murzin
` (14 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
PSTATE.ALLINT is always set to the inverse of SCTLR_ELx.SPINTMASK,
regardless of the value of SCTLR_ELx.NMI. SCTLR_ELx.NMI is initialised
to 0 by default, so PSTATE.ALLINT does not cause any interrupt
masking. With upcoming FEAT_NMI support, SCTLR_ELx.NMI will be set as
part of the enable sequence.
However, during CPU suspend/resume, we reinitialise PSTATE to
INIT_PSTATE_EL1. INIT_PSTATE_EL1 currently does not set
PSR_ALLINT_BIT, but SCTLR_ELx.NMI is restored from the saved
value. Since all exceptions are masked during CPU suspend/resume,
avoid a state where SCTLR_ELx.NMI is restored to 1 but PSTATE.ALLINT
is clear, since that would permit an NMI during resume.
Include PSR_ALLINT_BIT in INIT_PSTATE_EL1 so PSTATE.ALLINT is always
set.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/ptrace.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index 2c42296b7675..d0cb50bcda68 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -17,7 +17,7 @@
#define CurrentEL_EL2 (2 << 2)
#define INIT_PSTATE_EL1 \
- (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL1h)
+ (PSR_ALLINT_BIT | PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL1h)
#include <linux/irqchip/arm-gic-v3-prio.h>
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 28/39] arm64/efi: Add ALLINT to IRQ flags checks
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (26 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 27/39] arm64: suspend: Always initialise PSTATE.ALLINT Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 29/39] arm64: kprobes: Disable NMIs Vladimir Murzin
` (13 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
EFI is not expected to modify PSTATE.ALLINT, so check that it remains
unchanged across EFI runtime calls.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/include/asm/efi.h | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h
index 0fc5a8cd5850..c7fb21c00690 100644
--- a/arch/arm64/include/asm/efi.h
+++ b/arch/arm64/include/asm/efi.h
@@ -78,6 +78,10 @@ do { \
pr_err_ratelimited(FW_BUG "DAIF+PMR corrupted (0x%x + 0x%x=>0x%x + 0x%x) by EFI call from %pS\n", \
__expected.daif, __expected.pmr, __actual.daif, __actual.pmr, \
caller ?: __builtin_return_address(0)); \
+ } else if (system_uses_nmi()) { \
+ pr_err_ratelimited(FW_BUG "DAIF+ALLINT corrupted (0x%x + 0x%x=>0x%x + 0x%x) by EFI call from %pS\n", \
+ __expected.daif, __expected.allint, __actual.daif, __actual.allint, \
+ caller ?: __builtin_return_address(0)); \
} else { \
pr_err_ratelimited(FW_BUG "DAIF corrupted (0x%x=>0x%x) by EFI call from %pS\n", \
__expected.daif, __actual.daif, \
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 29/39] arm64: kprobes: Disable NMIs
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (27 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 28/39] arm64/efi: Add ALLINT to IRQ flags checks Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 30/39] arm64: smp: Abstract SGI and LPI operations Vladimir Murzin
` (12 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
Kprobes masks all DAIF exceptions to keep things simple and avoid
nested exceptions. With FEAT_NMI, that is no longer enough. Mask
ALLINT as well to ensure things stay simple.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/kernel/probes/kprobes.c | 22 ++++++++++++++++------
1 file changed, 16 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c
index 68f4271ce02f..0e9bb98d2b27 100644
--- a/arch/arm64/kernel/probes/kprobes.c
+++ b/arch/arm64/kernel/probes/kprobes.c
@@ -208,17 +208,27 @@ static void __kprobes set_current_kprobe(struct kprobe *p)
* the kprobe state is per-CPU and doesn't get migrated.
*/
static void __kprobes kprobes_save_local_irqflag(struct kprobe_ctlblk *kcb,
- struct pt_regs *regs)
+ struct pt_regs *regs)
{
- kcb->saved_irqflag = regs->pstate & DAIF_MASK;
- regs->pstate |= DAIF_MASK;
+ arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
+
+ kcb->saved_irqflag = arm64_exc_hwstate_of_regs(regs).flags;
+
+ regs->pstate &= ~(DAIF_MASK | PSR_ALLINT_BIT);
+
+ regs->pstate |= next.daif;
+ regs->pstate |= next.allint;
}
static void __kprobes kprobes_restore_local_irqflag(struct kprobe_ctlblk *kcb,
- struct pt_regs *regs)
+ struct pt_regs *regs)
{
- regs->pstate &= ~DAIF_MASK;
- regs->pstate |= kcb->saved_irqflag;
+ arm64_exc_hwstate_t prev = {.flags = kcb->saved_irqflag};
+
+ regs->pstate &= ~(DAIF_MASK | PSR_ALLINT_BIT);
+
+ regs->pstate |= prev.daif;
+ regs->pstate |= prev.allint;
}
static void __kprobes setup_singlestep(struct kprobe *p,
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 30/39] arm64: smp: Abstract SGI and LPI operations
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (28 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 29/39] arm64: kprobes: Disable NMIs Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 31/39] arm64: smp: Fall back to IRQ when IPI NMI request fails Vladimir Murzin
` (11 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
SGI and LPI backed IPIs require different setup, enable, disable and
send operations. These differences are currently handled by repeatedly
checking percpu_ipi_descs. As the implementation specific logic grows,
these checks make the common IPI code increasingly difficult to
follow.
Introduce an operations structure for each implementation to
encapsulate the specific of SGI and LPI handling, leaving the common
IPI paths generic.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/kernel/smp.c | 136 ++++++++++++++++++++++++++--------------
1 file changed, 89 insertions(+), 47 deletions(-)
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 22274d775990..35182f06b34c 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -77,11 +77,18 @@ static DEFINE_PER_CPU_READ_MOSTLY(struct ipi_descs, pcpu_ipi_desc);
#define get_ipi_desc(__cpu, __ipi) (per_cpu_ptr(&pcpu_ipi_desc, __cpu)->descs[__ipi])
-static bool percpu_ipi_descs __ro_after_init;
+struct ipi_irq_ops {
+ void (*setup)(int ipi, int ncpus);
+ void (*disable)(int cpu, int ipi);
+ void (*enable)(int cpu, int ipi);
+ void (*send)(const cpumask_t *mask, unsigned int nr);
+};
+
+static const struct ipi_irq_ops *ipi_ops __ro_after_init;
static bool crash_stop;
-static void ipi_setup(int cpu);
+static void ipi_enable(int cpu);
#ifdef CONFIG_HOTPLUG_CPU
static void ipi_teardown(int cpu);
@@ -244,7 +251,7 @@ asmlinkage notrace void secondary_start_kernel(void)
notify_cpu_starting(cpu);
trace_hardirqs_off_finish();
- ipi_setup(cpu);
+ ipi_enable(cpu);
numa_add_cpu(cpu);
@@ -937,13 +944,7 @@ void __noreturn panic_smp_self_stop(void)
static void arm64_send_ipi(const cpumask_t *mask, unsigned int nr)
{
- unsigned int cpu;
-
- if (!percpu_ipi_descs)
- __ipi_send_mask(get_ipi_desc(0, nr), mask);
- else
- for_each_cpu(cpu, mask)
- __ipi_send_single(get_ipi_desc(cpu, nr), cpu);
+ ipi_ops->send(mask, nr);
}
static void arm64_backtrace_ipi(cpumask_t *mask)
@@ -1074,54 +1075,37 @@ static bool ipi_should_be_nmi(enum ipi_msg_type ipi)
}
}
-static void ipi_setup(int cpu)
+static void ipi_enable(int cpu)
{
- int i;
+ int ipi;
if (WARN_ON_ONCE(!ipi_irq_base))
return;
- for (i = 0; i < nr_ipi; i++) {
- if (!percpu_ipi_descs) {
- if (ipi_should_be_nmi(i)) {
- prepare_percpu_nmi(ipi_irq_base + i);
- enable_percpu_nmi(ipi_irq_base + i, 0);
- } else {
- enable_percpu_irq(ipi_irq_base + i, 0);
- }
- } else {
- enable_irq(irq_desc_get_irq(get_ipi_desc(cpu, i)));
- }
- }
+ for (ipi = 0; ipi < nr_ipi; ipi++)
+ ipi_ops->enable(cpu, ipi);
}
#ifdef CONFIG_HOTPLUG_CPU
static void ipi_teardown(int cpu)
{
- int i;
+ int ipi;
if (WARN_ON_ONCE(!ipi_irq_base))
return;
- for (i = 0; i < nr_ipi; i++) {
- if (!percpu_ipi_descs) {
- if (ipi_should_be_nmi(i)) {
- disable_percpu_nmi(ipi_irq_base + i);
- teardown_percpu_nmi(ipi_irq_base + i);
- } else {
- disable_percpu_irq(ipi_irq_base + i);
- }
- } else {
- disable_irq_nosync(irq_desc_get_irq(get_ipi_desc(cpu, i)));
- }
- }
+ for (ipi = 0; ipi < nr_ipi; ipi++)
+ ipi_ops->disable(cpu, ipi);
}
#endif
-static void ipi_setup_sgi(int ipi)
+static void ipi_sgi_setup(int ipi, int ncpus)
{
int err, irq, cpu;
+ if (WARN_ON_ONCE(ncpus))
+ return;
+
irq = ipi_irq_base + ipi;
if (ipi_should_be_nmi(ipi)) {
@@ -1138,7 +1122,53 @@ static void ipi_setup_sgi(int ipi)
irq_set_status_flags(irq, IRQ_HIDDEN);
}
-static void ipi_setup_lpi(int ipi, int ncpus)
+static void ipi_sgi_enable(int cpu, int ipi)
+{
+ int irq = ipi_irq_base + ipi;
+
+ if (ipi_should_be_nmi(ipi)) {
+ prepare_percpu_nmi(irq);
+ enable_percpu_nmi(irq, 0);
+ } else {
+ enable_percpu_irq(irq, 0);
+ }
+}
+
+static void ipi_sgi_disable(int cpu, int ipi)
+{
+ int irq = ipi_irq_base + ipi;
+
+ if (ipi_should_be_nmi(ipi)) {
+ disable_percpu_nmi(irq);
+ teardown_percpu_nmi(irq);
+ } else {
+ disable_percpu_irq(irq);
+ }
+}
+
+static void ipi_sgi_send(const cpumask_t *mask, unsigned int nr)
+{
+ __ipi_send_mask(get_ipi_desc(0, nr), mask);
+}
+
+static const struct ipi_irq_ops ipi_sgi_ops = {
+ .setup = ipi_sgi_setup,
+ .disable = ipi_sgi_disable,
+ .enable = ipi_sgi_enable,
+ .send = ipi_sgi_send,
+};
+
+static void ipi_lpi_enable(int cpu, int ipi)
+{
+ enable_irq(irq_desc_get_irq(get_ipi_desc(cpu, ipi)));
+}
+
+static void ipi_lpi_disable(int cpu, int ipi)
+{
+ disable_irq_nosync(irq_desc_get_irq(get_ipi_desc(cpu, ipi)));
+}
+
+static void ipi_lpi_setup(int ipi, int ncpus)
{
for (int cpu = 0; cpu < ncpus; cpu++) {
int err, irq;
@@ -1158,6 +1188,21 @@ static void ipi_setup_lpi(int ipi, int ncpus)
}
}
+static void ipi_lpi_send(const cpumask_t *mask, unsigned int nr)
+{
+ int cpu;
+
+ for_each_cpu(cpu, mask)
+ __ipi_send_single(get_ipi_desc(cpu, nr), cpu);
+}
+
+static const struct ipi_irq_ops ipi_lpi_ops = {
+ .setup = ipi_lpi_setup,
+ .disable = ipi_lpi_disable,
+ .enable = ipi_lpi_enable,
+ .send = ipi_lpi_send,
+};
+
void __init set_smp_ipi_range_percpu(int ipi_base, int n, int ncpus)
{
int i;
@@ -1165,18 +1210,15 @@ void __init set_smp_ipi_range_percpu(int ipi_base, int n, int ncpus)
WARN_ON(n < MAX_IPI);
nr_ipi = min(n, MAX_IPI);
- percpu_ipi_descs = !!ncpus;
ipi_irq_base = ipi_base;
- for (i = 0; i < nr_ipi; i++) {
- if (!percpu_ipi_descs)
- ipi_setup_sgi(i);
- else
- ipi_setup_lpi(i, ncpus);
- }
+ ipi_ops = ncpus ? &ipi_lpi_ops : &ipi_sgi_ops;
+
+ for (i = 0; i < nr_ipi; i++)
+ ipi_ops->setup(i, ncpus);
/* Setup the boot CPU immediately */
- ipi_setup(smp_processor_id());
+ ipi_enable(smp_processor_id());
}
void arch_smp_send_reschedule(int cpu)
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 31/39] arm64: smp: Fall back to IRQ when IPI NMI request fails
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (29 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 30/39] arm64: smp: Abstract SGI and LPI operations Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 32/39] arm64: nmi: Add Kconfig for NMI Vladimir Murzin
` (10 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
Existing IPI setup logic relies on ipi_should_be_nmi() to decide
whether to request an NMI (if supported) or an ordinary IRQ. If NMI is
not supported, using an ordinary IRQ is acceptable.
FEAT_NMI requires complementary support in the interrupt
controller. If FEAT_NMI is present but interrupt controller support is
absent, request_percpu_nmi() can fail.
One way to prevent this would be to extend ipi_should_be_nmi() to also
check for interrupt controller support. However, since falling back to
an ordinary IRQ is acceptable, treat ipi_should_be_nmi() as a hint
instead. Whether an IPI is actually registered as an NMI is determined
by the success or failure of request_percpu_nmi(). If the request
fails, fall back to request_irq().
Track which IPIs were actually registered as NMIs and use that state
in the per-CPU enable and disable paths. This avoids calling the
percpu NMI helpers for an IPI that fell back to regular IRQ handling.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/kernel/smp.c | 37 +++++++++++++++++++++++++++----------
1 file changed, 27 insertions(+), 10 deletions(-)
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 35182f06b34c..58b705426120 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -71,11 +71,14 @@ static int nr_ipi __ro_after_init = NR_IPI;
struct ipi_descs {
struct irq_desc *descs[MAX_IPI];
+ DECLARE_BITMAP(nmi_bitmap, MAX_IPI);
};
static DEFINE_PER_CPU_READ_MOSTLY(struct ipi_descs, pcpu_ipi_desc);
#define get_ipi_desc(__cpu, __ipi) (per_cpu_ptr(&pcpu_ipi_desc, __cpu)->descs[__ipi])
+#define get_ipi_nmi_bitmap(__cpu) (per_cpu_ptr(&pcpu_ipi_desc, __cpu)->nmi_bitmap)
+#define ipi_is_nmi(__cpu, __ipi) test_bit(__ipi, get_ipi_nmi_bitmap(__cpu))
struct ipi_irq_ops {
void (*setup)(int ipi, int ncpus);
@@ -967,8 +970,8 @@ void arch_trigger_cpumask_backtrace(const cpumask_t *mask, int exclude_cpu)
/*
* NOTE: though nmi_trigger_cpumask_backtrace() has "nmi_" in the name,
* nothing about it truly needs to be implemented using an NMI, it's
- * just that it's _allowed_ to work with NMIs. If ipi_should_be_nmi()
- * returned false our backtrace attempt will just use a regular IPI.
+ * just that it's _allowed_ to work with NMIs. If we failed request
+ * NMI our backtrace attempt will just use a regular IPI.
*/
nmi_trigger_cpumask_backtrace(mask, exclude_cpu, arm64_backtrace_ipi);
}
@@ -1101,6 +1104,7 @@ static void ipi_teardown(int cpu)
static void ipi_sgi_setup(int ipi, int ncpus)
{
+ bool is_nmi = ipi_should_be_nmi(ipi);
int err, irq, cpu;
if (WARN_ON_ONCE(ncpus))
@@ -1108,16 +1112,29 @@ static void ipi_sgi_setup(int ipi, int ncpus)
irq = ipi_irq_base + ipi;
- if (ipi_should_be_nmi(ipi)) {
- err = request_percpu_nmi(irq, ipi_handler, "IPI", NULL, &irq_stat);
- WARN(err, "Could not request IRQ %d as NMI, err=%d\n", irq, err);
- } else {
+ /*
+ * Try to request NMI if it should be
+ */
+ if (is_nmi && request_percpu_nmi(irq, ipi_handler, "IPI", NULL, &irq_stat)) {
+ /*
+ * Fallback to regular IRQ
+ */
+ is_nmi = false;
+ }
+
+ /*
+ * Otherwise regular IRQ
+ */
+ if (!is_nmi) {
err = request_percpu_irq(irq, ipi_handler, "IPI", &irq_stat);
WARN(err, "Could not request IRQ %d as IRQ, err=%d\n", irq, err);
}
- for_each_possible_cpu(cpu)
+ for_each_possible_cpu(cpu) {
get_ipi_desc(cpu, ipi) = irq_to_desc(irq);
+ if (is_nmi)
+ set_bit(ipi, get_ipi_nmi_bitmap(cpu));
+ }
irq_set_status_flags(irq, IRQ_HIDDEN);
}
@@ -1126,7 +1143,7 @@ static void ipi_sgi_enable(int cpu, int ipi)
{
int irq = ipi_irq_base + ipi;
- if (ipi_should_be_nmi(ipi)) {
+ if (ipi_is_nmi(cpu, ipi)) {
prepare_percpu_nmi(irq);
enable_percpu_nmi(irq, 0);
} else {
@@ -1138,7 +1155,7 @@ static void ipi_sgi_disable(int cpu, int ipi)
{
int irq = ipi_irq_base + ipi;
- if (ipi_should_be_nmi(ipi)) {
+ if (ipi_is_nmi(cpu, ipi)) {
disable_percpu_nmi(irq);
teardown_percpu_nmi(irq);
} else {
@@ -1311,7 +1328,7 @@ void smp_send_stop(void)
* If CPUs are still online, try an NMI. There's no excuse for this to
* be slow, so we only give them an extra 10 ms to respond.
*/
- if (num_other_online_cpus() && ipi_should_be_nmi(IPI_CPU_STOP_NMI)) {
+ if (num_other_online_cpus() && ipi_is_nmi(smp_processor_id(), IPI_CPU_STOP_NMI)) {
smp_rmb();
cpumask_copy(&mask, cpu_online_mask);
cpumask_clear_cpu(smp_processor_id(), &mask);
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 32/39] arm64: nmi: Add Kconfig for NMI
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (30 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 31/39] arm64: smp: Fall back to IRQ when IPI NMI request fails Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 33/39] arm64: nmi: Select HAVE_PERF_EVENTS_NMI for FEAT_NMI Vladimir Murzin
` (9 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
Mark Brown, maz, will, ruanjinjie
From: Mark Brown <broonie@kernel.org>
Since NMI handling is in some fairly hot paths we provide a Kconfig
option which allows support to be compiled out when not needed.
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/Kconfig | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 4f8b0b294fe1..12cad0650863 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2293,6 +2293,24 @@ endmenu # "ARMv8.7 architectural features"
config AS_HAS_MOPS
def_bool $(as-instr,.arch_extension mops)
+menu "ARMv8.8 architectural features"
+
+config ARM64_NMI
+ bool "Enable support for Non-maskable Interrupts (NMI)"
+ default y
+ help
+ Non-maskable interrupts are an architecture and GIC feature
+ which allow the system to configure some interrupts to have
+ superpriority, allowing them to be handled before other interrupts
+ and masked for shorter periods of time.
+
+ The feature is detected at runtime, and will remain disabled
+ if the cpu does not implement the feature. It will also be
+ disabled if pseudo NMIs are enabled at runtime.
+
+endmenu # "ARMv8.8 architectural features"
+
+
menu "ARMv8.9 architectural features"
config ARM64_POE
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 33/39] arm64: nmi: Select HAVE_PERF_EVENTS_NMI for FEAT_NMI
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (31 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 32/39] arm64: nmi: Add Kconfig for NMI Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 34/39] irqchip/gic-v3: Rename nmi to pseudo_nmi Vladimir Murzin
` (8 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
HAVE_PERF_EVENTS_NMI is currently selected only when pseudo-NMIs are
enabled. FEAT_NMI also allows performance events to use NMI context,
so select HAVE_PERF_EVENTS_NMI when ARM64_NMI is enabled.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
arch/arm64/Kconfig | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 12cad0650863..5f8c71e44af5 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -216,7 +216,7 @@ config ARM64
select HAVE_MOD_ARCH_SPECIFIC
select HAVE_NMI
select HAVE_PERF_EVENTS
- select HAVE_PERF_EVENTS_NMI if ARM64_PSEUDO_NMI
+ select HAVE_PERF_EVENTS_NMI if (ARM64_PSEUDO_NMI || ARM64_NMI)
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
select HAVE_REGS_AND_STACK_ACCESS_API
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 34/39] irqchip/gic-v3: Rename nmi to pseudo_nmi
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (32 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 33/39] arm64: nmi: Select HAVE_PERF_EVENTS_NMI for FEAT_NMI Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 35/39] irqchip/gic-v3: Implement FEAT_GICv3_NMI support Vladimir Murzin
` (7 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
Mark Brown, maz, will, ruanjinjie
From: Lorenzo Pieralisi <lpieralisi@kernel.org>
Currently, the terms "NMI" and "pseudo-NMI" are used
interchangeably. That creates confusion with upcoming FEAT_GICv3_NMI
support. Rename helper functions and variables related to pseudo-NMI
to make the code paths clearer and more explicit.
Signed-off-by: Lorenzo Pieralisi <lpieralisi@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
drivers/irqchip/irq-gic-v3.c | 28 ++++++++++++++--------------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 94873646d46f..d9049c0679ff 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -87,7 +87,7 @@ static DEFINE_STATIC_KEY_TRUE(supports_deactivate_key);
#define GIC_LINE_NR min(GICD_TYPER_SPIS(gic_data.rdists.gicd_typer), 1020U)
#define GIC_ESPI_NR GICD_TYPER_ESPIS(gic_data.rdists.gicd_typer)
-static bool nmi_support_forbidden;
+static bool pseudo_nmi_support_forbidden;
/*
* There are 16 SGIs, though we only actually use 8 in Linux. The other 8 SGIs
@@ -117,7 +117,7 @@ static bool nmi_support_forbidden;
* - Figure 4-7 Secure read of the priority field for a Non-secure Group 1
* interrupt.
*/
-static DEFINE_STATIC_KEY_FALSE(supports_pseudo_nmis);
+static DEFINE_STATIC_KEY_FALSE(supports_pseudo_nmi);
static u32 gic_get_pribits(void)
{
@@ -187,7 +187,7 @@ static void __init gic_prio_init(void)
pr_warn("Broken GIC integration, security disabled\n");
} else {
pr_warn("Broken GIC integration, pNMI forbidden\n");
- nmi_support_forbidden = true;
+ pseudo_nmi_support_forbidden = true;
}
}
@@ -514,10 +514,10 @@ static void gic_unmask_irq(struct irq_data *d)
gic_poke_irq(d, GICD_ISENABLER);
}
-static inline bool gic_supports_nmi(void)
+static inline bool gic_supports_pseudo_nmi(void)
{
return IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) &&
- static_branch_likely(&supports_pseudo_nmis);
+ static_branch_likely(&supports_pseudo_nmi);
}
static int gic_irq_set_irqchip_state(struct irq_data *d,
@@ -600,7 +600,7 @@ static int gic_irq_nmi_setup(struct irq_data *d)
{
struct irq_desc *desc = irq_to_desc(d->irq);
- if (!gic_supports_nmi())
+ if (!gic_supports_pseudo_nmi())
return -EINVAL;
if (gic_peek_irq(d, GICD_ISENABLER)) {
@@ -628,7 +628,7 @@ static void gic_irq_nmi_teardown(struct irq_data *d)
{
struct irq_desc *desc = irq_to_desc(d->irq);
- if (WARN_ON(!gic_supports_nmi()))
+ if (WARN_ON(!gic_supports_pseudo_nmi()))
return;
if (gic_peek_irq(d, GICD_ISENABLER)) {
@@ -806,7 +806,7 @@ static inline void gic_complete_ack(u32 irqnr)
static bool gic_rpr_is_nmi_prio(void)
{
- if (!gic_supports_nmi())
+ if (!gic_supports_pseudo_nmi())
return false;
return unlikely(gic_read_rpr() == GICV3_PRIO_NMI);
@@ -913,7 +913,7 @@ static void __gic_handle_irq_from_irqsoff(struct pt_regs *regs)
static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
{
- if (unlikely(gic_supports_nmi() && !interrupts_enabled(regs)))
+ if (unlikely(gic_supports_pseudo_nmi() && !interrupts_enabled(regs)))
__gic_handle_irq_from_irqsoff(regs);
else
__gic_handle_irq_from_irqson(regs);
@@ -1164,7 +1164,7 @@ static void gic_cpu_sys_reg_init(void)
/* Set priority mask register */
if (!gic_prio_masking_enabled()) {
write_gicreg(DEFAULT_PMR_VALUE, ICC_PMR_EL1);
- } else if (gic_supports_nmi()) {
+ } else if (gic_supports_pseudo_nmi()) {
/*
* Check that all CPUs use the same priority space.
*
@@ -1958,15 +1958,15 @@ static const struct gic_quirk gic_quirks[] = {
}
};
-static void gic_enable_nmi_support(void)
+static void gic_enable_pseudo_nmi(void)
{
- if (!gic_prio_masking_enabled() || nmi_support_forbidden)
+ if (!gic_prio_masking_enabled() || pseudo_nmi_support_forbidden)
return;
pr_info("Pseudo-NMIs enabled using %s ICC_PMR_EL1 synchronisation\n",
gic_has_relaxed_pmr_sync() ? "relaxed" : "forced");
- static_branch_enable(&supports_pseudo_nmis);
+ static_branch_enable(&supports_pseudo_nmi);
if (static_branch_likely(&supports_deactivate_key))
gic_eoimode1_chip.flags |= IRQCHIP_SUPPORTS_NMI;
@@ -2050,7 +2050,7 @@ static int __init gic_init_bases(phys_addr_t dist_phys_base,
gic_prio_init();
gic_dist_init();
gic_cpu_init();
- gic_enable_nmi_support();
+ gic_enable_pseudo_nmi();
gic_smp_init();
gic_cpu_pm_init();
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 35/39] irqchip/gic-v3: Implement FEAT_GICv3_NMI support
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (33 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 34/39] irqchip/gic-v3: Rename nmi to pseudo_nmi Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 36/39] arm64: smp: Add NMI support for LPI-backed IPIs Vladimir Murzin
` (6 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, catalin.marinas, lpieralisi, liaochang1, ashok.raj,
Mark Brown, maz, will, ruanjinjie
From: Lorenzo Pieralisi <lpieralisi@kernel.org>
The FEAT_GICv3_NMI GIC feature coupled with the CPU FEAT_NMI enables
handling NMI interrupts in HW on aarch64, by adding a superpriority
interrupt to the existing GIC priority scheme.
Implement GIC driver support for the FEAT_GICv3_NMI feature.
Check, through the ARM64 capabilitity infrastructure, if support
for FEAT_NMI was detected on the core and the system has not overridden
the detection and forced pseudo-NMIs enablement.
If FEAT_NMI is detected, it was not overridden (check embedded in the
system_uses_nmi() call) and the GIC supports the FEAT_GICv3_NMI
feature, initialize NMIs related HW GIC registers and route irq to NMI
handling logic.
Signed-off-by: Lorenzo Pieralisi <lpieralisi@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
drivers/irqchip/irq-gic-v3.c | 127 +++++++++++++++++++++++++++--
include/linux/irqchip/arm-gic-v3.h | 4 +
2 files changed, 124 insertions(+), 7 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index d9049c0679ff..853d12951288 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -64,6 +64,7 @@ struct gic_chip_data {
u32 nr_redist_regions;
u64 flags;
bool has_rss;
+ bool has_nmi;
unsigned int ppi_nr;
struct partition_affinity *parts;
unsigned int nr_parts;
@@ -89,6 +90,9 @@ static DEFINE_STATIC_KEY_TRUE(supports_deactivate_key);
static bool pseudo_nmi_support_forbidden;
+static DEFINE_RAW_SPINLOCK(gic_nmi_lock);
+
+
/*
* There are 16 SGIs, though we only actually use 8 in Linux. The other 8 SGIs
* are potentially stolen by the secure side. Some code, especially code dealing
@@ -254,6 +258,37 @@ enum gic_intid_range {
__INVALID_RANGE__
};
+#ifdef CONFIG_ARM64
+#include <asm/cpufeature.h>
+
+static inline bool gic_supports_v3_3_nmi(void)
+{
+ return gic_data.has_nmi && system_uses_nmi();
+}
+
+static inline u64 gic_read_nmiar(void)
+{
+ u64 irqstat;
+
+ irqstat = read_sysreg_s(SYS_ICC_NMIAR1_EL1);
+
+ dsb(sy);
+
+ return irqstat;
+}
+
+#else
+static inline bool gic_supports_v3_3_nmi(void)
+{
+ return false;
+}
+
+static inline u64 gic_read_nmiar(void)
+{
+ BUG();
+}
+#endif
+
static enum gic_intid_range __get_intid_range(irq_hw_number_t hwirq)
{
switch (hwirq) {
@@ -440,6 +475,8 @@ static u32 convert_offset_index(struct irq_data *d, u32 offset, u32 *index)
return GICD_ICFGRnE;
case GICD_IROUTER:
return GICD_IROUTERnE;
+ case GICD_INMIR:
+ return GICD_INMIRnE;
default:
break;
}
@@ -469,6 +506,42 @@ static int gic_peek_irq(struct irq_data *d, u32 offset)
return !!(readl_relaxed(base + offset + (index / 32) * 4) & mask);
}
+static void gic_irq_configure_nmi(struct irq_data *d, bool enable)
+{
+ void __iomem *base, *addr;
+ unsigned long val;
+ u32 offset, index;
+
+ offset = convert_offset_index(d, GICD_INMIR, &index);
+
+ if (gic_irq_in_rdist(d))
+ base = gic_data_rdist_sgi_base();
+ else
+ base = gic_dist_base_alias(d);
+
+ addr = base + offset + (index / 32) * 4;
+
+ raw_spin_lock(&gic_nmi_lock);
+
+ val = readl_relaxed(addr);
+
+ __assign_bit(index % 32, &val, enable);
+
+ writel_relaxed(val, addr);
+
+ raw_spin_unlock(&gic_nmi_lock);
+}
+
+static void gic_irq_enable_nmi(struct irq_data *d)
+{
+ gic_irq_configure_nmi(d, true);
+}
+
+static void gic_irq_disable_nmi(struct irq_data *d)
+{
+ gic_irq_configure_nmi(d, false);
+}
+
static void gic_poke_irq(struct irq_data *d, u32 offset)
{
void __iomem *base;
@@ -600,7 +673,7 @@ static int gic_irq_nmi_setup(struct irq_data *d)
{
struct irq_desc *desc = irq_to_desc(d->irq);
- if (!gic_supports_pseudo_nmi())
+ if (!gic_supports_pseudo_nmi() && !gic_supports_v3_3_nmi())
return -EINVAL;
if (gic_peek_irq(d, GICD_ISENABLER)) {
@@ -619,7 +692,10 @@ static int gic_irq_nmi_setup(struct irq_data *d)
if (!gic_irq_in_rdist(d))
desc->handle_irq = handle_fasteoi_nmi;
- gic_irq_set_prio(d, dist_prio_nmi);
+ if (gic_supports_v3_3_nmi())
+ gic_irq_enable_nmi(d);
+ else
+ gic_irq_set_prio(d, dist_prio_nmi);
return 0;
}
@@ -628,7 +704,7 @@ static void gic_irq_nmi_teardown(struct irq_data *d)
{
struct irq_desc *desc = irq_to_desc(d->irq);
- if (WARN_ON(!gic_supports_pseudo_nmi()))
+ if (WARN_ON(!gic_supports_pseudo_nmi() && !gic_supports_v3_3_nmi()))
return;
if (gic_peek_irq(d, GICD_ISENABLER)) {
@@ -647,7 +723,10 @@ static void gic_irq_nmi_teardown(struct irq_data *d)
if (!gic_irq_in_rdist(d))
desc->handle_irq = handle_fasteoi_irq;
- gic_irq_set_prio(d, dist_prio_irq);
+ if (gic_supports_v3_3_nmi())
+ gic_irq_disable_nmi(d);
+ else
+ gic_irq_set_prio(d, dist_prio_irq);
}
static bool gic_arm64_erratum_2941627_needed(struct irq_data *d)
@@ -838,7 +917,8 @@ static void __gic_handle_nmi(u32 irqnr, struct pt_regs *regs)
gic_complete_ack(irqnr);
if (generic_handle_domain_nmi(gic_data.domain, irqnr)) {
- WARN_ONCE(true, "Unexpected pseudo-NMI (irqnr %u)\n", irqnr);
+ WARN_ONCE(true, "Unexpected %sNMI (irqnr %u)\n",
+ gic_supports_pseudo_nmi() ? "pseudo-" : "", irqnr);
gic_deactivate_unhandled(irqnr);
}
}
@@ -913,7 +993,11 @@ static void __gic_handle_irq_from_irqsoff(struct pt_regs *regs)
static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
{
- if (unlikely(gic_supports_pseudo_nmi() && !interrupts_enabled(regs)))
+ if (gic_supports_v3_3_nmi() && in_nmi()) {
+ u32 irqnr = gic_read_nmiar();
+
+ __gic_handle_nmi(irqnr, regs);
+ } else if (unlikely(gic_supports_pseudo_nmi() && !interrupts_enabled(regs)))
__gic_handle_irq_from_irqsoff(regs);
else
__gic_handle_irq_from_irqson(regs);
@@ -955,6 +1039,14 @@ static void __init gic_dist_init(void)
writel_relaxed(REPEAT_BYTE_U32(dist_prio_irq),
base + GICD_IPRIORITYRnE + i);
+ if (gic_data.has_nmi) {
+ for (i = 0; i < GIC_LINE_NR; i += 32)
+ writel_relaxed(0, base + GICD_INMIR + i / 8);
+
+ for (i = 0; i < GIC_ESPI_NR; i += 32)
+ writel_relaxed(0, base + GICD_INMIRnE + i / 8);
+ }
+
/* Now do the common stuff */
gic_dist_config(base, GIC_LINE_NR, dist_prio_irq);
@@ -1292,6 +1384,11 @@ static void gic_cpu_init(void)
for (i = 0; i < gic_data.ppi_nr + SGI_NR; i += 32)
writel_relaxed(~0, rbase + GICR_IGROUPR0 + i / 8);
+ if (gic_data.has_nmi) {
+ for (i = 0; i < gic_data.ppi_nr + SGI_NR; i += 32)
+ writel_relaxed(0, rbase + GICR_INMIR0 + i / 8);
+ }
+
gic_cpu_config(rbase, gic_data.ppi_nr + SGI_NR, dist_prio_irq);
gic_redist_wait_for_rwp();
@@ -1967,6 +2064,19 @@ static void gic_enable_pseudo_nmi(void)
gic_has_relaxed_pmr_sync() ? "relaxed" : "forced");
static_branch_enable(&supports_pseudo_nmi);
+}
+
+static void gic_enable_nmi_support(void)
+{
+ if ((!gic_prio_masking_enabled() || pseudo_nmi_support_forbidden) && !gic_supports_v3_3_nmi())
+ return;
+
+ /*
+ * Initialize pseudo-NMIs only if GIC driver cannot take advantage
+ * of core (FEAT_NMI) and GIC (FEAT_GICv3_NMI) in HW
+ */
+ if (!gic_supports_v3_3_nmi())
+ gic_enable_pseudo_nmi();
if (static_branch_likely(&supports_deactivate_key))
gic_eoimode1_chip.flags |= IRQCHIP_SUPPORTS_NMI;
@@ -2035,6 +2145,9 @@ static int __init gic_init_bases(phys_addr_t dist_phys_base,
irq_domain_update_bus_token(gic_data.domain, DOMAIN_BUS_WIRED);
gic_data.has_rss = !!(typer & GICD_TYPER_RSS);
+ gic_data.has_nmi = !!(typer & GICD_TYPER_NMI);
+
+ pr_info("Non-maskable interrupt property %s supported\n", gic_data.has_nmi ? "is" : "not");
if (typer & GICD_TYPER_MBIS) {
err = mbi_init(handle, gic_data.domain);
@@ -2050,7 +2163,7 @@ static int __init gic_init_bases(phys_addr_t dist_phys_base,
gic_prio_init();
gic_dist_init();
gic_cpu_init();
- gic_enable_pseudo_nmi();
+ gic_enable_nmi_support();
gic_smp_init();
gic_cpu_pm_init();
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index ea5fd2374ebe..ebccd76cadda 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -30,6 +30,7 @@
#define GICD_ICFGR 0x0C00
#define GICD_IGRPMODR 0x0D00
#define GICD_NSACR 0x0E00
+#define GICD_INMIR 0x0F80
#define GICD_IGROUPRnE 0x1000
#define GICD_ISENABLERnE 0x1200
#define GICD_ICENABLERnE 0x1400
@@ -39,6 +40,7 @@
#define GICD_ICACTIVERnE 0x1C00
#define GICD_IPRIORITYRnE 0x2000
#define GICD_ICFGRnE 0x3000
+#define GICD_INMIRnE 0x3B00
#define GICD_IROUTER 0x6000
#define GICD_IROUTERnE 0x8000
#define GICD_IDREGS 0xFFD0
@@ -83,6 +85,7 @@
#define GICD_TYPER_LPIS (1U << 17)
#define GICD_TYPER_MBIS (1U << 16)
#define GICD_TYPER_ESPI (1U << 8)
+#define GICD_TYPER_NMI (1U << 9)
#define GICD_TYPER_ID_BITS(typer) ((((typer) >> 19) & 0x1f) + 1)
#define GICD_TYPER_NUM_LPIS(typer) ((((typer) >> 11) & 0x1f) + 1)
@@ -238,6 +241,7 @@
#define GICR_ICFGR0 GICD_ICFGR
#define GICR_IGRPMODR0 GICD_IGRPMODR
#define GICR_NSACR GICD_NSACR
+#define GICR_INMIR0 GICD_INMIR
#define GICR_TYPER_PLPIS (1U << 0)
#define GICR_TYPER_VLPIS (1U << 1)
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 36/39] arm64: smp: Add NMI support for LPI-backed IPIs
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (34 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 35/39] irqchip/gic-v3: Implement FEAT_GICv3_NMI support Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 37/39] genirq: Trust local irqchip when considering IRQ for NMI Vladimir Murzin
` (5 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
LPI-backed IPIs are currently always requested and managed as regular
IRQs. Allow IPIs selected by ipi_should_be_nmi() to be requested as
NMIs, falling back to regular IRQs when the NMI request fails.
Track which IPIs were successfully requested as NMIs so that the
per-CPU enable and disable paths use the appropriate NMI helpers.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
arch/arm64/kernel/smp.c | 38 +++++++++++++++++++++++++++++++++-----
1 file changed, 33 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 58b705426120..c4d1aff31258 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -1177,16 +1177,28 @@ static const struct ipi_irq_ops ipi_sgi_ops = {
static void ipi_lpi_enable(int cpu, int ipi)
{
- enable_irq(irq_desc_get_irq(get_ipi_desc(cpu, ipi)));
+ int irq = irq_desc_get_irq(get_ipi_desc(cpu, ipi));
+
+ if (ipi_is_nmi(cpu, ipi))
+ enable_nmi(irq);
+ else
+ enable_irq(irq);
}
static void ipi_lpi_disable(int cpu, int ipi)
{
- disable_irq_nosync(irq_desc_get_irq(get_ipi_desc(cpu, ipi)));
+ int irq = irq_desc_get_irq(get_ipi_desc(cpu, ipi));
+
+ if (ipi_is_nmi(cpu, ipi))
+ disable_nmi_nosync(irq);
+ else
+ disable_irq_nosync(irq);
}
static void ipi_lpi_setup(int ipi, int ncpus)
{
+ bool is_nmi = ipi_should_be_nmi(ipi);
+
for (int cpu = 0; cpu < ncpus; cpu++) {
int err, irq;
@@ -1195,13 +1207,29 @@ static void ipi_lpi_setup(int ipi, int ncpus)
err = irq_force_affinity(irq, cpumask_of(cpu));
WARN(err, "Could not force affinity IRQ %d, err=%d\n", irq, err);
- err = request_irq(irq, ipi_handler, IRQF_NO_AUTOEN, "IPI",
- NULL);
- WARN(err, "Could not request IRQ %d, err=%d\n", irq, err);
+ /*
+ * Try to request NMI if it should be
+ */
+ if (is_nmi && request_nmi(irq, ipi_handler, IRQF_PERCPU | IRQF_NO_AUTOEN, "IPI", NULL)) {
+ /*
+ * Fallback to regular IRQ
+ */
+ is_nmi = false;
+ }
+ /*
+ * Otherwise regular IRQ
+ */
+ if (!is_nmi) {
+ err = request_irq(irq, ipi_handler, IRQF_NO_AUTOEN, "IPI", NULL);
+ WARN(err, "Could not request IRQ %d, err=%d\n", irq, err);
+ }
irq_set_status_flags(irq, (IRQ_HIDDEN | IRQ_NO_BALANCING_MASK));
get_ipi_desc(cpu, ipi) = irq_to_desc(irq);
+
+ if (is_nmi)
+ set_bit(ipi, get_ipi_nmi_bitmap(cpu));
}
}
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 37/39] genirq: Trust local irqchip when considering IRQ for NMI
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (35 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 36/39] arm64: smp: Add NMI support for LPI-backed IPIs Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 38/39] genirq: Do not call add_interrupt_randomness() in NMI context Vladimir Murzin
` (4 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
Generic IRQ code sanitises whether an IRQ can be considered for NMI
use. Currently, it only considers IRQs managed by the root irqchip.
GICv5 IPIs belong to a logical domain built on top of the LPI domain.
Consequently, generic IRQ code rejects these IPIs for NMI use, even
though their hardware state is managed by the root LPI domain.
One way to allow GICv5 IPIs to be used as NMIs would be to adjust the
sanitisation logic to account for this domain hierarchy. However, the
generic IRQ code can instead trust the local irqchip.
Remove the root irqchip check and use the local irqchip when
considering an IRQ for NMI use.
Suggested-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
include/linux/irq.h | 2 +-
kernel/irq/manage.c | 5 -----
2 files changed, 1 insertion(+), 6 deletions(-)
diff --git a/include/linux/irq.h b/include/linux/irq.h
index f485369b1b4f..92d3ad639ae8 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -563,7 +563,7 @@ struct irq_chip {
* IRQCHIP_ONESHOT_SAFE: One shot does not require mask/unmask
* IRQCHIP_EOI_THREADED: Chip requires eoi() on unmask in threaded mode
* IRQCHIP_SUPPORTS_LEVEL_MSI: Chip can provide two doorbells for Level MSIs
- * IRQCHIP_SUPPORTS_NMI: Chip can deliver NMIs, only for root irqchips
+ * IRQCHIP_SUPPORTS_NMI: Chip can deliver NMIs
* IRQCHIP_ENABLE_WAKEUP_ON_SUSPEND: Invokes __enable_irq()/__disable_irq() for wake irqs
* in the suspend path if they are in disabled state
* IRQCHIP_AFFINITY_PRE_STARTUP: Default affinity update before startup
diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index ec3ecb0e48ce..9c32607f1ac3 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -1377,11 +1377,6 @@ static bool irq_supports_nmi(struct irq_desc *desc)
{
struct irq_data *d = irq_desc_get_irq_data(desc);
-#ifdef CONFIG_IRQ_DOMAIN_HIERARCHY
- /* Only IRQs directly managed by the root irqchip can be set as NMI */
- if (d->parent_data)
- return false;
-#endif
/* Don't support NMIs for chips behind a slow bus */
if (d->chip->irq_bus_lock || d->chip->irq_bus_sync_unlock)
return false;
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 38/39] genirq: Do not call add_interrupt_randomness() in NMI context
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (36 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 37/39] genirq: Trust local irqchip when considering IRQ for NMI Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 39/39] irqchip/gic-v5: Add NMI support Vladimir Murzin
` (3 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
handle_irq_event_percpu() calls add_interrupt_randomness(). Similarly
to 512718bbc51b ("genirq/chip: Don't call add_interrupt_randomness()
for NMIs"), calling it while handling an NMI is not safe.
Stop calling add_interrupt_randomness() when handle_irq_event_percpu()
runs in NMI context.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
kernel/irq/handle.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
index b7d52821837b..114fd63b0210 100644
--- a/kernel/irq/handle.c
+++ b/kernel/irq/handle.c
@@ -245,7 +245,8 @@ irqreturn_t handle_irq_event_percpu(struct irq_desc *desc)
retval = __handle_irq_event_percpu(desc);
- add_interrupt_randomness(desc->irq_data.irq);
+ if (!in_nmi())
+ add_interrupt_randomness(desc->irq_data.irq);
if (!irq_settings_no_debug(desc))
note_interrupt(desc, retval);
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* [PATCH v2 39/39] irqchip/gic-v5: Add NMI support
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (37 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 38/39] genirq: Do not call add_interrupt_randomness() in NMI context Vladimir Murzin
@ 2026-11-12 13:20 ` Vladimir Murzin
2026-11-14 5:03 ` [PATCH v2 00/39] arm64: Add support for FEAT_NMI Ashok Raj
` (2 subsequent siblings)
41 siblings, 0 replies; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-12 13:20 UTC (permalink / raw)
To: linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, ruanjinjie,
liaochang1, ashok.raj, maz
Implement NMI support for PPIs, SPIs, LPIs and IPIs by toggling their
priority between the superpriority and regular interrupt levels.
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
drivers/irqchip/irq-gic-v5.c | 244 +++++++++++++++++++++++++++++++++--
1 file changed, 234 insertions(+), 10 deletions(-)
diff --git a/drivers/irqchip/irq-gic-v5.c b/drivers/irqchip/irq-gic-v5.c
index 8c0303af71e7..585c8e5fcd68 100644
--- a/drivers/irqchip/irq-gic-v5.c
+++ b/drivers/irqchip/irq-gic-v5.c
@@ -22,6 +22,7 @@
static u8 pri_bits __ro_after_init = 5;
#define GICV5_IRQ_PRI_MASK 0x1f
+#define GICV5_IRQ_PRI_NMI 0
#define GICV5_IRQ_PRI_MI (GICV5_IRQ_PRI_MASK & GENMASK(4, 5 - pri_bits))
#define PPI_NR 128
@@ -97,6 +98,56 @@ static void gicv5_hwirq_init(irq_hw_number_t hwirq, u8 priority, u8 hwirq_type)
}
}
+static void gicv5_ppi_irq_set_priority(struct irq_data *d, u8 priority)
+{
+ u64 hwirq_id_offset = d->hwirq / 8;
+ u64 hwirq_id_index = d->hwirq % 8;
+ u64 hwirq_id_clear = (u64)GICV5_IRQ_PRI_MASK << (hwirq_id_index * 8);
+ u64 hwirq_id_set = (u64)priority << (hwirq_id_index * 8);
+
+#define SET_ICC_PPI_PRIORITYRn_EL1(cnt) \
+ case cnt: sysreg_clear_set_s(SYS_ICC_PPI_PRIORITYR ## cnt ## _EL1, hwirq_id_clear, hwirq_id_set); break
+
+ switch (hwirq_id_offset) {
+ SET_ICC_PPI_PRIORITYRn_EL1(0);
+ SET_ICC_PPI_PRIORITYRn_EL1(1);
+ SET_ICC_PPI_PRIORITYRn_EL1(2);
+ SET_ICC_PPI_PRIORITYRn_EL1(3);
+ SET_ICC_PPI_PRIORITYRn_EL1(4);
+ SET_ICC_PPI_PRIORITYRn_EL1(5);
+ SET_ICC_PPI_PRIORITYRn_EL1(6);
+ SET_ICC_PPI_PRIORITYRn_EL1(7);
+ SET_ICC_PPI_PRIORITYRn_EL1(8);
+ SET_ICC_PPI_PRIORITYRn_EL1(9);
+ SET_ICC_PPI_PRIORITYRn_EL1(10);
+ SET_ICC_PPI_PRIORITYRn_EL1(11);
+ SET_ICC_PPI_PRIORITYRn_EL1(12);
+ SET_ICC_PPI_PRIORITYRn_EL1(13);
+ SET_ICC_PPI_PRIORITYRn_EL1(14);
+ SET_ICC_PPI_PRIORITYRn_EL1(15);
+#undef SET_ICC_PPI_PRIORITYRn_EL1
+ default:
+ WARN_ON(1);
+ }
+ /*
+ * Context syncronization required to make sure system register writes
+ * effects are synchronised.
+ */
+ isb();
+}
+
+static void gicv5_hwirq_set_priority(irq_hw_number_t hwirq, u8 priority, u8 hwirq_type) {
+ u64 cdpri;
+
+ cdpri = FIELD_PREP(GICV5_GIC_CDPRI_PRIORITY_MASK, priority) |
+ FIELD_PREP(GICV5_GIC_CDPRI_TYPE_MASK, hwirq_type) |
+ FIELD_PREP(GICV5_GIC_CDPRI_ID_MASK, hwirq);
+
+ gic_insn(cdpri, CDPRI);
+
+ return;
+}
+
static void gicv5_ppi_irq_mask(struct irq_data *d)
{
u64 hwirq_id_bit = BIT_ULL(d->hwirq % 64);
@@ -512,7 +563,152 @@ static int gicv5_ppi_irq_set_vcpu_affinity(struct irq_data *d, void *vcpu)
return 0;
}
-static const struct irq_chip gicv5_ppi_irq_chip = {
+static int gicv5_hwirq_peek_irq(irq_hw_number_t hwirq, u8 hwirq_type)
+{
+ u64 icsr, cdrcfg;
+
+ cdrcfg = hwirq | FIELD_PREP(GICV5_GIC_CDRCFG_TYPE_MASK, hwirq_type);
+
+ gic_insn(cdrcfg, CDRCFG);
+ isb();
+ icsr = read_sysreg_s(SYS_ICC_ICSR_EL1);
+
+ if (FIELD_GET(ICC_ICSR_EL1_F, icsr)) {
+ pr_err("ICSR_EL1 is invalid\n");
+ return -EINVAL;
+ }
+
+ return !!(FIELD_GET(ICC_ICSR_EL1_Enabled, icsr));
+
+}
+
+static int gicv5_ppi_irq_nmi_setup(struct irq_data *d)
+{
+ unsigned long hwirq_id_bit = d->hwirq % 64;
+ unsigned long enable;
+
+ if (!system_uses_nmi())
+ return -EINVAL;
+
+ if (d->hwirq < 64)
+ enable = read_sysreg_s(SYS_ICC_PPI_ENABLER0_EL1);
+ else
+ enable = read_sysreg_s(SYS_ICC_PPI_ENABLER1_EL1);
+
+ if (test_bit(hwirq_id_bit, &enable))
+ return -EINVAL;
+
+ gicv5_ppi_irq_set_priority(d, GICV5_IRQ_PRI_NMI);
+
+ return 0;
+}
+
+static void gicv5_ppi_irq_nmi_teardown(struct irq_data *d)
+{
+ unsigned long hwirq_id_bit = d->hwirq % 64;
+ unsigned long enable;
+
+ if (WARN_ON(!system_uses_nmi()))
+ return;
+
+ if (d->hwirq < 64)
+ enable = read_sysreg_s(SYS_ICC_PPI_ENABLER0_EL1);
+ else
+ enable = read_sysreg_s(SYS_ICC_PPI_ENABLER1_EL1);
+
+ if (test_bit(hwirq_id_bit, &enable))
+ return;
+
+ gicv5_ppi_irq_set_priority(d, GICV5_IRQ_PRI_MI);
+
+ return;
+}
+
+static int gicv5_hwirq_irq_nmi_setup(irq_hw_number_t hwirq, u8 hwirq_type)
+{
+ if (!system_uses_nmi())
+ return -EINVAL;
+
+ if (WARN_ON(hwirq_type != GICV5_HWIRQ_TYPE_LPI && hwirq_type != GICV5_HWIRQ_TYPE_SPI))
+ return -EINVAL;
+
+ if (gicv5_hwirq_peek_irq(hwirq, hwirq_type))
+ return -EINVAL;
+
+ gicv5_hwirq_set_priority(hwirq, GICV5_IRQ_PRI_NMI, hwirq_type);
+
+ return 0;
+}
+
+static void gicv5_hwirq_irq_nmi_teardown(irq_hw_number_t hwirq, u8 hwirq_type)
+{
+ if (WARN_ON(!system_uses_nmi()))
+ return;
+
+ if (WARN_ON(hwirq_type != GICV5_HWIRQ_TYPE_LPI && hwirq_type != GICV5_HWIRQ_TYPE_SPI))
+ return;
+
+ if (gicv5_hwirq_peek_irq(hwirq, hwirq_type))
+ return;
+
+ gicv5_hwirq_set_priority(hwirq, GICV5_IRQ_PRI_MI, hwirq_type);
+
+ return;
+}
+
+static int gicv5_lpi_irq_nmi_setup(struct irq_data *d)
+{
+ int ret = gicv5_hwirq_irq_nmi_setup(d->hwirq, GICV5_HWIRQ_TYPE_LPI);
+
+ if (!ret)
+ irq_to_desc(d->irq)->handle_irq = handle_fasteoi_nmi;
+
+ return ret;
+}
+
+static void gicv5_lpi_irq_nmi_teardown(struct irq_data *d)
+{
+ gicv5_hwirq_irq_nmi_teardown(d->hwirq, GICV5_HWIRQ_TYPE_LPI);
+ irq_to_desc(d->irq)->handle_irq = handle_fasteoi_irq;
+}
+
+static int gicv5_spi_irq_nmi_setup(struct irq_data *d)
+{
+ int ret = gicv5_hwirq_irq_nmi_setup(d->hwirq, GICV5_HWIRQ_TYPE_SPI);
+
+ if (!ret)
+ irq_to_desc(d->irq)->handle_irq = handle_fasteoi_nmi;
+
+ return ret;
+}
+
+static void gicv5_spi_irq_nmi_teardown(struct irq_data *d)
+{
+ gicv5_hwirq_irq_nmi_teardown(d->hwirq, GICV5_HWIRQ_TYPE_SPI);
+ irq_to_desc(d->irq)->handle_irq = handle_fasteoi_irq;
+}
+
+static int gicv5_ipi_irq_nmi_setup(struct irq_data *d)
+{
+ if (WARN_ON(!d->parent_data))
+ return -EINVAL;
+
+ d = d->parent_data;
+
+ return gicv5_hwirq_irq_nmi_setup(d->hwirq, GICV5_HWIRQ_TYPE_LPI);
+}
+
+static void gicv5_ipi_irq_nmi_teardown(struct irq_data *d)
+{
+ if (WARN_ON(!d->parent_data))
+ return;
+
+ d = d->parent_data;
+
+ gicv5_hwirq_irq_nmi_teardown(d->hwirq, GICV5_HWIRQ_TYPE_LPI);
+}
+
+static struct irq_chip gicv5_ppi_irq_chip __ro_after_init = {
.name = "GICv5-PPI",
.irq_mask = gicv5_ppi_irq_mask,
.irq_unmask = gicv5_ppi_irq_unmask,
@@ -521,11 +717,13 @@ static const struct irq_chip gicv5_ppi_irq_chip = {
.irq_get_irqchip_state = gicv5_ppi_irq_get_irqchip_state,
.irq_set_irqchip_state = gicv5_ppi_irq_set_irqchip_state,
.irq_set_vcpu_affinity = gicv5_ppi_irq_set_vcpu_affinity,
+ .irq_nmi_setup = gicv5_ppi_irq_nmi_setup,
+ .irq_nmi_teardown = gicv5_ppi_irq_nmi_teardown,
.flags = IRQCHIP_SKIP_SET_WAKE |
IRQCHIP_MASK_ON_SUSPEND,
};
-static const struct irq_chip gicv5_spi_irq_chip = {
+static struct irq_chip gicv5_spi_irq_chip __ro_after_init = {
.name = "GICv5-SPI",
.irq_mask = gicv5_spi_irq_mask,
.irq_unmask = gicv5_spi_irq_unmask,
@@ -535,13 +733,15 @@ static const struct irq_chip gicv5_spi_irq_chip = {
.irq_retrigger = gicv5_spi_irq_retrigger,
.irq_get_irqchip_state = gicv5_spi_irq_get_irqchip_state,
.irq_set_irqchip_state = gicv5_spi_irq_set_irqchip_state,
+ .irq_nmi_setup = gicv5_spi_irq_nmi_setup,
+ .irq_nmi_teardown = gicv5_spi_irq_nmi_teardown,
.flags = IRQCHIP_SET_TYPE_MASKED |
IRQCHIP_AFFINITY_PRE_STARTUP |
IRQCHIP_SKIP_SET_WAKE |
IRQCHIP_MASK_ON_SUSPEND,
};
-static const struct irq_chip gicv5_lpi_irq_chip = {
+static struct irq_chip gicv5_lpi_irq_chip __ro_after_init = {
.name = "GICv5-LPI",
.irq_mask = gicv5_lpi_irq_mask,
.irq_unmask = gicv5_lpi_irq_unmask,
@@ -550,12 +750,14 @@ static const struct irq_chip gicv5_lpi_irq_chip = {
.irq_retrigger = gicv5_lpi_irq_retrigger,
.irq_get_irqchip_state = gicv5_lpi_irq_get_irqchip_state,
.irq_set_irqchip_state = gicv5_lpi_irq_set_irqchip_state,
+ .irq_nmi_setup = gicv5_lpi_irq_nmi_setup,
+ .irq_nmi_teardown = gicv5_lpi_irq_nmi_teardown,
.flags = IRQCHIP_AFFINITY_PRE_STARTUP |
IRQCHIP_SKIP_SET_WAKE |
IRQCHIP_MASK_ON_SUSPEND,
};
-static const struct irq_chip gicv5_ipi_irq_chip = {
+static struct irq_chip gicv5_ipi_irq_chip __ro_after_init = {
.name = "GICv5-IPI",
.irq_mask = irq_chip_mask_parent,
.irq_unmask = irq_chip_unmask_parent,
@@ -563,6 +765,8 @@ static const struct irq_chip gicv5_ipi_irq_chip = {
.irq_set_affinity = irq_chip_set_affinity_parent,
.irq_get_irqchip_state = irq_chip_get_parent_state,
.irq_set_irqchip_state = irq_chip_set_parent_state,
+ .irq_nmi_setup = gicv5_ipi_irq_nmi_setup,
+ .irq_nmi_teardown = gicv5_ipi_irq_nmi_teardown,
.ipi_send_single = gicv5_ipi_send_single,
.flags = IRQCHIP_SKIP_SET_WAKE |
IRQCHIP_MASK_ON_SUSPEND,
@@ -907,11 +1111,12 @@ static const struct irq_domain_ops gicv5_irq_ipi_domain_ops = {
.free = gicv5_irq_ipi_domain_free,
};
-static void handle_irq_per_domain(u32 hwirq)
+static void handle_irq_per_domain(u32 hwirq, bool is_nmi)
{
u8 hwirq_type = FIELD_GET(GICV5_HWIRQ_TYPE, hwirq);
u32 hwirq_id = FIELD_GET(GICV5_HWIRQ_ID, hwirq);
struct irq_domain *domain;
+ int ret;
switch (hwirq_type) {
case GICV5_HWIRQ_TYPE_PPI:
@@ -928,26 +1133,32 @@ static void handle_irq_per_domain(u32 hwirq)
return;
}
- if (generic_handle_domain_irq(domain, hwirq_id)) {
- pr_err_once("Could not handle, hwirq = 0x%x", hwirq_id);
+ if (is_nmi)
+ ret = generic_handle_domain_nmi(domain, hwirq_id);
+ else
+ ret = generic_handle_domain_irq(domain, hwirq_id);
+
+ if (ret) {
+ pr_err_once("Could not handle, hwirq = 0x%x (%d)", hwirq_id, ret);
gicv5_hwirq_eoi(hwirq_id, hwirq_type);
}
}
static void __exception_irq_entry gicv5_handle_irq(struct pt_regs *regs)
{
+ bool is_nmi = in_nmi();
bool valid;
u32 hwirq;
u64 ia;
- ia = gicr_insn(CDIA);
+ ia = is_nmi ? gicr_insn(CDNMIA) : gicr_insn(CDIA);
valid = GICV5_GICR_CDIA_VALID(ia);
if (!valid)
return;
/*
- * Ensure that the CDIA instruction effects (ie IRQ activation) are
+ * Ensure that the CD{NM}IA instruction effects (ie IRQ activation) are
* completed before handling the interrupt.
*/
gsb_ack();
@@ -967,7 +1178,7 @@ static void __exception_irq_entry gicv5_handle_irq(struct pt_regs *regs)
hwirq = FIELD_GET(GICV5_HWIRQ_INTID, ia);
- handle_irq_per_domain(hwirq);
+ handle_irq_per_domain(hwirq, is_nmi);
}
static void gicv5_cpu_disable_interrupts(void)
@@ -1010,6 +1221,17 @@ static int gicv5_starting_cpu(unsigned int cpu)
return gicv5_irs_register_cpu(cpu);
}
+static void gicv5_enable_nmi_support(void)
+{
+ if (!system_uses_nmi())
+ return;
+
+ gicv5_ppi_irq_chip.flags |= IRQCHIP_SUPPORTS_NMI;
+ gicv5_spi_irq_chip.flags |= IRQCHIP_SUPPORTS_NMI;
+ gicv5_lpi_irq_chip.flags |= IRQCHIP_SUPPORTS_NMI;
+ gicv5_ipi_irq_chip.flags |= IRQCHIP_SUPPORTS_NMI;
+}
+
static void __init gicv5_smp_init(void)
{
unsigned int num_ipis = GICV5_IPIS_PER_CPU * nr_cpu_ids;
@@ -1272,6 +1494,8 @@ static int __init gicv5_init_common(struct fwnode_handle *parent_domain)
if (set_handle_irq(gicv5_handle_irq))
panic("GICv5: unable to install root IRQ handler\n");
+ gicv5_enable_nmi_support();
+
gicv5_smp_init();
gicv5_irs_its_probe();
--
2.34.1
^ permalink raw reply related [flat|nested] 49+ messages in thread
* Re: [PATCH v2 17/39] arm64/efi: Implement override for IRQ flags checks
2026-11-12 13:20 ` [PATCH v2 17/39] arm64/efi: Implement override for " Vladimir Murzin
@ 2026-11-13 13:22 ` Mark Brown
0 siblings, 0 replies; 49+ messages in thread
From: Mark Brown @ 2026-11-13 13:22 UTC (permalink / raw)
To: Vladimir Murzin
Cc: linux-arm-kernel, catalin.marinas, will, mark.rutland, lpieralisi,
ruanjinjie, liaochang1, ashok.raj, maz
[-- Attachment #1: Type: text/plain, Size: 496 bytes --]
On Thu, Nov 12, 2026 at 01:20:20PM +0000, Vladimir Murzin wrote:
> +#define arch_efi_check_flags(state_flags, caller) \
> +do { \
> + arm64_exc_hwstate_t __actual = {.flags = arch_local_save_flags() }; \
> + arm64_exc_hwstate_t __expected = {.flags=state_flags}; \
The generic version has an:
if (!WARN_ON_ONCE(mismatch & ARCH_EFI_IRQ_FLAGS_MASK))
break;
which will generate a backtrace the first time things go wrong, this
doesn't.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v2 00/39] arm64: Add support for FEAT_NMI
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (38 preceding siblings ...)
2026-11-12 13:20 ` [PATCH v2 39/39] irqchip/gic-v5: Add NMI support Vladimir Murzin
@ 2026-11-14 5:03 ` Ashok Raj
2026-11-16 6:58 ` Jinjie Ruan
2026-11-18 12:17 ` Jinjie Ruan
41 siblings, 0 replies; 49+ messages in thread
From: Ashok Raj @ 2026-11-14 5:03 UTC (permalink / raw)
To: Vladimir Murzin
Cc: linux-arm-kernel, catalin.marinas, will, mark.rutland, lpieralisi,
ruanjinjie, liaochang1, maz, Ashok Raj, Ashok Raj,
Jonathan Cameron
On Thu, Nov 12, 2026 at 01:20:03PM +0000, Vladimir Murzin wrote:
> FEAT_NMI provides an architected mechanism for supporting non-maskable
> interrupts (NMIs) and less-masked interrupts (LMIs).
>
> Since we already support pseudo-NMIs via priority masking, introducing
> another flavour of NMI on top of the existing infrastructure could
> easily become messy, making the code harder to follow and reason
> about.
>
> To avoid that, this series first makes room for the new NMI "tenant"
> by restructuring the existing exception masking logic.
>
> The main idea is to separate the logical view of exception state from
> its hardware representation. To achieve this, we introduce logical
> exception contexts that can be mapped onto the corresponding hardware
> state. This naturally consolidates the hardware-specific handling into
> a small number of places, while allowing the rest of the code to
> operate purely in terms of logical exception contexts.
>
> Since this restructuring is non-trivial and carries a risk of subtle
> behavioural changes, the series adds extensive debug checks to verify
> that the hardware state always matches the expected logical state.
>
> With this restructuring in place, wiring FEAT_NMI into the new
> framework becomes much more manageable. GIC support for NMI is build
> on top.
>
> This work would not have been possible without the contributions of
> Ada Couprie Diaz, Mark Brown, and Lorenzo Pieralisi - please credit
> them for everything that works well. Any remaining bugs or issues are
> entirely my own.
>
> Most of the testing has been done on QEMU, Fast Model and pre-silicon
> environment, so the series definitely benefit from extensive testing
> on real hardware (without NMI, with pseudo-NMI, and with FEAT_NMI).
Thanks Vladimir
I have tested this series on QEMU and also another dev platform and they
work.
I have a small change, I'll submit after sanitizing my patch and running
couple more tests.
I haven't reviewed the series which I'll do next :-)
Cheers,
Ashok
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v2 00/39] arm64: Add support for FEAT_NMI
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (39 preceding siblings ...)
2026-11-14 5:03 ` [PATCH v2 00/39] arm64: Add support for FEAT_NMI Ashok Raj
@ 2026-11-16 6:58 ` Jinjie Ruan
2026-11-16 10:02 ` Vladimir Murzin
2026-11-18 12:17 ` Jinjie Ruan
41 siblings, 1 reply; 49+ messages in thread
From: Jinjie Ruan @ 2026-11-16 6:58 UTC (permalink / raw)
To: Vladimir Murzin, linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, liaochang1,
ashok.raj, maz
Hi, Vladimir,
While testing this patch set, I discovered IPI as NMI now defaults to
using SDEI, and the FEAT_NMI cannot cover this path..
In the case where FEAT_NMI is supported, should it default to using
FEAT_NMI instead?
It seems that the motivation behind the cross-CPU NMI via SDEI patch was
a transitional solution in the absence of FEAT_NMI.
Link: https://lore.kernel.org/all/cover.1782744912.git.kas@kernel.org/
Best Regards,
Jinjie
在 2026/11/12 21:20, Vladimir Murzin 写道:
> FEAT_NMI provides an architected mechanism for supporting non-maskable
> interrupts (NMIs) and less-masked interrupts (LMIs).
>
> Since we already support pseudo-NMIs via priority masking, introducing
> another flavour of NMI on top of the existing infrastructure could
> easily become messy, making the code harder to follow and reason
> about.
>
> To avoid that, this series first makes room for the new NMI "tenant"
> by restructuring the existing exception masking logic.
>
> The main idea is to separate the logical view of exception state from
> its hardware representation. To achieve this, we introduce logical
> exception contexts that can be mapped onto the corresponding hardware
> state. This naturally consolidates the hardware-specific handling into
> a small number of places, while allowing the rest of the code to
> operate purely in terms of logical exception contexts.
>
> Since this restructuring is non-trivial and carries a risk of subtle
> behavioural changes, the series adds extensive debug checks to verify
> that the hardware state always matches the expected logical state.
>
> With this restructuring in place, wiring FEAT_NMI into the new
> framework becomes much more manageable. GIC support for NMI is build
> on top.
>
> This work would not have been possible without the contributions of
> Ada Couprie Diaz, Mark Brown, and Lorenzo Pieralisi - please credit
> them for everything that works well. Any remaining bugs or issues are
> entirely my own.
>
> Most of the testing has been done on QEMU, Fast Model and pre-silicon
> environment, so the series definitely benefit from extensive testing
> on real hardware (without NMI, with pseudo-NMI, and with FEAT_NMI).
>
> Patches are organized as follow
>
> 1 - 15
> Rework of existing pseudo-NMI logic to make room for FEAT_NMI
>
> 16 - 33
> FEAT_NMI support including some preparation patches which have
> no dependencies:
> - efi/runtime-wrappers: Permit architectures to override IRQ flags checks
> - arm64: smp: Abstract SGI and LPI operations
>
> 34 - 35
> GICv3 support for NMI
>
> 36 - 39
> GICv5 support for NMI (including some preparation patches)
>
> Changelog
>
> v1 -> v2
>
> - Rebased on v7.4-rc2
> - Select HAVE_PERF_EVENTS_NMI when FEAT_NMI is enabled
> - Reworked APEI SEA handling to mask exceptions sequentially
> - Updated GICv3 initialization to clear NMI configuration
> - Added Tested-by and Reviewed-by tags where approriate
> - Droped tags due to meaningful changes for:
> + "arm64: replace local_daif helpers"
> + "irqchip/gic-v3: Implement FEAT_GICv3_NMI support"
>
> RFC v2 -> v1
>
> - Rebased on v7.3-rc3.
> - Dropped RFC tag, have not received objections, at the same time
> got support from Jinjie
> - Dropped prerequisite patches already present in the newer base
> - Added logic to reject probing/stepping over ALLINT
> - Moved addition of arm64_debug_exc_context() in separate patch
> - Reworked debug logic for validating direction of exception
> transition
> - Split support for NMI in GICv5 in 3 separate patches
> - Added review tags
>
> RFC v1 -> RFC v2
>
> - Rebased on v7.2-rc5.
> - Improved commit descriptions.
> - Split some patches into smaller ones.
> - Added a graceful fallback for IPI requests when FEAT_NMI is
> supported by the CPU but unsupported by the interrupt
> controller.
> - Split the GICv3 preparation from the FEAT_GICv3_NMI
> implementation.
> - Added NMI support to GICv5, primarily as an illustration of the
> implementation, although it also happens to work on Fast Model.
> - Added review tags only to unchanged patches. Review tags were
> intentionally dropped from modified patches to reflect those
> changes and encourage re-review.
>
> RFC v1:
> https://lore.kernel.org/linux-arm-kernel/20260709121333.23507-1-vladimir.murzin@arm.com/
>
> RFC v2:
> https://lore.kernel.org/linux-arm-kernel/20260727163453.7969-1-vladimir.murzin@arm.com/
>
> v1:
> https://lore.kernel.org/linux-arm-kernel/20260914102102.68597-1-vladimir.murzin@arm.com/
>
> Thanks!
> Vladimir
>
> Ada Couprie Diaz (15):
> arm64: irqflags: Introduce arm64-specific irqflags type
> arm64: irqflags: save and use both DAIF and PMR
> arm64: interrupts: Add common exception state helpers
> arm64: entry: Introduce entry specific exception masking helpers
> arm64: entry: replace DAIF helpers with entry helpers
> arm64: interrupts: Introduce exception masking save/restore helpers
> arm64: interrupts: introduce a helper for GIC priority initialization
> arm64: replace local_daif helpers
> arm64: cpuidle: use new helpers to bypass interrupt priority masking
> arm64: remove daifflags.h
> arm64: gicv3: remove GIC_PRIO_PSR_I_SET
> arm64: ptrace: Add PSR_ALLINT_BIT
> arm64: cpufeature: Detect PE support for FEAT_NMI
> arm64: nmi: Manage masking for superpriority interrupts
> arm64: irq: Report FEAT_NMI masking local IRQs
>
> Lorenzo Pieralisi (2):
> irqchip/gic-v3: Rename nmi to pseudo_nmi
> irqchip/gic-v3: Implement FEAT_GICv3_NMI support
>
> Mark Brown (5):
> arm64: booting: Document boot requirements for FEAT_NMI
> arm64: sysreg: Add definitions for immediate versions of MSR ALLINT
> arm64: idreg: Add an override for FEAT_NMI
> arm64: nmi: Add handling of superpriority interrupts as NMIs
> arm64: nmi: Add Kconfig for NMI
>
> Vladimir Murzin (17):
> arm64: process: Lift check for exception state to generic
> infrastructure
> arm64: cpufeature: Remove system_has_prio_mask_debugging()
> arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS
> arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING
> efi/runtime-wrappers: Permit architectures to override IRQ flags
> checks
> arm64/efi: Implement override for IRQ flags checks
> arm64: probes: Reject stepping over ALLINT
> arm64: suspend: Always initialise PSTATE.ALLINT
> arm64/efi: Add ALLINT to IRQ flags checks
> arm64: kprobes: Disable NMIs
> arm64: smp: Abstract SGI and LPI operations
> arm64: smp: Fall back to IRQ when IPI NMI request fails
> arm64: nmi: Select HAVE_PERF_EVENTS_NMI for FEAT_NMI
> arm64: smp: Add NMI support for LPI-backed IPIs
> genirq: Trust local irqchip when considering IRQ for NMI
> genirq: Do not call add_interrupt_randomness() in NMI context
> irqchip/gic-v5: Add NMI support
>
> Documentation/arch/arm64/booting.rst | 6 +
> arch/arm64/Kconfig | 31 +-
> arch/arm64/include/asm/assembler.h | 20 +-
> arch/arm64/include/asm/cpucaps.h | 2 +
> arch/arm64/include/asm/cpufeature.h | 14 +-
> arch/arm64/include/asm/cpuidle.h | 18 +-
> arch/arm64/include/asm/daifflags.h | 144 --------
> arch/arm64/include/asm/efi.h | 42 ++-
> arch/arm64/include/asm/entry-common.h | 8 +-
> arch/arm64/include/asm/insn.h | 1 +
> .../include/asm/interrupts/common_flags.h | 255 +++++++++++++
> arch/arm64/include/asm/interrupts/entry.h | 102 ++++++
> arch/arm64/include/asm/interrupts/masking.h | 166 +++++++++
> arch/arm64/include/asm/irqflags.h | 150 ++++----
> arch/arm64/include/asm/kvm_host.h | 1 -
> arch/arm64/include/asm/mmu_context.h | 1 -
> arch/arm64/include/asm/ptrace.h | 25 +-
> arch/arm64/include/asm/sysreg.h | 2 +
> arch/arm64/include/uapi/asm/ptrace.h | 1 +
> arch/arm64/kernel/acpi.c | 43 +--
> arch/arm64/kernel/cpufeature.c | 68 +++-
> arch/arm64/kernel/debug-monitors.c | 1 -
> arch/arm64/kernel/entry-common.c | 335 +++++++++++++-----
> arch/arm64/kernel/entry.S | 14 +-
> arch/arm64/kernel/hibernate.c | 14 +-
> arch/arm64/kernel/hw_breakpoint.c | 16 +-
> arch/arm64/kernel/irq.c | 45 ++-
> arch/arm64/kernel/machine_kexec.c | 4 +-
> arch/arm64/kernel/pi/idreg-override.c | 1 +
> arch/arm64/kernel/probes/decode-insn.c | 11 +-
> arch/arm64/kernel/probes/kprobes.c | 24 +-
> arch/arm64/kernel/process.c | 24 +-
> arch/arm64/kernel/setup.c | 4 +-
> arch/arm64/kernel/signal.c | 1 -
> arch/arm64/kernel/smp.c | 217 ++++++++----
> arch/arm64/kernel/suspend.c | 14 +-
> arch/arm64/kernel/traps.c | 1 -
> arch/arm64/kvm/hyp/nvhe/switch.c | 2 +-
> arch/arm64/kvm/hyp/vgic-v3-sr.c | 7 +-
> arch/arm64/kvm/hyp/vhe/switch.c | 14 +-
> arch/arm64/mm/fault.c | 1 -
> arch/arm64/mm/mmu.c | 7 +-
> arch/arm64/tools/cpucaps | 2 +
> drivers/firmware/efi/runtime-wrappers.c | 32 +-
> drivers/irqchip/irq-gic-v3.c | 145 +++++++-
> drivers/irqchip/irq-gic-v5.c | 244 ++++++++++++-
> include/linux/irq.h | 2 +-
> include/linux/irqchip/arm-gic-v3-prio.h | 8 -
> include/linux/irqchip/arm-gic-v3.h | 4 +
> kernel/irq/handle.c | 3 +-
> kernel/irq/manage.c | 5 -
> 51 files changed, 1696 insertions(+), 606 deletions(-)
> delete mode 100644 arch/arm64/include/asm/daifflags.h
> create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
> create mode 100644 arch/arm64/include/asm/interrupts/entry.h
> create mode 100644 arch/arm64/include/asm/interrupts/masking.h
>
--
Best regards,
Jinjie
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v2 00/39] arm64: Add support for FEAT_NMI
2026-11-16 6:58 ` Jinjie Ruan
@ 2026-11-16 10:02 ` Vladimir Murzin
2026-11-16 11:01 ` Jinjie Ruan
0 siblings, 1 reply; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-16 10:02 UTC (permalink / raw)
To: Jinjie Ruan, linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, liaochang1,
ashok.raj, maz
Hi Jinjie,
On 11/16/26 06:58, Jinjie Ruan wrote:
> Hi, Vladimir,
>
> While testing this patch set, I discovered IPI as NMI now defaults to
> using SDEI, and the FEAT_NMI cannot cover this path..
>
Can you please expand on that? Sashiko has several comments related to
interaction with SDEI which I'm planing to dig into. However, glancing
over smp_send_stop() IPI as NMI is kind of try and see approach:
try normal IPI
if CPU is still online
try NMI IPI
if CPU is still online
try SDEI NMI
if CPU is still online
give up
So, if NMI IPI did not trigger in your test, I'm really keen on details.
Thanks
Vladimir
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v2 00/39] arm64: Add support for FEAT_NMI
2026-11-16 10:02 ` Vladimir Murzin
@ 2026-11-16 11:01 ` Jinjie Ruan
2026-11-18 13:22 ` Vladimir Murzin
0 siblings, 1 reply; 49+ messages in thread
From: Jinjie Ruan @ 2026-11-16 11:01 UTC (permalink / raw)
To: Vladimir Murzin, linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, liaochang1,
ashok.raj, maz
在 2026/11/16 18:02, Vladimir Murzin 写道:
> Hi Jinjie,
>
> On 11/16/26 06:58, Jinjie Ruan wrote:
>> Hi, Vladimir,
>>
>> While testing this patch set, I discovered IPI as NMI now defaults to
>> using SDEI, and the FEAT_NMI cannot cover this path..
>>
>
> Can you please expand on that? Sashiko has several comments related to
> interaction with SDEI which I'm planing to dig into. However, glancing
I mean the following code now prefer sdei nmi now, we can not use FEAT_NMI:
954 void arch_trigger_cpumask_backtrace(const cpumask_t *mask, int
exclude_cpu)
955 {
956 /*
957 * Prefer the SDEI cross-CPU NMI provider when active: firmware
958 * dispatches the event out of EL3 and reaches CPUs that have
959 * interrupts locally masked, without the per-IRQ-mask cost
that
960 * pseudo-NMI pays for the same reach. The plain IPI path below
961 * can't reach such a CPU unless pseudo-NMI is enabled.
962 */
963 if (sdei_nmi_trigger_cpumask_backtrace(mask, exclude_cpu))
964 return;
965
966 /*
967 * NOTE: though nmi_trigger_cpumask_backtrace() has "nmi_"
in the name,
968 * nothing about it truly needs to be implemented using an
NMI, it's
969 * just that it's _allowed_ to work with NMIs. If
ipi_should_be_nmi()
970 * returned false our backtrace attempt will just use a
regular IPI.
971 */
972 nmi_trigger_cpumask_backtrace(mask, exclude_cpu,
arm64_backtrace_ipi);
973 }
> over smp_send_stop() IPI as NMI is kind of try and see approach:
>
> try normal IPI
> if CPU is still online
> try NMI IPI
> if CPU is still online
> try SDEI NMI
> if CPU is still online
> give up
>
> So, if NMI IPI did not trigger in your test, I'm really keen on details.
>
> Thanks
> Vladimir
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v2 00/39] arm64: Add support for FEAT_NMI
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
` (40 preceding siblings ...)
2026-11-16 6:58 ` Jinjie Ruan
@ 2026-11-18 12:17 ` Jinjie Ruan
2026-11-19 1:54 ` Jinjie Ruan
41 siblings, 1 reply; 49+ messages in thread
From: Jinjie Ruan @ 2026-11-18 12:17 UTC (permalink / raw)
To: Vladimir Murzin, linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, liaochang1,
ashok.raj, maz
Hi Vladimir,
I gave the series a test on a Kunpeng HIP12 machine with 384 CPUs,
GICv4.2, firmware with SDEI event 0. The PE implements FEAT_NMI and the
GIC implements FEAT_GICv3_NMI.
Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
Rather than a single boot, I varied the four dimensions that select a
different masking path:
- PE FEAT_NMI : present / masked with id_aa64pfr1.nmi=0
- GIC superpriority : present / disabled
- Pseudo-NMI : irqchip.gicv3_pseudo_nmi=0/1
- SDEI cross-CPU : firmware event 0 active / disabled
The last one needs a patch to switch attached at the end.
# The Results are as below:
Grp PE GIC pse SDEI | supports active pseudo IRQS_NMI fast result
------------------------------------------------------------------
A y y n y | 1 1 0 4 y PASS
A2 y y n n | 1 1 0 4 y PASS
A3 y y y n | 1 0 1 4 y PASS
B y y y y | 1 0 1 4 y PASS
C n y n y | 0 0 0 4 y PASS
D n y y y | 0 0 1 4 y PASS
E y y n y | 0 0 0 0 y * PASS
F y n n y | 1 1 0 0 n PASS
G y n n n | 0 0 0 0 n PASS
H y n y n | 0 0 1 4 y PASS
supports system_supports_nmi()
active system_uses_nmi()
pseudo system_uses_irq_prio_masking()
IRQS_NMI IRQ descriptors with IRQS_NMI set (IPIs + arm-pmu)
fast cpumask_backtrace() at a CPU spinning with local IRQs
disabled returns in ms (NMI delivered) rather than
~1.5 s (plain IPI)
## IRQS_NMI is 4 means 4 types of NMI Interrupt Tests:
IRQ hwirq purpose
---------------------------------------------------------------
4 0x3 SGI IPI_CPU_STOP_NMI - stop a CPU on panic
7 0x6 SGI IPI_KGDB_ROUNDUP - kgdb roundup
8 0x7 SGI IPI_CPU_BACKTRACE - hardlockup all-CPU backtrace
42 0x17 PPI arm-pmu (PMUv3) - perf sampling
* E is CONFIG_ARM64_NMI=n plus SDEI. "fast" there comes from SDEI,
not from the architected path; IRQS_NMI=0 says so.
# Checks that run on every boot and pass in all applicable groups:
irqflags / exception state
save, restore and nesting round trip for DAIF, PMR and ALLINT;
arch_irqs_disabled_flags() takes ALLINT into account;
local_irq_disable() leaves ALLINT clear, and an NMI is still
delivered inside that window.
exception contexts
arm64_exc_hwstate_of_context() for PROCESS / NOIRQ / NONMI /
ERROR / CRITICAL, including that CRITICAL sets ALLINT and NOIRQ
does not; local_exceptions_save_mask()/restore() for NOIRQ and
CRITICAL; local_exceptions_force_daif_save()/restore() on the
idle/WFI path (pseudo-NMI only).
pstate
regs_irqs_disabled() honours PSR_ALLINT_BIT; the bit survives the
ptrace path.
kprobes
registering a probe does not disturb ALLINT, including when it is
set (so an NMI cannot become deliverable inside a probe); a probe
on the real "msr allint" / "mrs xN, allint" instruction is
rejected with -EINVAL.
adjacent subsystems
an EFI runtime call does not clobber ALLINT; perf events keep
counting across an ALLINT toggle.
genirq
add_interrupt_randomness() is never called in NMI context.
CPU stop
spin one CPU with IRQs disabled, then panic: the stop escalation
reaches the NMI rung, "SMP: retry stop with NMI for CPUs 0".
# The problem
SDEI wins over the architected IPI path, unconditionally
arch_trigger_cpumask_backtrace() returns as soon as
sdei_nmi_trigger_cpumask_backtrace() succeeds, so on a platform where
firmware offers SDEI event 0 the irqchip NMI path is never reached.
More importantly PSTATE.ALLINT has no effect on the backtrace:
SDEI active : cpumask_backtrace() returns in ~14 ms with ALLINT set
SDEI disabled : the same call takes ~1.5 s with ALLINT set
That is, the NMI delivered by SDEI bypasses the PSTATE.ALLINT masking
(the interruption is cleared within 14 ms). The NMI that goes through
the GIC superpriority is restricted by ALLINT (the masking is cleared
after 1.5s).
And there is no equivalent of "irqchip.gicv3_pseudo_nmi=" for SDEI NMI.
The patch used for the SDEI switch is as below.
Best regards,
Jinjie
diff --git a/drivers/firmware/arm_sdei_nmi.c
b/drivers/firmware/arm_sdei_nmi.c
index 038eb79e28ac..22e8c3e7aa75 100644
--- a/drivers/firmware/arm_sdei_nmi.c
+++ b/drivers/firmware/arm_sdei_nmi.c
@@ -62,6 +62,24 @@
static bool sdei_nmi_available;
+static bool sdei_nmi_disabled __ro_after_init;
+
+static int __init sdei_nmi_disable_setup(char *str)
+{
+ return kstrtobool(str, &sdei_nmi_disabled);
+}
+early_param("sdei_nmi.disabled", sdei_nmi_disable_setup);
+
#define SDEI_NMI_EVENT 0
/*
@@ -168,7 +186,7 @@ static void sdei_nmi_raise_backtrace(cpumask_t *mask)
*/
bool sdei_nmi_trigger_cpumask_backtrace(const cpumask_t *mask, int
exclude_cpu)
{
- if (!sdei_nmi_available)
+ if (!sdei_nmi_available || sdei_nmi_disabled)
return false;
nmi_trigger_cpumask_backtrace(mask, exclude_cpu,
@@ -194,6 +212,9 @@ void sdei_nmi_stop_cpus(const cpumask_t *mask)
{
unsigned int cpu;
+ if (sdei_nmi_disabled)
+ return;
+
WRITE_ONCE(sdei_nmi_stopping, true);
/*
@@ -238,6 +259,10 @@ static int __init sdei_nmi_init(void)
}
sdei_nmi_available = true;
+ if (sdei_nmi_disabled) {
+ pr_info("Sdei nmi disabled by sdei_nmi.disabled\n");
+ return 0;
+ }
pr_info("using SDEI cross-CPU NMI (SDEI_EVENT_SIGNAL, event %u)\n",
SDEI_NMI_EVENT);
在 2026/11/12 21:20, Vladimir Murzin 写道:
> FEAT_NMI provides an architected mechanism for supporting non-maskable
> interrupts (NMIs) and less-masked interrupts (LMIs).
>
> Since we already support pseudo-NMIs via priority masking, introducing
> another flavour of NMI on top of the existing infrastructure could
> easily become messy, making the code harder to follow and reason
> about.
>
> To avoid that, this series first makes room for the new NMI "tenant"
> by restructuring the existing exception masking logic.
>
> The main idea is to separate the logical view of exception state from
> its hardware representation. To achieve this, we introduce logical
> exception contexts that can be mapped onto the corresponding hardware
> state. This naturally consolidates the hardware-specific handling into
> a small number of places, while allowing the rest of the code to
> operate purely in terms of logical exception contexts.
>
> Since this restructuring is non-trivial and carries a risk of subtle
> behavioural changes, the series adds extensive debug checks to verify
> that the hardware state always matches the expected logical state.
>
> With this restructuring in place, wiring FEAT_NMI into the new
> framework becomes much more manageable. GIC support for NMI is build
> on top.
>
> This work would not have been possible without the contributions of
> Ada Couprie Diaz, Mark Brown, and Lorenzo Pieralisi - please credit
> them for everything that works well. Any remaining bugs or issues are
> entirely my own.
>
> Most of the testing has been done on QEMU, Fast Model and pre-silicon
> environment, so the series definitely benefit from extensive testing
> on real hardware (without NMI, with pseudo-NMI, and with FEAT_NMI).
>
> Patches are organized as follow
>
> 1 - 15
> Rework of existing pseudo-NMI logic to make room for FEAT_NMI
>
> 16 - 33
> FEAT_NMI support including some preparation patches which have
> no dependencies:
> - efi/runtime-wrappers: Permit architectures to override IRQ flags checks
> - arm64: smp: Abstract SGI and LPI operations
>
> 34 - 35
> GICv3 support for NMI
>
> 36 - 39
> GICv5 support for NMI (including some preparation patches)
>
> Changelog
>
> v1 -> v2
>
> - Rebased on v7.4-rc2
> - Select HAVE_PERF_EVENTS_NMI when FEAT_NMI is enabled
> - Reworked APEI SEA handling to mask exceptions sequentially
> - Updated GICv3 initialization to clear NMI configuration
> - Added Tested-by and Reviewed-by tags where approriate
> - Droped tags due to meaningful changes for:
> + "arm64: replace local_daif helpers"
> + "irqchip/gic-v3: Implement FEAT_GICv3_NMI support"
>
> RFC v2 -> v1
>
> - Rebased on v7.3-rc3.
> - Dropped RFC tag, have not received objections, at the same time
> got support from Jinjie
> - Dropped prerequisite patches already present in the newer base
> - Added logic to reject probing/stepping over ALLINT
> - Moved addition of arm64_debug_exc_context() in separate patch
> - Reworked debug logic for validating direction of exception
> transition
> - Split support for NMI in GICv5 in 3 separate patches
> - Added review tags
>
> RFC v1 -> RFC v2
>
> - Rebased on v7.2-rc5.
> - Improved commit descriptions.
> - Split some patches into smaller ones.
> - Added a graceful fallback for IPI requests when FEAT_NMI is
> supported by the CPU but unsupported by the interrupt
> controller.
> - Split the GICv3 preparation from the FEAT_GICv3_NMI
> implementation.
> - Added NMI support to GICv5, primarily as an illustration of the
> implementation, although it also happens to work on Fast Model.
> - Added review tags only to unchanged patches. Review tags were
> intentionally dropped from modified patches to reflect those
> changes and encourage re-review.
>
> RFC v1:
> https://lore.kernel.org/linux-arm-kernel/20260709121333.23507-1-vladimir.murzin@arm.com/
>
> RFC v2:
> https://lore.kernel.org/linux-arm-kernel/20260727163453.7969-1-vladimir.murzin@arm.com/
>
> v1:
> https://lore.kernel.org/linux-arm-kernel/20260914102102.68597-1-vladimir.murzin@arm.com/
>
> Thanks!
> Vladimir
>
> Ada Couprie Diaz (15):
> arm64: irqflags: Introduce arm64-specific irqflags type
> arm64: irqflags: save and use both DAIF and PMR
> arm64: interrupts: Add common exception state helpers
> arm64: entry: Introduce entry specific exception masking helpers
> arm64: entry: replace DAIF helpers with entry helpers
> arm64: interrupts: Introduce exception masking save/restore helpers
> arm64: interrupts: introduce a helper for GIC priority initialization
> arm64: replace local_daif helpers
> arm64: cpuidle: use new helpers to bypass interrupt priority masking
> arm64: remove daifflags.h
> arm64: gicv3: remove GIC_PRIO_PSR_I_SET
> arm64: ptrace: Add PSR_ALLINT_BIT
> arm64: cpufeature: Detect PE support for FEAT_NMI
> arm64: nmi: Manage masking for superpriority interrupts
> arm64: irq: Report FEAT_NMI masking local IRQs
>
> Lorenzo Pieralisi (2):
> irqchip/gic-v3: Rename nmi to pseudo_nmi
> irqchip/gic-v3: Implement FEAT_GICv3_NMI support
>
> Mark Brown (5):
> arm64: booting: Document boot requirements for FEAT_NMI
> arm64: sysreg: Add definitions for immediate versions of MSR ALLINT
> arm64: idreg: Add an override for FEAT_NMI
> arm64: nmi: Add handling of superpriority interrupts as NMIs
> arm64: nmi: Add Kconfig for NMI
>
> Vladimir Murzin (17):
> arm64: process: Lift check for exception state to generic
> infrastructure
> arm64: cpufeature: Remove system_has_prio_mask_debugging()
> arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS
> arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING
> efi/runtime-wrappers: Permit architectures to override IRQ flags
> checks
> arm64/efi: Implement override for IRQ flags checks
> arm64: probes: Reject stepping over ALLINT
> arm64: suspend: Always initialise PSTATE.ALLINT
> arm64/efi: Add ALLINT to IRQ flags checks
> arm64: kprobes: Disable NMIs
> arm64: smp: Abstract SGI and LPI operations
> arm64: smp: Fall back to IRQ when IPI NMI request fails
> arm64: nmi: Select HAVE_PERF_EVENTS_NMI for FEAT_NMI
> arm64: smp: Add NMI support for LPI-backed IPIs
> genirq: Trust local irqchip when considering IRQ for NMI
> genirq: Do not call add_interrupt_randomness() in NMI context
> irqchip/gic-v5: Add NMI support
>
> Documentation/arch/arm64/booting.rst | 6 +
> arch/arm64/Kconfig | 31 +-
> arch/arm64/include/asm/assembler.h | 20 +-
> arch/arm64/include/asm/cpucaps.h | 2 +
> arch/arm64/include/asm/cpufeature.h | 14 +-
> arch/arm64/include/asm/cpuidle.h | 18 +-
> arch/arm64/include/asm/daifflags.h | 144 --------
> arch/arm64/include/asm/efi.h | 42 ++-
> arch/arm64/include/asm/entry-common.h | 8 +-
> arch/arm64/include/asm/insn.h | 1 +
> .../include/asm/interrupts/common_flags.h | 255 +++++++++++++
> arch/arm64/include/asm/interrupts/entry.h | 102 ++++++
> arch/arm64/include/asm/interrupts/masking.h | 166 +++++++++
> arch/arm64/include/asm/irqflags.h | 150 ++++----
> arch/arm64/include/asm/kvm_host.h | 1 -
> arch/arm64/include/asm/mmu_context.h | 1 -
> arch/arm64/include/asm/ptrace.h | 25 +-
> arch/arm64/include/asm/sysreg.h | 2 +
> arch/arm64/include/uapi/asm/ptrace.h | 1 +
> arch/arm64/kernel/acpi.c | 43 +--
> arch/arm64/kernel/cpufeature.c | 68 +++-
> arch/arm64/kernel/debug-monitors.c | 1 -
> arch/arm64/kernel/entry-common.c | 335 +++++++++++++-----
> arch/arm64/kernel/entry.S | 14 +-
> arch/arm64/kernel/hibernate.c | 14 +-
> arch/arm64/kernel/hw_breakpoint.c | 16 +-
> arch/arm64/kernel/irq.c | 45 ++-
> arch/arm64/kernel/machine_kexec.c | 4 +-
> arch/arm64/kernel/pi/idreg-override.c | 1 +
> arch/arm64/kernel/probes/decode-insn.c | 11 +-
> arch/arm64/kernel/probes/kprobes.c | 24 +-
> arch/arm64/kernel/process.c | 24 +-
> arch/arm64/kernel/setup.c | 4 +-
> arch/arm64/kernel/signal.c | 1 -
> arch/arm64/kernel/smp.c | 217 ++++++++----
> arch/arm64/kernel/suspend.c | 14 +-
> arch/arm64/kernel/traps.c | 1 -
> arch/arm64/kvm/hyp/nvhe/switch.c | 2 +-
> arch/arm64/kvm/hyp/vgic-v3-sr.c | 7 +-
> arch/arm64/kvm/hyp/vhe/switch.c | 14 +-
> arch/arm64/mm/fault.c | 1 -
> arch/arm64/mm/mmu.c | 7 +-
> arch/arm64/tools/cpucaps | 2 +
> drivers/firmware/efi/runtime-wrappers.c | 32 +-
> drivers/irqchip/irq-gic-v3.c | 145 +++++++-
> drivers/irqchip/irq-gic-v5.c | 244 ++++++++++++-
> include/linux/irq.h | 2 +-
> include/linux/irqchip/arm-gic-v3-prio.h | 8 -
> include/linux/irqchip/arm-gic-v3.h | 4 +
> kernel/irq/handle.c | 3 +-
> kernel/irq/manage.c | 5 -
> 51 files changed, 1696 insertions(+), 606 deletions(-)
> delete mode 100644 arch/arm64/include/asm/daifflags.h
> create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
> create mode 100644 arch/arm64/include/asm/interrupts/entry.h
> create mode 100644 arch/arm64/include/asm/interrupts/masking.h
>
^ permalink raw reply related [flat|nested] 49+ messages in thread
* Re: [PATCH v2 00/39] arm64: Add support for FEAT_NMI
2026-11-16 11:01 ` Jinjie Ruan
@ 2026-11-18 13:22 ` Vladimir Murzin
2026-11-19 2:33 ` Jinjie Ruan
0 siblings, 1 reply; 49+ messages in thread
From: Vladimir Murzin @ 2026-11-18 13:22 UTC (permalink / raw)
To: Jinjie Ruan, linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, liaochang1,
ashok.raj, maz
Hi Jinjie,
On 11/16/26 11:01, Jinjie Ruan wrote:
>
> 在 2026/11/16 18:02, Vladimir Murzin 写道:
>> Hi Jinjie,
>>
>> On 11/16/26 06:58, Jinjie Ruan wrote:
>>> Hi, Vladimir,
>>>
>>> While testing this patch set, I discovered IPI as NMI now defaults to
>>> using SDEI, and the FEAT_NMI cannot cover this path..
>>>
>> Can you please expand on that? Sashiko has several comments related to
>> interaction with SDEI which I'm planing to dig into. However, glancing
> I mean the following code now prefer sdei nmi now, we can not use FEAT_NMI:
>
> 954 void arch_trigger_cpumask_backtrace(const cpumask_t *mask, int
> exclude_cpu)
> 955 {
> 956 /*
> 957 * Prefer the SDEI cross-CPU NMI provider when active: firmware
> 958 * dispatches the event out of EL3 and reaches CPUs that have
> 959 * interrupts locally masked, without the per-IRQ-mask cost
> that
> 960 * pseudo-NMI pays for the same reach. The plain IPI path below
> 961 * can't reach such a CPU unless pseudo-NMI is enabled.
> 962 */
> 963 if (sdei_nmi_trigger_cpumask_backtrace(mask, exclude_cpu))
> 964 return;
> 965
> 966 /*
> 967 * NOTE: though nmi_trigger_cpumask_backtrace() has "nmi_"
> in the name,
> 968 * nothing about it truly needs to be implemented using an
> NMI, it's
> 969 * just that it's _allowed_ to work with NMIs. If
> ipi_should_be_nmi()
> 970 * returned false our backtrace attempt will just use a
> regular IPI.
> 971 */
> 972 nmi_trigger_cpumask_backtrace(mask, exclude_cpu,
> arm64_backtrace_ipi);
> 973 }
>
>
Ahh, I did not notice that. Perhaps we can do something like bellow
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index c4d1aff31258..17da29f4a7ec 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -955,8 +955,24 @@ static void arm64_backtrace_ipi(cpumask_t *mask)
arm64_send_ipi(mask, IPI_CPU_BACKTRACE);
}
+static bool arm64_nmi_trigger_cpumask_backtrace(const cpumask_t *mask, int exclude_cpu)
+{
+ if (!ipi_is_nmi(smp_processor_id(), IPI_CPU_BACKTRACE))
+ return false;
+
+ nmi_trigger_cpumask_backtrace(mask, exclude_cpu, arm64_backtrace_ipi);
+
+ return true;
+}
+
void arch_trigger_cpumask_backtrace(const cpumask_t *mask, int exclude_cpu)
{
+ /*
+ * Backtrace IPI is NMI, use that.
+ */
+ if (arm64_nmi_trigger_cpumask_backtrace(mask, exclude_cpu))
+ return;
+
/*
* Prefer the SDEI cross-CPU NMI provider when active: firmware
* dispatches the event out of EL3 and reaches CPUs that have
So, preference order would be NMI -> SDEI NMI -> IRQ. My understanding that
users of SDEI NMI do not opt-in for pseudo-NMI so they get what they want, and
if they do not want FEAT_NMI they can opt-out via id_aa64pfr1.nmi=0.
If it makes sense to you I'll include it in the series as separate patch.
Cheers
Vladimir
>> over smp_send_stop() IPI as NMI is kind of try and see approach:
>>
>> try normal IPI
>> if CPU is still online
>> try NMI IPI
>> if CPU is still online
>> try SDEI NMI
>> if CPU is still online
>> give up
>>
>> So, if NMI IPI did not trigger in your test, I'm really keen on details.
>>
>> Thanks
>> Vladimir
>
^ permalink raw reply related [flat|nested] 49+ messages in thread
* Re: [PATCH v2 00/39] arm64: Add support for FEAT_NMI
2026-11-18 12:17 ` Jinjie Ruan
@ 2026-11-19 1:54 ` Jinjie Ruan
0 siblings, 0 replies; 49+ messages in thread
From: Jinjie Ruan @ 2026-11-19 1:54 UTC (permalink / raw)
To: Vladimir Murzin, linux-arm-kernel, kirill
Cc: catalin.marinas, will, mark.rutland, lpieralisi, liaochang1,
ashok.raj, maz
在 2026/11/18 20:17, Jinjie Ruan 写道:
> Hi Vladimir,
>
> I gave the series a test on a Kunpeng HIP12 machine with 384 CPUs,
> GICv4.2, firmware with SDEI event 0. The PE implements FEAT_NMI and the
> GIC implements FEAT_GICv3_NMI.
>
> Tested-by: Jinjie Ruan <ruanjinjie@huawei.com>
>
> Rather than a single boot, I varied the four dimensions that select a
> different masking path:
>
> - PE FEAT_NMI : present / masked with id_aa64pfr1.nmi=0
> - GIC superpriority : present / disabled
> - Pseudo-NMI : irqchip.gicv3_pseudo_nmi=0/1
> - SDEI cross-CPU : firmware event 0 active / disabled
Hi Vladimir,
It appears that SDEI cross-CPU has more issues. On our 384-CPU HIP12
with SDEI event 0 active, a single
echo l > /proc/sysrq-trigger
stuck the machine. Concretely, we see:
- only 8-23 of the 384 CPUs answer the broadcast, the rest never do;
- the framework's 10-second wait then expires for them:
After 10 seconds, these CPUS still haven't responded to the NMI:
3-9,11-45,47-261,263-339,341-351,353-383
- the hardlockup detector fires on the stuck CPUs one watchdog
period later, and with hardlockup_panic=1 the box panics:
watchdog: CPU9: Watchdog detected hard LOCKUP on cpu 9
Kernel panic - not syncing: Hard LOCKUP
- the stop escalation cannot rescue it either:
SMP: retry stop with SDEI NMI for CPUs 194
SMP: failed to stop secondary CPUs 194
so kdump reports "Some CPUs may be stale, kdump will be unreliable".
The same kernel booted with sdei_nmi.disabled=1 answers 384/384 in 47-60
ms with no warnings at all, and a single-target SDEI backtrace is fine
at ~14 ms -- so what breaks is the broadcast form specifically, not SDEI
delivery as such.
Best regards,
Jinjie
>
> The last one needs a patch to switch attached at the end.
>
> # The Results are as below:
>
> Grp PE GIC pse SDEI | supports active pseudo IRQS_NMI fast result
> ------------------------------------------------------------------
> A y y n y | 1 1 0 4 y PASS
> A2 y y n n | 1 1 0 4 y PASS
> A3 y y y n | 1 0 1 4 y PASS
> B y y y y | 1 0 1 4 y PASS
> C n y n y | 0 0 0 4 y PASS
> D n y y y | 0 0 1 4 y PASS
> E y y n y | 0 0 0 0 y * PASS
> F y n n y | 1 1 0 0 n PASS
> G y n n n | 0 0 0 0 n PASS
> H y n y n | 0 0 1 4 y PASS
>
> supports system_supports_nmi()
> active system_uses_nmi()
> pseudo system_uses_irq_prio_masking()
> IRQS_NMI IRQ descriptors with IRQS_NMI set (IPIs + arm-pmu)
> fast cpumask_backtrace() at a CPU spinning with local IRQs
> disabled returns in ms (NMI delivered) rather than
> ~1.5 s (plain IPI)
>
> ## IRQS_NMI is 4 means 4 types of NMI Interrupt Tests:
>
> IRQ hwirq purpose
> ---------------------------------------------------------------
> 4 0x3 SGI IPI_CPU_STOP_NMI - stop a CPU on panic
> 7 0x6 SGI IPI_KGDB_ROUNDUP - kgdb roundup
> 8 0x7 SGI IPI_CPU_BACKTRACE - hardlockup all-CPU backtrace
> 42 0x17 PPI arm-pmu (PMUv3) - perf sampling
>
>
> * E is CONFIG_ARM64_NMI=n plus SDEI. "fast" there comes from SDEI,
> not from the architected path; IRQS_NMI=0 says so.
>
> # Checks that run on every boot and pass in all applicable groups:
>
> irqflags / exception state
> save, restore and nesting round trip for DAIF, PMR and ALLINT;
> arch_irqs_disabled_flags() takes ALLINT into account;
> local_irq_disable() leaves ALLINT clear, and an NMI is still
> delivered inside that window.
>
> exception contexts
> arm64_exc_hwstate_of_context() for PROCESS / NOIRQ / NONMI /
> ERROR / CRITICAL, including that CRITICAL sets ALLINT and NOIRQ
> does not; local_exceptions_save_mask()/restore() for NOIRQ and
> CRITICAL; local_exceptions_force_daif_save()/restore() on the
> idle/WFI path (pseudo-NMI only).
>
> pstate
> regs_irqs_disabled() honours PSR_ALLINT_BIT; the bit survives the
> ptrace path.
>
> kprobes
> registering a probe does not disturb ALLINT, including when it is
> set (so an NMI cannot become deliverable inside a probe); a probe
> on the real "msr allint" / "mrs xN, allint" instruction is
> rejected with -EINVAL.
>
> adjacent subsystems
> an EFI runtime call does not clobber ALLINT; perf events keep
> counting across an ALLINT toggle.
>
> genirq
> add_interrupt_randomness() is never called in NMI context.
>
> CPU stop
> spin one CPU with IRQs disabled, then panic: the stop escalation
> reaches the NMI rung, "SMP: retry stop with NMI for CPUs 0".
>
> # The problem
>
> SDEI wins over the architected IPI path, unconditionally
>
> arch_trigger_cpumask_backtrace() returns as soon as
> sdei_nmi_trigger_cpumask_backtrace() succeeds, so on a platform where
> firmware offers SDEI event 0 the irqchip NMI path is never reached.
>
> More importantly PSTATE.ALLINT has no effect on the backtrace:
>
> SDEI active : cpumask_backtrace() returns in ~14 ms with ALLINT set
> SDEI disabled : the same call takes ~1.5 s with ALLINT set
>
> That is, the NMI delivered by SDEI bypasses the PSTATE.ALLINT masking
> (the interruption is cleared within 14 ms). The NMI that goes through
> the GIC superpriority is restricted by ALLINT (the masking is cleared
> after 1.5s).
>
> And there is no equivalent of "irqchip.gicv3_pseudo_nmi=" for SDEI NMI.
>
> The patch used for the SDEI switch is as below.
>
> Best regards,
> Jinjie
>
> diff --git a/drivers/firmware/arm_sdei_nmi.c
> b/drivers/firmware/arm_sdei_nmi.c
> index 038eb79e28ac..22e8c3e7aa75 100644
> --- a/drivers/firmware/arm_sdei_nmi.c
> +++ b/drivers/firmware/arm_sdei_nmi.c
> @@ -62,6 +62,24 @@
>
> static bool sdei_nmi_available;
>
> +static bool sdei_nmi_disabled __ro_after_init;
> +
> +static int __init sdei_nmi_disable_setup(char *str)
> +{
> + return kstrtobool(str, &sdei_nmi_disabled);
> +}
> +early_param("sdei_nmi.disabled", sdei_nmi_disable_setup);
> +
> #define SDEI_NMI_EVENT 0
>
> /*
> @@ -168,7 +186,7 @@ static void sdei_nmi_raise_backtrace(cpumask_t *mask)
> */
> bool sdei_nmi_trigger_cpumask_backtrace(const cpumask_t *mask, int
> exclude_cpu)
> {
> - if (!sdei_nmi_available)
> + if (!sdei_nmi_available || sdei_nmi_disabled)
> return false;
>
> nmi_trigger_cpumask_backtrace(mask, exclude_cpu,
> @@ -194,6 +212,9 @@ void sdei_nmi_stop_cpus(const cpumask_t *mask)
> {
> unsigned int cpu;
>
> + if (sdei_nmi_disabled)
> + return;
> +
> WRITE_ONCE(sdei_nmi_stopping, true);
>
> /*
> @@ -238,6 +259,10 @@ static int __init sdei_nmi_init(void)
> }
>
> sdei_nmi_available = true;
> + if (sdei_nmi_disabled) {
> + pr_info("Sdei nmi disabled by sdei_nmi.disabled\n");
> + return 0;
> + }
> pr_info("using SDEI cross-CPU NMI (SDEI_EVENT_SIGNAL, event %u)\n",
> SDEI_NMI_EVENT);
>
> 在 2026/11/12 21:20, Vladimir Murzin 写道:
>> FEAT_NMI provides an architected mechanism for supporting non-maskable
>> interrupts (NMIs) and less-masked interrupts (LMIs).
>>
>> Since we already support pseudo-NMIs via priority masking, introducing
>> another flavour of NMI on top of the existing infrastructure could
>> easily become messy, making the code harder to follow and reason
>> about.
>>
>> To avoid that, this series first makes room for the new NMI "tenant"
>> by restructuring the existing exception masking logic.
>>
>> The main idea is to separate the logical view of exception state from
>> its hardware representation. To achieve this, we introduce logical
>> exception contexts that can be mapped onto the corresponding hardware
>> state. This naturally consolidates the hardware-specific handling into
>> a small number of places, while allowing the rest of the code to
>> operate purely in terms of logical exception contexts.
>>
>> Since this restructuring is non-trivial and carries a risk of subtle
>> behavioural changes, the series adds extensive debug checks to verify
>> that the hardware state always matches the expected logical state.
>>
>> With this restructuring in place, wiring FEAT_NMI into the new
>> framework becomes much more manageable. GIC support for NMI is build
>> on top.
>>
>> This work would not have been possible without the contributions of
>> Ada Couprie Diaz, Mark Brown, and Lorenzo Pieralisi - please credit
>> them for everything that works well. Any remaining bugs or issues are
>> entirely my own.
>>
>> Most of the testing has been done on QEMU, Fast Model and pre-silicon
>> environment, so the series definitely benefit from extensive testing
>> on real hardware (without NMI, with pseudo-NMI, and with FEAT_NMI).
>>
>> Patches are organized as follow
>>
>> 1 - 15
>> Rework of existing pseudo-NMI logic to make room for FEAT_NMI
>>
>> 16 - 33
>> FEAT_NMI support including some preparation patches which have
>> no dependencies:
>> - efi/runtime-wrappers: Permit architectures to override IRQ flags checks
>> - arm64: smp: Abstract SGI and LPI operations
>>
>> 34 - 35
>> GICv3 support for NMI
>>
>> 36 - 39
>> GICv5 support for NMI (including some preparation patches)
>>
>> Changelog
>>
>> v1 -> v2
>>
>> - Rebased on v7.4-rc2
>> - Select HAVE_PERF_EVENTS_NMI when FEAT_NMI is enabled
>> - Reworked APEI SEA handling to mask exceptions sequentially
>> - Updated GICv3 initialization to clear NMI configuration
>> - Added Tested-by and Reviewed-by tags where approriate
>> - Droped tags due to meaningful changes for:
>> + "arm64: replace local_daif helpers"
>> + "irqchip/gic-v3: Implement FEAT_GICv3_NMI support"
>>
>> RFC v2 -> v1
>>
>> - Rebased on v7.3-rc3.
>> - Dropped RFC tag, have not received objections, at the same time
>> got support from Jinjie
>> - Dropped prerequisite patches already present in the newer base
>> - Added logic to reject probing/stepping over ALLINT
>> - Moved addition of arm64_debug_exc_context() in separate patch
>> - Reworked debug logic for validating direction of exception
>> transition
>> - Split support for NMI in GICv5 in 3 separate patches
>> - Added review tags
>>
>> RFC v1 -> RFC v2
>>
>> - Rebased on v7.2-rc5.
>> - Improved commit descriptions.
>> - Split some patches into smaller ones.
>> - Added a graceful fallback for IPI requests when FEAT_NMI is
>> supported by the CPU but unsupported by the interrupt
>> controller.
>> - Split the GICv3 preparation from the FEAT_GICv3_NMI
>> implementation.
>> - Added NMI support to GICv5, primarily as an illustration of the
>> implementation, although it also happens to work on Fast Model.
>> - Added review tags only to unchanged patches. Review tags were
>> intentionally dropped from modified patches to reflect those
>> changes and encourage re-review.
>>
>> RFC v1:
>> https://lore.kernel.org/linux-arm-kernel/20260709121333.23507-1-vladimir.murzin@arm.com/
>>
>> RFC v2:
>> https://lore.kernel.org/linux-arm-kernel/20260727163453.7969-1-vladimir.murzin@arm.com/
>>
>> v1:
>> https://lore.kernel.org/linux-arm-kernel/20260914102102.68597-1-vladimir.murzin@arm.com/
>>
>> Thanks!
>> Vladimir
>>
>> Ada Couprie Diaz (15):
>> arm64: irqflags: Introduce arm64-specific irqflags type
>> arm64: irqflags: save and use both DAIF and PMR
>> arm64: interrupts: Add common exception state helpers
>> arm64: entry: Introduce entry specific exception masking helpers
>> arm64: entry: replace DAIF helpers with entry helpers
>> arm64: interrupts: Introduce exception masking save/restore helpers
>> arm64: interrupts: introduce a helper for GIC priority initialization
>> arm64: replace local_daif helpers
>> arm64: cpuidle: use new helpers to bypass interrupt priority masking
>> arm64: remove daifflags.h
>> arm64: gicv3: remove GIC_PRIO_PSR_I_SET
>> arm64: ptrace: Add PSR_ALLINT_BIT
>> arm64: cpufeature: Detect PE support for FEAT_NMI
>> arm64: nmi: Manage masking for superpriority interrupts
>> arm64: irq: Report FEAT_NMI masking local IRQs
>>
>> Lorenzo Pieralisi (2):
>> irqchip/gic-v3: Rename nmi to pseudo_nmi
>> irqchip/gic-v3: Implement FEAT_GICv3_NMI support
>>
>> Mark Brown (5):
>> arm64: booting: Document boot requirements for FEAT_NMI
>> arm64: sysreg: Add definitions for immediate versions of MSR ALLINT
>> arm64: idreg: Add an override for FEAT_NMI
>> arm64: nmi: Add handling of superpriority interrupts as NMIs
>> arm64: nmi: Add Kconfig for NMI
>>
>> Vladimir Murzin (17):
>> arm64: process: Lift check for exception state to generic
>> infrastructure
>> arm64: cpufeature: Remove system_has_prio_mask_debugging()
>> arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS
>> arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING
>> efi/runtime-wrappers: Permit architectures to override IRQ flags
>> checks
>> arm64/efi: Implement override for IRQ flags checks
>> arm64: probes: Reject stepping over ALLINT
>> arm64: suspend: Always initialise PSTATE.ALLINT
>> arm64/efi: Add ALLINT to IRQ flags checks
>> arm64: kprobes: Disable NMIs
>> arm64: smp: Abstract SGI and LPI operations
>> arm64: smp: Fall back to IRQ when IPI NMI request fails
>> arm64: nmi: Select HAVE_PERF_EVENTS_NMI for FEAT_NMI
>> arm64: smp: Add NMI support for LPI-backed IPIs
>> genirq: Trust local irqchip when considering IRQ for NMI
>> genirq: Do not call add_interrupt_randomness() in NMI context
>> irqchip/gic-v5: Add NMI support
>>
>> Documentation/arch/arm64/booting.rst | 6 +
>> arch/arm64/Kconfig | 31 +-
>> arch/arm64/include/asm/assembler.h | 20 +-
>> arch/arm64/include/asm/cpucaps.h | 2 +
>> arch/arm64/include/asm/cpufeature.h | 14 +-
>> arch/arm64/include/asm/cpuidle.h | 18 +-
>> arch/arm64/include/asm/daifflags.h | 144 --------
>> arch/arm64/include/asm/efi.h | 42 ++-
>> arch/arm64/include/asm/entry-common.h | 8 +-
>> arch/arm64/include/asm/insn.h | 1 +
>> .../include/asm/interrupts/common_flags.h | 255 +++++++++++++
>> arch/arm64/include/asm/interrupts/entry.h | 102 ++++++
>> arch/arm64/include/asm/interrupts/masking.h | 166 +++++++++
>> arch/arm64/include/asm/irqflags.h | 150 ++++----
>> arch/arm64/include/asm/kvm_host.h | 1 -
>> arch/arm64/include/asm/mmu_context.h | 1 -
>> arch/arm64/include/asm/ptrace.h | 25 +-
>> arch/arm64/include/asm/sysreg.h | 2 +
>> arch/arm64/include/uapi/asm/ptrace.h | 1 +
>> arch/arm64/kernel/acpi.c | 43 +--
>> arch/arm64/kernel/cpufeature.c | 68 +++-
>> arch/arm64/kernel/debug-monitors.c | 1 -
>> arch/arm64/kernel/entry-common.c | 335 +++++++++++++-----
>> arch/arm64/kernel/entry.S | 14 +-
>> arch/arm64/kernel/hibernate.c | 14 +-
>> arch/arm64/kernel/hw_breakpoint.c | 16 +-
>> arch/arm64/kernel/irq.c | 45 ++-
>> arch/arm64/kernel/machine_kexec.c | 4 +-
>> arch/arm64/kernel/pi/idreg-override.c | 1 +
>> arch/arm64/kernel/probes/decode-insn.c | 11 +-
>> arch/arm64/kernel/probes/kprobes.c | 24 +-
>> arch/arm64/kernel/process.c | 24 +-
>> arch/arm64/kernel/setup.c | 4 +-
>> arch/arm64/kernel/signal.c | 1 -
>> arch/arm64/kernel/smp.c | 217 ++++++++----
>> arch/arm64/kernel/suspend.c | 14 +-
>> arch/arm64/kernel/traps.c | 1 -
>> arch/arm64/kvm/hyp/nvhe/switch.c | 2 +-
>> arch/arm64/kvm/hyp/vgic-v3-sr.c | 7 +-
>> arch/arm64/kvm/hyp/vhe/switch.c | 14 +-
>> arch/arm64/mm/fault.c | 1 -
>> arch/arm64/mm/mmu.c | 7 +-
>> arch/arm64/tools/cpucaps | 2 +
>> drivers/firmware/efi/runtime-wrappers.c | 32 +-
>> drivers/irqchip/irq-gic-v3.c | 145 +++++++-
>> drivers/irqchip/irq-gic-v5.c | 244 ++++++++++++-
>> include/linux/irq.h | 2 +-
>> include/linux/irqchip/arm-gic-v3-prio.h | 8 -
>> include/linux/irqchip/arm-gic-v3.h | 4 +
>> kernel/irq/handle.c | 3 +-
>> kernel/irq/manage.c | 5 -
>> 51 files changed, 1696 insertions(+), 606 deletions(-)
>> delete mode 100644 arch/arm64/include/asm/daifflags.h
>> create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
>> create mode 100644 arch/arm64/include/asm/interrupts/entry.h
>> create mode 100644 arch/arm64/include/asm/interrupts/masking.h
>>
>
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [PATCH v2 00/39] arm64: Add support for FEAT_NMI
2026-11-18 13:22 ` Vladimir Murzin
@ 2026-11-19 2:33 ` Jinjie Ruan
0 siblings, 0 replies; 49+ messages in thread
From: Jinjie Ruan @ 2026-11-19 2:33 UTC (permalink / raw)
To: Vladimir Murzin, linux-arm-kernel
Cc: catalin.marinas, will, mark.rutland, lpieralisi, liaochang1,
ashok.raj, maz
在 2026/11/18 21:22, Vladimir Murzin 写道:
> Hi Jinjie,
Hi Vladimir,
>
> On 11/16/26 11:01, Jinjie Ruan wrote:
>>
>> 在 2026/11/16 18:02, Vladimir Murzin 写道:
>>> Hi Jinjie,
>>>
>>> On 11/16/26 06:58, Jinjie Ruan wrote:
>>>> Hi, Vladimir,
>>>>
>>>> While testing this patch set, I discovered IPI as NMI now defaults to
>>>> using SDEI, and the FEAT_NMI cannot cover this path..
>>>>
>>> Can you please expand on that? Sashiko has several comments related to
>>> interaction with SDEI which I'm planing to dig into. However, glancing
>> I mean the following code now prefer sdei nmi now, we can not use FEAT_NMI:
>>
>> 954 void arch_trigger_cpumask_backtrace(const cpumask_t *mask, int
>> exclude_cpu)
>> 955 {
>> 956 /*
>> 957 * Prefer the SDEI cross-CPU NMI provider when active: firmware
>> 958 * dispatches the event out of EL3 and reaches CPUs that have
>> 959 * interrupts locally masked, without the per-IRQ-mask cost
>> that
>> 960 * pseudo-NMI pays for the same reach. The plain IPI path below
>> 961 * can't reach such a CPU unless pseudo-NMI is enabled.
>> 962 */
>> 963 if (sdei_nmi_trigger_cpumask_backtrace(mask, exclude_cpu))
>> 964 return;
>> 965
>> 966 /*
>> 967 * NOTE: though nmi_trigger_cpumask_backtrace() has "nmi_"
>> in the name,
>> 968 * nothing about it truly needs to be implemented using an
>> NMI, it's
>> 969 * just that it's _allowed_ to work with NMIs. If
>> ipi_should_be_nmi()
>> 970 * returned false our backtrace attempt will just use a
>> regular IPI.
>> 971 */
>> 972 nmi_trigger_cpumask_backtrace(mask, exclude_cpu,
>> arm64_backtrace_ipi);
>> 973 }
>>
>>
>
> Ahh, I did not notice that. Perhaps we can do something like bellow
>
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index c4d1aff31258..17da29f4a7ec 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -955,8 +955,24 @@ static void arm64_backtrace_ipi(cpumask_t *mask)
> arm64_send_ipi(mask, IPI_CPU_BACKTRACE);
> }
>
> +static bool arm64_nmi_trigger_cpumask_backtrace(const cpumask_t *mask, int exclude_cpu)
> +{
> + if (!ipi_is_nmi(smp_processor_id(), IPI_CPU_BACKTRACE))
> + return false;
This false case should be Pseudo-NMI and IRQ.
> +
> + nmi_trigger_cpumask_backtrace(mask, exclude_cpu, arm64_backtrace_ipi);
> +
> + return true;
This is FEAT_NMI
> +}
> +
> void arch_trigger_cpumask_backtrace(const cpumask_t *mask, int exclude_cpu)
> {
> + /*
> + * Backtrace IPI is NMI, use that.
> + */
> + if (arm64_nmi_trigger_cpumask_backtrace(mask, exclude_cpu))
> + return;
> +
> /*
> * Prefer the SDEI cross-CPU NMI provider when active: firmware
> * dispatches the event out of EL3 and reaches CPUs that have
>
>
> So, preference order would be NMI -> SDEI NMI -> IRQ. My understanding that
Based on the motivation of the cross CPU SDEI patch, I guess the order
should be as follows:
FEAT_NMI > SDEI NMI > Pseudo-NMI > IRQ
> users of SDEI NMI do not opt-in for pseudo-NMI so they get what they want, and
My understanding is the same.
> if they do not want FEAT_NMI they can opt-out via id_aa64pfr1.nmi=0.
I believe this situation is relatively rare and does not seem to align
with the original intent of SDEI NMI. In its cover letter, there is the
following statement, and I understand that if hardware supports FEAT_NMI
should be used as the default first priority.
"FEAT_NMI (Armv8.8) -- the architectural fix, but absent from deployed
silicon and from most of the fleet for years to come."
Link: https://lore.kernel.org/all/cover.1782744912.git.kas@kernel.org/
>
> If it makes sense to you I'll include it in the series as separate patch.
>
> Cheers
> Vladimir
>
>>> over smp_send_stop() IPI as NMI is kind of try and see approach:
>>>
>>> try normal IPI
>>> if CPU is still online
>>> try NMI IPI
>>> if CPU is still online
>>> try SDEI NMI
>>> if CPU is still online
>>> give up
>>>
>>> So, if NMI IPI did not trigger in your test, I'm really keen on details.
>>>
>>> Thanks
>>> Vladimir
>>
>
^ permalink raw reply [flat|nested] 49+ messages in thread
end of thread, other threads:[~2026-11-19 2:33 UTC | newest]
Thread overview: 49+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-12 13:20 [PATCH v2 00/39] arm64: Add support for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 01/39] arm64: irqflags: Introduce arm64-specific irqflags type Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 02/39] arm64: irqflags: save and use both DAIF and PMR Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 03/39] arm64: interrupts: Add common exception state helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 04/39] arm64: process: Lift check for exception state to generic infrastructure Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 05/39] arm64: entry: Introduce entry specific exception masking helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 06/39] arm64: entry: replace DAIF helpers with entry helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 07/39] arm64: interrupts: Introduce exception masking save/restore helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 08/39] arm64: interrupts: introduce a helper for GIC priority initialization Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 09/39] arm64: replace local_daif helpers Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 10/39] arm64: cpuidle: use new helpers to bypass interrupt priority masking Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 11/39] arm64: remove daifflags.h Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 12/39] arm64: gicv3: remove GIC_PRIO_PSR_I_SET Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 13/39] arm64: cpufeature: Remove system_has_prio_mask_debugging() Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 14/39] arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 15/39] arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 16/39] efi/runtime-wrappers: Permit architectures to override IRQ flags checks Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 17/39] arm64/efi: Implement override for " Vladimir Murzin
2026-11-13 13:22 ` Mark Brown
2026-11-12 13:20 ` [PATCH v2 18/39] arm64: booting: Document boot requirements for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 19/39] arm64: sysreg: Add definitions for immediate versions of MSR ALLINT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 20/39] arm64: probes: Reject stepping over ALLINT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 21/39] arm64: ptrace: Add PSR_ALLINT_BIT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 22/39] arm64: idreg: Add an override for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 23/39] arm64: cpufeature: Detect PE support " Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 24/39] arm64: nmi: Manage masking for superpriority interrupts Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 25/39] arm64: irq: Report FEAT_NMI masking local IRQs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 26/39] arm64: nmi: Add handling of superpriority interrupts as NMIs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 27/39] arm64: suspend: Always initialise PSTATE.ALLINT Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 28/39] arm64/efi: Add ALLINT to IRQ flags checks Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 29/39] arm64: kprobes: Disable NMIs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 30/39] arm64: smp: Abstract SGI and LPI operations Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 31/39] arm64: smp: Fall back to IRQ when IPI NMI request fails Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 32/39] arm64: nmi: Add Kconfig for NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 33/39] arm64: nmi: Select HAVE_PERF_EVENTS_NMI for FEAT_NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 34/39] irqchip/gic-v3: Rename nmi to pseudo_nmi Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 35/39] irqchip/gic-v3: Implement FEAT_GICv3_NMI support Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 36/39] arm64: smp: Add NMI support for LPI-backed IPIs Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 37/39] genirq: Trust local irqchip when considering IRQ for NMI Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 38/39] genirq: Do not call add_interrupt_randomness() in NMI context Vladimir Murzin
2026-11-12 13:20 ` [PATCH v2 39/39] irqchip/gic-v5: Add NMI support Vladimir Murzin
2026-11-14 5:03 ` [PATCH v2 00/39] arm64: Add support for FEAT_NMI Ashok Raj
2026-11-16 6:58 ` Jinjie Ruan
2026-11-16 10:02 ` Vladimir Murzin
2026-11-16 11:01 ` Jinjie Ruan
2026-11-18 13:22 ` Vladimir Murzin
2026-11-19 2:33 ` Jinjie Ruan
2026-11-18 12:17 ` Jinjie Ruan
2026-11-19 1:54 ` Jinjie Ruan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox