[PATCH 00/38] arm64: Add support for FEAT_NMI

[PATCH 00/38] arm64: Add support for FEAT_NMI

[Vladimir Murzin]

FEAT_NMI provides an architected mechanism for supporting non-maskable
interrupts (NMIs) and less-masked interrupts (LMIs).

Since we already support pseudo-NMIs via priority masking, introducing
another flavour of NMI on top of the existing infrastructure could
easily become messy, making the code harder to follow and reason
about.

To avoid that, this series first makes room for the new NMI "tenant"
by restructuring the existing exception masking logic (patches 01/38 -
15/38)

The main idea is to separate the logical view of exception state from
its hardware representation. To achieve this, we introduce logical
exception contexts that can be mapped onto the corresponding hardware
state. This naturally consolidates the hardware-specific handling into
a small number of places, while allowing the rest of the code to
operate purely in terms of logical exception contexts.

Since this restructuring is non-trivial and carries a risk of subtle
behavioural changes, the series adds extensive debug checks to verify
that the hardware state always matches the expected logical state.

With this restructuring in place, wiring FEAT_NMI into the new
framework becomes much more manageable (pathes 16/38 - 32/38). GIC
support for NMI is build on top (pathes 33/38 - 38/38)

This work would not have been possible without the contributions of
Ada Couprie Diaz, Mark Brown, and Lorenzo Pieralisi - please credit
them for everything that works well. Any remaining bugs or issues are
entirely my own.

Although I droped RFC tag, I'd still appreciate feedback on the
overall approach.

Most of the testing has been done on QEMU, Fast Model and pre-silicon
environment, so the series definitely benefit from extensive testing
on real hardware (without NMI, with pseudo-NMI, and with FEAT_NMI).

Changelog
  RFC v2 -> v1

    - Rebased on v7.3-rc3.
    - Dropped RFC tag, have not received objections, at the same time
      got support from Jinjie
    - Dropped prerequisite patches already present in the newer base
    - Added logic to reject probing/stepping over ALLINT
    - Moved addition of arm64_debug_exc_context() in separate patch
    - Reworked debug logic for validating direction of exception
      transition
    - Split support for NMI in GICv5 in 3 separate patches
    - Added review tags

  RFC v1 -> RFC v2

    - Rebased on v7.2-rc5.
    - Improved commit descriptions.
    - Split some patches into smaller ones.
    - Added a graceful fallback for IPI requests when FEAT_NMI is
      supported by the CPU but unsupported by the interrupt
      controller.
    - Split the GICv3 preparation from the FEAT_GICv3_NMI
      implementation.
    - Added NMI support to GICv5, primarily as an illustration of the
      implementation, although it also happens to work on Fast Model.
    - Added review tags only to unchanged patches. Review tags were
      intentionally dropped from modified patches to reflect those
      changes and encourage re-review.

RFC v1:
  https://lore.kernel.org/linux-arm-kernel/20260709121333.23507-1-vladimir.murzin@arm.com/

RFC v2:
  https://lore.kernel.org/linux-arm-kernel/20260727163453.7969-1-vladimir.murzin@arm.com/

Thanks!
Vladimir

Ada Couprie Diaz (15):
  arm64: irqflags: Introduce arm64-specific irqflags type
  arm64: irqflags: save and use both DAIF and PMR
  arm64: interrupts: Add common exception state helpers
  arm64: entry: Introduce entry specific exception masking helpers
  arm64: entry: replace DAIF helpers with entry helpers
  arm64: interrupts: Introduce exception masking save/restore helpers
  arm64: interrupts: introduce a helper for GIC priority initialization
  arm64: replace local_daif helpers
  arm64: cpuidle: use new helpers to bypass interrupt priority masking
  arm64: remove daifflags.h
  arm64: gicv3: remove GIC_PRIO_PSR_I_SET
  arm64: ptrace: Add PSR_ALLINT_BIT
  arm64: cpufeature: Detect PE support for FEAT_NMI
  arm64: nmi: Manage masking for superpriority interrupts
  arm64: irq: Report FEAT_NMI masking local IRQs

Lorenzo Pieralisi (2):
  irqchip/gic-v3: Rename nmi to pseudo_nmi
  irqchip/gic-v3: Implement FEAT_GICv3_NMI support

Mark Brown (5):
  arm64: booting: Document boot requirements for FEAT_NMI
  arm64: sysreg: Add definitions for immediate versions of MSR ALLINT
  arm64: idreg: Add an override for FEAT_NMI
  arm64: nmi: Add handling of superpriority interrupts as NMIs
  arm64: nmi: Add Kconfig for NMI

Vladimir Murzin (16):
  arm64: process: Lift check for exception state to generic
    infrastructure
  arm64: cpufeature: Remove system_has_prio_mask_debugging()
  arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS
  arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING
  efi/runtime-wrappers: Permit architectures to override IRQ flags
    checks
  arm64/efi: Implement override for IRQ flags checks
  arm64: probes: Reject stepping over ALLINT
  arm64: suspend: Always initialise PSTATE.ALLINT
  arm64/efi: Add ALLINT to IRQ flags checks
  arm64: kprobes: Disable NMIs
  arm64: smp: Abstract SGI and LPI operations
  arm64: smp: Fall back to IRQ when IPI NMI request fails
  arm64: smp: Add NMI support for LPI-backed IPIs
  genirq: Trust local irqchip when considering IRQ for NMI
  genirq: Do not call add_interrupt_randomness() in NMI context
  irqchip/gic-v5: Add NMI support

 Documentation/arch/arm64/booting.rst          |   6 +
 arch/arm64/Kconfig                            |  29 +-
 arch/arm64/include/asm/assembler.h            |  20 +-
 arch/arm64/include/asm/cpucaps.h              |   2 +
 arch/arm64/include/asm/cpufeature.h           |  14 +-
 arch/arm64/include/asm/cpuidle.h              |  18 +-
 arch/arm64/include/asm/daifflags.h            | 144 --------
 arch/arm64/include/asm/efi.h                  |  42 ++-
 arch/arm64/include/asm/entry-common.h         |   8 +-
 arch/arm64/include/asm/insn.h                 |   1 +
 .../include/asm/interrupts/common_flags.h     | 256 +++++++++++++
 arch/arm64/include/asm/interrupts/entry.h     | 102 ++++++
 arch/arm64/include/asm/interrupts/masking.h   | 166 +++++++++
 arch/arm64/include/asm/irqflags.h             | 150 ++++----
 arch/arm64/include/asm/kvm_host.h             |   1 -
 arch/arm64/include/asm/mmu_context.h          |   1 -
 arch/arm64/include/asm/ptrace.h               |  25 +-
 arch/arm64/include/asm/sysreg.h               |   2 +
 arch/arm64/include/uapi/asm/ptrace.h          |   1 +
 arch/arm64/kernel/acpi.c                      |  15 +-
 arch/arm64/kernel/cpufeature.c                |  68 +++-
 arch/arm64/kernel/debug-monitors.c            |   1 -
 arch/arm64/kernel/entry-common.c              | 335 +++++++++++++-----
 arch/arm64/kernel/entry.S                     |  14 +-
 arch/arm64/kernel/hibernate.c                 |  14 +-
 arch/arm64/kernel/irq.c                       |  45 ++-
 arch/arm64/kernel/machine_kexec.c             |   4 +-
 arch/arm64/kernel/pi/idreg-override.c         |   1 +
 arch/arm64/kernel/probes/decode-insn.c        |  11 +-
 arch/arm64/kernel/probes/kprobes.c            |  24 +-
 arch/arm64/kernel/process.c                   |  24 +-
 arch/arm64/kernel/setup.c                     |   4 +-
 arch/arm64/kernel/signal.c                    |   1 -
 arch/arm64/kernel/smp.c                       | 216 +++++++----
 arch/arm64/kernel/suspend.c                   |  14 +-
 arch/arm64/kernel/traps.c                     |   1 -
 arch/arm64/kvm/hyp/nvhe/switch.c              |   2 +-
 arch/arm64/kvm/hyp/vgic-v3-sr.c               |   7 +-
 arch/arm64/kvm/hyp/vhe/switch.c               |  14 +-
 arch/arm64/mm/fault.c                         |   1 -
 arch/arm64/mm/mmu.c                           |   7 +-
 arch/arm64/tools/cpucaps                      |   2 +
 drivers/firmware/efi/runtime-wrappers.c       |  32 +-
 drivers/irqchip/irq-gic-v3.c                  | 132 ++++++-
 drivers/irqchip/irq-gic-v5.c                  | 244 ++++++++++++-
 include/linux/irq.h                           |   2 +-
 include/linux/irqchip/arm-gic-v3-prio.h       |   8 -
 include/linux/irqchip/arm-gic-v3.h            |   4 +
 kernel/irq/handle.c                           |   3 +-
 kernel/irq/manage.c                           |   5 -
 50 files changed, 1661 insertions(+), 582 deletions(-)
 delete mode 100644 arch/arm64/include/asm/daifflags.h
 create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
 create mode 100644 arch/arm64/include/asm/interrupts/entry.h
 create mode 100644 arch/arm64/include/asm/interrupts/masking.h

-- 
2.34.1

[PATCH 01/38] arm64: irqflags: Introduce arm64-specific irqflags type

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

With pseudo-NMIs enabled, we have two mechanisms that control
interrupt masking in parallel :
- The DAIF flags, masking at the CPU
- The GIC PMR, masking before the CPU

However, our irqflags implementation currently assumes that only one
of the two is used at a time, so both DAIF and PMR masking use the same
`unsigned long flags` in their own way.
This is incorrect, as some parts of the kernel will mask interrupts
with DAIF directly or bypass the local_irq masking via the PMR,
and makes tracking the state and changes of both in parallel impossible.

The irqflags API expects `unsigned long`s to be passed around, but
they should not be manipulated outside of the arch-specific code.
So, we can encode the information we need however we want as long as
we return and accept `unsigned long`s.

Introduce a union type for arm64 irqflags whose first member is
a struct allowing us to track DAIF and PMR in parallel, and the second
is the `unsigned long` expected by the irqflags API.

DAIF is a two byte value, to maintain compatibility with existing defines.
PMR is a one byte value, which is the maximum amount of priority bits
allowed by the GICv3 architecture.

Update the internal irqflags functions to use this new union and convert
back and forth with the irqflags unsigned long.
There should be no functional changes.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/irqflags.h | 44 ++++++++++++++++++++++++++-----
 1 file changed, 38 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index a8cb5a5c93b7..5c17e26727f9 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -9,6 +9,8 @@
 #include <asm/ptrace.h>
 #include <asm/sysreg.h>
 
+#include <linux/compiler.h>
+
 /*
  * Aarch64 has flags for masking: Debug, Asynchronous (serror), Interrupts and
  * FIQ exceptions, in the 'daif' register. We mask and unmask them in 'daif'
@@ -20,6 +22,24 @@
  * exceptions should be unmasked.
  */
 
+ /*
+  * Internally, we want to independently manipulate and track the different
+  * interrupt masking mechanisms.
+  * Externally, the generic irqflags API expects unsgined longs to represent
+  * the state of interrupts, which are treated as obscure arch-specific data.
+  */
+typedef union arm64_exc_hwstate {
+	struct {
+		u16 daif;
+		u8 pmr;
+		u8 __pad0;
+		u32 __pad1;
+	};
+	unsigned long flags;
+} arm64_exc_hwstate_t;
+
+static_assert(sizeof(arm64_exc_hwstate_t) == sizeof(unsigned long));
+
 static __always_inline void __daif_local_irq_enable(void)
 {
 	barrier();
@@ -79,12 +99,16 @@ static __always_inline void arch_local_irq_disable(void)
 
 static __always_inline unsigned long __daif_local_save_flags(void)
 {
-	return read_sysreg(daif);
+	arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
+
+	return hwstate.flags;
 }
 
 static __always_inline unsigned long __pmr_local_save_flags(void)
 {
-	return read_sysreg_s(SYS_ICC_PMR_EL1);
+	arm64_exc_hwstate_t hwstate = { .pmr = read_sysreg_s(SYS_ICC_PMR_EL1) };
+
+	return hwstate.flags;
 }
 
 /*
@@ -101,12 +125,16 @@ static __always_inline unsigned long arch_local_save_flags(void)
 
 static __always_inline bool __daif_irqs_disabled_flags(unsigned long flags)
 {
-	return flags & PSR_I_BIT;
+	arm64_exc_hwstate_t hwstate = { .flags = flags };
+
+	return hwstate.daif & PSR_I_BIT;
 }
 
 static __always_inline bool __pmr_irqs_disabled_flags(unsigned long flags)
 {
-	return flags != GIC_PRIO_IRQON;
+	arm64_exc_hwstate_t hwstate = { .flags = flags };
+
+	return hwstate.pmr != GIC_PRIO_IRQON;
 }
 
 static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
@@ -171,15 +199,19 @@ static __always_inline unsigned long arch_local_irq_save(void)
 
 static __always_inline void __daif_local_irq_restore(unsigned long flags)
 {
+	arm64_exc_hwstate_t hwstate = { .flags = flags };
+
 	barrier();
-	write_sysreg(flags, daif);
+	write_sysreg(hwstate.daif, daif);
 	barrier();
 }
 
 static __always_inline void __pmr_local_irq_restore(unsigned long flags)
 {
+	arm64_exc_hwstate_t hwstate = { .flags = flags };
+
 	barrier();
-	write_sysreg_s(flags, SYS_ICC_PMR_EL1);
+	write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
 	pmr_sync();
 	barrier();
 }
-- 
2.34.1

[PATCH 02/38] arm64: irqflags: save and use both DAIF and PMR

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

With pseudo-NMIs enabled, both DAIF and the PMR affect interrupt masking.
Now that we have a type which can track both of them at the same time,
update our irqflags implementation to use it.

Save DAIF flags in all cases, as they can be manipulated directly by other
code, and the PMR if it is in use.

When checking if IRQs are disabled, now that we always save DAIF we
can check that the I flag is set and bypass checking the PMR if it is.
We can also properly check if PMR masks interrupts, now that we don't
need to rely on the GIC_PRIO_PSR_I_SET bit being set in the PMR to
know if DAIF is already masking interrupts.  Update
`irqs_priority_unmasked()` to align with this change.

This allows us to remove the `__daif_...` and `__pmr_...` versions
of the save and check functions, as they are now unified.

We can reasonably merge the two `__{daif,pmr}_irq_restore()` functions
in the main one, as the DAIF and PMR values are properly split now.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/irqflags.h | 118 +++++++-----------------------
 arch/arm64/include/asm/ptrace.h   |  18 +++--
 2 files changed, 38 insertions(+), 98 deletions(-)

diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index 5c17e26727f9..9f5c774cbe68 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -97,135 +97,69 @@ static __always_inline void arch_local_irq_disable(void)
 	}
 }
 
-static __always_inline unsigned long __daif_local_save_flags(void)
-{
-	arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
-
-	return hwstate.flags;
-}
-
-static __always_inline unsigned long __pmr_local_save_flags(void)
-{
-	arm64_exc_hwstate_t hwstate = { .pmr = read_sysreg_s(SYS_ICC_PMR_EL1) };
-
-	return hwstate.flags;
-}
-
 /*
  * Save the current interrupt enable state.
  */
 static __always_inline unsigned long arch_local_save_flags(void)
 {
-	if (system_uses_irq_prio_masking()) {
-		return __pmr_local_save_flags();
-	} else {
-		return __daif_local_save_flags();
-	}
-}
+	arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
 
-static __always_inline bool __daif_irqs_disabled_flags(unsigned long flags)
-{
-	arm64_exc_hwstate_t hwstate = { .flags = flags };
+	if (system_uses_irq_prio_masking())
+		hwstate.pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
 
-	return hwstate.daif & PSR_I_BIT;
+	return hwstate.flags;
 }
 
-static __always_inline bool __pmr_irqs_disabled_flags(unsigned long flags)
+static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
 {
 	arm64_exc_hwstate_t hwstate = { .flags = flags };
 
-	return hwstate.pmr != GIC_PRIO_IRQON;
-}
+	if (hwstate.daif & PSR_I_BIT)
+		return true;
 
-static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
-{
 	if (system_uses_irq_prio_masking()) {
-		return __pmr_irqs_disabled_flags(flags);
-	} else {
-		return __daif_irqs_disabled_flags(flags);
-	}
-}
+		WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_IRQFLAGS) &&
+			     hwstate.pmr != GIC_PRIO_IRQOFF &&
+			     hwstate.pmr != GIC_PRIO_IRQON);
 
-static __always_inline bool __daif_irqs_disabled(void)
-{
-	return __daif_irqs_disabled_flags(__daif_local_save_flags());
-}
+		return hwstate.pmr != GIC_PRIO_IRQON;
+	}
 
-static __always_inline bool __pmr_irqs_disabled(void)
-{
-	return __pmr_irqs_disabled_flags(__pmr_local_save_flags());
+	return false;
 }
 
 static __always_inline bool arch_irqs_disabled(void)
 {
-	if (system_uses_irq_prio_masking()) {
-		return __pmr_irqs_disabled();
-	} else {
-		return __daif_irqs_disabled();
-	}
+	return arch_irqs_disabled_flags(arch_local_save_flags());
 }
 
-static __always_inline unsigned long __daif_local_irq_save(void)
-{
-	unsigned long flags = __daif_local_save_flags();
-
-	__daif_local_irq_disable();
-
-	return flags;
-}
-
-static __always_inline unsigned long __pmr_local_irq_save(void)
+static __always_inline unsigned long arch_local_irq_save(void)
 {
-	unsigned long flags = __pmr_local_save_flags();
+	unsigned long flags = arch_local_save_flags();
 
-	/*
-	 * There are too many states with IRQs disabled, just keep the current
-	 * state if interrupts are already disabled/masked.
-	 */
-	if (!__pmr_irqs_disabled_flags(flags))
+	if (system_uses_irq_prio_masking())
 		__pmr_local_irq_disable();
+	else
+		__daif_local_irq_disable();
 
 	return flags;
 }
 
-static __always_inline unsigned long arch_local_irq_save(void)
-{
-	if (system_uses_irq_prio_masking()) {
-		return __pmr_local_irq_save();
-	} else {
-		return __daif_local_irq_save();
-	}
-}
-
-static __always_inline void __daif_local_irq_restore(unsigned long flags)
-{
-	arm64_exc_hwstate_t hwstate = { .flags = flags };
-
-	barrier();
-	write_sysreg(hwstate.daif, daif);
-	barrier();
-}
-
-static __always_inline void __pmr_local_irq_restore(unsigned long flags)
-{
-	arm64_exc_hwstate_t hwstate = { .flags = flags };
-
-	barrier();
-	write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
-	pmr_sync();
-	barrier();
-}
-
 /*
  * restore saved IRQ state
  */
 static __always_inline void arch_local_irq_restore(unsigned long flags)
 {
+	arm64_exc_hwstate_t hwstate = { .flags = flags };
+
+	barrier();
 	if (system_uses_irq_prio_masking()) {
-		__pmr_local_irq_restore(flags);
+		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
+		pmr_sync();
 	} else {
-		__daif_local_irq_restore(flags);
+		write_sysreg(hwstate.daif, daif);
 	}
+	barrier();
 }
 
 #endif /* __ASM_IRQFLAGS_H */
diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index f7dc5fb9427d..e4c7e6bafa3d 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -203,14 +203,20 @@ static inline void forget_syscall(struct pt_regs *regs)
 #define processor_mode(regs) \
 	((regs)->pstate & PSR_MODE_MASK)
 
-#define irqs_priority_unmasked(regs)					\
-	(system_uses_irq_prio_masking() ?				\
-		(regs)->pmr == GIC_PRIO_IRQON :				\
-		true)
-
 static __always_inline bool regs_irqs_disabled(const struct pt_regs *regs)
 {
-	return (regs->pstate & PSR_I_BIT) || !irqs_priority_unmasked(regs);
+	if (regs->pstate & PSR_I_BIT)
+		return true;
+
+	if (system_uses_irq_prio_masking()) {
+		WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_IRQFLAGS) &&
+			     regs->pmr != GIC_PRIO_IRQOFF &&
+			     regs->pmr != GIC_PRIO_IRQON);
+
+		return regs->pmr != GIC_PRIO_IRQON;
+	}
+
+	return false;
 }
 
 #define interrupts_enabled(regs)	(!regs_irqs_disabled(regs))
-- 
2.34.1

[PATCH 03/38] arm64: interrupts: Add common exception state helpers

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

The entry code and the rest of the kernel require different interrupt
masking APIs, but all need a common representation of the hardware
exception state.

Introduce exception contexts that map DAIF and PMR state to the
corresponding masking context. Provide helpers to translate between
exception contexts and hardware state and update the hardware state.

Updating PMR is unnecessary when a caller knows that its value has not
changed. Allow such callers to avoid the update, while providing a
force option for callers that cannot rely on the previous PMR state.

Entry specific and general purpose masking APIs will be built on these
helpers in subsequent patches.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 .../include/asm/interrupts/common_flags.h     | 174 ++++++++++++++++++
 1 file changed, 174 insertions(+)
 create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h

diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
new file mode 100644
index 000000000000..0de5ba1ae155
--- /dev/null
+++ b/arch/arm64/include/asm/interrupts/common_flags.h
@@ -0,0 +1,174 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2025 Arm Ltd.
+ */
+#ifndef __ASM_INTERRUPTS_COMMON_FLAGS_H
+#define __ASM_INTERRUPTS_COMMON_FLAGS_H
+
+#include <asm/arch_gicv3.h>
+#include <asm/bug.h>
+#include <asm/cpufeature.h>
+#include <asm/ptrace.h>
+#include <asm/sysreg.h>
+#include <asm/irqflags.h>
+
+#define DAIF_PROCCTX		0
+#define DAIF_PROCCTX_NOIRQ	(PSR_I_BIT | PSR_F_BIT)
+#define DAIF_ERRCTX		(PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
+#define DAIF_MASK		(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
+
+/*
+ * Exception context mapping
+ *
+ * pseudo-NMI
+ *
+ * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
+ * ERROR    ->  AIF + IRQON
+ * NONMI    ->   IF + IRQON
+ * NOIRQ    ->    0 + IRQOFF
+ * PROCESS  ->    0 + IRQON
+ *
+ * Otherwise
+ *
+ * CRITICAL -> DAIF           (corresponds to the state on exception entry)
+ * ERROR    ->  AIF
+ * NONMI    ->   IF
+ * NOIRQ    ->   IF
+ * PROCESS  ->    0
+ */
+typedef enum arm64_exc_context {
+	PROCESS_CONTEXT,
+	NOIRQ_CONTEXT,
+	NONMI_CONTEXT,
+	ERROR_CONTEXT,
+	CRITICAL_CONTEXT,
+} arm64_exc_context_t;
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_process_context(void)
+{
+	if (system_uses_irq_prio_masking())
+		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQON};
+
+	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX};
+}
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
+{
+	if (system_uses_irq_prio_masking())
+		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
+
+	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
+}
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
+{
+	if (system_uses_irq_prio_masking())
+		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
+
+	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
+}
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
+{
+	if (system_uses_irq_prio_masking())
+		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
+
+	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
+}
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
+{
+	if (system_uses_irq_prio_masking())
+		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
+
+	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_exc_hwstate_of_context(arm64_exc_context_t context) {
+	switch (context) {
+	case PROCESS_CONTEXT:
+		return __arm64_exc_hwstate_of_process_context();
+	case NOIRQ_CONTEXT:
+		return __arm64_exc_hwstate_of_noirq_context();
+	case NONMI_CONTEXT:
+		return __arm64_exc_hwstate_of_nonmi_context();
+	case ERROR_CONTEXT:
+		return __arm64_exc_hwstate_of_error_context();
+	case CRITICAL_CONTEXT:
+		return __arm64_exc_hwstate_of_critical_context();
+	default:
+		BUG();
+	}
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
+{
+	arm64_exc_hwstate_t state = {.daif=regs->pstate & DAIF_MASK};
+
+	if (system_uses_irq_prio_masking())
+		state.pmr = regs->pmr;
+
+	return state;
+}
+
+static __always_inline
+void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
+{
+	bool pseudo_nmi = system_uses_irq_prio_masking();
+
+	barrier();
+
+	if (pseudo_nmi && hwstate.pmr == GIC_PRIO_IRQOFF && force) {
+		/*
+		 * There has been concern that the write to daif
+		 * might be reordered before this write to PMR.
+		 * From the ARM ARM DDI 0487D.a, section D1.7.1
+		 * "Accessing PSTATE fields":
+		 *   Writes to the PSTATE fields have side-effects on
+		 *   various aspects of the PE operation. All of these
+		 *   side-effects are guaranteed:
+		 *     - Not to be visible to earlier instructions in
+		 *       the execution stream.
+		 *     - To be visible to later instructions in the
+		 *       execution stream
+		 *
+		 * Also, writes to PMR are self-synchronizing, so no
+		 * interrupts with a lower priority than PMR is signaled
+		 * to the PE after the write.
+		 *
+		 * So we don't need additional synchronization here.
+		 */
+		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
+	}
+
+	write_sysreg(hwstate.daif, daif);
+
+	if (pseudo_nmi && hwstate.pmr == GIC_PRIO_IRQON && force) {
+		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
+		pmr_sync();
+	}
+
+	barrier();
+}
+
+static __always_inline
+void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate)
+{
+	__arm64_update_exc_hwstate(hwstate, true);
+}
+
+static __always_inline
+void arm64_update_exc_context(arm64_exc_context_t context)
+{
+	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(context);
+
+	arm64_update_exc_hwstate(hwstate);
+}
+#endif /* __ASM_INTERRUPTS_COMMON_FLAGS_H */
-- 
2.34.1

[PATCH 04/38] arm64: process: Lift check for exception state to generic infrastructure

[Vladimir Murzin]

debug_switch_state() explicitly checks the hardware exception state
for both the pseudo-NMI and non-NMI paths. The expected state matches
the NOIRQ_CONTEXT exception context introduced alongside the exception
masking helpers and state checkers. Such checks are handy outside of
task switch, so turn them into generic helpers available for wider
use. Make debug_switch_state() the first user of newly introduced
helpers.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 .../include/asm/interrupts/common_flags.h     | 30 +++++++++++++++++++
 arch/arm64/kernel/process.c                   | 21 ++-----------
 2 files changed, 32 insertions(+), 19 deletions(-)

diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
index 0de5ba1ae155..03cfde24a530 100644
--- a/arch/arm64/include/asm/interrupts/common_flags.h
+++ b/arch/arm64/include/asm/interrupts/common_flags.h
@@ -118,6 +118,36 @@ arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
 	return state;
 }
 
+static __always_inline
+void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
+{
+	arm64_exc_hwstate_t actual;
+
+	if (!IS_ENABLED(CONFIG_DEBUG_IRQFLAGS))
+		return;
+
+	actual.flags = arch_local_save_flags();
+
+	if (expected.flags == actual.flags)
+		return;
+
+	if (system_uses_irq_prio_masking()) {
+		WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
+			  actual.daif, actual.pmr, expected.daif, expected.pmr);
+	} else {
+		WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
+			  actual.daif, expected.daif);
+	}
+}
+
+static __always_inline
+void arm64_debug_exc_context(arm64_exc_context_t context)
+{
+	arm64_exc_hwstate_t expected = arm64_exc_hwstate_of_context(context);
+
+	arm64_debug_exc_hwstate(expected);
+}
+
 static __always_inline
 void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
 {
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 581f80e9b9b7..0fd09136d398 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -51,6 +51,7 @@
 #include <asm/exec.h>
 #include <asm/fpsimd.h>
 #include <asm/gcs.h>
+#include <asm/interrupts/common_flags.h>
 #include <asm/mmu_context.h>
 #include <asm/mpam.h>
 #include <asm/mte.h>
@@ -703,25 +704,7 @@ void update_sctlr_el1(u64 sctlr)
 
 static inline void debug_switch_state(void)
 {
-	if (system_uses_irq_prio_masking()) {
-		unsigned long daif_expected = 0;
-		unsigned long daif_actual = read_sysreg(daif);
-		unsigned long pmr_expected = GIC_PRIO_IRQOFF;
-		unsigned long pmr_actual = read_sysreg_s(SYS_ICC_PMR_EL1);
-
-		WARN_ONCE(daif_actual != daif_expected ||
-			  pmr_actual != pmr_expected,
-			  "Unexpected DAIF + PMR: 0x%lx + 0x%lx (expected 0x%lx + 0x%lx)\n",
-			  daif_actual, pmr_actual,
-			  daif_expected, pmr_expected);
-	} else {
-		unsigned long daif_expected = DAIF_PROCCTX_NOIRQ;
-		unsigned long daif_actual = read_sysreg(daif);
-
-		WARN_ONCE(daif_actual != daif_expected,
-			  "Unexpected DAIF value: 0x%lx (expected 0x%lx)\n",
-			  daif_actual, daif_expected);
-	}
+	arm64_debug_exc_context(NOIRQ_CONTEXT);
 }
 
 /*
-- 
2.34.1

[PATCH 05/38] arm64: entry: Introduce entry specific exception masking helpers

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

The entry code handles interrupt masking differently from the rest of
the kernel. Exception handlers enter and exit with all exceptions
masked, but they must temporarily unmask the appropriate set of
exceptions so that the rest of the handler executes with the expected
exception state.

For EL0 handlers, this means dropping to masking context appropriate
for the work to be performed. For EL1 handlers, this means restoring
the masking context of the interrupted task. In both cases, all
exceptions must be masked again before returning from the exception
handler.

The rest of the kernel typically follows the opposite pattern: it
raises the masking context to protect a critical section and later
restores the previous context.

Given these different usage patterns, introduce a dedicated set of
exception masking helpers for the entry code. Keeping these helpers
separate from the generic interrupt masking APIs makes the intended
usage explicit and helps avoid mixing the two masking models.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/interrupts/entry.h | 100 ++++++++++++++++++++++
 1 file changed, 100 insertions(+)
 create mode 100644 arch/arm64/include/asm/interrupts/entry.h

diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
new file mode 100644
index 000000000000..0cfca62aa25b
--- /dev/null
+++ b/arch/arm64/include/asm/interrupts/entry.h
@@ -0,0 +1,100 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2025 Arm Ltd.
+ */
+#ifndef __ASM_INTERRUPTS_ENTRY_H
+#define __ASM_INTERRUPTS_ENTRY_H
+
+#include <asm/arch_gicv3.h>
+#include <asm/bug.h>
+#include <asm/cpufeature.h>
+#include <asm/interrupts/common_flags.h>
+
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
+						  arm64_exc_hwstate_t next)
+{
+	bool irqs_disabled = arch_irqs_disabled_flags(next.flags);
+	bool force;
+
+	arm64_debug_exc_hwstate(prev);
+
+	if (prev.flags == next.flags)
+		return next;
+
+	if (!irqs_disabled)
+		trace_hardirqs_on();
+
+	force = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
+
+	__arm64_update_exc_hwstate(next, force);
+
+	if (irqs_disabled)
+		trace_hardirqs_off();
+
+	return next;
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_inherit_exc_context(struct pt_regs *regs)
+{
+	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
+	arm64_exc_hwstate_t next = arm64_inherit_exc_hwstate(regs);
+
+	return __arm64_switch_exc_hwstate_to(prev, next);
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
+{
+	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
+
+	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
+		WARN_ON_ONCE(prev.daif < next.daif);
+
+		if (prev.daif == next.daif) {
+			/*
+			 * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so larger PMR value is weaker
+			 */
+			WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr > next.pmr);
+		}
+	}
+
+	return __arm64_switch_exc_hwstate_to(prev, next);
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
+{
+	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
+
+	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
+		WARN_ON_ONCE(prev.daif > next.daif);
+
+		if (prev.daif == next.daif) {
+			/*
+			 * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so smaller PMR value is stronger
+			 */
+			WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr < next.pmr);
+		}
+	}
+
+	return __arm64_switch_exc_hwstate_to(prev, next);
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_unmask_exc_context(arm64_exc_context_t context)
+{
+	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
+
+	return arm64_drop_exc_context(prev, context);
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_mask_exc_context(arm64_exc_hwstate_t prev)
+{
+	return arm64_lift_exc_context(prev, CRITICAL_CONTEXT);
+}
+
+#endif /* __ASM_INTERRUPTS_ENTRY_H */
-- 
2.34.1

[PATCH 06/38] arm64: entry: replace DAIF helpers with entry helpers

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Replace all uses of the `local_daif_...` helpers in `entry-common.c`
with the new entry-specific exception masking helpers.

Also replace `local_irq_disable()` with a switch to NOIRQ context
using the new helpers. This provides:

- consistency checks for hardware state
- a unified style for managing exception context

Now that both the irqflags API and entry code handle DAIF and PMR
correctly, remove `GIC_PRIO_PSR_I_SET` from the PMR value set by
`kernel_entry` in `entry.S` and by `init_gic_priority_masking()` in
`kernel/smp.c`.

The `local_daif_...` helpers and other specialized code still use
`GIC_PRIO_PSR_I_SET` for now; this commit does not change their
behavior.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/kernel/entry-common.c | 258 ++++++++++++++++++++-----------
 arch/arm64/kernel/entry.S        |   2 +-
 arch/arm64/kernel/smp.c          |   2 +-
 3 files changed, 173 insertions(+), 89 deletions(-)

diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 72c03ccea59f..fd0bfead6e70 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -18,11 +18,11 @@
 #include <linux/thread_info.h>
 
 #include <asm/cpufeature.h>
-#include <asm/daifflags.h>
 #include <asm/esr.h>
 #include <asm/exception.h>
 #include <asm/fpsimd.h>
 #include <asm/irq_regs.h>
+#include <asm/interrupts/entry.h>
 #include <asm/kprobes.h>
 #include <asm/mmu.h>
 #include <asm/processor.h>
@@ -53,9 +53,10 @@ static noinstr irqentry_state_t arm64_enter_from_kernel_mode(struct pt_regs *reg
  * instrumentable code, or any code which may trigger an exception.
  */
 static void noinstr __arm64_exit_to_kernel_mode(struct pt_regs *regs,
-						irqentry_state_t state)
+						irqentry_state_t state,
+						arm64_exc_hwstate_t hwstate)
 {
-	local_daif_mask();
+	arm64_mask_exc_context(hwstate);
 	mte_check_tfsr_exit();
 	irqentry_exit_to_kernel_mode_after_preempt(regs, state);
 }
@@ -64,22 +65,24 @@ static void noinstr __arm64_exit_to_kernel_mode(struct pt_regs *regs,
  * We are returning from the context which allows involuntary kernel preemption
  */
 static void noinstr arm64_exit_to_kernel_mode_preempt(struct pt_regs *regs,
-						      irqentry_state_t state)
+						      irqentry_state_t state,
+						      arm64_exc_hwstate_t hwstate)
 {
 	irqentry_exit_to_kernel_mode_preempt(regs, state);
-	__arm64_exit_to_kernel_mode(regs, state);
+	__arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
-					      irqentry_state_t state)
+					      irqentry_state_t state,
+					      arm64_exc_hwstate_t hwstate)
 {
 	if (!regs_irqs_disabled(regs)) {
-		local_irq_disable();
-		arm64_exit_to_kernel_mode_preempt(regs, state);
+		hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
+		arm64_exit_to_kernel_mode_preempt(regs, state, hwstate);
 		return;
 	}
 
-	__arm64_exit_to_kernel_mode(regs, state);
+	__arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static __always_inline void arm64_syscall_enter_from_user_mode(struct pt_regs *regs)
@@ -102,11 +105,12 @@ static __always_inline void arm64_enter_from_user_mode(struct pt_regs *regs)
 	sme_enter_from_user_mode();
 }
 
-static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs)
+static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs,
+							    arm64_exc_hwstate_t hwstate)
 {
-	local_irq_disable();
+	hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
 	syscall_exit_to_user_mode_prepare(regs);
-	local_daif_mask();
+	arm64_mask_exc_context(hwstate);
 	sme_exit_to_user_mode();
 	mte_check_tfsr_exit();
 	exit_to_user_mode();
@@ -117,11 +121,12 @@ static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs
  * After this function returns it is not safe to call regular kernel code,
  * instrumentable code, or any code which may trigger an exception.
  */
-static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
+static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs,
+						    arm64_exc_hwstate_t hwstate)
 {
-	local_irq_disable();
+	hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
 	irqentry_exit_to_user_mode_prepare(regs);
-	local_daif_mask();
+	arm64_mask_exc_context(hwstate);
 	sme_exit_to_user_mode();
 	mte_check_tfsr_exit();
 	exit_to_user_mode();
@@ -129,7 +134,10 @@ static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
 
 asmlinkage void noinstr asm_exit_to_user_mode(struct pt_regs *regs)
 {
-	arm64_syscall_exit_to_user_mode(regs);
+	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(PROCESS_CONTEXT);
+
+	arm64_syscall_exit_to_user_mode(regs, hwstate);
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 /*
@@ -335,63 +343,69 @@ UNHANDLED(el1t, 64, error)
 static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
 {
 	unsigned long far = read_sysreg(far_el1);
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_mem_abort(far, esr, regs);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static void noinstr el1_pc(struct pt_regs *regs, unsigned long esr)
 {
 	unsigned long far = read_sysreg(far_el1);
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_sp_pc_abort(far, esr, regs);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static void noinstr el1_undef(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_el1_undef(regs, esr);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static void noinstr el1_bti(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_el1_bti(regs, esr);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static void noinstr el1_gcs(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_el1_gcs(regs, esr);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static void noinstr el1_mops(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_el1_mops(regs, esr);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static void noinstr el1_breakpt(struct pt_regs *regs, unsigned long esr)
@@ -451,12 +465,13 @@ static void noinstr el1_brk64(struct pt_regs *regs, unsigned long esr)
 
 static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_el1_fpac(regs, esr);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
@@ -506,16 +521,20 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
 	default:
 		__panic_unhandled(regs, "64-bit el1h sync", esr);
 	}
+
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 static __always_inline void __el1_pnmi(struct pt_regs *regs,
 				       void (*handler)(struct pt_regs *))
 {
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = irqentry_nmi_enter(regs);
+	hwstate = arm64_unmask_exc_context(NONMI_CONTEXT);
 	do_interrupt_handler(regs, handler);
-	local_daif_mask();
+	arm64_mask_exc_context(hwstate);
 	irqentry_nmi_exit(regs, state);
 }
 
@@ -526,21 +545,32 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
 
 	state = arm64_enter_from_kernel_mode(regs);
 
+	arm64_unmask_exc_context(NONMI_CONTEXT);
+
 	irq_enter_rcu();
 	do_interrupt_handler(regs, handler);
 	irq_exit_rcu();
 
-	arm64_exit_to_kernel_mode_preempt(regs, state);
+	/*
+	 * If pseudo-NMIs are enabled and the interrupted context had
+	 * IRQs unmasked, the interrupt handler will have cleared DAIF
+	 * and switched to PMR masking in order to handle
+	 * NMIs. Otherwise it would keep IF.  In both cases on return
+	 * we effectivly have NOIRQ_CONTEXT - keep track of it
+	 */
+	arm64_debug_exc_context(NOIRQ_CONTEXT);
+	arm64_exit_to_kernel_mode_preempt(regs, state, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
 }
+
 static void noinstr el1_interrupt(struct pt_regs *regs,
 				  void (*handler)(struct pt_regs *))
 {
-	write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
-
 	if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && regs_irqs_disabled(regs))
 		__el1_pnmi(regs, handler);
 	else
 		__el1_irq(regs, handler);
+
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
@@ -556,28 +586,31 @@ asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
 asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
 {
 	unsigned long esr = read_sysreg(esr_el1);
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
-	local_daif_restore(DAIF_ERRCTX);
 	state = irqentry_nmi_enter(regs);
+	hwstate = arm64_unmask_exc_context(ERROR_CONTEXT);
 	do_serror(regs, esr);
-	local_daif_mask();
+	arm64_mask_exc_context(hwstate);
 	irqentry_nmi_exit(regs, state);
 }
 
 static void noinstr el0_da(struct pt_regs *regs, unsigned long esr)
 {
 	unsigned long far = read_sysreg(far_el1);
+	arm64_exc_hwstate_t hwstate;
 
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_mem_abort(far, esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
 {
 	unsigned long far = read_sysreg(far_el1);
+	arm64_exc_hwstate_t hwstate;
 
 	/*
 	 * We've taken an instruction abort from userspace and not yet
@@ -588,114 +621,139 @@ static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
 		arm64_apply_bp_hardening();
 
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_mem_abort(far, esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_fpsimd_acc(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_fpsimd_acc(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_sve_acc(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_sve_acc(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_sme_acc(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_sme_acc(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_fpsimd_exc(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_fpsimd_exc(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_sys(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_sys(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_pc(struct pt_regs *regs, unsigned long esr)
 {
 	unsigned long far = read_sysreg(far_el1);
+	arm64_exc_hwstate_t hwstate;
 
 	if (!is_ttbr0_addr(instruction_pointer(regs)))
 		arm64_apply_bp_hardening();
 
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_sp_pc_abort(far, esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_sp(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_sp_pc_abort(regs->sp, esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_undef(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_undef(regs, esr);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_bti(struct pt_regs *regs)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_bti(regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_mops(regs, esr);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_gcs(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_gcs(regs, esr);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_inv(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	bad_el0_sync(regs, 0, esr);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	if (!is_ttbr0_addr(regs->pc))
 		arm64_apply_bp_hardening();
 
@@ -703,12 +761,13 @@ static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
 	debug_exception_enter(regs);
 	do_breakpoint(esr, regs);
 	debug_exception_exit(regs);
-	local_daif_restore(DAIF_PROCCTX);
-	arm64_exit_to_user_mode(regs);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
 	bool step_done;
 
 	if (!is_ttbr0_addr(regs->pc))
@@ -722,50 +781,56 @@ static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
 	 * the single-step is complete.
 	 */
 	step_done = try_step_suspended_breakpoints(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	if (!step_done)
 		do_el0_softstep(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_watchpt(struct pt_regs *regs, unsigned long esr)
 {
 	/* Watchpoints are the only debug exception to write FAR_EL1 */
 	unsigned long far = read_sysreg(far_el1);
+	arm64_exc_hwstate_t hwstate;
 
 	arm64_enter_from_user_mode(regs);
 	debug_exception_enter(regs);
 	do_watchpoint(far, esr, regs);
 	debug_exception_exit(regs);
-	local_daif_restore(DAIF_PROCCTX);
-	arm64_exit_to_user_mode(regs);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_brk64(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_brk64(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_svc(struct pt_regs *regs)
 {
+	arm64_exc_hwstate_t hwstate;
 	arm64_syscall_enter_from_user_mode(regs);
 	cortex_a76_erratum_1463225_svc_handler();
 	fpsimd_syscall_enter();
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_svc(regs);
-	arm64_syscall_exit_to_user_mode(regs);
+	arm64_syscall_exit_to_user_mode(regs, hwstate);
 	fpsimd_syscall_exit();
 }
 
 static void noinstr el0_fpac(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_fpac(regs, esr);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
@@ -834,6 +899,8 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
 	default:
 		el0_inv(regs, esr);
 	}
+
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 static void noinstr el0_interrupt(struct pt_regs *regs,
@@ -841,7 +908,7 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
 {
 	arm64_enter_from_user_mode(regs);
 
-	write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
+	arm64_unmask_exc_context(NONMI_CONTEXT);
 
 	if (regs->pc & BIT(55))
 		arm64_apply_bp_hardening();
@@ -850,7 +917,14 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
 	do_interrupt_handler(regs, handler);
 	irq_exit_rcu();
 
-	arm64_exit_to_user_mode(regs);
+	/*
+	 * For the same reason as in el1_irq() we effectivly
+	 * have NOIRQ_CONTEXT on return from handler - keep
+	 * track of it
+	 */
+	arm64_debug_exc_context(NOIRQ_CONTEXT);
+	arm64_exit_to_user_mode(regs, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 static void noinstr __el0_irq_handler_common(struct pt_regs *regs)
@@ -876,15 +950,17 @@ asmlinkage void noinstr el0t_64_fiq_handler(struct pt_regs *regs)
 static void noinstr __el0_error_handler_common(struct pt_regs *regs)
 {
 	unsigned long esr = read_sysreg(esr_el1);
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_ERRCTX);
+	hwstate = arm64_unmask_exc_context(ERROR_CONTEXT);
 	state = irqentry_nmi_enter(regs);
 	do_serror(regs, esr);
 	irqentry_nmi_exit(regs, state);
-	local_daif_restore(DAIF_PROCCTX);
-	arm64_exit_to_user_mode(regs);
+	hwstate = arm64_drop_exc_context(hwstate, PROCESS_CONTEXT);
+	arm64_exit_to_user_mode(regs, hwstate);
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
@@ -895,27 +971,33 @@ asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
 #ifdef CONFIG_COMPAT
 static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_cp15(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_svc_compat(struct pt_regs *regs)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_syscall_enter_from_user_mode(regs);
 	cortex_a76_erratum_1463225_svc_handler();
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_svc_compat(regs);
-	arm64_syscall_exit_to_user_mode(regs);
+	arm64_syscall_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_bkpt32(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_bkpt32(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
@@ -966,6 +1048,8 @@ asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
 	default:
 		el0_inv(regs, esr);
 	}
+
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 asmlinkage void noinstr el0t_32_irq_handler(struct pt_regs *regs)
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index f63049ac32dc..cb3be770f2d0 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -316,7 +316,7 @@ alternative_else_nop_endif
 
 	mrs_s	x20, SYS_ICC_PMR_EL1
 	str	w20, [sp, #S_PMR]
-	mov	x20, #GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET
+	mov	x20, #GIC_PRIO_IRQON
 	msr_s	SYS_ICC_PMR_EL1, x20
 
 .Lskip_pmr_save\@:
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index a61dc3016a11..17e4557404f2 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -187,7 +187,7 @@ static void init_gic_priority_masking(void)
 	WARN_ON(!(cpuflags & PSR_I_BIT));
 	WARN_ON(!(cpuflags & PSR_F_BIT));
 
-	gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
+	gic_write_pmr(GIC_PRIO_IRQON);
 }
 
 /*
-- 
2.34.1

[PATCH 07/38] arm64: interrupts: Introduce exception masking save/restore helpers

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Outside the entry code, the local_daif_... helpers need to be replaced
so that both DAIF and PMR are handled correctly and their use can be
controlled more strongly.

Introduce local_exceptions_save_mask() and local_exceptions_restore(),
which must be used as a pair. Save both the original and requested
exception state so that inconsistent changes between the save and
restore can be detected.

There are two cases where paired save and restore operations do not
make sense:

- when initializing a CPU, and
- when preparing to turn it off.

Handle those cases with local_exceptions_cpu_init_mask() and
local_exceptions_final_mask(). When CONFIG_DEBUG_IRQFLAGS is enabled,
enforce that these helpers are not used outside their intended
contexts.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/interrupts/masking.h | 92 +++++++++++++++++++++
 arch/arm64/kernel/irq.c                     | 31 +++++++
 2 files changed, 123 insertions(+)
 create mode 100644 arch/arm64/include/asm/interrupts/masking.h

diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
new file mode 100644
index 000000000000..db3439853b64
--- /dev/null
+++ b/arch/arm64/include/asm/interrupts/masking.h
@@ -0,0 +1,92 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2025 Arm Ltd.
+ */
+#ifndef __ASM_INTERRUPTS_MASKING_H
+#define __ASM_INTERRUPTS_MASKING_H
+
+#include <asm/arch_gicv3.h>
+#include <asm/bug.h>
+#include <asm/cpufeature.h>
+#include <asm/interrupts/common_flags.h>
+#include <asm/ptrace.h>
+
+typedef struct arm64_exc_hwstates {
+	arm64_exc_hwstate_t saved;
+	arm64_exc_hwstate_t expected;
+} arm64_exc_hwstates_t;
+
+static inline
+arm64_exc_hwstates_t local_exceptions_save_mask(arm64_exc_context_t new)
+{
+	arm64_exc_hwstate_t actual = {.flags = arch_local_save_flags()};
+	arm64_exc_hwstate_t state = arm64_exc_hwstate_of_context(new);
+	bool irqs_disabled = arch_irqs_disabled_flags(state.flags);
+	bool force;
+
+	/*
+	 * We've just got actual HW state so we can rely on that to
+	 * optimize some unnecessary updates.
+	 */
+	force = system_uses_irq_prio_masking() && actual.pmr != state.pmr;
+
+	if (!irqs_disabled)
+		trace_hardirqs_on();
+
+	__arm64_update_exc_hwstate(state, force);
+
+	if (irqs_disabled)
+		trace_hardirqs_off();
+
+	return (arm64_exc_hwstates_t){.saved = actual, .expected = state};
+}
+
+static inline void local_exceptions_restore(arm64_exc_hwstates_t states)
+{
+	bool irqs_disabled = arch_irqs_disabled_flags(states.saved.flags);
+
+	arm64_debug_exc_hwstate(states.expected);
+
+	if (!irqs_disabled)
+		trace_hardirqs_on();
+
+	arm64_update_exc_hwstate(states.saved);
+
+	if (irqs_disabled)
+		trace_hardirqs_off();
+}
+
+static __always_inline
+void __local_exceptions_cpu_init_mask(arm64_exc_context_t context)
+{
+	if (context == PROCESS_CONTEXT)
+		trace_hardirqs_on();
+
+	arm64_update_exc_context(context);
+
+	if (context != PROCESS_CONTEXT)
+		trace_hardirqs_off();
+}
+
+static __always_inline void __local_exceptions_final_mask(void)
+{
+	arm64_update_exc_context(CRITICAL_CONTEXT);
+	trace_hardirqs_off();
+}
+
+#ifdef CONFIG_DEBUG_IRQFLAGS
+void local_exceptions_cpu_init_mask(arm64_exc_context_t context);
+void local_exceptions_final_mask(void);
+#else
+static inline
+void local_exceptions_cpu_init_mask(arm64_exc_context_t context)
+{
+	__local_exceptions_cpu_init_mask(context);
+}
+
+static inline void local_exceptions_final_mask(void)
+{
+	__local_exceptions_final_mask();
+}
+#endif /* CONFIG_DEBUG_IRQFLAGS */
+#endif /* __ASM_INTERRUPTS_MASKING_H */
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index 9fafd826002b..58b778083576 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -22,6 +22,7 @@
 #include <linux/smp.h>
 #include <linux/vmalloc.h>
 #include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/exception.h>
 #include <asm/numa.h>
 #include <asm/softirq_stack.h>
@@ -39,6 +40,12 @@ DECLARE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr);
 DEFINE_PER_CPU(unsigned long *, irq_shadow_call_stack_ptr);
 #endif
 
+#ifdef CONFIG_DEBUG_IRQFLAGS
+/* Make sure the CPU init/tear down masking functions are only used once. */
+static cpumask_t irqs_masks_init_done;
+static cpumask_t irqs_masks_final_done;
+#endif
+
 static int __init init_irq_scs(void)
 {
 	int cpu;
@@ -133,3 +140,27 @@ void __init init_IRQ(void)
 		local_daif_restore(DAIF_PROCCTX_NOIRQ);
 	}
 }
+
+#ifdef CONFIG_DEBUG_IRQFLAGS
+void local_exceptions_cpu_init_mask(arm64_exc_context_t context)
+{
+	int cpu = smp_processor_id();
+
+	WARN_ON(cpumask_test_cpu(cpu, &irqs_masks_init_done));
+
+	__local_exceptions_cpu_init_mask(context);
+	cpumask_set_cpu(cpu, &irqs_masks_init_done);
+	cpumask_clear_cpu(cpu, &irqs_masks_final_done);
+}
+
+void local_exceptions_final_mask(void)
+{
+	int cpu = smp_processor_id();
+
+	WARN_ON(cpumask_test_cpu(cpu, &irqs_masks_final_done));
+
+	__local_exceptions_final_mask();
+	cpumask_set_cpu(cpu, &irqs_masks_final_done);
+	cpumask_clear_cpu(cpu, &irqs_masks_init_done);
+}
+#endif
-- 
2.34.1

[PATCH 08/38] arm64: interrupts: introduce a helper for GIC priority initialization

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Arm64's `init_IRQ()` calls `local_daif_restore()` to synchronize
interrupt masking via DAIF and switch to masking via PMR.  This
depends on a very specific behaviour of `local_daif_restore()` which
will clear DAIF if masking interrupts via PMR, which will get removed
in the future.

As `setup_arch()` cleared DA only earlier, introduce a dedicated
helper that explicitly initializes the PMR to mask interrupts and
clears DAIF, switching to IRQ priority masking.

Given it is a dedicated helper, add a lockdep assertion as
`setup_arch()` should always have called `trace_hardirqs_off()` when
clearing DA, otherwise something bad happened.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/interrupts/masking.h | 19 +++++++++++++++++++
 arch/arm64/kernel/irq.c                     | 14 +++++---------
 2 files changed, 24 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
index db3439853b64..9e589bf11ec8 100644
--- a/arch/arm64/include/asm/interrupts/masking.h
+++ b/arch/arm64/include/asm/interrupts/masking.h
@@ -5,6 +5,8 @@
 #ifndef __ASM_INTERRUPTS_MASKING_H
 #define __ASM_INTERRUPTS_MASKING_H
 
+#include <linux/lockdep.h>
+
 #include <asm/arch_gicv3.h>
 #include <asm/bug.h>
 #include <asm/cpufeature.h>
@@ -89,4 +91,21 @@ static inline void local_exceptions_final_mask(void)
 	__local_exceptions_final_mask();
 }
 #endif /* CONFIG_DEBUG_IRQFLAGS */
+
+/*
+ * During early boot, we unmask PSR.DA before the GIC has been set up.
+ * If we use IRQ priority masking, the PMR and PSR will be out of sync
+ * after the GIC is enabled : sync them up.
+ */
+static inline void local_interrupt_priority_init(void)
+{
+	if (!system_uses_irq_prio_masking())
+		return;
+
+	WARN_ON(read_sysreg(daif) & PSR_A_BIT);
+	lockdep_assert_irqs_disabled();
+
+	arm64_update_exc_context(NOIRQ_CONTEXT);
+}
+
 #endif /* __ASM_INTERRUPTS_MASKING_H */
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index 58b778083576..a3d9128ba195 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -21,7 +21,6 @@
 #include <linux/seq_file.h>
 #include <linux/smp.h>
 #include <linux/vmalloc.h>
-#include <asm/daifflags.h>
 #include <asm/interrupts/masking.h>
 #include <asm/exception.h>
 #include <asm/numa.h>
@@ -131,14 +130,11 @@ void __init init_IRQ(void)
 
 	irqchip_init();
 
-	if (system_uses_irq_prio_masking()) {
-		/*
-		 * Now that we have a stack for our IRQ handler, set
-		 * the PMR/PSR pair to a consistent state.
-		 */
-		WARN_ON(read_sysreg(daif) & PSR_A_BIT);
-		local_daif_restore(DAIF_PROCCTX_NOIRQ);
-	}
+	/*
+	 * Now that we have a stack for our IRQ handler,
+	 * let's mask interrupts via the PMR.
+	 */
+	local_interrupt_priority_init();
 }
 
 #ifdef CONFIG_DEBUG_IRQFLAGS
-- 
2.34.1

[PATCH 09/38] arm64: replace local_daif helpers

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Replace existing `local_daif_...` usage with the new `local_all_irqs...`
helpers, adding the necessary save-state struct where required.

Slightly rework `apei_claim_sea()`'s handling of interrupt masking to
fit with the new APIs.  Currently, it saves the DAIF flags to restore
later and the `local_irq` ones to check if interrupts are disabled.
It then masks AIF to signal the SEA (forcing DAIF masking), and will
unmask to IF (re-enabling PMR masking if pseudo-NMIs are in use) if it
interrupted a task with interrupts unmaksed, restoring with the DAIF
flags from the beginning afterwards.

As we want to preserve a "mask/unmask" process, change the logic from
"mask AIF/unmask A/unmask IF" to "mask IF/mask A/unmask A/unmask IF" :
first mask with `local_irq_save()`, mask/unmask further around the
SEA signaling, and restore at the end with `local_irq_restore()`.
Now that both APIs properly work with DAIF and PMR, this works
as expected and also removes the need for a "save flags without masking"
function for the new helpers, as this was the only user.

Remove the use of GIC_PRIO_PSR_I_SET in `kvm/hyp/nvhe/switch.c`,
as we now check both PMR and DAIF everywhere and we don't need to signal
interrupts being masked in DAIF in the PMR anymore.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/kernel/acpi.c          | 15 +++++++--------
 arch/arm64/kernel/hibernate.c     | 14 +++++++-------
 arch/arm64/kernel/machine_kexec.c |  4 ++--
 arch/arm64/kernel/setup.c         |  4 ++--
 arch/arm64/kernel/smp.c           | 10 +++++-----
 arch/arm64/kernel/suspend.c       | 14 +++++++-------
 arch/arm64/kvm/hyp/nvhe/switch.c  |  2 +-
 arch/arm64/kvm/hyp/vgic-v3-sr.c   |  7 ++++---
 arch/arm64/kvm/hyp/vhe/switch.c   | 14 ++++++++------
 arch/arm64/mm/mmu.c               |  7 ++++---
 10 files changed, 47 insertions(+), 44 deletions(-)

diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
index 681aa2bbc399..ea9a9da7e412 100644
--- a/arch/arm64/kernel/acpi.c
+++ b/arch/arm64/kernel/acpi.c
@@ -33,7 +33,7 @@
 #include <acpi/processor.h>
 #include <asm/cputype.h>
 #include <asm/cpu_ops.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/smp_plat.h>
 
 int acpi_noirq = 1;		/* skip ACPI IRQ initialization */
@@ -391,15 +391,14 @@ int apei_claim_sea(struct pt_regs *regs)
 	int err = -ENOENT;
 	bool return_to_irqs_enabled;
 	unsigned long current_flags;
+	arm64_exc_hwstates_t hwstate;
 
 	if (!IS_ENABLED(CONFIG_ACPI_APEI_GHES))
 		return err;
 
-	current_flags = local_daif_save_flags();
-
-	/* current_flags isn't useful here as daif doesn't tell us about pNMI */
-	return_to_irqs_enabled = !irqs_disabled_flags(arch_local_save_flags());
+	local_irq_save(current_flags);
 
+	return_to_irqs_enabled = !irqs_disabled_flags(current_flags);
 	if (regs)
 		return_to_irqs_enabled = !regs_irqs_disabled(regs);
 
@@ -407,10 +406,11 @@ int apei_claim_sea(struct pt_regs *regs)
 	 * SEA can interrupt SError, mask it and describe this as an NMI so
 	 * that APEI defers the handling.
 	 */
-	local_daif_restore(DAIF_ERRCTX);
+	hwstate = local_exceptions_save_mask(ERROR_CONTEXT);
 	nmi_enter();
 	err = ghes_notify_sea();
 	nmi_exit();
+	local_exceptions_restore(hwstate);
 
 	/*
 	 * APEI NMI-like notifications are deferred to irq_work. Unless
@@ -418,7 +418,6 @@ int apei_claim_sea(struct pt_regs *regs)
 	 */
 	if (!err) {
 		if (return_to_irqs_enabled) {
-			local_daif_restore(DAIF_PROCCTX_NOIRQ);
 			__irq_enter();
 			irq_work_run();
 			__irq_exit();
@@ -428,7 +427,7 @@ int apei_claim_sea(struct pt_regs *regs)
 		}
 	}
 
-	local_daif_restore(current_flags);
+	local_irq_restore(current_flags);
 
 	return err;
 }
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 7bf117427777..74848a89ba37 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -20,7 +20,7 @@
 #include <asm/barrier.h>
 #include <asm/cacheflush.h>
 #include <asm/cputype.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/irqflags.h>
 #include <asm/kexec.h>
 #include <asm/memory.h>
@@ -332,16 +332,16 @@ static void swsusp_mte_restore_tags(void)
 
 int swsusp_arch_suspend(void)
 {
-	int ret = 0;
-	unsigned long flags;
 	struct sleep_stack_data state;
+	arm64_exc_hwstates_t hwstate;
+	int ret = 0;
 
 	if (cpus_are_stuck_in_kernel()) {
 		pr_err("Can't hibernate: no mechanism to offline secondary CPUs.\n");
 		return -EBUSY;
 	}
 
-	flags = local_daif_save();
+	hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
 
 	if (__cpu_suspend_enter(&state)) {
 		/* make the crash dump kernel image visible/saveable */
@@ -349,7 +349,7 @@ int swsusp_arch_suspend(void)
 
 		ret = swsusp_mte_save_tags();
 		if (ret) {
-			local_daif_restore(flags);
+			local_exceptions_restore(hwstate);
 			return ret;
 		}
 
@@ -393,7 +393,7 @@ int swsusp_arch_suspend(void)
 		spectre_v4_enable_mitigation(NULL);
 	}
 
-	local_daif_restore(flags);
+	local_exceptions_restore(hwstate);
 
 	return ret;
 }
@@ -477,7 +477,7 @@ int __nocfi swsusp_arch_resume(void)
 	 *   swsusp_arch_resume(), and expects to be re-entered in the
 	 *   same state : with all DAIF exceptions masked.
 	 */
-	local_daif_save();
+	local_exceptions_save_mask(CRITICAL_CONTEXT);
 	hibernate_exit(virt_to_phys(tmp_pg_dir), resume_hdr.ttbr1_el1,
 		       resume_hdr.reenter_kernel, restore_pblist,
 		       resume_hdr.__hyp_stub_vectors, virt_to_phys(zero_page));
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index 8f9bc2327dc8..f1353ca6c86b 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -17,7 +17,7 @@
 
 #include <asm/cacheflush.h>
 #include <asm/cpu_ops.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/memory.h>
 #include <asm/mmu.h>
 #include <asm/mmu_context.h>
@@ -174,7 +174,7 @@ void machine_kexec(struct kimage *kimage)
 
 	pr_info("Bye!\n");
 
-	local_daif_mask();
+	local_exceptions_final_mask();
 
 	/*
 	 * Both restart and kernel_reloc will shutdown the MMU, disable data
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 29c6100f0c50..7f4cf0e50551 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -37,7 +37,7 @@
 #include <asm/fixmap.h>
 #include <asm/cpu.h>
 #include <asm/cputype.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/elf.h>
 #include <asm/cpufeature.h>
 #include <asm/cpu_ops.h>
@@ -311,7 +311,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 	 * IRQ and FIQ will be unmasked after the root irqchip has been
 	 * detected and initialized.
 	 */
-	local_daif_restore(DAIF_PROCCTX_NOIRQ);
+	local_exceptions_cpu_init_mask(NOIRQ_CONTEXT);
 
 	/*
 	 * TTBR0 is only used for the identity mapping at this stage. Make it
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 17e4557404f2..f29094ec3843 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -43,7 +43,7 @@
 #include <asm/cpu.h>
 #include <asm/cputype.h>
 #include <asm/cpu_ops.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/kvm_mmu.h>
 #include <asm/mmu_context.h>
 #include <asm/nmi.h>
@@ -265,7 +265,7 @@ asmlinkage notrace void secondary_start_kernel(void)
 	 * as the root irqchip has already been detected and initialized we can
 	 * unmask IRQ and FIQ at the same time.
 	 */
-	local_daif_restore(DAIF_PROCCTX);
+	local_exceptions_cpu_init_mask(PROCESS_CONTEXT);
 
 	/*
 	 * OK, it's off to the idle thread for us
@@ -372,7 +372,7 @@ void __noreturn cpu_die(void)
 
 	idle_task_exit();
 
-	local_daif_mask();
+	local_exceptions_final_mask();
 
 	/* Tell cpuhp_bp_sync_dead() that this CPU is now safe to dispose of */
 	cpuhp_ap_report_dead();
@@ -898,14 +898,14 @@ void __noreturn arm64_nmi_cpu_stop(struct pt_regs *regs, bool die_on_crash)
 	bool crash = IS_ENABLED(CONFIG_KEXEC_CORE) && crash_stop;
 
 	/*
-	 * Use local_daif_mask() instead of local_irq_disable() to make sure
+	 * Use local_exceptions_final_mask() instead of local_irq_disable() to make sure
 	 * that pseudo-NMIs are disabled. The "stop" code starts with an IRQ
 	 * and falls back to NMI (which might be pseudo). If the IRQ finally
 	 * goes through right as we're timing out then the NMI could interrupt
 	 * us. It's better to prevent the NMI and let the IRQ finish since the
 	 * pt_regs will be better.
 	 */
-	local_daif_mask();
+	local_exceptions_final_mask();
 
 #ifdef CONFIG_KEXEC_CORE
 	if (crash && regs)
diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c
index c41724a40b75..61012b783279 100644
--- a/arch/arm64/kernel/suspend.c
+++ b/arch/arm64/kernel/suspend.c
@@ -9,7 +9,7 @@
 #include <asm/cacheflush.h>
 #include <asm/cpufeature.h>
 #include <asm/cpuidle.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/debug-monitors.h>
 #include <asm/exec.h>
 #include <asm/fpsimd.h>
@@ -69,7 +69,7 @@ void notrace __cpu_suspend_exit(void)
 	/*
 	 * Restore HW breakpoint registers to sane values
 	 * before debug exceptions are possibly reenabled
-	 * by cpu_suspend()s local_daif_restore() call.
+	 * by cpu_suspend()s local_exceptions_restore() call.
 	 */
 	if (hw_breakpoint_restore)
 		hw_breakpoint_restore(cpu);
@@ -97,7 +97,7 @@ void notrace __cpu_suspend_exit(void)
 int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 {
 	int ret = 0;
-	unsigned long flags;
+	arm64_exc_hwstates_t hwstate;
 	struct sleep_stack_data state;
 
 	/*
@@ -119,12 +119,12 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 	 *
 	 * Strictly speaking the trace_hardirqs_off() here is superfluous,
 	 * hardirqs should be firmly off by now. This really ought to use
-	 * something like raw_local_daif_save().
+	 * something like raw_local_exceptions_save_mask().
 	 *
 	 * This also unmasks interrupts in PMR in order to reliably
 	 * resume if we're using pseudo-NMIs.
 	 */
-	flags = local_daif_save();
+	hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
 
 	/*
 	 * Function graph tracer state gets inconsistent when the kernel
@@ -158,11 +158,11 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 	unpause_graph_tracing();
 
 	/*
-	 * Restore pstate flags. OS lock and mdscr have been already
+	 * Restore interrupt masks. OS lock and mdscr have been already
 	 * restored, so from this point onwards, debugging is fully
 	 * reenabled if it was enabled when core started shutdown.
 	 */
-	local_daif_restore(flags);
+	local_exceptions_restore(hwstate);
 
 	return ret;
 }
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index 7318e3e6a5f3..50566b455c80 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -270,7 +270,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
 	 * Naturally, we want to avoid this.
 	 */
 	if (system_uses_irq_prio_masking()) {
-		gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
+		gic_write_pmr(GIC_PRIO_IRQON);
 		pmr_sync();
 	}
 
diff --git a/arch/arm64/kvm/hyp/vgic-v3-sr.c b/arch/arm64/kvm/hyp/vgic-v3-sr.c
index 74d4a509f1de..ee0d829a0642 100644
--- a/arch/arm64/kvm/hyp/vgic-v3-sr.c
+++ b/arch/arm64/kvm/hyp/vgic-v3-sr.c
@@ -10,6 +10,7 @@
 #include <linux/irqchip/arm-gic-v3.h>
 #include <linux/kvm_host.h>
 
+#include <asm/interrupts/masking.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_hyp.h>
 #include <asm/kvm_mmu.h>
@@ -440,8 +441,8 @@ void __vgic_v3_init_lrs(void)
 /* Return true if GICv3 is MMIO (GICv2) capable, false otherwise */
 bool __vgic_v3_get_gic_config(void)
 {
+	arm64_exc_hwstates_t hwstate;
 	u64 val, sre;
-	unsigned long flags = 0;
 
 	sre = read_gicreg(ICC_SRE_EL1);
 	/*
@@ -462,7 +463,7 @@ bool __vgic_v3_get_gic_config(void)
 	 * of the exception entry to EL2.
 	 */
 	if (has_vhe()) {
-		flags = local_daif_save();
+		hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
 	} else {
 		sysreg_clear_set_hcr(0, HCR_AMO | HCR_FMO | HCR_IMO);
 		isb();
@@ -477,7 +478,7 @@ bool __vgic_v3_get_gic_config(void)
 	isb();
 
 	if (has_vhe()) {
-		local_daif_restore(flags);
+		local_exceptions_restore(hwstate);
 	} else {
 		sysreg_clear_set_hcr(HCR_AMO | HCR_FMO | HCR_IMO, 0);
 		isb();
diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
index 7875911c0506..11d9feea7d20 100644
--- a/arch/arm64/kvm/hyp/vhe/switch.c
+++ b/arch/arm64/kvm/hyp/vhe/switch.c
@@ -17,6 +17,7 @@
 
 #include <asm/barrier.h>
 #include <asm/cpufeature.h>
+#include <asm/interrupts/masking.h>
 #include <asm/kprobes.h>
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
@@ -661,9 +662,10 @@ NOKPROBE_SYMBOL(__kvm_vcpu_run_vhe);
 
 int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
 {
+	arm64_exc_hwstates_t hwstate;
 	int ret;
 
-	local_daif_mask();
+	hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
 
 	/*
 	 * Having IRQs masked via PMR when entering the guest means the GIC
@@ -671,18 +673,18 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
 	 * only way to get out will be via guest exceptions.
 	 * Naturally, we want to avoid this.
 	 *
-	 * local_daif_mask() already sets GIC_PRIO_PSR_I_SET, we just need a
-	 * dsb to ensure the redistributor is forwards EL2 IRQs to the CPU.
+	 * local_exceptions_save_mask(), via helpers, sets
+	 * GIC_PRIO_IRQON along with pmr_sync(), so we can be shure
+	 * that redistributor forwards EL2 IRQs to the CPU.
 	 */
-	pmr_sync();
 
 	ret = __kvm_vcpu_run_vhe(vcpu);
 
 	/*
-	 * local_daif_restore() takes care to properly restore PSTATE.DAIF
+	 * local_exceptions_restore() takes care to properly restore PSTATE.DAIF
 	 * and the GIC PMR if the host is using IRQ priorities.
 	 */
-	local_daif_restore(DAIF_PROCCTX_NOIRQ);
+	local_exceptions_restore(hwstate);
 
 	return ret;
 }
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 79d90226fd5d..abe89b60a64a 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -33,6 +33,7 @@
 
 #include <asm/barrier.h>
 #include <asm/cputype.h>
+#include <asm/interrupts/masking.h>
 #include <asm/fixmap.h>
 #include <asm/kasan.h>
 #include <asm/kernel-pgtable.h>
@@ -2329,7 +2330,7 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
 	typedef void (ttbr_replace_func)(phys_addr_t);
 	extern ttbr_replace_func idmap_cpu_replace_ttbr1;
 	ttbr_replace_func *replace_phys;
-	unsigned long daif;
+	arm64_exc_hwstates_t hwstate;
 
 	/* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */
 	phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp));
@@ -2345,9 +2346,9 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
 	 * We really don't want to take *any* exceptions while TTBR1 is
 	 * in the process of being replaced so mask everything.
 	 */
-	daif = local_daif_save();
+	hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
 	replace_phys(ttbr1);
-	local_daif_restore(daif);
+	local_exceptions_restore(hwstate);
 
 	cpu_uninstall_idmap();
 }
-- 
2.34.1

[PATCH 10/38] arm64: cpuidle: use new helpers to bypass interrupt priority masking

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

When the CPU goes into WFI and pseudo-NMIs are in use, we need to make
sure that interrupts are not masked by the PMR otherwise they would not
reach the CPU and wake it up.

Now that we have a proper API that handles both DAIF and PMR, extend it
to provide helpers allowing to temporarily switch to masking interrupts
via DAIF only when interrupt priorities are in use.
This could allow other parts of the code to make use of it and makes it
easier to check for proper use.

Replace the custom cpuidle helper with those new helpers.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/cpuidle.h            | 18 ++-----
 arch/arm64/include/asm/interrupts/masking.h | 54 +++++++++++++++++++++
 2 files changed, 58 insertions(+), 14 deletions(-)

diff --git a/arch/arm64/include/asm/cpuidle.h b/arch/arm64/include/asm/cpuidle.h
index 2047713e097d..b70ef5fd5e57 100644
--- a/arch/arm64/include/asm/cpuidle.h
+++ b/arch/arm64/include/asm/cpuidle.h
@@ -5,32 +5,22 @@
 #include <asm/proc-fns.h>
 
 #ifdef CONFIG_ARM64_PSEUDO_NMI
-#include <asm/arch_gicv3.h>
+#include <asm/interrupts/masking.h>
 
 struct arm_cpuidle_irq_context {
-	unsigned long pmr;
-	unsigned long daif_bits;
+	arm64_exc_hwstates_t arm64_context;
 };
 
 #define arm_cpuidle_save_irq_context(__c)				\
 	do {								\
 		struct arm_cpuidle_irq_context *c = __c;		\
-		if (system_uses_irq_prio_masking()) {			\
-			c->daif_bits = read_sysreg(daif);		\
-			write_sysreg(c->daif_bits | PSR_I_BIT | PSR_F_BIT, \
-				     daif);				\
-			c->pmr = gic_read_pmr();			\
-			gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); \
-		}							\
+		c->arm64_context = local_exceptions_force_daif_save(); 	\
 	} while (0)
 
 #define arm_cpuidle_restore_irq_context(__c)				\
 	do {								\
 		struct arm_cpuidle_irq_context *c = __c;		\
-		if (system_uses_irq_prio_masking()) {			\
-			gic_write_pmr(c->pmr);				\
-			write_sysreg(c->daif_bits, daif);		\
-		}							\
+		local_exceptions_force_daif_restore(c->arm64_context);	\
 	} while (0)
 #else
 struct arm_cpuidle_irq_context { };
diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
index 9e589bf11ec8..8729922974b0 100644
--- a/arch/arm64/include/asm/interrupts/masking.h
+++ b/arch/arm64/include/asm/interrupts/masking.h
@@ -92,6 +92,60 @@ static inline void local_exceptions_final_mask(void)
 }
 #endif /* CONFIG_DEBUG_IRQFLAGS */
 
+/*
+ * In some cases, WFI or guest entry for example, we always want interrupts
+ * to reach the CPU even if masked. Masking via the PMR prevents them from
+ * reaching the CPU and waking it up.
+ * Force IRQ masking using DAIF by raising the priority mask
+ * and setting the IF flags.
+ *
+ * Should only be called when IRQs are already masked.
+ */
+static inline arm64_exc_hwstates_t local_exceptions_force_daif_save(void)
+{
+	arm64_exc_hwstates_t states = {};
+
+	if (!system_uses_irq_prio_masking())
+		return states;
+
+	/*
+	 * Cannot use lockdep_assert here as idle entry enables hardirqs
+	 * while keeping interrupts masked.
+	 */
+	WARN_ON_ONCE(!irqs_disabled());
+
+	states.saved.daif = read_sysreg(daif);
+	states.saved.pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
+	states.expected.daif = states.saved.daif | DAIF_PROCCTX_NOIRQ;
+	states.expected.pmr = GIC_PRIO_IRQON;
+
+	arm64_update_exc_hwstate(states.expected);
+
+	return states;
+}
+
+/*
+ * Return to masking with the PMR, restoring previously saved DAIF and PMR.
+ *
+ * IRQs or interrupt priority masking should not have been re-enabled in between
+ * the save and restore.
+ */
+static inline
+void local_exceptions_force_daif_restore(arm64_exc_hwstates_t states)
+{
+	if (!system_uses_irq_prio_masking())
+		return;
+
+	/*
+	 * Cannot use lockdep_assert here as idle entry enables hardirqs
+	 * while keeping interrupts masked.
+	 */
+	WARN_ON_ONCE(!irqs_disabled());
+
+	arm64_debug_exc_hwstate(states.expected);
+	arm64_update_exc_hwstate(states.saved);
+}
+
 /*
  * During early boot, we unmask PSR.DA before the GIC has been set up.
  * If we use IRQ priority masking, the PMR and PSR will be out of sync
-- 
2.34.1

[PATCH 11/38] arm64: remove daifflags.h

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Interrupt masking is now split in different APIs and `local_daif_...`
functions are not in use anymore.
The `DAIF_...` masks defines are now in `asm/interrupts/common_flags.h`,
so we can safely remove `daifflags.h` and all `#include`s referring to it.

Only kprobes.c uses `DAIF_...` masks defines, so add the new header
there.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/daifflags.h    | 144 --------------------------
 arch/arm64/include/asm/entry-common.h |   1 -
 arch/arm64/include/asm/kvm_host.h     |   1 -
 arch/arm64/include/asm/mmu_context.h  |   1 -
 arch/arm64/kernel/debug-monitors.c    |   1 -
 arch/arm64/kernel/probes/kprobes.c    |   2 +-
 arch/arm64/kernel/signal.c            |   1 -
 arch/arm64/kernel/traps.c             |   1 -
 arch/arm64/mm/fault.c                 |   1 -
 9 files changed, 1 insertion(+), 152 deletions(-)
 delete mode 100644 arch/arm64/include/asm/daifflags.h

diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
deleted file mode 100644
index 795b35128467..000000000000
--- a/arch/arm64/include/asm/daifflags.h
+++ /dev/null
@@ -1,144 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (C) 2017 ARM Ltd.
- */
-#ifndef __ASM_DAIFFLAGS_H
-#define __ASM_DAIFFLAGS_H
-
-#include <linux/irqflags.h>
-
-#include <asm/arch_gicv3.h>
-#include <asm/barrier.h>
-#include <asm/cpufeature.h>
-#include <asm/ptrace.h>
-
-#define DAIF_PROCCTX		0
-#define DAIF_PROCCTX_NOIRQ	(PSR_I_BIT | PSR_F_BIT)
-#define DAIF_ERRCTX		(PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
-#define DAIF_MASK		(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
-
-
-/* mask/save/unmask/restore all exceptions, including interrupts. */
-static __always_inline void local_daif_mask(void)
-{
-	WARN_ON(system_has_prio_mask_debugging() &&
-		(read_sysreg_s(SYS_ICC_PMR_EL1) == (GIC_PRIO_IRQOFF |
-						    GIC_PRIO_PSR_I_SET)));
-
-	asm volatile(
-		"msr	daifset, #0xf		// local_daif_mask\n"
-		:
-		:
-		: "memory");
-
-	/* Don't really care for a dsb here, we don't intend to enable IRQs */
-	if (system_uses_irq_prio_masking())
-		gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
-
-	trace_hardirqs_off();
-}
-
-static __always_inline unsigned long local_daif_save_flags(void)
-{
-	unsigned long flags;
-
-	flags = read_sysreg(daif);
-
-	if (system_uses_irq_prio_masking()) {
-		/* If IRQs are masked with PMR, reflect it in the flags */
-		if (read_sysreg_s(SYS_ICC_PMR_EL1) != GIC_PRIO_IRQON)
-			flags |= PSR_I_BIT | PSR_F_BIT;
-	}
-
-	return flags;
-}
-
-static __always_inline unsigned long local_daif_save(void)
-{
-	unsigned long flags;
-
-	flags = local_daif_save_flags();
-
-	local_daif_mask();
-
-	return flags;
-}
-
-static __always_inline void local_daif_restore(unsigned long flags)
-{
-	bool irq_disabled = flags & PSR_I_BIT;
-
-	WARN_ON(system_has_prio_mask_debugging() &&
-		(read_sysreg(daif) & (PSR_I_BIT | PSR_F_BIT)) != (PSR_I_BIT | PSR_F_BIT));
-
-	if (!irq_disabled) {
-		trace_hardirqs_on();
-
-		if (system_uses_irq_prio_masking()) {
-			gic_write_pmr(GIC_PRIO_IRQON);
-			pmr_sync();
-		}
-	} else if (system_uses_irq_prio_masking()) {
-		u64 pmr;
-
-		if (!(flags & PSR_A_BIT)) {
-			/*
-			 * If interrupts are disabled but we can take
-			 * asynchronous errors, we can take NMIs
-			 */
-			flags &= ~(PSR_I_BIT | PSR_F_BIT);
-			pmr = GIC_PRIO_IRQOFF;
-		} else {
-			pmr = GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET;
-		}
-
-		/*
-		 * There has been concern that the write to daif
-		 * might be reordered before this write to PMR.
-		 * From the ARM ARM DDI 0487D.a, section D1.7.1
-		 * "Accessing PSTATE fields":
-		 *   Writes to the PSTATE fields have side-effects on
-		 *   various aspects of the PE operation. All of these
-		 *   side-effects are guaranteed:
-		 *     - Not to be visible to earlier instructions in
-		 *       the execution stream.
-		 *     - To be visible to later instructions in the
-		 *       execution stream
-		 *
-		 * Also, writes to PMR are self-synchronizing, so no
-		 * interrupts with a lower priority than PMR is signaled
-		 * to the PE after the write.
-		 *
-		 * So we don't need additional synchronization here.
-		 */
-		gic_write_pmr(pmr);
-	}
-
-	write_sysreg(flags, daif);
-
-	if (irq_disabled)
-		trace_hardirqs_off();
-}
-
-/*
- * Called by synchronous exception handlers to restore the DAIF bits that were
- * modified by taking an exception.
- */
-static __always_inline void local_daif_inherit(struct pt_regs *regs)
-{
-	unsigned long flags = regs->pstate & DAIF_MASK;
-
-	if (!regs_irqs_disabled(regs))
-		trace_hardirqs_on();
-
-	if (system_uses_irq_prio_masking())
-		gic_write_pmr(regs->pmr);
-
-	/*
-	 * We can't use local_daif_restore(regs->pstate) here as
-	 * system_has_prio_mask_debugging() won't restore the I bit if it can
-	 * use the pmr instead.
-	 */
-	write_sysreg(flags, daif);
-}
-#endif
diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
index 1905765159aa..73d82a8d8e95 100644
--- a/arch/arm64/include/asm/entry-common.h
+++ b/arch/arm64/include/asm/entry-common.h
@@ -6,7 +6,6 @@
 #include <linux/thread_info.h>
 
 #include <asm/cpufeature.h>
-#include <asm/daifflags.h>
 #include <asm/fpsimd.h>
 #include <asm/mte.h>
 #include <asm/stacktrace.h>
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 27fe0cd5b2d7..72113427f48a 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -23,7 +23,6 @@
 #include <asm/barrier.h>
 #include <asm/cpufeature.h>
 #include <asm/cputype.h>
-#include <asm/daifflags.h>
 #include <asm/fpsimd.h>
 #include <asm/kvm.h>
 #include <asm/kvm_asm.h>
diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index 803b68758152..b3354a1350cf 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -19,7 +19,6 @@
 
 #include <asm/cacheflush.h>
 #include <asm/cpufeature.h>
-#include <asm/daifflags.h>
 #include <asm/gcs.h>
 #include <asm/proc-fns.h>
 #include <asm/cputype.h>
diff --git a/arch/arm64/kernel/debug-monitors.c b/arch/arm64/kernel/debug-monitors.c
index e271fbac5f82..5db27c7e86bb 100644
--- a/arch/arm64/kernel/debug-monitors.c
+++ b/arch/arm64/kernel/debug-monitors.c
@@ -19,7 +19,6 @@
 
 #include <asm/cpufeature.h>
 #include <asm/cputype.h>
-#include <asm/daifflags.h>
 #include <asm/debug-monitors.h>
 #include <asm/exception.h>
 #include <asm/kgdb.h>
diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c
index 4e0efad5caf2..68f4271ce02f 100644
--- a/arch/arm64/kernel/probes/kprobes.c
+++ b/arch/arm64/kernel/probes/kprobes.c
@@ -24,9 +24,9 @@
 #include <linux/vmalloc.h>
 
 #include <asm/cacheflush.h>
-#include <asm/daifflags.h>
 #include <asm/debug-monitors.h>
 #include <asm/insn.h>
+#include <asm/interrupts/common_flags.h>
 #include <asm/irq.h>
 #include <asm/text-patching.h>
 #include <asm/ptrace.h>
diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
index 38e6fa204c17..2276d316cc9e 100644
--- a/arch/arm64/kernel/signal.c
+++ b/arch/arm64/kernel/signal.c
@@ -22,7 +22,6 @@
 #include <linux/syscalls.h>
 #include <linux/pkeys.h>
 
-#include <asm/daifflags.h>
 #include <asm/debug-monitors.h>
 #include <asm/elf.h>
 #include <asm/exception.h>
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index 914282016069..77b6be6fc4c1 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -33,7 +33,6 @@
 #include <asm/atomic.h>
 #include <asm/bug.h>
 #include <asm/cpufeature.h>
-#include <asm/daifflags.h>
 #include <asm/debug-monitors.h>
 #include <asm/efi.h>
 #include <asm/esr.h>
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 75c3e463df2e..1d8f6089c787 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -35,7 +35,6 @@
 #include <asm/cpufeature.h>
 #include <asm/efi.h>
 #include <asm/exception.h>
-#include <asm/daifflags.h>
 #include <asm/debug-monitors.h>
 #include <asm/esr.h>
 #include <asm/kprobes.h>
-- 
2.34.1

[PATCH 12/38] arm64: gicv3: remove GIC_PRIO_PSR_I_SET

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

`GIC_PRIO_PSR_I_SET` was used to keep track of sections of the kernel
where interrupts always need to reach the CPU and could not be masked
via interrupt priorities, or DAIF bypassed the status of the PMR.

This flag unmasked IRQs when OR'd to GIC_PRIO_IRQON and ensured that
that IRQs were seen as disabled by the `local_irqs_...` functions,
as they would only check the PMR if pseudo-NMIs were enabled.

This is not needed anymore as our IRQ masking APIs now know how to handle
interrupt priorities properly and in parallel with DAIF : remove it.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/ptrace.h         | 2 --
 include/linux/irqchip/arm-gic-v3-prio.h | 8 --------
 2 files changed, 10 deletions(-)

diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index e4c7e6bafa3d..cf7e51b8ef0b 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -24,8 +24,6 @@
 #define GIC_PRIO_IRQON		GICV3_PRIO_UNMASKED
 #define GIC_PRIO_IRQOFF		GICV3_PRIO_IRQ
 
-#define GIC_PRIO_PSR_I_SET	GICV3_PRIO_PSR_I_SET
-
 /* Additional SPSR bits not exposed in the UABI */
 #define PSR_MODE_THREAD_BIT	(1 << 0)
 #define PSR_IL_BIT		(1 << 20)
diff --git a/include/linux/irqchip/arm-gic-v3-prio.h b/include/linux/irqchip/arm-gic-v3-prio.h
index 44157c9abb78..464bf1d29453 100644
--- a/include/linux/irqchip/arm-gic-v3-prio.h
+++ b/include/linux/irqchip/arm-gic-v3-prio.h
@@ -20,17 +20,11 @@
  * GICV3_PRIO_NMI is the PMR view of the priority of pseudo-NMIs. This can be
  * written to the PMR to mask pseudo-NMIs.
  *
- * On arm64 some code sections either automatically switch back to PSR.I or
- * explicitly require to not use priority masking. If bit GICV3_PRIO_PSR_I_SET
- * is included in the priority mask, it indicates that PSR.I should be set and
- * interrupt disabling temporarily does not rely on IRQ priorities.
  */
 #define GICV3_PRIO_UNMASKED	0xe0
 #define GICV3_PRIO_IRQ		0xc0
 #define GICV3_PRIO_NMI		0x80
 
-#define GICV3_PRIO_PSR_I_SET	(1 << 4)
-
 #ifndef __ASSEMBLER__
 
 #define __gicv3_prio_to_ns(p)	(0xff & ((p) << 1))
@@ -45,8 +39,6 @@ static_assert(__gicv3_prio_valid_ns(GICV3_PRIO_IRQ));
 static_assert(GICV3_PRIO_NMI < GICV3_PRIO_IRQ);
 static_assert(GICV3_PRIO_IRQ < GICV3_PRIO_UNMASKED);
 
-static_assert(GICV3_PRIO_IRQ < (GICV3_PRIO_IRQ | GICV3_PRIO_PSR_I_SET));
-
 #endif /* __ASSEMBLER */
 
 #endif /* __LINUX_IRQCHIP_ARM_GIC_V3_PRIO_H */
-- 
2.34.1

[PATCH 13/38] arm64: cpufeature: Remove system_has_prio_mask_debugging()

[Vladimir Murzin]

Users of system_has_prio_mask_debugging() have gone, so remove it.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/cpufeature.h | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index 4f04ad82ea34..09aa80df602a 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -814,12 +814,6 @@ static inline bool system_supports_mte(void)
 	return alternative_has_cap_unlikely(ARM64_MTE);
 }
 
-static inline bool system_has_prio_mask_debugging(void)
-{
-	return IS_ENABLED(CONFIG_ARM64_DEBUG_PRIORITY_MASKING) &&
-	       system_uses_irq_prio_masking();
-}
-
 static inline bool system_supports_bti(void)
 {
 	return cpus_have_final_cap(ARM64_BTI);
-- 
2.34.1

[PATCH 14/38] arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS

[Vladimir Murzin]

All exception state debug logic now lives under the generic
CONFIG_DEBUG_IRQFLAGS. Move the two remaining cases from the dedicated
config option to CONFIG_DEBUG_IRQFLAGS as well, keeping all related
debug logic under a single configuration option.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/irqflags.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index 9f5c774cbe68..eedbdf7358e6 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -49,7 +49,7 @@ static __always_inline void __daif_local_irq_enable(void)
 
 static __always_inline void __pmr_local_irq_enable(void)
 {
-	if (IS_ENABLED(CONFIG_ARM64_DEBUG_PRIORITY_MASKING)) {
+	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
 		u32 pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
 		WARN_ON_ONCE(pmr != GIC_PRIO_IRQON && pmr != GIC_PRIO_IRQOFF);
 	}
@@ -78,7 +78,7 @@ static __always_inline void __daif_local_irq_disable(void)
 
 static __always_inline void __pmr_local_irq_disable(void)
 {
-	if (IS_ENABLED(CONFIG_ARM64_DEBUG_PRIORITY_MASKING)) {
+	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
 		u32 pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
 		WARN_ON_ONCE(pmr != GIC_PRIO_IRQON && pmr != GIC_PRIO_IRQOFF);
 	}
-- 
2.34.1

[PATCH 15/38] arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING

[Vladimir Murzin]

Last users of CONFIG_ARM64_DEBUG_PRIORITY_MASKING have gone, so remove
it.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/Kconfig | 11 -----------
 1 file changed, 11 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b5a51b0ef944..2738637ccb92 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2390,17 +2390,6 @@ config ARM64_PSEUDO_NMI
 
 	  If unsure, say N
 
-if ARM64_PSEUDO_NMI
-config ARM64_DEBUG_PRIORITY_MASKING
-	bool "Debug interrupt priority masking"
-	help
-	  This adds runtime checks to functions enabling/disabling
-	  interrupts when using priority masking. The additional checks verify
-	  the validity of ICC_PMR_EL1 when calling concerned functions.
-
-	  If unsure, say N
-endif # ARM64_PSEUDO_NMI
-
 config RELOCATABLE
 	bool "Build a relocatable kernel image" if EXPERT
 	select ARCH_HAS_RELR
-- 
2.34.1

[PATCH 16/38] efi/runtime-wrappers: Permit architectures to override IRQ flags checks

[Vladimir Murzin]

Representation of IRQ flags on arm64 has recently changed. They now
track both DAIF and PMR state while remaining compatible with the
irqflags API by encoding the state in an unsigned long.

While the generic check can still detect corruption of the IRQ flags
caused by EFI runtime services, the resulting error message is cryptic
because it does not understand the arm64-specific representation.

Permit architectures to override the IRQ flags checks and associated
error messages, while keeping the existing implementation as the
default.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 drivers/firmware/efi/runtime-wrappers.c | 32 +++++++++++++++----------
 1 file changed, 20 insertions(+), 12 deletions(-)

diff --git a/drivers/firmware/efi/runtime-wrappers.c b/drivers/firmware/efi/runtime-wrappers.c
index 2344b9d1e81f..b42f8aa18da7 100644
--- a/drivers/firmware/efi/runtime-wrappers.c
+++ b/drivers/firmware/efi/runtime-wrappers.c
@@ -145,6 +145,25 @@ struct efi_runtime_work efi_rts_work;
 #define arch_efi_restore_flags(state_flags)	local_irq_restore(state_flags)
 #endif
 
+#ifndef arch_efi_check_flags
+#define arch_efi_check_flags(state_flags, caller) 							\
+do {													\
+	unsigned long cur_flags, mismatch;								\
+													\
+	cur_flags = efi_call_virt_save_flags();								\
+													\
+	mismatch = state_flags ^ cur_flags;								\
+	if (!WARN_ON_ONCE(mismatch & ARCH_EFI_IRQ_FLAGS_MASK))						\
+		break;											\
+													\
+	add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_NOW_UNRELIABLE); 					\
+	pr_err_ratelimited(FW_BUG "IRQ flags corrupted (0x%08lx=>0x%08lx) by EFI call from %pS\n",	\
+			   state_flags, cur_flags,							\
+			   caller ?: __builtin_return_address(0));					\
+	arch_efi_restore_flags(state_flags);								\
+} while(0)
+#endif
+
 unsigned long efi_call_virt_save_flags(void)
 {
 	unsigned long flags;
@@ -155,18 +174,7 @@ unsigned long efi_call_virt_save_flags(void)
 
 void efi_call_virt_check_flags(unsigned long flags, const void *caller)
 {
-	unsigned long cur_flags, mismatch;
-
-	cur_flags = efi_call_virt_save_flags();
-
-	mismatch = flags ^ cur_flags;
-	if (!WARN_ON_ONCE(mismatch & ARCH_EFI_IRQ_FLAGS_MASK))
-		return;
-
-	add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_NOW_UNRELIABLE);
-	pr_err_ratelimited(FW_BUG "IRQ flags corrupted (0x%08lx=>0x%08lx) by EFI call from %pS\n",
-			   flags, cur_flags, caller ?: __builtin_return_address(0));
-	arch_efi_restore_flags(flags);
+	arch_efi_check_flags(flags, caller);
 }
 
 /*
-- 
2.34.1

[PATCH 17/38] arm64/efi: Implement override for IRQ flags checks

[Vladimir Murzin]

Take advantage of the recent changes to the way arm64 represents IRQ
flags and implement an architecture-specific override for the EFI IRQ
flags check.

IRQ flags now carry both DAIF and PMR state. EFI is not expected to
modify either of them, so check both when verifying that IRQ state is
preserved across EFI runtime calls.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/efi.h | 38 ++++++++++++++++++++++++++++++------
 1 file changed, 32 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h
index e8a9783235cb..0fc5a8cd5850 100644
--- a/arch/arm64/include/asm/efi.h
+++ b/arch/arm64/include/asm/efi.h
@@ -5,6 +5,7 @@
 #include <asm/boot.h>
 #include <asm/cpufeature.h>
 #include <asm/fpsimd.h>
+#include <asm/interrupts/common_flags.h>
 #include <asm/io.h>
 #include <asm/memory.h>
 #include <asm/mmu_context.h>
@@ -48,18 +49,43 @@ void arch_efi_call_virt_teardown(void);
 	(efi_rt_stack_top != NULL &&					\
 	 on_task_stack(current, READ_ONCE(efi_rt_stack_top[-1]), 1))
 
-#define ARCH_EFI_IRQ_FLAGS_MASK (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
-
 /*
  * Even when Linux uses IRQ priorities for IRQ disabling, EFI does not.
  * And EFI shouldn't really play around with priority masking as it is not aware
  * which priorities the OS has assigned to its interrupts.
  */
 #define arch_efi_save_flags(state_flags)		\
-	((void)((state_flags) = read_sysreg(daif)))
-
-#define arch_efi_restore_flags(state_flags)	write_sysreg(state_flags, daif)
-
+	((void)((state_flags) = arch_local_save_flags()))
+
+#define arch_efi_restore_flags(state_flags)			\
+do {								\
+	arm64_exc_hwstate_t __st = {.flags=state_flags};	\
+								\
+	arm64_update_exc_hwstate(__st);				\
+} while(0)
+
+#define arch_efi_check_flags(state_flags, caller)									\
+do {															\
+	arm64_exc_hwstate_t __actual = {.flags =  arch_local_save_flags() };						\
+	arm64_exc_hwstate_t __expected = {.flags=state_flags};								\
+															\
+	if (__expected.flags == __actual.flags)										\
+		break;													\
+															\
+	add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_NOW_UNRELIABLE);							\
+															\
+	if (system_uses_irq_prio_masking()) {										\
+		pr_err_ratelimited(FW_BUG "DAIF+PMR corrupted (0x%x + 0x%x=>0x%x + 0x%x) by EFI call from %pS\n",	\
+				   __expected.daif, __expected.pmr, __actual.daif, __actual.pmr,			\
+				   caller ?: __builtin_return_address(0));						\
+	} else {													\
+		pr_err_ratelimited(FW_BUG "DAIF corrupted (0x%x=>0x%x) by EFI call from %pS\n",				\
+				   __expected.daif, __actual.daif,							\
+				   caller ?: __builtin_return_address(0));						\
+	}														\
+															\
+	arch_efi_restore_flags(state_flags);										\
+} while(0)
 
 /* arch specific definitions used by the stub code */
 
-- 
2.34.1

[PATCH 18/38] arm64: booting: Document boot requirements for FEAT_NMI

[Vladimir Murzin]

From: Mark Brown <broonie@kernel.org>

In order to use FEAT_NMI we must be able to use ALLINT, require that
it behave as though not trapped when it is present.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 Documentation/arch/arm64/booting.rst | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Documentation/arch/arm64/booting.rst b/Documentation/arch/arm64/booting.rst
index 13ef311dace8..3170b26ae981 100644
--- a/Documentation/arch/arm64/booting.rst
+++ b/Documentation/arch/arm64/booting.rst
@@ -568,6 +568,12 @@ Before jumping into the kernel, the following conditions must be met:
 
     - HCRX_EL2.EnASR (bit 2) must be initialised to 0b1.
 
+  For CPUs with support for Non-maskable Interrupts (FEAT_NMI):
+
+  - If the kernel is entered at EL1 and EL2 is present:
+
+    - HCRX_EL2.TALLINT (bit 6) must be initialised to 0b0.
+
 The requirements described above for CPU mode, caches, MMUs, architected
 timers, coherency and system registers apply to all CPUs.  All CPUs must
 enter the kernel in the same exception level.  Where the values documented
-- 
2.34.1

[PATCH 19/38] arm64: sysreg: Add definitions for immediate versions of MSR ALLINT

[Vladimir Murzin]

From: Mark Brown <broonie@kernel.org>

Encodings are provided for ALLINT which allow setting of ALLINT.ALLINT
using an immediate rather than requiring that a register be loaded with
the value to write. Since these don't currently fit within the scheme we
have for sysreg generation add manual encodings like we currently do for
other similar registers such as SVCR.

Since it is required that these immediate versions be encoded with xzr
as the source register, provide asm wrappers which ensure this is the
case.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/interrupts/common_flags.h | 16 ++++++++++++++++
 arch/arm64/include/asm/sysreg.h                  |  2 ++
 2 files changed, 18 insertions(+)

diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
index 03cfde24a530..c50c6e6473ec 100644
--- a/arch/arm64/include/asm/interrupts/common_flags.h
+++ b/arch/arm64/include/asm/interrupts/common_flags.h
@@ -44,6 +44,22 @@ typedef enum arm64_exc_context {
 	CRITICAL_CONTEXT,
 } arm64_exc_context_t;
 
+
+#ifdef CONFIG_ARM64_NMI
+static __always_inline void _allint_clear(void)
+{
+	asm volatile(__msr_s(SYS_ALLINT_CLR, "xzr"));
+}
+
+static __always_inline void _allint_set(void)
+{
+	asm volatile(__msr_s(SYS_ALLINT_SET, "xzr"));
+}
+#else
+static __always_inline void _allint_clear(void) {}
+static __always_inline void _allint_set(void) {}
+#endif /* CONFIG_ARM64_NMI */
+
 static __always_inline
 arm64_exc_hwstate_t __arm64_exc_hwstate_of_process_context(void)
 {
diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index 7aa08d59d494..6d2c6bc1b985 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -179,6 +179,8 @@
  * System registers, organised loosely by encoding but grouped together
  * where the architected name contains an index. e.g. ID_MMFR<n>_EL1.
  */
+#define SYS_ALLINT_CLR			sys_reg(0, 1, 4, 0, 0)
+#define SYS_ALLINT_SET			sys_reg(0, 1, 4, 1, 0)
 #define SYS_SVCR_SMSTOP_SM_EL0		sys_reg(0, 3, 4, 2, 3)
 #define SYS_SVCR_SMSTART_SM_EL0		sys_reg(0, 3, 4, 3, 3)
 #define SYS_SVCR_SMSTOP_SMZA_EL0	sys_reg(0, 3, 4, 6, 3)
-- 
2.34.1

[PATCH 20/38] arm64: probes: Reject stepping over ALLINT

[Vladimir Murzin]

ALLINT instructions are not safe for probing. Both the register and
immediate forms of MSR for ALLINT are currently rejected because they
match aarch64_insn_is_msr_{imm,reg}(). However, MRS instructions
accessing ALLINT need to be rejected explicitly.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/insn.h          |  1 +
 arch/arm64/kernel/probes/decode-insn.c | 11 +++++++----
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 1ce75a442638..0a81b5feb2ed 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -118,6 +118,7 @@ enum aarch64_insn_special_register {
 	AARCH64_INSN_SPCLREG_SP_EL0	= 0xC208,
 	AARCH64_INSN_SPCLREG_SPSEL	= 0xC210,
 	AARCH64_INSN_SPCLREG_CURRENTEL	= 0xC212,
+	AARCH64_INSN_SPCLREG_ALLINT	= 0xC218,
 	AARCH64_INSN_SPCLREG_DAIF	= 0xDA11,
 	AARCH64_INSN_SPCLREG_NZCV	= 0xDA10,
 	AARCH64_INSN_SPCLREG_FPCR	= 0xDA20,
diff --git a/arch/arm64/kernel/probes/decode-insn.c b/arch/arm64/kernel/probes/decode-insn.c
index 4137cc5ef031..a98d9d30fb4a 100644
--- a/arch/arm64/kernel/probes/decode-insn.c
+++ b/arch/arm64/kernel/probes/decode-insn.c
@@ -36,11 +36,14 @@ static bool __kprobes aarch64_insn_is_steppable(u32 insn)
 		/*
 		 * The MRS instruction may not return a correct value when
 		 * executing in the single-stepping environment. We do make one
-		 * exception, for reading the DAIF bits.
+		 * exception, for reading the DAIF and ALLINT bits.
 		 */
-		if (aarch64_insn_is_mrs(insn))
-			return aarch64_insn_extract_system_reg(insn)
-			     != AARCH64_INSN_SPCLREG_DAIF;
+		if (aarch64_insn_is_mrs(insn)) {
+			u32 sysreg = aarch64_insn_extract_system_reg(insn);
+
+			return (sysreg != AARCH64_INSN_SPCLREG_DAIF) &&
+			       (sysreg != AARCH64_INSN_SPCLREG_ALLINT);
+		}
 
 		/*
 		 * The HINT instruction is steppable only if it is in whitelist
-- 
2.34.1

[PATCH 21/38] arm64: ptrace: Add PSR_ALLINT_BIT

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

When FEAT_NMI is implemented, ALLINT is part of PSTATE and will be saved
to SPSR on exception entry.
As it can mask interrupts, depending on configuration, we will need to
check its value in interruted tasks, so add the corresponding bit
to ptrace.h.

Given its impact on interrupt and NMI masking, use it to update
`print_pstate()` so that it displays ALLINT state.
Display it after DAIF so that interrupt masking fields are together.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/uapi/asm/ptrace.h | 1 +
 arch/arm64/kernel/process.c          | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/uapi/asm/ptrace.h b/arch/arm64/include/uapi/asm/ptrace.h
index 15649a253a57..c4042da27d80 100644
--- a/arch/arm64/include/uapi/asm/ptrace.h
+++ b/arch/arm64/include/uapi/asm/ptrace.h
@@ -48,6 +48,7 @@
 #define PSR_D_BIT	0x00000200
 #define PSR_BTYPE_MASK	0x00000c00
 #define PSR_SSBS_BIT	0x00001000
+#define PSR_ALLINT_BIT	0x00002000
 #define PSR_PAN_BIT	0x00400000
 #define PSR_UAO_BIT	0x00800000
 #define PSR_DIT_BIT	0x01000000
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 0fd09136d398..c77d1e2b1b3a 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -183,7 +183,7 @@ static void print_pstate(struct pt_regs *regs)
 		const char *btype_str = btypes[(pstate & PSR_BTYPE_MASK) >>
 					       PSR_BTYPE_SHIFT];
 
-		printk("pstate: %08llx (%c%c%c%c %c%c%c%c %cPAN %cUAO %cTCO %cDIT %cSSBS BTYPE=%s)\n",
+		printk("pstate: %08llx (%c%c%c%c %c%c%c%c %cALLINT %cPAN %cUAO %cTCO %cDIT %cSSBS BTYPE=%s)\n",
 			pstate,
 			pstate & PSR_N_BIT ? 'N' : 'n',
 			pstate & PSR_Z_BIT ? 'Z' : 'z',
@@ -193,6 +193,7 @@ static void print_pstate(struct pt_regs *regs)
 			pstate & PSR_A_BIT ? 'A' : 'a',
 			pstate & PSR_I_BIT ? 'I' : 'i',
 			pstate & PSR_F_BIT ? 'F' : 'f',
+			pstate & PSR_ALLINT_BIT ? '+' : '-',
 			pstate & PSR_PAN_BIT ? '+' : '-',
 			pstate & PSR_UAO_BIT ? '+' : '-',
 			pstate & PSR_TCO_BIT ? '+' : '-',
-- 
2.34.1

[PATCH 22/38] arm64: idreg: Add an override for FEAT_NMI

[Vladimir Murzin]

From: Mark Brown <broonie@kernel.org>

Add a named override for FEAT_NMI, allowing it to be explicitly disabled
in case of problems.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/kernel/pi/idreg-override.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c
index 274bc72ba98d..ccda6b2ad69e 100644
--- a/arch/arm64/kernel/pi/idreg-override.c
+++ b/arch/arm64/kernel/pi/idreg-override.c
@@ -164,6 +164,7 @@ static const struct ftr_set_desc pfr1 __prel64_initconst = {
 		FIELD("bt", ID_AA64PFR1_EL1_BT_SHIFT, NULL ),
 		FIELD("gcs", ID_AA64PFR1_EL1_GCS_SHIFT, NULL),
 		FIELD("mte", ID_AA64PFR1_EL1_MTE_SHIFT, NULL),
+		FIELD("nmi", ID_AA64PFR1_EL1_NMI_SHIFT, NULL),
 		FIELD("sme", ID_AA64PFR1_EL1_SME_SHIFT, pfr1_sme_filter),
 		FIELD("mpam_frac", ID_AA64PFR1_EL1_MPAM_frac_SHIFT, NULL),
 		{}
-- 
2.34.1

[PATCH 23/38] arm64: cpufeature: Detect PE support for FEAT_NMI

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Use of FEAT_NMI requires that all the PEs in the system and the GIC
have NMI support. This patch implements the PE part of that detection.

In order to avoid problematic interactions between real and pseudo NMIs
we disable the architected feature if the user has enabled pseudo NMIs
on the command line. If this is done on a system where support for the
architected feature is detected then a warning is printed during boot in
order to help users spot what is likely to be a misconfiguration.

As KVM does not care about the host kernel supporting FEAT_NMI or not
to allow guests to use it, split the CPU cap in two : ARM64_HAS_NMI is
the hardware support, ARM64_NMI is the kernel making use of it.

Co-developed-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/cpucaps.h    |  2 +
 arch/arm64/include/asm/cpufeature.h | 10 +++++
 arch/arm64/kernel/cpufeature.c      | 68 ++++++++++++++++++++++++++++-
 arch/arm64/tools/cpucaps            |  2 +
 4 files changed, 81 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 76350b38f0d7..1a4ef4e94997 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -75,6 +75,8 @@ cpucap_is_possible(const unsigned int cap)
 		return IS_ENABLED(CONFIG_HW_PERF_EVENTS);
 	case ARM64_HAS_LSUI:
 		return IS_ENABLED(CONFIG_ARM64_LSUI);
+	case ARM64_NMI:
+		return IS_ENABLED(CONFIG_ARM64_NMI);
 	}
 
 	return true;
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index 09aa80df602a..c8b1bacdd161 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -809,6 +809,16 @@ static __always_inline bool system_uses_irq_prio_masking(void)
 	return alternative_has_cap_unlikely(ARM64_HAS_GIC_PRIO_MASKING);
 }
 
+static __always_inline bool system_supports_nmi(void)
+{
+	return alternative_has_cap_unlikely(ARM64_HAS_NMI);
+}
+
+static __always_inline bool system_uses_nmi(void)
+{
+	return alternative_has_cap_unlikely(ARM64_NMI);
+}
+
 static inline bool system_supports_mte(void)
 {
 	return alternative_has_cap_unlikely(ARM64_MTE);
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 32102c3912fa..f9fe7a875f85 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -83,13 +83,14 @@
 #include <asm/cpu_ops.h>
 #include <asm/fpsimd.h>
 #include <asm/hwcap.h>
+#include <asm/hypervisor.h>
 #include <asm/insn.h>
+#include <asm/interrupts/common_flags.h>
 #include <asm/kvm_host.h>
 #include <asm/mmu.h>
 #include <asm/mmu_context.h>
 #include <asm/mpam.h>
 #include <asm/mte.h>
-#include <asm/hypervisor.h>
 #include <asm/processor.h>
 #include <asm/smp.h>
 #include <asm/sysreg.h>
@@ -315,6 +316,7 @@ static const struct arm64_ftr_bits ftr_id_aa64pfr1[] = {
 	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_GCS),
 		       FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_GCS_SHIFT, 4, 0),
 	S_ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MTE_frac_SHIFT, 4, 0),
+	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_NMI_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SME),
 		       FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_SME_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MPAM_frac_SHIFT, 4, 0),
@@ -2341,6 +2343,51 @@ static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry
 }
 #endif
 
+#ifdef CONFIG_ARM64_NMI
+static bool can_use_nmi(const struct arm64_cpu_capabilities *entry, int scope)
+{
+	/*
+	 * ARM64_HAS_NMI has a lower index, and is a boot CPU
+	 * feature, so will be detected earlier.
+	 */
+	BUILD_BUG_ON(ARM64_NMI <= ARM64_HAS_NMI);
+	if (!cpus_have_cap(ARM64_HAS_NMI))
+		return false;
+
+	/*
+	 * Having both real and pseudo NMIs enabled simultaneously is
+	 * likely to cause confusion.  Since pseudo NMIs must be
+	 * enabled with an explicit command line option, if the user
+	 * has set that option on a system with real NMIs for some
+	 * reason assume they know what they're doing.
+	 *
+	 * ARM64_HAS_GIC_PRIO_MASKING has a lower index, and is a boot CPU
+	 * feature, so will be detected earlier.
+	 */
+	BUILD_BUG_ON(IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) &&
+			(ARM64_NMI <= ARM64_HAS_GIC_PRIO_MASKING));
+	if (cpus_have_cap(ARM64_HAS_GIC_PRIO_MASKING)) {
+		pr_info_once("Pseudo NMI enabled, not using architected NMI\n");
+		return false;
+	}
+
+	return true;
+}
+
+static void nmi_enable(const struct arm64_cpu_capabilities *__unused)
+{
+	/*
+	 * Enable use of NMIs controlled by ALLINT, SPINTMASK should
+	 * be clear by default but make it explicit that we are using
+	 * this mode.  Ensure that ALLINT is clear first in order to
+	 * avoid leaving things masked.
+	 */
+	_allint_clear();
+	sysreg_clear_set(sctlr_el1, SCTLR_EL1_SPINTMASK, SCTLR_EL1_NMI);
+	isb();
+}
+#endif
+
 static bool can_trap_icv_dir_el1(const struct arm64_cpu_capabilities *entry,
 				 int scope)
 {
@@ -3233,6 +3280,25 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.matches = has_cpuid_feature,
 		ARM64_CPUID_FIELDS(ID_AA64ISAR3_EL1, LSUI, IMP)
 	},
+#endif
+	{
+		.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
+		.capability = ARM64_HAS_NMI,
+		.matches = has_cpuid_feature,
+		ARM64_CPUID_FIELDS(ID_AA64PFR1_EL1, NMI, IMP)
+	},
+#ifdef CONFIG_ARM64_NMI
+	/*
+	 * Depends on ARM64_HAS_NMI
+	 * Checks for conflict with pseudo-NMIs, giving them priority.
+	 */
+	{
+		.desc = "Non-maskable Interrupts",
+		.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
+		.capability = ARM64_NMI,
+		.matches = can_use_nmi,
+		.cpu_enable = nmi_enable,
+	},
 #endif
 	{},
 };
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index 2775ba3359cf..531583822ec2 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -52,6 +52,7 @@ HAS_LS64_V
 HAS_LSUI
 HAS_MOPS
 HAS_NESTED_VIRT
+HAS_NMI
 HAS_NV2P1
 HAS_NV3
 HAS_PAN
@@ -82,6 +83,7 @@ MTE
 MTE_ASYMM
 MTE_FAR
 MTE_STORE_ONLY
+NMI
 SME
 SME_FA64
 SME2
-- 
2.34.1

[PATCH 24/38] arm64: nmi: Manage masking for superpriority interrupts

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Extend logic to handle and debug exception context/state with knowlage
of FEAT_NMI.

Take care to order writes to ALLINT relative to DAIF, as clearing
ALLINT before DAIF could result in taking an NMI while DAIF is fully
masked, as could setting it after DAIF.

Since superpriority interrupts are not masked through DAIF like pseduo
NMIs are, we also need to modify the assembler macros for managing
DAIF to ensure that the masking is done in the assembly code.

Note that save_and_disable_irq/restore_irq and
save_and_disable_daif/restore_irq pairs are used in distinct
contextes:

- former is used in context of SW PAN to quickly disable/enable
  preemption

- latter is used to completely mask all exceptions.

For that reason split save_and_disable_daif/restore_irq into more
generic exception save restore pair and plumb with FEAT_NMI logic.

Co-developed-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/assembler.h            | 20 +++++++++--
 .../include/asm/interrupts/common_flags.h     | 36 +++++++++++++++++++
 arch/arm64/include/asm/interrupts/entry.h     |  6 ++--
 arch/arm64/include/asm/interrupts/masking.h   |  3 +-
 arch/arm64/include/asm/irqflags.h             |  3 +-
 arch/arm64/kernel/entry.S                     | 12 +++----
 6 files changed, 68 insertions(+), 12 deletions(-)

diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 0b58b550e8dc..bcdbc308afba 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -37,11 +37,27 @@
 /*
  * Save/restore interrupts.
  */
-	.macro save_and_disable_daif, flags
-	mrs	\flags, daif
+	.macro save_and_disable_exceptions, flags, tmp
+	mrs     \flags, daif		// updates flags[9:6] with DAIF
+#ifdef CONFIG_ARM64_NMI
+alternative_if ARM64_NMI
+	mrs_s	\tmp, SYS_ALLINT	// updates tmp[13] with AllInt
+	msr_s	SYS_ALLINT_SET, xzr
+	orr	\flags, \flags, \tmp	// now flags[13,9:6] carry pair of AllInt,DAIF
+alternative_else_nop_endif
+#endif
 	msr	daifset, #0xf
 	.endm
 
+	.macro  restore_exceptions, flags
+	msr     daif, \flags		// bits other than flags[9:6] are ignored
+#ifdef CONFIG_ARM64_NMI
+alternative_if ARM64_NMI
+	msr_s	SYS_ALLINT, \flags	// bits other than flags[13] are ignored
+alternative_else_nop_endif
+#endif
+	.endm
+
 	.macro	save_and_disable_irq, flags
 	mrs	\flags, daif
 	msr	daifset, #3
diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
index c50c6e6473ec..18c9a392028a 100644
--- a/arch/arm64/include/asm/interrupts/common_flags.h
+++ b/arch/arm64/include/asm/interrupts/common_flags.h
@@ -20,6 +20,14 @@
 /*
  * Exception context mapping
  *
+ * FEAT_NMI
+ *
+ * CRITICAL -> DAIF + AllInt (corresponds to the state on exception entry)
+ * ERROR    ->  AIF + AllInt
+ * NONMI    ->   IF + AllInt
+ * NOIRQ    ->   IF
+ * PROCESS  ->    0
+ *
  * pseudo-NMI
  *
  * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
@@ -76,6 +84,7 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
 		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
 
 	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
+
 }
 
 static __always_inline
@@ -84,6 +93,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
 	if (system_uses_irq_prio_masking())
 		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
 
+	if (system_uses_nmi())
+               return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .allint=ALLINT_ALLINT};
+
 	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
 }
 
@@ -93,6 +105,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
 	if (system_uses_irq_prio_masking())
 		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
 
+	if (system_uses_nmi())
+		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .allint=ALLINT_ALLINT};
+
 	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
 }
 
@@ -102,6 +117,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
 	if (system_uses_irq_prio_masking())
 		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
 
+	if (system_uses_nmi())
+		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .allint=ALLINT_ALLINT};
+
 	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
 }
 
@@ -131,6 +149,9 @@ arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
 	if (system_uses_irq_prio_masking())
 		state.pmr = regs->pmr;
 
+	if (system_uses_nmi())
+		state.allint = regs->pstate & PSR_ALLINT_BIT;
+
 	return state;
 }
 
@@ -150,6 +171,9 @@ void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
 	if (system_uses_irq_prio_masking()) {
 		WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
 			  actual.daif, actual.pmr, expected.daif, expected.pmr);
+	} else if (system_uses_nmi()) {
+		WARN_ONCE(1, "Unexpected DAIF+ALLINT: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
+			  actual.daif, actual.allint, expected.daif, expected.allint);
 	} else {
 		WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
 			  actual.daif, expected.daif);
@@ -168,6 +192,7 @@ static __always_inline
 void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
 {
 	bool pseudo_nmi = system_uses_irq_prio_masking();
+	bool nmi = system_uses_nmi();
 
 	barrier();
 
@@ -194,8 +219,19 @@ void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
 		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
 	}
 
+	/*
+	 * Try to order ALLINT writes to be consistent with the DAIF state :
+	 * we don't want to take an NMI with DAIF masked or when it should
+	 * be masked but isn't yet.
+	 */
+	if (nmi && (hwstate.allint & ALLINT_ALLINT) && force)
+		_allint_set();
+
 	write_sysreg(hwstate.daif, daif);
 
+	if (nmi && !(hwstate.allint & ALLINT_ALLINT) && force)
+		_allint_clear();
+
 	if (pseudo_nmi && hwstate.pmr == GIC_PRIO_IRQON && force) {
 		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
 		pmr_sync();
diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
index 0cfca62aa25b..2a522dd15556 100644
--- a/arch/arm64/include/asm/interrupts/entry.h
+++ b/arch/arm64/include/asm/interrupts/entry.h
@@ -10,7 +10,6 @@
 #include <asm/cpufeature.h>
 #include <asm/interrupts/common_flags.h>
 
-
 static __always_inline
 arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
 						  arm64_exc_hwstate_t next)
@@ -26,7 +25,8 @@ arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
 	if (!irqs_disabled)
 		trace_hardirqs_on();
 
-	force = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
+	force = (system_uses_irq_prio_masking() && prev.pmr != next.pmr) ||
+		(system_uses_nmi() && prev.allint != next.allint);
 
 	__arm64_update_exc_hwstate(next, force);
 
@@ -58,6 +58,7 @@ arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
 			 * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so larger PMR value is weaker
 			 */
 			WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr > next.pmr);
+			WARN_ON_ONCE(system_uses_nmi() && prev.allint < next.allint);
 		}
 	}
 
@@ -77,6 +78,7 @@ arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
 			 * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so smaller PMR value is stronger
 			 */
 			WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr < next.pmr);
+			WARN_ON_ONCE(system_uses_nmi() && prev.allint > next.allint);
 		}
 	}
 
diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
index 8729922974b0..6c5ca5befbc1 100644
--- a/arch/arm64/include/asm/interrupts/masking.h
+++ b/arch/arm64/include/asm/interrupts/masking.h
@@ -30,7 +30,8 @@ arm64_exc_hwstates_t local_exceptions_save_mask(arm64_exc_context_t new)
 	 * We've just got actual HW state so we can rely on that to
 	 * optimize some unnecessary updates.
 	 */
-	force = system_uses_irq_prio_masking() && actual.pmr != state.pmr;
+	force = (system_uses_irq_prio_masking() && actual.pmr != state.pmr) ||
+		(system_uses_nmi() && actual.allint != state.allint);
 
 	if (!irqs_disabled)
 		trace_hardirqs_on();
diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index eedbdf7358e6..31b8f4e99532 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -31,9 +31,10 @@
 typedef union arm64_exc_hwstate {
 	struct {
 		u16 daif;
+		u16 allint;
 		u8 pmr;
 		u8 __pad0;
-		u32 __pad1;
+		u16 __pad1;
 	};
 	unsigned long flags;
 } arm64_exc_hwstate_t;
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index cb3be770f2d0..39ea3fdeb03a 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -815,7 +815,7 @@ SYM_CODE_END(__bp_harden_el1_vectors)
  *
  */
 SYM_FUNC_START(cpu_switch_to)
-	save_and_disable_daif x11
+	save_and_disable_exceptions x11, x12
 	mov	x10, #THREAD_CPU_CONTEXT
 	add	x8, x0, x10
 	mov	x9, sp
@@ -839,7 +839,7 @@ SYM_FUNC_START(cpu_switch_to)
 	ptrauth_keys_install_kernel x1, x8, x9, x10
 	scs_save x0
 	scs_load_current
-	restore_irq x11
+	restore_exceptions x11
 	ret
 SYM_FUNC_END(cpu_switch_to)
 NOKPROBE(cpu_switch_to)
@@ -866,7 +866,7 @@ NOKPROBE(ret_from_fork)
  * Calls func(regs) using this CPU's irq stack and shadow irq stack.
  */
 SYM_FUNC_START(call_on_irq_stack)
-	save_and_disable_daif x9
+	save_and_disable_exceptions x9, x10
 #ifdef CONFIG_SHADOW_CALL_STACK
 	get_current_task x16
 	scs_save x16
@@ -881,10 +881,10 @@ SYM_FUNC_START(call_on_irq_stack)
 
 	/* Move to the new stack and call the function there */
 	add	sp, x16, #IRQ_STACK_SIZE
-	restore_irq x9
+	restore_exceptions x9
 	blr	x1
 
-	save_and_disable_daif x9
+	save_and_disable_exceptions x9, x10
 	/*
 	 * Restore the SP from the FP, and restore the FP and LR from the frame
 	 * record.
@@ -892,7 +892,7 @@ SYM_FUNC_START(call_on_irq_stack)
 	mov	sp, x29
 	ldp	x29, x30, [sp], #16
 	scs_load_current
-	restore_irq x9
+	restore_exceptions x9
 	ret
 SYM_FUNC_END(call_on_irq_stack)
 NOKPROBE(call_on_irq_stack)
-- 
2.34.1

[PATCH 25/38] arm64: irq: Report FEAT_NMI masking local IRQs

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

As we clear SCTLR_EL1.SPINTMASK when enabling FEAT_NMI, ALLINT masks
IRQs and FIQs regardless of superpriority.
Update irqflags.h and `regs_irqs_disabled()` to take it into account,
so we properly keep track of the local IRQ masking state.

We have documentation at the top of irqflags.h which explains the DAIF
masking. Since the additional masking with NMIs is related and also
covers the IF in DAIF, extend the comment to note what's going on
with NMIs.

Co-developed-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/irqflags.h | 17 +++++++++++++++++
 arch/arm64/include/asm/ptrace.h   |  3 +++
 2 files changed, 20 insertions(+)

diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index 31b8f4e99532..d64d89a8f31a 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -20,6 +20,16 @@
  * always masked and unmasked together, and have no side effects for other
  * flags. Keeping to this order makes it easier for entry.S to know which
  * exceptions should be unmasked.
+ *
+ * With the addition of the FEAT_NMI extension we gain an additional
+ * class of superpriority IRQ/FIQ which is separately masked with a
+ * choice of modes controlled by SCTLR_ELn.{SPINTMASK,NMI}.
+ * Linux sets SPINTMASK to 0 and NMI to 1 which results in ALLINT.ALLINT
+ * masking both superpriority interrupts and IRQ/FIQ regardless of the
+ * I and F settings. Since these superpriority interrupts are being
+ * used as NMIs we do not include them in the interrupt masking here,
+ * anything that requires that NMIs be masked needs to explicitly do so,
+ * but we do check for ALLINT masking IRQs/FIQs.
  */
 
  /*
@@ -105,6 +115,9 @@ static __always_inline unsigned long arch_local_save_flags(void)
 {
 	arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
 
+	if (system_uses_nmi())
+		hwstate.allint = read_sysreg_s(SYS_ALLINT);
+
 	if (system_uses_irq_prio_masking())
 		hwstate.pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
 
@@ -118,6 +131,10 @@ static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
 	if (hwstate.daif & PSR_I_BIT)
 		return true;
 
+	/* SCTLR_EL1.SPINTMASK is clear, so ALLINT masks *all* IRQs/FIQs. */
+	if (system_uses_nmi())
+		return !!(hwstate.allint & ALLINT_ALLINT);
+
 	if (system_uses_irq_prio_masking()) {
 		WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_IRQFLAGS) &&
 			     hwstate.pmr != GIC_PRIO_IRQOFF &&
diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index cf7e51b8ef0b..16ba164b890e 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -206,6 +206,9 @@ static __always_inline bool regs_irqs_disabled(const struct pt_regs *regs)
 	if (regs->pstate & PSR_I_BIT)
 		return true;
 
+	if (system_uses_nmi())
+		return !!(regs->pstate & PSR_ALLINT_BIT);
+
 	if (system_uses_irq_prio_masking()) {
 		WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_IRQFLAGS) &&
 			     regs->pmr != GIC_PRIO_IRQOFF &&
-- 
2.34.1

[PATCH 26/38] arm64: nmi: Add handling of superpriority interrupts as NMIs

[Vladimir Murzin]

From: Mark Brown <broonie@kernel.org>

Our goal with superpriority interrupts is to use them as NMIs, taking
advantage of the much smaller regions where they are masked to allow
prompt handling of the most time-critical interrupts.

When an interrupt is configured with superpriority, we enter EL1 as we
do for any other interrupt. The presence of a superpriority interrupt
is indicated by a status bit in ISR_EL1. We check this bit before
unmasking interrupts in elX_interrupt(), and if a superpriority
interrupt is pending, we handle it as an NMI. Otherwise, normal
interrupts are handled as usual.

Since superpriority interrupts are always handled as NMIs, the
interrupt controller can rely on in_nmi() to distinguish them from
ordinary interrupts.

Enable IPIs to use superpriority interrupts as NMIs, matching the
existing pseudo-NMI behaviour.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/entry-common.h |  7 +++
 arch/arm64/kernel/entry-common.c      | 83 +++++++++++++++++++++++----
 arch/arm64/kernel/smp.c               |  2 +-
 3 files changed, 79 insertions(+), 13 deletions(-)

diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
index 73d82a8d8e95..0681ba91ac3b 100644
--- a/arch/arm64/include/asm/entry-common.h
+++ b/arch/arm64/include/asm/entry-common.h
@@ -37,6 +37,13 @@ static inline bool arch_irqentry_exit_need_resched(void)
 	if (system_uses_irq_prio_masking() && read_sysreg(daif))
 		return false;
 
+	/*
+	 * If AllInt is set then we must have handled an NMI, so skip
+	 * preemption
+	 */
+	if (system_uses_nmi() && read_sysreg_s(SYS_ALLINT))
+		return false;
+
 	/*
 	 * Preempting a task from an IRQ means we leave copies of PSTATE
 	 * on the stack. cpufeature's enable calls may modify PSTATE, but
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index fd0bfead6e70..31dae3b48f23 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -31,6 +31,14 @@
 #include <asm/sysreg.h>
 #include <asm/system_misc.h>
 
+static __always_inline bool is_nmi(void)
+{
+	if (!system_uses_nmi())
+		return false;
+
+	return !!(read_sysreg(isr_el1) & ISR_EL1_IS);
+}
+
 /*
  * Handle IRQ/context state management when entering from kernel mode.
  * Before this function is called it is not safe to call regular kernel code,
@@ -525,8 +533,8 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
 	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
-static __always_inline void __el1_pnmi(struct pt_regs *regs,
-				       void (*handler)(struct pt_regs *))
+static __always_inline void __el1_nmi(struct pt_regs *regs,
+				      void (*handler)(struct pt_regs *))
 {
 	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
@@ -545,7 +553,23 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
 
 	state = arm64_enter_from_kernel_mode(regs);
 
-	arm64_unmask_exc_context(NONMI_CONTEXT);
+	/*
+	 * For non-NMI systems, we can switch to NOIRQ context since
+	 * NMIs cannot occur.
+	 *
+	 * For pseudo-NMI, we must keep NONMI context until we can
+	 * determine the interrupt priority, which is done by the IRQ
+	 * handler.
+	 *
+	 * For FEAT_NMI, we already know the interrupt is an IRQ, so we
+	 * can switch to NOIRQ context immediately. NMIs do not
+	 * interfere because they follow a separate handling path and do
+	 * not share the IRQ handling state.
+	 */
+	if (system_uses_irq_prio_masking())
+		arm64_unmask_exc_context(NONMI_CONTEXT);
+	else
+		arm64_unmask_exc_context(NOIRQ_CONTEXT);
 
 	irq_enter_rcu();
 	do_interrupt_handler(regs, handler);
@@ -565,8 +589,9 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
 static void noinstr el1_interrupt(struct pt_regs *regs,
 				  void (*handler)(struct pt_regs *))
 {
-	if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && regs_irqs_disabled(regs))
-		__el1_pnmi(regs, handler);
+	/* Is there a NMI to handle? */
+	if (regs_irqs_disabled(regs) || is_nmi())
+		__el1_nmi(regs, handler);
 	else
 		__el1_irq(regs, handler);
 
@@ -580,6 +605,8 @@ asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
 
 asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
 {
+	WARN_ON_ONCE(system_uses_nmi() && (read_sysreg(isr_el1) & ISR_EL1_FS));
+
 	el1_interrupt(regs, handle_arch_fiq);
 }
 
@@ -903,20 +930,36 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
 	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
-static void noinstr el0_interrupt(struct pt_regs *regs,
-				  void (*handler)(struct pt_regs *))
+static __always_inline void __el0_nmi(struct pt_regs *regs,
+				      void (*handler)(struct pt_regs *))
 {
-	arm64_enter_from_user_mode(regs);
+	irqentry_state_t state;
+	arm64_exc_hwstate_t hwstate;
 
-	arm64_unmask_exc_context(NONMI_CONTEXT);
+	state = irqentry_nmi_enter(regs);
+	hwstate = arm64_unmask_exc_context(NONMI_CONTEXT);
+	do_interrupt_handler(regs, handler);
+	arm64_mask_exc_context(hwstate);
+	irqentry_nmi_exit(regs, state);
+}
 
-	if (regs->pc & BIT(55))
-		arm64_apply_bp_hardening();
+static __always_inline void __el0_irq(struct pt_regs *regs,
+				      void (*handler)(struct pt_regs *))
+{
+	arm64_enter_from_user_mode(regs);
+	/*
+	 * For the same reason as in el1_irq() keep NONMI
+	 * context for pesudo-NMI, and switch to NOIRQ
+	 * otherwise
+	 */
+	if (system_uses_irq_prio_masking())
+		arm64_unmask_exc_context(NONMI_CONTEXT);
+	else
+		arm64_unmask_exc_context(NOIRQ_CONTEXT);
 
 	irq_enter_rcu();
 	do_interrupt_handler(regs, handler);
 	irq_exit_rcu();
-
 	/*
 	 * For the same reason as in el1_irq() we effectivly
 	 * have NOIRQ_CONTEXT on return from handler - keep
@@ -924,6 +967,20 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
 	 */
 	arm64_debug_exc_context(NOIRQ_CONTEXT);
 	arm64_exit_to_user_mode(regs, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
+}
+
+static void noinstr el0_interrupt(struct pt_regs *regs,
+				  void (*handler)(struct pt_regs *))
+{
+	if (regs->pc & BIT(55))
+		arm64_apply_bp_hardening();
+
+	/* Is there a NMI to handle? */
+	if (is_nmi())
+		__el0_nmi(regs, handler);
+	else
+		__el0_irq(regs, handler);
+
 	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
@@ -939,6 +996,8 @@ asmlinkage void noinstr el0t_64_irq_handler(struct pt_regs *regs)
 
 static void noinstr __el0_fiq_handler_common(struct pt_regs *regs)
 {
+	WARN_ON_ONCE(system_uses_nmi() && (read_sysreg(isr_el1) & ISR_EL1_FS));
+
 	el0_interrupt(regs, handle_arch_fiq);
 }
 
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index f29094ec3843..3d7060aedd66 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -1061,7 +1061,7 @@ static void smp_cross_call(const struct cpumask *target, unsigned int ipinr)
 
 static bool ipi_should_be_nmi(enum ipi_msg_type ipi)
 {
-	if (!system_uses_irq_prio_masking())
+	if (!system_uses_nmi() && !system_uses_irq_prio_masking())
 		return false;
 
 	switch (ipi) {
-- 
2.34.1

[PATCH 27/38] arm64: suspend: Always initialise PSTATE.ALLINT

[Vladimir Murzin]

PSTATE.ALLINT is always set to the inverse of SCTLR_ELx.SPINTMASK,
regardless of the value of SCTLR_ELx.NMI. SCTLR_ELx.NMI is initialised
to 0 by default, so PSTATE.ALLINT does not cause any interrupt
masking. With upcoming FEAT_NMI support, SCTLR_ELx.NMI will be set as
part of the enable sequence.

However, during CPU suspend/resume, we reinitialise PSTATE to
INIT_PSTATE_EL1. INIT_PSTATE_EL1 currently does not set
PSR_ALLINT_BIT, but SCTLR_ELx.NMI is restored from the saved
value. Since all exceptions are masked during CPU suspend/resume,
avoid a state where SCTLR_ELx.NMI is restored to 1 but PSTATE.ALLINT
is clear, since that would permit an NMI during resume.

Include PSR_ALLINT_BIT in INIT_PSTATE_EL1 so PSTATE.ALLINT is always
set.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/ptrace.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index 16ba164b890e..ad5dcaf66a8f 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -17,7 +17,7 @@
 #define CurrentEL_EL2		(2 << 2)
 
 #define INIT_PSTATE_EL1 \
-	(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL1h)
+	(PSR_ALLINT_BIT | PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL1h)
 
 #include <linux/irqchip/arm-gic-v3-prio.h>
 
-- 
2.34.1

[PATCH 28/38] arm64/efi: Add ALLINT to IRQ flags checks

[Vladimir Murzin]

EFI is not expected to modify PSTATE.ALLINT, so check that it remains
unchanged across EFI runtime calls.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/efi.h | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h
index 0fc5a8cd5850..c7fb21c00690 100644
--- a/arch/arm64/include/asm/efi.h
+++ b/arch/arm64/include/asm/efi.h
@@ -78,6 +78,10 @@ do {															\
 		pr_err_ratelimited(FW_BUG "DAIF+PMR corrupted (0x%x + 0x%x=>0x%x + 0x%x) by EFI call from %pS\n",	\
 				   __expected.daif, __expected.pmr, __actual.daif, __actual.pmr,			\
 				   caller ?: __builtin_return_address(0));						\
+	} else if (system_uses_nmi()) {											\
+		pr_err_ratelimited(FW_BUG "DAIF+ALLINT corrupted (0x%x + 0x%x=>0x%x + 0x%x) by EFI call from %pS\n",	\
+				   __expected.daif, __expected.allint, __actual.daif, __actual.allint,			\
+				   caller ?: __builtin_return_address(0));						\
 	} else {													\
 		pr_err_ratelimited(FW_BUG "DAIF corrupted (0x%x=>0x%x) by EFI call from %pS\n",				\
 				   __expected.daif, __actual.daif,							\
-- 
2.34.1

[PATCH 29/38] arm64: kprobes: Disable NMIs

[Vladimir Murzin]

Kprobes masks all DAIF exceptions to keep things simple and avoid
nested exceptions. With FEAT_NMI, that is no longer enough. Mask
ALLINT as well to ensure things stay simple.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/kernel/probes/kprobes.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c
index 68f4271ce02f..ac14022ff4aa 100644
--- a/arch/arm64/kernel/probes/kprobes.c
+++ b/arch/arm64/kernel/probes/kprobes.c
@@ -208,17 +208,27 @@ static void __kprobes set_current_kprobe(struct kprobe *p)
  * the kprobe state is per-CPU and doesn't get migrated.
  */
 static void __kprobes kprobes_save_local_irqflag(struct kprobe_ctlblk *kcb,
-						struct pt_regs *regs)
+						 struct pt_regs *regs)
 {
-	kcb->saved_irqflag = regs->pstate & DAIF_MASK;
-	regs->pstate |= DAIF_MASK;
+	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
+
+	kcb->saved_irqflag = arm64_inherit_exc_hwstate(regs).flags;
+
+	regs->pstate &= ~(DAIF_MASK | PSR_ALLINT_BIT);
+
+	regs->pstate |= next.daif;
+	regs->pstate |= next.allint;
 }
 
 static void __kprobes kprobes_restore_local_irqflag(struct kprobe_ctlblk *kcb,
-						struct pt_regs *regs)
+						    struct pt_regs *regs)
 {
-	regs->pstate &= ~DAIF_MASK;
-	regs->pstate |= kcb->saved_irqflag;
+	arm64_exc_hwstate_t prev = {.flags = kcb->saved_irqflag};
+
+	regs->pstate &= ~(DAIF_MASK | PSR_ALLINT_BIT);
+
+	regs->pstate |= prev.daif;
+	regs->pstate |= prev.allint;
 }
 
 static void __kprobes setup_singlestep(struct kprobe *p,
-- 
2.34.1

[PATCH 30/38] arm64: smp: Abstract SGI and LPI operations

[Vladimir Murzin]

SGI and LPI backed IPIs require different setup, enable, disable and
send operations. These differences are currently handled by repeatedly
checking percpu_ipi_descs. As the implementation specific logic grows,
these checks make the common IPI code increasingly difficult to
follow.

Introduce an operations structure for each implementation to
encapsulate the specific of SGI and LPI handling, leaving the common
IPI paths generic.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/kernel/smp.c | 135 ++++++++++++++++++++++++++--------------
 1 file changed, 88 insertions(+), 47 deletions(-)

diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 3d7060aedd66..46cdcc114997 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -77,11 +77,18 @@ static DEFINE_PER_CPU_READ_MOSTLY(struct ipi_descs, pcpu_ipi_desc);
 
 #define get_ipi_desc(__cpu, __ipi) (per_cpu_ptr(&pcpu_ipi_desc, __cpu)->descs[__ipi])
 
-static bool percpu_ipi_descs __ro_after_init;
+struct ipi_irq_ops {
+	void (*setup)(int ipi, int ncpus);
+	void (*disable)(int cpu, int ipi);
+	void (*enable)(int cpu, int ipi);
+	void (*send)(const cpumask_t *mask, unsigned int nr);
+};
+
+static const struct ipi_irq_ops *ipi_ops __ro_after_init;
 
 static bool crash_stop;
 
-static void ipi_setup(int cpu);
+static void ipi_enable(int cpu);
 
 #ifdef CONFIG_HOTPLUG_CPU
 static void ipi_teardown(int cpu);
@@ -242,7 +249,7 @@ asmlinkage notrace void secondary_start_kernel(void)
 	 */
 	notify_cpu_starting(cpu);
 
-	ipi_setup(cpu);
+	ipi_enable(cpu);
 
 	numa_add_cpu(cpu);
 
@@ -937,13 +944,7 @@ void __noreturn panic_smp_self_stop(void)
 
 static void arm64_send_ipi(const cpumask_t *mask, unsigned int nr)
 {
-	unsigned int cpu;
-
-	if (!percpu_ipi_descs)
-		__ipi_send_mask(get_ipi_desc(0, nr), mask);
-	else
-		for_each_cpu(cpu, mask)
-			__ipi_send_single(get_ipi_desc(cpu, nr), cpu);
+	ipi_ops->send(mask, nr);
 }
 
 static void arm64_backtrace_ipi(cpumask_t *mask)
@@ -1074,54 +1075,37 @@ static bool ipi_should_be_nmi(enum ipi_msg_type ipi)
 	}
 }
 
-static void ipi_setup(int cpu)
+static void ipi_enable(int cpu)
 {
-	int i;
+	int ipi;
 
 	if (WARN_ON_ONCE(!ipi_irq_base))
 		return;
 
-	for (i = 0; i < nr_ipi; i++) {
-		if (!percpu_ipi_descs) {
-			if (ipi_should_be_nmi(i)) {
-				prepare_percpu_nmi(ipi_irq_base + i);
-				enable_percpu_nmi(ipi_irq_base + i, 0);
-			} else {
-				enable_percpu_irq(ipi_irq_base + i, 0);
-			}
-		} else {
-			enable_irq(irq_desc_get_irq(get_ipi_desc(cpu, i)));
-		}
-	}
+	for (ipi = 0; ipi < nr_ipi; ipi++)
+		ipi_ops->enable(cpu, ipi);
 }
 
 #ifdef CONFIG_HOTPLUG_CPU
 static void ipi_teardown(int cpu)
 {
-	int i;
+	int ipi;
 
 	if (WARN_ON_ONCE(!ipi_irq_base))
 		return;
 
-	for (i = 0; i < nr_ipi; i++) {
-		if (!percpu_ipi_descs) {
-			if (ipi_should_be_nmi(i)) {
-				disable_percpu_nmi(ipi_irq_base + i);
-				teardown_percpu_nmi(ipi_irq_base + i);
-			} else {
-				disable_percpu_irq(ipi_irq_base + i);
-			}
-		} else {
-			disable_irq_nosync(irq_desc_get_irq(get_ipi_desc(cpu, i)));
-		}
-	}
+	for (ipi = 0; ipi < nr_ipi; ipi++)
+		ipi_ops->disable(cpu, ipi);
 }
 #endif
 
-static void ipi_setup_sgi(int ipi)
+static void ipi_sgi_setup(int ipi, int ncpus)
 {
 	int err, irq, cpu;
 
+	if (WARN_ON_ONCE(ncpus))
+		return;
+
 	irq = ipi_irq_base + ipi;
 
 	if (ipi_should_be_nmi(ipi)) {
@@ -1138,7 +1122,53 @@ static void ipi_setup_sgi(int ipi)
 	irq_set_status_flags(irq, IRQ_HIDDEN);
 }
 
-static void ipi_setup_lpi(int ipi, int ncpus)
+static void ipi_sgi_enable(int cpu, int ipi)
+{
+	int irq = ipi_irq_base + ipi;
+
+	if (ipi_should_be_nmi(ipi)) {
+		prepare_percpu_nmi(irq);
+		enable_percpu_nmi(irq, 0);
+	} else {
+		enable_percpu_irq(irq, 0);
+	}
+}
+
+static void ipi_sgi_disable(int cpu, int ipi)
+{
+	int irq = ipi_irq_base + ipi;
+
+	if (ipi_should_be_nmi(ipi)) {
+		disable_percpu_nmi(irq);
+		teardown_percpu_nmi(irq);
+	} else {
+		disable_percpu_irq(irq);
+	}
+}
+
+static void ipi_sgi_send(const cpumask_t *mask, unsigned int nr)
+{
+	__ipi_send_mask(get_ipi_desc(0, nr), mask);
+}
+
+static const struct ipi_irq_ops ipi_sgi_ops = {
+	.setup = ipi_sgi_setup,
+	.disable = ipi_sgi_disable,
+	.enable = ipi_sgi_enable,
+	.send = ipi_sgi_send,
+};
+
+static void ipi_lpi_enable(int cpu, int ipi)
+{
+	enable_irq(irq_desc_get_irq(get_ipi_desc(cpu, ipi)));
+}
+
+static void ipi_lpi_disable(int cpu, int ipi)
+{
+	disable_irq_nosync(irq_desc_get_irq(get_ipi_desc(cpu, ipi)));
+}
+
+static void ipi_lpi_setup(int ipi, int ncpus)
 {
 	for (int cpu = 0; cpu < ncpus; cpu++) {
 		int err, irq;
@@ -1158,6 +1188,20 @@ static void ipi_setup_lpi(int ipi, int ncpus)
 	}
 }
 
+static void ipi_lpi_send(const cpumask_t *mask, unsigned int nr) {
+	int cpu;
+
+	for_each_cpu(cpu, mask)
+		__ipi_send_single(get_ipi_desc(cpu, nr), cpu);
+}
+
+static const struct ipi_irq_ops ipi_lpi_ops = {
+	.setup = ipi_lpi_setup,
+	.disable = ipi_lpi_disable,
+	.enable = ipi_lpi_enable,
+	.send = ipi_lpi_send,
+};
+
 void __init set_smp_ipi_range_percpu(int ipi_base, int n, int ncpus)
 {
 	int i;
@@ -1165,18 +1209,15 @@ void __init set_smp_ipi_range_percpu(int ipi_base, int n, int ncpus)
 	WARN_ON(n < MAX_IPI);
 	nr_ipi = min(n, MAX_IPI);
 
-	percpu_ipi_descs = !!ncpus;
 	ipi_irq_base = ipi_base;
 
-	for (i = 0; i < nr_ipi; i++) {
-		if (!percpu_ipi_descs)
-			ipi_setup_sgi(i);
-		else
-			ipi_setup_lpi(i, ncpus);
-	}
+	ipi_ops = ncpus ? &ipi_lpi_ops : &ipi_sgi_ops;
+
+	for (i = 0; i < nr_ipi; i++)
+		ipi_ops->setup(i, ncpus);
 
 	/* Setup the boot CPU immediately */
-	ipi_setup(smp_processor_id());
+	ipi_enable(smp_processor_id());
 }
 
 void arch_smp_send_reschedule(int cpu)
-- 
2.34.1

[PATCH 31/38] arm64: smp: Fall back to IRQ when IPI NMI request fails

[Vladimir Murzin]

Existing IPI setup logic relies on ipi_should_be_nmi() to decide
whether to request an NMI (if supported) or an ordinary IRQ. If NMI is
not supported, using an ordinary IRQ is acceptable.

FEAT_NMI requires complementary support in the interrupt
controller. If FEAT_NMI is present but interrupt controller support is
absent, request_percpu_nmi() can fail.

One way to prevent this would be to extend ipi_should_be_nmi() to also
check for interrupt controller support. However, since falling back to
an ordinary IRQ is acceptable, treat ipi_should_be_nmi() as a hint
instead. Whether an IPI is actually registered as an NMI is determined
by the success or failure of request_percpu_nmi(). If the request
fails, fall back to request_irq().

Track which IPIs were actually registered as NMIs and use that state
in the per-CPU enable and disable paths. This avoids calling the
percpu NMI helpers for an IPI that fell back to regular IRQ handling.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/kernel/smp.c | 37 +++++++++++++++++++++++++++----------
 1 file changed, 27 insertions(+), 10 deletions(-)

diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 46cdcc114997..dec075bfbf3b 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -71,11 +71,14 @@ static int nr_ipi __ro_after_init = NR_IPI;
 
 struct ipi_descs {
 	struct irq_desc *descs[MAX_IPI];
+	DECLARE_BITMAP(nmi_bitmap, MAX_IPI);
 };
 
 static DEFINE_PER_CPU_READ_MOSTLY(struct ipi_descs, pcpu_ipi_desc);
 
 #define get_ipi_desc(__cpu, __ipi) (per_cpu_ptr(&pcpu_ipi_desc, __cpu)->descs[__ipi])
+#define get_ipi_nmi_bitmap(__cpu) (per_cpu_ptr(&pcpu_ipi_desc, __cpu)->nmi_bitmap)
+#define ipi_is_nmi(__cpu, __ipi) test_bit(__ipi, get_ipi_nmi_bitmap(__cpu))
 
 struct ipi_irq_ops {
 	void (*setup)(int ipi, int ncpus);
@@ -967,8 +970,8 @@ void arch_trigger_cpumask_backtrace(const cpumask_t *mask, int exclude_cpu)
 	/*
 	 * NOTE: though nmi_trigger_cpumask_backtrace() has "nmi_" in the name,
 	 * nothing about it truly needs to be implemented using an NMI, it's
-	 * just that it's _allowed_ to work with NMIs. If ipi_should_be_nmi()
-	 * returned false our backtrace attempt will just use a regular IPI.
+	 * just that it's _allowed_ to work with NMIs. If we failed request
+	 * NMI our backtrace attempt will just use a regular IPI.
 	 */
 	nmi_trigger_cpumask_backtrace(mask, exclude_cpu, arm64_backtrace_ipi);
 }
@@ -1101,6 +1104,7 @@ static void ipi_teardown(int cpu)
 
 static void ipi_sgi_setup(int ipi, int ncpus)
 {
+	bool is_nmi = ipi_should_be_nmi(ipi);
 	int err, irq, cpu;
 
 	if (WARN_ON_ONCE(ncpus))
@@ -1108,16 +1112,29 @@ static void ipi_sgi_setup(int ipi, int ncpus)
 
 	irq = ipi_irq_base + ipi;
 
-	if (ipi_should_be_nmi(ipi)) {
-		err = request_percpu_nmi(irq, ipi_handler, "IPI", NULL, &irq_stat);
-		WARN(err, "Could not request IRQ %d as NMI, err=%d\n", irq, err);
-	} else {
+	/*
+	 * Try to request NMI if it should be
+	 */
+	if (is_nmi && request_percpu_nmi(irq, ipi_handler, "IPI", NULL, &irq_stat)) {
+		/*
+		 * Fallback to regular IRQ
+		 */
+		is_nmi = false;
+	}
+
+       /*
+        * Otherwise regular IRQ
+        */
+	if (!is_nmi) {
 		err = request_percpu_irq(irq, ipi_handler, "IPI", &irq_stat);
 		WARN(err, "Could not request IRQ %d as IRQ, err=%d\n", irq, err);
 	}
 
-	for_each_possible_cpu(cpu)
+	for_each_possible_cpu(cpu) {
 		get_ipi_desc(cpu, ipi) = irq_to_desc(irq);
+		if (is_nmi)
+			set_bit(ipi, get_ipi_nmi_bitmap(cpu));
+	}
 
 	irq_set_status_flags(irq, IRQ_HIDDEN);
 }
@@ -1126,7 +1143,7 @@ static void ipi_sgi_enable(int cpu, int ipi)
 {
 	int irq = ipi_irq_base + ipi;
 
-	if (ipi_should_be_nmi(ipi)) {
+	if (ipi_is_nmi(cpu, ipi)) {
 		prepare_percpu_nmi(irq);
 		enable_percpu_nmi(irq, 0);
 	} else {
@@ -1138,7 +1155,7 @@ static void ipi_sgi_disable(int cpu, int ipi)
 {
 	int irq = ipi_irq_base + ipi;
 
-	if (ipi_should_be_nmi(ipi)) {
+	if (ipi_is_nmi(cpu, ipi)) {
 		disable_percpu_nmi(irq);
 		teardown_percpu_nmi(irq);
 	} else {
@@ -1310,7 +1327,7 @@ void smp_send_stop(void)
 	 * If CPUs are still online, try an NMI. There's no excuse for this to
 	 * be slow, so we only give them an extra 10 ms to respond.
 	 */
-	if (num_other_online_cpus() && ipi_should_be_nmi(IPI_CPU_STOP_NMI)) {
+	if (num_other_online_cpus() && ipi_is_nmi(smp_processor_id(), IPI_CPU_STOP_NMI)) {
 		smp_rmb();
 		cpumask_copy(&mask, cpu_online_mask);
 		cpumask_clear_cpu(smp_processor_id(), &mask);
-- 
2.34.1

[PATCH 32/38] arm64: nmi: Add Kconfig for NMI

[Vladimir Murzin]

From: Mark Brown <broonie@kernel.org>

Since NMI handling is in some fairly hot paths we provide a Kconfig
option which allows support to be compiled out when not needed.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/Kconfig | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 2738637ccb92..df5ff59715d1 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2251,6 +2251,24 @@ endmenu # "ARMv8.7 architectural features"
 config AS_HAS_MOPS
 	def_bool $(as-instr,.arch_extension mops)
 
+menu "ARMv8.8 architectural features"
+
+config ARM64_NMI
+	bool "Enable support for Non-maskable Interrupts (NMI)"
+	default y
+	help
+	  Non-maskable interrupts are an architecture and GIC feature
+	  which allow the system to configure some interrupts to be
+	  configured to have superpriority, allowing them to be handled
+	  before other interrupts and masked for shorter periods of time.
+
+	  The feature is detected at runtime, and will remain disabled
+	  if the cpu does not implement the feature. It will also be
+	  disabled if pseudo NMIs are enabled at runtime.
+
+endmenu # "ARMv8.8 architectural features"
+
+
 menu "ARMv8.9 architectural features"
 
 config ARM64_POE
-- 
2.34.1

[PATCH 33/38] irqchip/gic-v3: Rename nmi to pseudo_nmi

[Vladimir Murzin]

From: Lorenzo Pieralisi <lpieralisi@kernel.org>

Currently, the terms "NMI" and "pseudo-NMI" are used
interchangeably. That creates confusion with upcoming FEAT_GICv3_NMI
support. Rename helper functions and variables related to pseudo-NMI
to make the code paths clearer and more explicit.

Signed-off-by: Lorenzo Pieralisi <lpieralisi@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 drivers/irqchip/irq-gic-v3.c | 28 ++++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 6e1fa5b247fc..6a807c226425 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -87,7 +87,7 @@ static DEFINE_STATIC_KEY_TRUE(supports_deactivate_key);
 #define GIC_LINE_NR	min(GICD_TYPER_SPIS(gic_data.rdists.gicd_typer), 1020U)
 #define GIC_ESPI_NR	GICD_TYPER_ESPIS(gic_data.rdists.gicd_typer)
 
-static bool nmi_support_forbidden;
+static bool pseudo_nmi_support_forbidden;
 
 /*
  * There are 16 SGIs, though we only actually use 8 in Linux. The other 8 SGIs
@@ -117,7 +117,7 @@ static bool nmi_support_forbidden;
  * - Figure 4-7 Secure read of the priority field for a Non-secure Group 1
  *   interrupt.
  */
-static DEFINE_STATIC_KEY_FALSE(supports_pseudo_nmis);
+static DEFINE_STATIC_KEY_FALSE(supports_pseudo_nmi);
 
 static u32 gic_get_pribits(void)
 {
@@ -185,7 +185,7 @@ static void __init gic_prio_init(void)
 				pr_warn("Broken GIC integration, security disabled\n");
 		} else {
 			pr_warn("Broken GIC integration, pNMI forbidden\n");
-			nmi_support_forbidden = true;
+			pseudo_nmi_support_forbidden = true;
 		}
 	}
 
@@ -512,10 +512,10 @@ static void gic_unmask_irq(struct irq_data *d)
 	gic_poke_irq(d, GICD_ISENABLER);
 }
 
-static inline bool gic_supports_nmi(void)
+static inline bool gic_supports_pseudo_nmi(void)
 {
 	return IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) &&
-	       static_branch_likely(&supports_pseudo_nmis);
+	       static_branch_likely(&supports_pseudo_nmi);
 }
 
 static int gic_irq_set_irqchip_state(struct irq_data *d,
@@ -598,7 +598,7 @@ static int gic_irq_nmi_setup(struct irq_data *d)
 {
 	struct irq_desc *desc = irq_to_desc(d->irq);
 
-	if (!gic_supports_nmi())
+	if (!gic_supports_pseudo_nmi())
 		return -EINVAL;
 
 	if (gic_peek_irq(d, GICD_ISENABLER)) {
@@ -626,7 +626,7 @@ static void gic_irq_nmi_teardown(struct irq_data *d)
 {
 	struct irq_desc *desc = irq_to_desc(d->irq);
 
-	if (WARN_ON(!gic_supports_nmi()))
+	if (WARN_ON(!gic_supports_pseudo_nmi()))
 		return;
 
 	if (gic_peek_irq(d, GICD_ISENABLER)) {
@@ -804,7 +804,7 @@ static inline void gic_complete_ack(u32 irqnr)
 
 static bool gic_rpr_is_nmi_prio(void)
 {
-	if (!gic_supports_nmi())
+	if (!gic_supports_pseudo_nmi())
 		return false;
 
 	return unlikely(gic_read_rpr() == GICV3_PRIO_NMI);
@@ -911,7 +911,7 @@ static void __gic_handle_irq_from_irqsoff(struct pt_regs *regs)
 
 static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
 {
-	if (unlikely(gic_supports_nmi() && !interrupts_enabled(regs)))
+	if (unlikely(gic_supports_pseudo_nmi() && !interrupts_enabled(regs)))
 		__gic_handle_irq_from_irqsoff(regs);
 	else
 		__gic_handle_irq_from_irqson(regs);
@@ -1162,7 +1162,7 @@ static void gic_cpu_sys_reg_init(void)
 	/* Set priority mask register */
 	if (!gic_prio_masking_enabled()) {
 		write_gicreg(DEFAULT_PMR_VALUE, ICC_PMR_EL1);
-	} else if (gic_supports_nmi()) {
+	} else if (gic_supports_pseudo_nmi()) {
 		/*
 		 * Check that all CPUs use the same priority space.
 		 *
@@ -1955,15 +1955,15 @@ static const struct gic_quirk gic_quirks[] = {
 	}
 };
 
-static void gic_enable_nmi_support(void)
+static void gic_enable_pseudo_nmi(void)
 {
-	if (!gic_prio_masking_enabled() || nmi_support_forbidden)
+	if (!gic_prio_masking_enabled() || pseudo_nmi_support_forbidden)
 		return;
 
 	pr_info("Pseudo-NMIs enabled using %s ICC_PMR_EL1 synchronisation\n",
 		gic_has_relaxed_pmr_sync() ? "relaxed" : "forced");
 
-	static_branch_enable(&supports_pseudo_nmis);
+	static_branch_enable(&supports_pseudo_nmi);
 
 	if (static_branch_likely(&supports_deactivate_key))
 		gic_eoimode1_chip.flags |= IRQCHIP_SUPPORTS_NMI;
@@ -2047,7 +2047,7 @@ static int __init gic_init_bases(phys_addr_t dist_phys_base,
 	gic_prio_init();
 	gic_dist_init();
 	gic_cpu_init();
-	gic_enable_nmi_support();
+	gic_enable_pseudo_nmi();
 	gic_smp_init();
 	gic_cpu_pm_init();
 
-- 
2.34.1

[PATCH 34/38] irqchip/gic-v3: Implement FEAT_GICv3_NMI support

[Vladimir Murzin]

From: Lorenzo Pieralisi <lpieralisi@kernel.org>

The FEAT_GICv3_NMI GIC feature coupled with the CPU FEAT_NMI enables
handling NMI interrupts in HW on aarch64, by adding a superpriority
interrupt to the existing GIC priority scheme.

Implement GIC driver support for the FEAT_GICv3_NMI feature.

Check, through the ARM64 capabilitity infrastructure, if support
for FEAT_NMI was detected on the core and the system has not overridden
the detection and forced pseudo-NMIs enablement.

If FEAT_NMI is detected, it was not overridden (check embedded in the
system_uses_nmi() call) and the GIC supports the FEAT_GICv3_NMI
feature, initialize NMIs related HW GIC registers and route irq to NMI
handling logic.

Signed-off-by: Lorenzo Pieralisi <lpieralisi@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 drivers/irqchip/irq-gic-v3.c       | 114 +++++++++++++++++++++++++++--
 include/linux/irqchip/arm-gic-v3.h |   4 +
 2 files changed, 111 insertions(+), 7 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 6a807c226425..e2f2e5f29818 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -64,6 +64,7 @@ struct gic_chip_data {
 	u32			nr_redist_regions;
 	u64			flags;
 	bool			has_rss;
+	bool			has_nmi;
 	unsigned int		ppi_nr;
 	struct partition_affinity *parts;
 	unsigned int		nr_parts;
@@ -89,6 +90,9 @@ static DEFINE_STATIC_KEY_TRUE(supports_deactivate_key);
 
 static bool pseudo_nmi_support_forbidden;
 
+static DEFINE_RAW_SPINLOCK(gic_nmi_lock);
+
+
 /*
  * There are 16 SGIs, though we only actually use 8 in Linux. The other 8 SGIs
  * are potentially stolen by the secure side. Some code, especially code dealing
@@ -252,6 +256,37 @@ enum gic_intid_range {
 	__INVALID_RANGE__
 };
 
+#ifdef CONFIG_ARM64
+#include <asm/cpufeature.h>
+
+static inline bool gic_supports_v3_3_nmi(void)
+{
+	return gic_data.has_nmi && system_uses_nmi();
+}
+
+static inline u64 gic_read_nmiar(void)
+{
+	u64 irqstat;
+
+	irqstat = read_sysreg_s(SYS_ICC_NMIAR1_EL1);
+
+	dsb(sy);
+
+	return irqstat;
+}
+
+#else
+static inline bool gic_supports_v3_3_nmi(void)
+{
+	return false;
+}
+
+static inline u64 gic_read_nmiar(void)
+{
+	BUG();
+}
+#endif
+
 static enum gic_intid_range __get_intid_range(irq_hw_number_t hwirq)
 {
 	switch (hwirq) {
@@ -438,6 +473,8 @@ static u32 convert_offset_index(struct irq_data *d, u32 offset, u32 *index)
 			return GICD_ICFGRnE;
 		case GICD_IROUTER:
 			return GICD_IROUTERnE;
+		case GICD_INMIR:
+			return GICD_INMIRnE;
 		default:
 			break;
 		}
@@ -467,6 +504,42 @@ static int gic_peek_irq(struct irq_data *d, u32 offset)
 	return !!(readl_relaxed(base + offset + (index / 32) * 4) & mask);
 }
 
+static void gic_irq_configure_nmi(struct irq_data *d, bool enable)
+{
+	void __iomem *base, *addr;
+	unsigned long val;
+	u32 offset, index;
+
+	offset = convert_offset_index(d, GICD_INMIR, &index);
+
+	if (gic_irq_in_rdist(d))
+		base = gic_data_rdist_sgi_base();
+	else
+		base = gic_dist_base_alias(d);
+
+	addr = base + offset + (index / 32) * 4;
+
+	raw_spin_lock(&gic_nmi_lock);
+
+	val = readl_relaxed(addr);
+
+	__assign_bit(index % 32, &val, enable);
+
+	writel_relaxed(val, addr);
+
+	raw_spin_unlock(&gic_nmi_lock);
+}
+
+static void gic_irq_enable_nmi(struct irq_data *d)
+{
+	gic_irq_configure_nmi(d, true);
+}
+
+static void gic_irq_disable_nmi(struct irq_data *d)
+{
+	gic_irq_configure_nmi(d, false);
+}
+
 static void gic_poke_irq(struct irq_data *d, u32 offset)
 {
 	void __iomem *base;
@@ -598,7 +671,7 @@ static int gic_irq_nmi_setup(struct irq_data *d)
 {
 	struct irq_desc *desc = irq_to_desc(d->irq);
 
-	if (!gic_supports_pseudo_nmi())
+	if (!gic_supports_pseudo_nmi() && !gic_supports_v3_3_nmi())
 		return -EINVAL;
 
 	if (gic_peek_irq(d, GICD_ISENABLER)) {
@@ -617,7 +690,10 @@ static int gic_irq_nmi_setup(struct irq_data *d)
 	if (!gic_irq_in_rdist(d))
 		desc->handle_irq = handle_fasteoi_nmi;
 
-	gic_irq_set_prio(d, dist_prio_nmi);
+	if (gic_supports_v3_3_nmi())
+		gic_irq_enable_nmi(d);
+	else
+		gic_irq_set_prio(d, dist_prio_nmi);
 
 	return 0;
 }
@@ -626,7 +702,7 @@ static void gic_irq_nmi_teardown(struct irq_data *d)
 {
 	struct irq_desc *desc = irq_to_desc(d->irq);
 
-	if (WARN_ON(!gic_supports_pseudo_nmi()))
+	if (WARN_ON(!gic_supports_pseudo_nmi() && !gic_supports_v3_3_nmi()))
 		return;
 
 	if (gic_peek_irq(d, GICD_ISENABLER)) {
@@ -645,7 +721,10 @@ static void gic_irq_nmi_teardown(struct irq_data *d)
 	if (!gic_irq_in_rdist(d))
 		desc->handle_irq = handle_fasteoi_irq;
 
-	gic_irq_set_prio(d, dist_prio_irq);
+	if (gic_supports_v3_3_nmi())
+		gic_irq_disable_nmi(d);
+	else
+		gic_irq_set_prio(d, dist_prio_irq);
 }
 
 static bool gic_arm64_erratum_2941627_needed(struct irq_data *d)
@@ -836,7 +915,8 @@ static void __gic_handle_nmi(u32 irqnr, struct pt_regs *regs)
 	gic_complete_ack(irqnr);
 
 	if (generic_handle_domain_nmi(gic_data.domain, irqnr)) {
-		WARN_ONCE(true, "Unexpected pseudo-NMI (irqnr %u)\n", irqnr);
+		WARN_ONCE(true, "Unexpected %sNMI (irqnr %u)\n",
+			  gic_supports_pseudo_nmi() ? "pseudo-" : "", irqnr);
 		gic_deactivate_unhandled(irqnr);
 	}
 }
@@ -911,7 +991,11 @@ static void __gic_handle_irq_from_irqsoff(struct pt_regs *regs)
 
 static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
 {
-	if (unlikely(gic_supports_pseudo_nmi() && !interrupts_enabled(regs)))
+	if (gic_supports_v3_3_nmi() && in_nmi()) {
+		u32 irqnr = gic_read_nmiar();
+
+		__gic_handle_nmi(irqnr, regs);
+	} else if (unlikely(gic_supports_pseudo_nmi() && !interrupts_enabled(regs)))
 		__gic_handle_irq_from_irqsoff(regs);
 	else
 		__gic_handle_irq_from_irqson(regs);
@@ -1964,6 +2048,19 @@ static void gic_enable_pseudo_nmi(void)
 		gic_has_relaxed_pmr_sync() ? "relaxed" : "forced");
 
 	static_branch_enable(&supports_pseudo_nmi);
+}
+
+static void gic_enable_nmi_support(void)
+{
+	if ((!gic_prio_masking_enabled() || pseudo_nmi_support_forbidden) && !gic_supports_v3_3_nmi())
+		return;
+
+	/*
+	 * Initialize pseudo-NMIs only if GIC driver cannot take advantage
+	 * of core (FEAT_NMI) and GIC (FEAT_GICv3_NMI) in HW
+	 */
+	if (!gic_supports_v3_3_nmi())
+		gic_enable_pseudo_nmi();
 
 	if (static_branch_likely(&supports_deactivate_key))
 		gic_eoimode1_chip.flags |= IRQCHIP_SUPPORTS_NMI;
@@ -2032,6 +2129,9 @@ static int __init gic_init_bases(phys_addr_t dist_phys_base,
 	irq_domain_update_bus_token(gic_data.domain, DOMAIN_BUS_WIRED);
 
 	gic_data.has_rss = !!(typer & GICD_TYPER_RSS);
+	gic_data.has_nmi = !!(typer & GICD_TYPER_NMI);
+
+	pr_info("Non-maskable interrupt property %s supported\n", gic_data.has_nmi ? "is" : "not");
 
 	if (typer & GICD_TYPER_MBIS) {
 		err = mbi_init(handle, gic_data.domain);
@@ -2047,7 +2147,7 @@ static int __init gic_init_bases(phys_addr_t dist_phys_base,
 	gic_prio_init();
 	gic_dist_init();
 	gic_cpu_init();
-	gic_enable_pseudo_nmi();
+	gic_enable_nmi_support();
 	gic_smp_init();
 	gic_cpu_pm_init();
 
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index ea5fd2374ebe..ebccd76cadda 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -30,6 +30,7 @@
 #define GICD_ICFGR			0x0C00
 #define GICD_IGRPMODR			0x0D00
 #define GICD_NSACR			0x0E00
+#define GICD_INMIR			0x0F80
 #define GICD_IGROUPRnE			0x1000
 #define GICD_ISENABLERnE		0x1200
 #define GICD_ICENABLERnE		0x1400
@@ -39,6 +40,7 @@
 #define GICD_ICACTIVERnE		0x1C00
 #define GICD_IPRIORITYRnE		0x2000
 #define GICD_ICFGRnE			0x3000
+#define GICD_INMIRnE			0x3B00
 #define GICD_IROUTER			0x6000
 #define GICD_IROUTERnE			0x8000
 #define GICD_IDREGS			0xFFD0
@@ -83,6 +85,7 @@
 #define GICD_TYPER_LPIS			(1U << 17)
 #define GICD_TYPER_MBIS			(1U << 16)
 #define GICD_TYPER_ESPI			(1U << 8)
+#define GICD_TYPER_NMI			(1U << 9)
 
 #define GICD_TYPER_ID_BITS(typer)	((((typer) >> 19) & 0x1f) + 1)
 #define GICD_TYPER_NUM_LPIS(typer)	((((typer) >> 11) & 0x1f) + 1)
@@ -238,6 +241,7 @@
 #define GICR_ICFGR0			GICD_ICFGR
 #define GICR_IGRPMODR0			GICD_IGRPMODR
 #define GICR_NSACR			GICD_NSACR
+#define GICR_INMIR0			GICD_INMIR
 
 #define GICR_TYPER_PLPIS		(1U << 0)
 #define GICR_TYPER_VLPIS		(1U << 1)
-- 
2.34.1

[PATCH 35/38] arm64: smp: Add NMI support for LPI-backed IPIs

[Vladimir Murzin]

LPI-backed IPIs are currently always requested and managed as regular
IRQs. Allow IPIs selected by ipi_should_be_nmi() to be requested as
NMIs, falling back to regular IRQs when the NMI request fails.

Track which IPIs were successfully requested as NMIs so that the
per-CPU enable and disable paths use the appropriate NMI helpers.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/kernel/smp.c | 38 +++++++++++++++++++++++++++++++++-----
 1 file changed, 33 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index dec075bfbf3b..151cc326dad9 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -1177,16 +1177,28 @@ static const struct ipi_irq_ops ipi_sgi_ops = {
 
 static void ipi_lpi_enable(int cpu, int ipi)
 {
-	enable_irq(irq_desc_get_irq(get_ipi_desc(cpu, ipi)));
+	int irq = irq_desc_get_irq(get_ipi_desc(cpu, ipi));
+
+	if (ipi_is_nmi(cpu, ipi))
+		enable_nmi(irq);
+	else
+		enable_irq(irq);
 }
 
 static void ipi_lpi_disable(int cpu, int ipi)
 {
-	disable_irq_nosync(irq_desc_get_irq(get_ipi_desc(cpu, ipi)));
+	int irq = irq_desc_get_irq(get_ipi_desc(cpu, ipi));
+
+	if (ipi_is_nmi(cpu, ipi))
+		disable_nmi_nosync(irq);
+	else
+		disable_irq_nosync(irq);
 }
 
 static void ipi_lpi_setup(int ipi, int ncpus)
 {
+	bool is_nmi = ipi_should_be_nmi(ipi);
+
 	for (int cpu = 0; cpu < ncpus; cpu++) {
 		int err, irq;
 
@@ -1195,13 +1207,29 @@ static void ipi_lpi_setup(int ipi, int ncpus)
 		err = irq_force_affinity(irq, cpumask_of(cpu));
 		WARN(err, "Could not force affinity IRQ %d, err=%d\n", irq, err);
 
-		err = request_irq(irq, ipi_handler, IRQF_NO_AUTOEN, "IPI",
-				  NULL);
-		WARN(err, "Could not request IRQ %d, err=%d\n", irq, err);
+		/*
+		 * Try to request NMI if it should be
+		 */
+		if (is_nmi && request_nmi(irq, ipi_handler, IRQF_PERCPU | IRQF_NO_AUTOEN, "IPI", NULL)) {
+			/*
+			 * Fallback to regular IRQ
+			 */
+			is_nmi = false;
+		}
+		/*
+		 * Otherwise regular IRQ
+		 */
+		if (!is_nmi) {
+			err = request_irq(irq, ipi_handler, IRQF_NO_AUTOEN, "IPI", NULL);
+			WARN(err, "Could not request IRQ %d, err=%d\n", irq, err);
+		}
 
 		irq_set_status_flags(irq, (IRQ_HIDDEN | IRQ_NO_BALANCING_MASK));
 
 		get_ipi_desc(cpu, ipi) = irq_to_desc(irq);
+
+		if (is_nmi)
+			set_bit(ipi, get_ipi_nmi_bitmap(cpu));
 	}
 }
 
-- 
2.34.1

[PATCH 36/38] genirq: Trust local irqchip when considering IRQ for NMI

[Vladimir Murzin]

Generic IRQ code sanitises whether an IRQ can be considered for NMI
use. Currently, it only considers IRQs managed by the root irqchip.

GICv5 IPIs belong to a logical domain built on top of the LPI domain.
Consequently, generic IRQ code rejects these IPIs for NMI use, even
though their hardware state is managed by the root LPI domain.

One way to allow GICv5 IPIs to be used as NMIs would be to adjust the
sanitisation logic to account for this domain hierarchy. However, the
generic IRQ code can instead trust the local irqchip.

Remove the root irqchip check and use the local irqchip when
considering an IRQ for NMI use.

Suggested-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 include/linux/irq.h | 2 +-
 kernel/irq/manage.c | 5 -----
 2 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/include/linux/irq.h b/include/linux/irq.h
index f485369b1b4f..92d3ad639ae8 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -563,7 +563,7 @@ struct irq_chip {
  * IRQCHIP_ONESHOT_SAFE:              One shot does not require mask/unmask
  * IRQCHIP_EOI_THREADED:              Chip requires eoi() on unmask in threaded mode
  * IRQCHIP_SUPPORTS_LEVEL_MSI:        Chip can provide two doorbells for Level MSIs
- * IRQCHIP_SUPPORTS_NMI:              Chip can deliver NMIs, only for root irqchips
+ * IRQCHIP_SUPPORTS_NMI:              Chip can deliver NMIs
  * IRQCHIP_ENABLE_WAKEUP_ON_SUSPEND:  Invokes __enable_irq()/__disable_irq() for wake irqs
  *                                    in the suspend path if they are in disabled state
  * IRQCHIP_AFFINITY_PRE_STARTUP:      Default affinity update before startup
diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index 57eff26fa646..c9ad79743ef6 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -1370,11 +1370,6 @@ static bool irq_supports_nmi(struct irq_desc *desc)
 {
 	struct irq_data *d = irq_desc_get_irq_data(desc);
 
-#ifdef CONFIG_IRQ_DOMAIN_HIERARCHY
-	/* Only IRQs directly managed by the root irqchip can be set as NMI */
-	if (d->parent_data)
-		return false;
-#endif
 	/* Don't support NMIs for chips behind a slow bus */
 	if (d->chip->irq_bus_lock || d->chip->irq_bus_sync_unlock)
 		return false;
-- 
2.34.1

[PATCH 37/38] genirq: Do not call add_interrupt_randomness() in NMI context

[Vladimir Murzin]

handle_irq_event_percpu() calls add_interrupt_randomness(). Similarly
to 512718bbc51b ("genirq/chip: Don't call add_interrupt_randomness()
for NMIs"), calling it while handling an NMI is not safe.

Stop calling add_interrupt_randomness() when handle_irq_event_percpu()
runs in NMI context.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 kernel/irq/handle.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
index b7d52821837b..114fd63b0210 100644
--- a/kernel/irq/handle.c
+++ b/kernel/irq/handle.c
@@ -245,7 +245,8 @@ irqreturn_t handle_irq_event_percpu(struct irq_desc *desc)
 
 	retval = __handle_irq_event_percpu(desc);
 
-	add_interrupt_randomness(desc->irq_data.irq);
+	if (!in_nmi())
+		add_interrupt_randomness(desc->irq_data.irq);
 
 	if (!irq_settings_no_debug(desc))
 		note_interrupt(desc, retval);
-- 
2.34.1

[PATCH 38/38] irqchip/gic-v5: Add NMI support

[Vladimir Murzin]

Implement NMI support for PPIs, SPIs, LPIs and IPIs by toggling their
priority between the superpriority and regular interrupt levels.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 drivers/irqchip/irq-gic-v5.c | 244 +++++++++++++++++++++++++++++++++--
 1 file changed, 234 insertions(+), 10 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v5.c b/drivers/irqchip/irq-gic-v5.c
index 5f2551cf077d..7e7d313e77bc 100644
--- a/drivers/irqchip/irq-gic-v5.c
+++ b/drivers/irqchip/irq-gic-v5.c
@@ -22,6 +22,7 @@
 static u8 pri_bits __ro_after_init = 5;
 
 #define GICV5_IRQ_PRI_MASK	0x1f
+#define GICV5_IRQ_PRI_NMI	0
 #define GICV5_IRQ_PRI_MI	(GICV5_IRQ_PRI_MASK & GENMASK(4, 5 - pri_bits))
 
 #define PPI_NR	128
@@ -97,6 +98,56 @@ static void gicv5_hwirq_init(irq_hw_number_t hwirq, u8 priority, u8 hwirq_type)
 	}
 }
 
+static void gicv5_ppi_irq_set_priority(struct irq_data *d, u8 priority)
+{
+	u64 hwirq_id_offset = d->hwirq / 8;
+	u64 hwirq_id_index = d->hwirq % 8;
+	u64 hwirq_id_clear = (u64)GICV5_IRQ_PRI_MASK << (hwirq_id_index * 8);
+	u64 hwirq_id_set = (u64)priority << (hwirq_id_index * 8);
+
+#define SET_ICC_PPI_PRIORITYRn_EL1(cnt)	\
+	case cnt: sysreg_clear_set_s(SYS_ICC_PPI_PRIORITYR ## cnt ## _EL1, hwirq_id_clear, hwirq_id_set); break
+
+	switch (hwirq_id_offset) {
+	SET_ICC_PPI_PRIORITYRn_EL1(0);
+	SET_ICC_PPI_PRIORITYRn_EL1(1);
+	SET_ICC_PPI_PRIORITYRn_EL1(2);
+	SET_ICC_PPI_PRIORITYRn_EL1(3);
+	SET_ICC_PPI_PRIORITYRn_EL1(4);
+	SET_ICC_PPI_PRIORITYRn_EL1(5);
+	SET_ICC_PPI_PRIORITYRn_EL1(6);
+	SET_ICC_PPI_PRIORITYRn_EL1(7);
+	SET_ICC_PPI_PRIORITYRn_EL1(8);
+	SET_ICC_PPI_PRIORITYRn_EL1(9);
+	SET_ICC_PPI_PRIORITYRn_EL1(10);
+	SET_ICC_PPI_PRIORITYRn_EL1(11);
+	SET_ICC_PPI_PRIORITYRn_EL1(12);
+	SET_ICC_PPI_PRIORITYRn_EL1(13);
+	SET_ICC_PPI_PRIORITYRn_EL1(14);
+	SET_ICC_PPI_PRIORITYRn_EL1(15);
+#undef SET_ICC_PPI_PRIORITYRn_EL1
+	default:
+		WARN_ON(1);
+	}
+	/*
+	 * Context syncronization required to make sure system register writes
+	 * effects are synchronised.
+	 */
+	isb();
+}
+
+static void gicv5_hwirq_set_priority(irq_hw_number_t hwirq, u8 priority, u8 hwirq_type) {
+	u64 cdpri;
+
+	cdpri = FIELD_PREP(GICV5_GIC_CDPRI_PRIORITY_MASK, priority)	|
+		FIELD_PREP(GICV5_GIC_CDPRI_TYPE_MASK, hwirq_type)	|
+		FIELD_PREP(GICV5_GIC_CDPRI_ID_MASK, hwirq);
+
+	gic_insn(cdpri, CDPRI);
+
+	return;
+}
+
 static void gicv5_ppi_irq_mask(struct irq_data *d)
 {
 	u64 hwirq_id_bit = BIT_ULL(d->hwirq % 64);
@@ -512,7 +563,152 @@ static int gicv5_ppi_irq_set_vcpu_affinity(struct irq_data *d, void *vcpu)
 	return 0;
 }
 
-static const struct irq_chip gicv5_ppi_irq_chip = {
+static int gicv5_hwirq_peek_irq(irq_hw_number_t hwirq, u8 hwirq_type)
+{
+	u64 icsr, cdrcfg;
+
+	cdrcfg = hwirq | FIELD_PREP(GICV5_GIC_CDRCFG_TYPE_MASK, hwirq_type);
+
+	gic_insn(cdrcfg, CDRCFG);
+	isb();
+	icsr = read_sysreg_s(SYS_ICC_ICSR_EL1);
+
+	if (FIELD_GET(ICC_ICSR_EL1_F, icsr)) {
+		pr_err("ICSR_EL1 is invalid\n");
+		return -EINVAL;
+	}
+
+	return !!(FIELD_GET(ICC_ICSR_EL1_Enabled, icsr));
+
+}
+
+static int gicv5_ppi_irq_nmi_setup(struct irq_data *d)
+{
+	unsigned long hwirq_id_bit = d->hwirq % 64;
+	unsigned long enable;
+
+	if (!system_uses_nmi())
+		return -EINVAL;
+
+	if (d->hwirq < 64)
+		enable = read_sysreg_s(SYS_ICC_PPI_ENABLER0_EL1);
+	else
+		enable = read_sysreg_s(SYS_ICC_PPI_ENABLER1_EL1);
+
+	if (test_bit(hwirq_id_bit, &enable))
+		return -EINVAL;
+
+	gicv5_ppi_irq_set_priority(d, GICV5_IRQ_PRI_NMI);
+
+	return 0;
+}
+
+static void gicv5_ppi_irq_nmi_teardown(struct irq_data *d)
+{
+	unsigned long hwirq_id_bit = d->hwirq % 64;
+	unsigned long enable;
+
+	if (WARN_ON(!system_uses_nmi()))
+		return;
+
+	if (d->hwirq < 64)
+		enable = read_sysreg_s(SYS_ICC_PPI_ENABLER0_EL1);
+	else
+		enable = read_sysreg_s(SYS_ICC_PPI_ENABLER1_EL1);
+
+	if (test_bit(hwirq_id_bit, &enable))
+		return;
+
+	gicv5_ppi_irq_set_priority(d, GICV5_IRQ_PRI_MI);
+
+	return;
+}
+
+static int gicv5_hwirq_irq_nmi_setup(irq_hw_number_t hwirq, u8 hwirq_type)
+{
+	if (!system_uses_nmi())
+		return -EINVAL;
+
+	if (WARN_ON(hwirq_type != GICV5_HWIRQ_TYPE_LPI && hwirq_type != GICV5_HWIRQ_TYPE_SPI))
+		return -EINVAL;
+
+	if (gicv5_hwirq_peek_irq(hwirq, hwirq_type))
+		return -EINVAL;
+
+	gicv5_hwirq_set_priority(hwirq, GICV5_IRQ_PRI_NMI, hwirq_type);
+
+	return 0;
+}
+
+static void gicv5_hwirq_irq_nmi_teardown(irq_hw_number_t hwirq, u8 hwirq_type)
+{
+	if (WARN_ON(!system_uses_nmi()))
+		return;
+
+	if (WARN_ON(hwirq_type != GICV5_HWIRQ_TYPE_LPI && hwirq_type != GICV5_HWIRQ_TYPE_SPI))
+		return;
+
+	if (gicv5_hwirq_peek_irq(hwirq, hwirq_type))
+		return;
+
+	gicv5_hwirq_set_priority(hwirq, GICV5_IRQ_PRI_MI, hwirq_type);
+
+	return;
+}
+
+static int gicv5_lpi_irq_nmi_setup(struct irq_data *d)
+{
+	int ret = gicv5_hwirq_irq_nmi_setup(d->hwirq, GICV5_HWIRQ_TYPE_LPI);
+
+	if (!ret)
+		irq_to_desc(d->irq)->handle_irq = handle_fasteoi_nmi;
+
+	return ret;
+}
+
+static void gicv5_lpi_irq_nmi_teardown(struct irq_data *d)
+{
+	gicv5_hwirq_irq_nmi_teardown(d->hwirq, GICV5_HWIRQ_TYPE_LPI);
+	irq_to_desc(d->irq)->handle_irq = handle_fasteoi_irq;
+}
+
+static int gicv5_spi_irq_nmi_setup(struct irq_data *d)
+{
+	int ret = gicv5_hwirq_irq_nmi_setup(d->hwirq, GICV5_HWIRQ_TYPE_SPI);
+
+	if (!ret)
+		irq_to_desc(d->irq)->handle_irq = handle_fasteoi_nmi;
+
+	return ret;
+}
+
+static void gicv5_spi_irq_nmi_teardown(struct irq_data *d)
+{
+	gicv5_hwirq_irq_nmi_teardown(d->hwirq, GICV5_HWIRQ_TYPE_SPI);
+	irq_to_desc(d->irq)->handle_irq = handle_fasteoi_irq;
+}
+
+static int gicv5_ipi_irq_nmi_setup(struct irq_data *d)
+{
+	if (WARN_ON(!d->parent_data))
+		return -EINVAL;
+
+	d = d->parent_data;
+
+	return gicv5_hwirq_irq_nmi_setup(d->hwirq, GICV5_HWIRQ_TYPE_LPI);
+}
+
+static void gicv5_ipi_irq_nmi_teardown(struct irq_data *d)
+{
+	if (WARN_ON(!d->parent_data))
+		return;
+
+	d = d->parent_data;
+
+	gicv5_hwirq_irq_nmi_teardown(d->hwirq, GICV5_HWIRQ_TYPE_LPI);
+}
+
+static struct irq_chip gicv5_ppi_irq_chip __ro_after_init = {
 	.name			= "GICv5-PPI",
 	.irq_mask		= gicv5_ppi_irq_mask,
 	.irq_unmask		= gicv5_ppi_irq_unmask,
@@ -521,11 +717,13 @@ static const struct irq_chip gicv5_ppi_irq_chip = {
 	.irq_get_irqchip_state	= gicv5_ppi_irq_get_irqchip_state,
 	.irq_set_irqchip_state	= gicv5_ppi_irq_set_irqchip_state,
 	.irq_set_vcpu_affinity	= gicv5_ppi_irq_set_vcpu_affinity,
+	.irq_nmi_setup		= gicv5_ppi_irq_nmi_setup,
+	.irq_nmi_teardown	= gicv5_ppi_irq_nmi_teardown,
 	.flags			= IRQCHIP_SKIP_SET_WAKE	  |
 				  IRQCHIP_MASK_ON_SUSPEND,
 };
 
-static const struct irq_chip gicv5_spi_irq_chip = {
+static struct irq_chip gicv5_spi_irq_chip __ro_after_init = {
 	.name			= "GICv5-SPI",
 	.irq_mask		= gicv5_spi_irq_mask,
 	.irq_unmask		= gicv5_spi_irq_unmask,
@@ -535,13 +733,15 @@ static const struct irq_chip gicv5_spi_irq_chip = {
 	.irq_retrigger		= gicv5_spi_irq_retrigger,
 	.irq_get_irqchip_state	= gicv5_spi_irq_get_irqchip_state,
 	.irq_set_irqchip_state	= gicv5_spi_irq_set_irqchip_state,
+	.irq_nmi_setup		= gicv5_spi_irq_nmi_setup,
+	.irq_nmi_teardown	= gicv5_spi_irq_nmi_teardown,
 	.flags			= IRQCHIP_SET_TYPE_MASKED |
 				  IRQCHIP_AFFINITY_PRE_STARTUP |
 				  IRQCHIP_SKIP_SET_WAKE	  |
 				  IRQCHIP_MASK_ON_SUSPEND,
 };
 
-static const struct irq_chip gicv5_lpi_irq_chip = {
+static struct irq_chip gicv5_lpi_irq_chip __ro_after_init = {
 	.name			= "GICv5-LPI",
 	.irq_mask		= gicv5_lpi_irq_mask,
 	.irq_unmask		= gicv5_lpi_irq_unmask,
@@ -550,12 +750,14 @@ static const struct irq_chip gicv5_lpi_irq_chip = {
 	.irq_retrigger		= gicv5_lpi_irq_retrigger,
 	.irq_get_irqchip_state	= gicv5_lpi_irq_get_irqchip_state,
 	.irq_set_irqchip_state	= gicv5_lpi_irq_set_irqchip_state,
+	.irq_nmi_setup		= gicv5_lpi_irq_nmi_setup,
+	.irq_nmi_teardown	= gicv5_lpi_irq_nmi_teardown,
 	.flags			= IRQCHIP_AFFINITY_PRE_STARTUP |
 				  IRQCHIP_SKIP_SET_WAKE	  |
 				  IRQCHIP_MASK_ON_SUSPEND,
 };
 
-static const struct irq_chip gicv5_ipi_irq_chip = {
+static struct irq_chip gicv5_ipi_irq_chip __ro_after_init = {
 	.name			= "GICv5-IPI",
 	.irq_mask		= irq_chip_mask_parent,
 	.irq_unmask		= irq_chip_unmask_parent,
@@ -563,6 +765,8 @@ static const struct irq_chip gicv5_ipi_irq_chip = {
 	.irq_set_affinity	= irq_chip_set_affinity_parent,
 	.irq_get_irqchip_state	= irq_chip_get_parent_state,
 	.irq_set_irqchip_state	= irq_chip_set_parent_state,
+	.irq_nmi_setup		= gicv5_ipi_irq_nmi_setup,
+	.irq_nmi_teardown	= gicv5_ipi_irq_nmi_teardown,
 	.ipi_send_single	= gicv5_ipi_send_single,
 	.flags			= IRQCHIP_SKIP_SET_WAKE	  |
 				  IRQCHIP_MASK_ON_SUSPEND,
@@ -907,11 +1111,12 @@ static const struct irq_domain_ops gicv5_irq_ipi_domain_ops = {
 	.free	= gicv5_irq_ipi_domain_free,
 };
 
-static void handle_irq_per_domain(u32 hwirq)
+static void handle_irq_per_domain(u32 hwirq, bool is_nmi)
 {
 	u8 hwirq_type = FIELD_GET(GICV5_HWIRQ_TYPE, hwirq);
 	u32 hwirq_id = FIELD_GET(GICV5_HWIRQ_ID, hwirq);
 	struct irq_domain *domain;
+	int ret;
 
 	switch (hwirq_type) {
 	case GICV5_HWIRQ_TYPE_PPI:
@@ -928,26 +1133,32 @@ static void handle_irq_per_domain(u32 hwirq)
 		return;
 	}
 
-	if (generic_handle_domain_irq(domain, hwirq_id)) {
-		pr_err_once("Could not handle, hwirq = 0x%x", hwirq_id);
+	if (is_nmi)
+		ret = generic_handle_domain_nmi(domain, hwirq_id);
+	else
+		ret = generic_handle_domain_irq(domain, hwirq_id);
+
+	if (ret) {
+		pr_err_once("Could not handle, hwirq = 0x%x (%d)", hwirq_id, ret);
 		gicv5_hwirq_eoi(hwirq_id, hwirq_type);
 	}
 }
 
 static void __exception_irq_entry gicv5_handle_irq(struct pt_regs *regs)
 {
+	bool is_nmi = in_nmi();
 	bool valid;
 	u32 hwirq;
 	u64 ia;
 
-	ia = gicr_insn(CDIA);
+	ia = is_nmi ? gicr_insn(CDNMIA) : gicr_insn(CDIA);
 	valid = GICV5_GICR_CDIA_VALID(ia);
 
 	if (!valid)
 		return;
 
 	/*
-	 * Ensure that the CDIA instruction effects (ie IRQ activation) are
+	 * Ensure that the CD{NM}IA instruction effects (ie IRQ activation) are
 	 * completed before handling the interrupt.
 	 */
 	gsb_ack();
@@ -967,7 +1178,7 @@ static void __exception_irq_entry gicv5_handle_irq(struct pt_regs *regs)
 
 	hwirq = FIELD_GET(GICV5_HWIRQ_INTID, ia);
 
-	handle_irq_per_domain(hwirq);
+	handle_irq_per_domain(hwirq, is_nmi);
 }
 
 static void gicv5_cpu_disable_interrupts(void)
@@ -1010,6 +1221,17 @@ static int gicv5_starting_cpu(unsigned int cpu)
 	return gicv5_irs_register_cpu(cpu);
 }
 
+static void gicv5_enable_nmi_support(void)
+{
+	if (!system_uses_nmi())
+		return;
+
+	gicv5_ppi_irq_chip.flags |= IRQCHIP_SUPPORTS_NMI;
+	gicv5_spi_irq_chip.flags |= IRQCHIP_SUPPORTS_NMI;
+	gicv5_lpi_irq_chip.flags |= IRQCHIP_SUPPORTS_NMI;
+	gicv5_ipi_irq_chip.flags |= IRQCHIP_SUPPORTS_NMI;
+}
+
 static void __init gicv5_smp_init(void)
 {
 	unsigned int num_ipis = GICV5_IPIS_PER_CPU * nr_cpu_ids;
@@ -1176,6 +1398,8 @@ static int __init gicv5_init_common(struct fwnode_handle *parent_domain)
 	if (ret)
 		goto out_handle;
 
+	gicv5_enable_nmi_support();
+
 	gicv5_smp_init();
 
 	gicv5_irs_its_probe();
-- 
2.34.1

Re: [PATCH 20/38] arm64: probes: Reject stepping over ALLINT

[Jinjie Ruan]

在 2026/9/14 18:20, Vladimir Murzin 写道:
> ALLINT instructions are not safe for probing. Both the register and
> immediate forms of MSR for ALLINT are currently rejected because they
> match aarch64_insn_is_msr_{imm,reg}(). However, MRS instructions
> accessing ALLINT need to be rejected explicitly.
> 
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/insn.h          |  1 +
>  arch/arm64/kernel/probes/decode-insn.c | 11 +++++++----
>  2 files changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
> index 1ce75a442638..0a81b5feb2ed 100644
> --- a/arch/arm64/include/asm/insn.h
> +++ b/arch/arm64/include/asm/insn.h
> @@ -118,6 +118,7 @@ enum aarch64_insn_special_register {
>  	AARCH64_INSN_SPCLREG_SP_EL0	= 0xC208,
>  	AARCH64_INSN_SPCLREG_SPSEL	= 0xC210,
>  	AARCH64_INSN_SPCLREG_CURRENTEL	= 0xC212,
> +	AARCH64_INSN_SPCLREG_ALLINT	= 0xC218,
>  	AARCH64_INSN_SPCLREG_DAIF	= 0xDA11,
>  	AARCH64_INSN_SPCLREG_NZCV	= 0xDA10,
>  	AARCH64_INSN_SPCLREG_FPCR	= 0xDA20,
> diff --git a/arch/arm64/kernel/probes/decode-insn.c b/arch/arm64/kernel/probes/decode-insn.c
> index 4137cc5ef031..a98d9d30fb4a 100644
> --- a/arch/arm64/kernel/probes/decode-insn.c
> +++ b/arch/arm64/kernel/probes/decode-insn.c
> @@ -36,11 +36,14 @@ static bool __kprobes aarch64_insn_is_steppable(u32 insn)
>  		/*
>  		 * The MRS instruction may not return a correct value when
>  		 * executing in the single-stepping environment. We do make one
> -		 * exception, for reading the DAIF bits.
> +		 * exception, for reading the DAIF and ALLINT bits.
>  		 */
> -		if (aarch64_insn_is_mrs(insn))
> -			return aarch64_insn_extract_system_reg(insn)
> -			     != AARCH64_INSN_SPCLREG_DAIF;
> +		if (aarch64_insn_is_mrs(insn)) {
> +			u32 sysreg = aarch64_insn_extract_system_reg(insn);
> +
> +			return (sysreg != AARCH64_INSN_SPCLREG_DAIF) &&
> +			       (sysreg != AARCH64_INSN_SPCLREG_ALLINT);

LGTM
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

> +		}
>  
>  		/*
>  		 * The HINT instruction is steppable only if it is in whitelist

Re: [PATCH 30/38] arm64: smp: Abstract SGI and LPI operations

[Jinjie Ruan]

在 2026/9/14 18:20, Vladimir Murzin 写道:
> SGI and LPI backed IPIs require different setup, enable, disable and
> send operations. These differences are currently handled by repeatedly
> checking percpu_ipi_descs. As the implementation specific logic grows,
> these checks make the common IPI code increasingly difficult to
> follow.
> 
> Introduce an operations structure for each implementation to
> encapsulate the specific of SGI and LPI handling, leaving the common
> IPI paths generic.
> 
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
> ---
>  arch/arm64/kernel/smp.c | 135 ++++++++++++++++++++++++++--------------
>  1 file changed, 88 insertions(+), 47 deletions(-)
> 
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index 3d7060aedd66..46cdcc114997 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -77,11 +77,18 @@ static DEFINE_PER_CPU_READ_MOSTLY(struct ipi_descs, pcpu_ipi_desc);
>  
>  #define get_ipi_desc(__cpu, __ipi) (per_cpu_ptr(&pcpu_ipi_desc, __cpu)->descs[__ipi])
>  
> -static bool percpu_ipi_descs __ro_after_init;
> +struct ipi_irq_ops {
> +	void (*setup)(int ipi, int ncpus);
> +	void (*disable)(int cpu, int ipi);
> +	void (*enable)(int cpu, int ipi);
> +	void (*send)(const cpumask_t *mask, unsigned int nr);
> +};
> +
> +static const struct ipi_irq_ops *ipi_ops __ro_after_init;
>  
>  static bool crash_stop;
>  
> -static void ipi_setup(int cpu);
> +static void ipi_enable(int cpu);
>  
>  #ifdef CONFIG_HOTPLUG_CPU
>  static void ipi_teardown(int cpu);
> @@ -242,7 +249,7 @@ asmlinkage notrace void secondary_start_kernel(void)
>  	 */
>  	notify_cpu_starting(cpu);
>  
> -	ipi_setup(cpu);
> +	ipi_enable(cpu);
>  
>  	numa_add_cpu(cpu);
>  
> @@ -937,13 +944,7 @@ void __noreturn panic_smp_self_stop(void)
>  
>  static void arm64_send_ipi(const cpumask_t *mask, unsigned int nr)
>  {
> -	unsigned int cpu;
> -
> -	if (!percpu_ipi_descs)
> -		__ipi_send_mask(get_ipi_desc(0, nr), mask);
> -	else
> -		for_each_cpu(cpu, mask)
> -			__ipi_send_single(get_ipi_desc(cpu, nr), cpu);
> +	ipi_ops->send(mask, nr);
>  }
>  
>  static void arm64_backtrace_ipi(cpumask_t *mask)
> @@ -1074,54 +1075,37 @@ static bool ipi_should_be_nmi(enum ipi_msg_type ipi)
>  	}
>  }
>  
> -static void ipi_setup(int cpu)
> +static void ipi_enable(int cpu)
>  {
> -	int i;
> +	int ipi;
>  
>  	if (WARN_ON_ONCE(!ipi_irq_base))
>  		return;
>  
> -	for (i = 0; i < nr_ipi; i++) {
> -		if (!percpu_ipi_descs) {
> -			if (ipi_should_be_nmi(i)) {
> -				prepare_percpu_nmi(ipi_irq_base + i);
> -				enable_percpu_nmi(ipi_irq_base + i, 0);
> -			} else {
> -				enable_percpu_irq(ipi_irq_base + i, 0);
> -			}
> -		} else {
> -			enable_irq(irq_desc_get_irq(get_ipi_desc(cpu, i)));
> -		}
> -	}
> +	for (ipi = 0; ipi < nr_ipi; ipi++)
> +		ipi_ops->enable(cpu, ipi);
>  }
>  
>  #ifdef CONFIG_HOTPLUG_CPU
>  static void ipi_teardown(int cpu)
>  {
> -	int i;
> +	int ipi;
>  
>  	if (WARN_ON_ONCE(!ipi_irq_base))
>  		return;
>  
> -	for (i = 0; i < nr_ipi; i++) {
> -		if (!percpu_ipi_descs) {
> -			if (ipi_should_be_nmi(i)) {
> -				disable_percpu_nmi(ipi_irq_base + i);
> -				teardown_percpu_nmi(ipi_irq_base + i);
> -			} else {
> -				disable_percpu_irq(ipi_irq_base + i);
> -			}
> -		} else {
> -			disable_irq_nosync(irq_desc_get_irq(get_ipi_desc(cpu, i)));
> -		}
> -	}
> +	for (ipi = 0; ipi < nr_ipi; ipi++)
> +		ipi_ops->disable(cpu, ipi);
>  }
>  #endif
>  
> -static void ipi_setup_sgi(int ipi)
> +static void ipi_sgi_setup(int ipi, int ncpus)
>  {
>  	int err, irq, cpu;
>  
> +	if (WARN_ON_ONCE(ncpus))
> +		return;
> +
>  	irq = ipi_irq_base + ipi;
>  
>  	if (ipi_should_be_nmi(ipi)) {
> @@ -1138,7 +1122,53 @@ static void ipi_setup_sgi(int ipi)
>  	irq_set_status_flags(irq, IRQ_HIDDEN);
>  }
>  
> -static void ipi_setup_lpi(int ipi, int ncpus)
> +static void ipi_sgi_enable(int cpu, int ipi)
> +{
> +	int irq = ipi_irq_base + ipi;
> +
> +	if (ipi_should_be_nmi(ipi)) {
> +		prepare_percpu_nmi(irq);
> +		enable_percpu_nmi(irq, 0);
> +	} else {
> +		enable_percpu_irq(irq, 0);
> +	}
> +}
> +
> +static void ipi_sgi_disable(int cpu, int ipi)
> +{
> +	int irq = ipi_irq_base + ipi;
> +
> +	if (ipi_should_be_nmi(ipi)) {
> +		disable_percpu_nmi(irq);
> +		teardown_percpu_nmi(irq);
> +	} else {
> +		disable_percpu_irq(irq);
> +	}
> +}
> +
> +static void ipi_sgi_send(const cpumask_t *mask, unsigned int nr)
> +{
> +	__ipi_send_mask(get_ipi_desc(0, nr), mask);
> +}
> +
> +static const struct ipi_irq_ops ipi_sgi_ops = {
> +	.setup = ipi_sgi_setup,
> +	.disable = ipi_sgi_disable,
> +	.enable = ipi_sgi_enable,
> +	.send = ipi_sgi_send,
> +};
> +
> +static void ipi_lpi_enable(int cpu, int ipi)
> +{
> +	enable_irq(irq_desc_get_irq(get_ipi_desc(cpu, ipi)));
> +}
> +
> +static void ipi_lpi_disable(int cpu, int ipi)
> +{
> +	disable_irq_nosync(irq_desc_get_irq(get_ipi_desc(cpu, ipi)));
> +}
> +
> +static void ipi_lpi_setup(int ipi, int ncpus)
>  {
>  	for (int cpu = 0; cpu < ncpus; cpu++) {
>  		int err, irq;
> @@ -1158,6 +1188,20 @@ static void ipi_setup_lpi(int ipi, int ncpus)
>  	}
>  }
>  
> +static void ipi_lpi_send(const cpumask_t *mask, unsigned int nr) {

Sashiko pointed out a typo in the formatting.

static void ipi_lpi_send(const cpumask_t *mask, unsigned int nr)
{

> +	int cpu;
> +
> +	for_each_cpu(cpu, mask)
> +		__ipi_send_single(get_ipi_desc(cpu, nr), cpu);
> +}
> +
> +static const struct ipi_irq_ops ipi_lpi_ops = {
> +	.setup = ipi_lpi_setup,
> +	.disable = ipi_lpi_disable,
> +	.enable = ipi_lpi_enable,
> +	.send = ipi_lpi_send,
> +};
> +
>  void __init set_smp_ipi_range_percpu(int ipi_base, int n, int ncpus)
>  {
>  	int i;
> @@ -1165,18 +1209,15 @@ void __init set_smp_ipi_range_percpu(int ipi_base, int n, int ncpus)
>  	WARN_ON(n < MAX_IPI);
>  	nr_ipi = min(n, MAX_IPI);
>  
> -	percpu_ipi_descs = !!ncpus;
>  	ipi_irq_base = ipi_base;
>  
> -	for (i = 0; i < nr_ipi; i++) {
> -		if (!percpu_ipi_descs)
> -			ipi_setup_sgi(i);
> -		else
> -			ipi_setup_lpi(i, ncpus);
> -	}
> +	ipi_ops = ncpus ? &ipi_lpi_ops : &ipi_sgi_ops;
> +
> +	for (i = 0; i < nr_ipi; i++)
> +		ipi_ops->setup(i, ncpus);
>  
>  	/* Setup the boot CPU immediately */
> -	ipi_setup(smp_processor_id());
> +	ipi_enable(smp_processor_id());
>  }
>  
>  void arch_smp_send_reschedule(int cpu)

Re: [PATCH 32/38] arm64: nmi: Add Kconfig for NMI

[Jinjie Ruan]

在 2026/9/14 18:20, Vladimir Murzin 写道:
> From: Mark Brown <broonie@kernel.org>
> 
> Since NMI handling is in some fairly hot paths we provide a Kconfig
> option which allows support to be compiled out when not needed.
> 
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/Kconfig | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
> 
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 2738637ccb92..df5ff59715d1 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -2251,6 +2251,24 @@ endmenu # "ARMv8.7 architectural features"
>  config AS_HAS_MOPS
>  	def_bool $(as-instr,.arch_extension mops)
>  
> +menu "ARMv8.8 architectural features"
> +
> +config ARM64_NMI
> +	bool "Enable support for Non-maskable Interrupts (NMI)"
> +	default y

We should select HAVE_PERF_EVENTS_NMI to enable perf-NMI-based
hardlockup detector as pseudo NMI do.

arch/arm64/Kconfig:     select HAVE_PERF_EVENTS_NMI if ARM64_PSEUDO_NMI
|| ARM64_NMI

> +	help
> +	  Non-maskable interrupts are an architecture and GIC feature
> +	  which allow the system to configure some interrupts to be
> +	  configured to have superpriority, allowing them to be handled
> +	  before other interrupts and masked for shorter periods of time.
> +
> +	  The feature is detected at runtime, and will remain disabled
> +	  if the cpu does not implement the feature. It will also be
> +	  disabled if pseudo NMIs are enabled at runtime.
> +
> +endmenu # "ARMv8.8 architectural features"
> +
> +
>  menu "ARMv8.9 architectural features"
>  
>  config ARM64_POE

Re: [PATCH 23/38] arm64: cpufeature: Detect PE support for FEAT_NMI

[Jinjie Ruan]

在 2026/9/14 18:20, Vladimir Murzin 写道:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Use of FEAT_NMI requires that all the PEs in the system and the GIC
> have NMI support. This patch implements the PE part of that detection.
> 
> In order to avoid problematic interactions between real and pseudo NMIs
> we disable the architected feature if the user has enabled pseudo NMIs
> on the command line. If this is done on a system where support for the
> architected feature is detected then a warning is printed during boot in
> order to help users spot what is likely to be a misconfiguration.
> 
> As KVM does not care about the host kernel supporting FEAT_NMI or not
> to allow guests to use it, split the CPU cap in two : ARM64_HAS_NMI is
> the hardware support, ARM64_NMI is the kernel making use of it.
> 
> Co-developed-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
> ---
>  arch/arm64/include/asm/cpucaps.h    |  2 +
>  arch/arm64/include/asm/cpufeature.h | 10 +++++
>  arch/arm64/kernel/cpufeature.c      | 68 ++++++++++++++++++++++++++++-
>  arch/arm64/tools/cpucaps            |  2 +
>  4 files changed, 81 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
> index 76350b38f0d7..1a4ef4e94997 100644
> --- a/arch/arm64/include/asm/cpucaps.h
> +++ b/arch/arm64/include/asm/cpucaps.h
> @@ -75,6 +75,8 @@ cpucap_is_possible(const unsigned int cap)
>  		return IS_ENABLED(CONFIG_HW_PERF_EVENTS);
>  	case ARM64_HAS_LSUI:
>  		return IS_ENABLED(CONFIG_ARM64_LSUI);
> +	case ARM64_NMI:
> +		return IS_ENABLED(CONFIG_ARM64_NMI);
>  	}
>  
>  	return true;
> diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
> index 09aa80df602a..c8b1bacdd161 100644
> --- a/arch/arm64/include/asm/cpufeature.h
> +++ b/arch/arm64/include/asm/cpufeature.h
> @@ -809,6 +809,16 @@ static __always_inline bool system_uses_irq_prio_masking(void)
>  	return alternative_has_cap_unlikely(ARM64_HAS_GIC_PRIO_MASKING);
>  }
>  
> +static __always_inline bool system_supports_nmi(void)
> +{
> +	return alternative_has_cap_unlikely(ARM64_HAS_NMI);
> +}
> +
> +static __always_inline bool system_uses_nmi(void)
> +{
> +	return alternative_has_cap_unlikely(ARM64_NMI);
> +}
> +
>  static inline bool system_supports_mte(void)
>  {
>  	return alternative_has_cap_unlikely(ARM64_MTE);
> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> index 32102c3912fa..f9fe7a875f85 100644
> --- a/arch/arm64/kernel/cpufeature.c
> +++ b/arch/arm64/kernel/cpufeature.c
> @@ -83,13 +83,14 @@
>  #include <asm/cpu_ops.h>
>  #include <asm/fpsimd.h>
>  #include <asm/hwcap.h>
> +#include <asm/hypervisor.h>
>  #include <asm/insn.h>
> +#include <asm/interrupts/common_flags.h>
>  #include <asm/kvm_host.h>
>  #include <asm/mmu.h>
>  #include <asm/mmu_context.h>
>  #include <asm/mpam.h>
>  #include <asm/mte.h>
> -#include <asm/hypervisor.h>
>  #include <asm/processor.h>
>  #include <asm/smp.h>
>  #include <asm/sysreg.h>
> @@ -315,6 +316,7 @@ static const struct arm64_ftr_bits ftr_id_aa64pfr1[] = {
>  	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_GCS),
>  		       FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_GCS_SHIFT, 4, 0),
>  	S_ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MTE_frac_SHIFT, 4, 0),
> +	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_NMI_SHIFT, 4, 0),
>  	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SME),
>  		       FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_SME_SHIFT, 4, 0),
>  	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MPAM_frac_SHIFT, 4, 0),
> @@ -2341,6 +2343,51 @@ static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry
>  }
>  #endif
>  
> +#ifdef CONFIG_ARM64_NMI
> +static bool can_use_nmi(const struct arm64_cpu_capabilities *entry, int scope)
> +{
> +	/*
> +	 * ARM64_HAS_NMI has a lower index, and is a boot CPU
> +	 * feature, so will be detected earlier.
> +	 */
> +	BUILD_BUG_ON(ARM64_NMI <= ARM64_HAS_NMI);
> +	if (!cpus_have_cap(ARM64_HAS_NMI))
> +		return false;
> +
> +	/*
> +	 * Having both real and pseudo NMIs enabled simultaneously is
> +	 * likely to cause confusion.  Since pseudo NMIs must be
> +	 * enabled with an explicit command line option, if the user
> +	 * has set that option on a system with real NMIs for some
> +	 * reason assume they know what they're doing.
> +	 *
> +	 * ARM64_HAS_GIC_PRIO_MASKING has a lower index, and is a boot CPU
> +	 * feature, so will be detected earlier.
> +	 */
> +	BUILD_BUG_ON(IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) &&
> +			(ARM64_NMI <= ARM64_HAS_GIC_PRIO_MASKING));
> +	if (cpus_have_cap(ARM64_HAS_GIC_PRIO_MASKING)) {
> +		pr_info_once("Pseudo NMI enabled, not using architected NMI\n");
> +		return false;

Hi, Vladimir,

The user's actual requirement may be to use FEAT_NMI by default when it
is available, and fall back to pseudo-NMI otherwise, since pseudo-NMI
has inherent noise and is only a second choice when there is no hardware
support. Can we design it this way?

Best regards,
Jinjie

> +	}
> +
> +	return true;
> +}
> +
> +static void nmi_enable(const struct arm64_cpu_capabilities *__unused)
> +{
> +	/*
> +	 * Enable use of NMIs controlled by ALLINT, SPINTMASK should
> +	 * be clear by default but make it explicit that we are using
> +	 * this mode.  Ensure that ALLINT is clear first in order to
> +	 * avoid leaving things masked.
> +	 */
> +	_allint_clear();
> +	sysreg_clear_set(sctlr_el1, SCTLR_EL1_SPINTMASK, SCTLR_EL1_NMI);
> +	isb();
> +}
> +#endif
> +
>  static bool can_trap_icv_dir_el1(const struct arm64_cpu_capabilities *entry,
>  				 int scope)
>  {
> @@ -3233,6 +3280,25 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
>  		.matches = has_cpuid_feature,
>  		ARM64_CPUID_FIELDS(ID_AA64ISAR3_EL1, LSUI, IMP)
>  	},
> +#endif
> +	{
> +		.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
> +		.capability = ARM64_HAS_NMI,
> +		.matches = has_cpuid_feature,
> +		ARM64_CPUID_FIELDS(ID_AA64PFR1_EL1, NMI, IMP)
> +	},
> +#ifdef CONFIG_ARM64_NMI
> +	/*
> +	 * Depends on ARM64_HAS_NMI
> +	 * Checks for conflict with pseudo-NMIs, giving them priority.
> +	 */
> +	{
> +		.desc = "Non-maskable Interrupts",
> +		.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
> +		.capability = ARM64_NMI,
> +		.matches = can_use_nmi,
> +		.cpu_enable = nmi_enable,
> +	},
>  #endif
>  	{},
>  };
> diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
> index 2775ba3359cf..531583822ec2 100644
> --- a/arch/arm64/tools/cpucaps
> +++ b/arch/arm64/tools/cpucaps
> @@ -52,6 +52,7 @@ HAS_LS64_V
>  HAS_LSUI
>  HAS_MOPS
>  HAS_NESTED_VIRT
> +HAS_NMI
>  HAS_NV2P1
>  HAS_NV3
>  HAS_PAN
> @@ -82,6 +83,7 @@ MTE
>  MTE_ASYMM
>  MTE_FAR
>  MTE_STORE_ONLY
> +NMI
>  SME
>  SME_FA64
>  SME2

Re: [PATCH 09/38] arm64: replace local_daif helpers

[Jinjie Ruan]

在 2026/9/14 18:20, Vladimir Murzin 写道:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Replace existing `local_daif_...` usage with the new `local_all_irqs...`
> helpers, adding the necessary save-state struct where required.
> 
> Slightly rework `apei_claim_sea()`'s handling of interrupt masking to
> fit with the new APIs.  Currently, it saves the DAIF flags to restore
> later and the `local_irq` ones to check if interrupts are disabled.
> It then masks AIF to signal the SEA (forcing DAIF masking), and will
> unmask to IF (re-enabling PMR masking if pseudo-NMIs are in use) if it
> interrupted a task with interrupts unmaksed, restoring with the DAIF
> flags from the beginning afterwards.
> 
> As we want to preserve a "mask/unmask" process, change the logic from
> "mask AIF/unmask A/unmask IF" to "mask IF/mask A/unmask A/unmask IF" :
> first mask with `local_irq_save()`, mask/unmask further around the
> SEA signaling, and restore at the end with `local_irq_restore()`.
> Now that both APIs properly work with DAIF and PMR, this works
> as expected and also removes the need for a "save flags without masking"
> function for the new helpers, as this was the only user.
> 
> Remove the use of GIC_PRIO_PSR_I_SET in `kvm/hyp/nvhe/switch.c`,
> as we now check both PMR and DAIF everywhere and we don't need to signal
> interrupts being masked in DAIF in the PMR anymore.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
> ---
>  arch/arm64/kernel/acpi.c          | 15 +++++++--------
>  arch/arm64/kernel/hibernate.c     | 14 +++++++-------
>  arch/arm64/kernel/machine_kexec.c |  4 ++--
>  arch/arm64/kernel/setup.c         |  4 ++--
>  arch/arm64/kernel/smp.c           | 10 +++++-----
>  arch/arm64/kernel/suspend.c       | 14 +++++++-------
>  arch/arm64/kvm/hyp/nvhe/switch.c  |  2 +-
>  arch/arm64/kvm/hyp/vgic-v3-sr.c   |  7 ++++---
>  arch/arm64/kvm/hyp/vhe/switch.c   | 14 ++++++++------
>  arch/arm64/mm/mmu.c               |  7 ++++---
>  10 files changed, 47 insertions(+), 44 deletions(-)
> 
> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> index 681aa2bbc399..ea9a9da7e412 100644
> --- a/arch/arm64/kernel/acpi.c
> +++ b/arch/arm64/kernel/acpi.c
> @@ -33,7 +33,7 @@
>  #include <acpi/processor.h>
>  #include <asm/cputype.h>
>  #include <asm/cpu_ops.h>
> -#include <asm/daifflags.h>
> +#include <asm/interrupts/masking.h>
>  #include <asm/smp_plat.h>
>  
>  int acpi_noirq = 1;		/* skip ACPI IRQ initialization */
> @@ -391,15 +391,14 @@ int apei_claim_sea(struct pt_regs *regs)
>  	int err = -ENOENT;
>  	bool return_to_irqs_enabled;
>  	unsigned long current_flags;
> +	arm64_exc_hwstates_t hwstate;
>  
>  	if (!IS_ENABLED(CONFIG_ACPI_APEI_GHES))
>  		return err;
>  
> -	current_flags = local_daif_save_flags();
> -
> -	/* current_flags isn't useful here as daif doesn't tell us about pNMI */
> -	return_to_irqs_enabled = !irqs_disabled_flags(arch_local_save_flags());
> +	local_irq_save(current_flags);
>  
> +	return_to_irqs_enabled = !irqs_disabled_flags(current_flags);
>  	if (regs)
>  		return_to_irqs_enabled = !regs_irqs_disabled(regs);
>  
> @@ -407,10 +406,11 @@ int apei_claim_sea(struct pt_regs *regs)
>  	 * SEA can interrupt SError, mask it and describe this as an NMI so
>  	 * that APEI defers the handling.
>  	 */
> -	local_daif_restore(DAIF_ERRCTX);
> +	hwstate = local_exceptions_save_mask(ERROR_CONTEXT);
>  	nmi_enter();
>  	err = ghes_notify_sea();
>  	nmi_exit();
> +	local_exceptions_restore(hwstate);

Hi Vladimir,

I think mixing local_irq_save()/local_irq_restore() and
local_exceptions_save_mask()/local_exceptions_restore here may be
problematic, because arch_local_irq_save() does not take into full
account the state of DAIF and the pseudo-NMI. For example, when the
pseudo-NMI is enabled, arch_local_irq_save() only sets PMR to
GIC_PRIO_IRQOFF without taking DAIF into consideration, whereas
local_exceptions_save_mask(NOIRQ_CONTEXT) additionally clears all the
DAIF bits.

But whereas local_exceptions_save_mask() returns exactly the current
hwstate arch_local_irq_save() sets, and local_exceptions_restore() also
needs to restore a unified state.

Best regards,
Jinjie

>  
>  	/*
>  	 * APEI NMI-like notifications are deferred to irq_work. Unless
> @@ -418,7 +418,6 @@ int apei_claim_sea(struct pt_regs *regs)
>  	 */
>  	if (!err) {
>  		if (return_to_irqs_enabled) {
> -			local_daif_restore(DAIF_PROCCTX_NOIRQ);
>  			__irq_enter();
>  			irq_work_run();
>  			__irq_exit();
> @@ -428,7 +427,7 @@ int apei_claim_sea(struct pt_regs *regs)
>  		}
>  	}
>  
> -	local_daif_restore(current_flags);
> +	local_irq_restore(current_flags);
>  
>  	return err;
>  }
> diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
> index 7bf117427777..74848a89ba37 100644
> --- a/arch/arm64/kernel/hibernate.c
> +++ b/arch/arm64/kernel/hibernate.c
> @@ -20,7 +20,7 @@
>  #include <asm/barrier.h>
>  #include <asm/cacheflush.h>
>  #include <asm/cputype.h>
> -#include <asm/daifflags.h>
> +#include <asm/interrupts/masking.h>
>  #include <asm/irqflags.h>
>  #include <asm/kexec.h>
>  #include <asm/memory.h>
> @@ -332,16 +332,16 @@ static void swsusp_mte_restore_tags(void)
>  
>  int swsusp_arch_suspend(void)
>  {
> -	int ret = 0;
> -	unsigned long flags;
>  	struct sleep_stack_data state;
> +	arm64_exc_hwstates_t hwstate;
> +	int ret = 0;
>  
>  	if (cpus_are_stuck_in_kernel()) {
>  		pr_err("Can't hibernate: no mechanism to offline secondary CPUs.\n");
>  		return -EBUSY;
>  	}
>  
> -	flags = local_daif_save();
> +	hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
>  
>  	if (__cpu_suspend_enter(&state)) {
>  		/* make the crash dump kernel image visible/saveable */
> @@ -349,7 +349,7 @@ int swsusp_arch_suspend(void)
>  
>  		ret = swsusp_mte_save_tags();
>  		if (ret) {
> -			local_daif_restore(flags);
> +			local_exceptions_restore(hwstate);
>  			return ret;
>  		}
>  
> @@ -393,7 +393,7 @@ int swsusp_arch_suspend(void)
>  		spectre_v4_enable_mitigation(NULL);
>  	}
>  
> -	local_daif_restore(flags);
> +	local_exceptions_restore(hwstate);
>  
>  	return ret;
>  }
> @@ -477,7 +477,7 @@ int __nocfi swsusp_arch_resume(void)
>  	 *   swsusp_arch_resume(), and expects to be re-entered in the
>  	 *   same state : with all DAIF exceptions masked.
>  	 */
> -	local_daif_save();
> +	local_exceptions_save_mask(CRITICAL_CONTEXT);
>  	hibernate_exit(virt_to_phys(tmp_pg_dir), resume_hdr.ttbr1_el1,
>  		       resume_hdr.reenter_kernel, restore_pblist,
>  		       resume_hdr.__hyp_stub_vectors, virt_to_phys(zero_page));
> diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
> index 8f9bc2327dc8..f1353ca6c86b 100644
> --- a/arch/arm64/kernel/machine_kexec.c
> +++ b/arch/arm64/kernel/machine_kexec.c
> @@ -17,7 +17,7 @@
>  
>  #include <asm/cacheflush.h>
>  #include <asm/cpu_ops.h>
> -#include <asm/daifflags.h>
> +#include <asm/interrupts/masking.h>
>  #include <asm/memory.h>
>  #include <asm/mmu.h>
>  #include <asm/mmu_context.h>
> @@ -174,7 +174,7 @@ void machine_kexec(struct kimage *kimage)
>  
>  	pr_info("Bye!\n");
>  
> -	local_daif_mask();
> +	local_exceptions_final_mask();
>  
>  	/*
>  	 * Both restart and kernel_reloc will shutdown the MMU, disable data
> diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
> index 29c6100f0c50..7f4cf0e50551 100644
> --- a/arch/arm64/kernel/setup.c
> +++ b/arch/arm64/kernel/setup.c
> @@ -37,7 +37,7 @@
>  #include <asm/fixmap.h>
>  #include <asm/cpu.h>
>  #include <asm/cputype.h>
> -#include <asm/daifflags.h>
> +#include <asm/interrupts/masking.h>
>  #include <asm/elf.h>
>  #include <asm/cpufeature.h>
>  #include <asm/cpu_ops.h>
> @@ -311,7 +311,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
>  	 * IRQ and FIQ will be unmasked after the root irqchip has been
>  	 * detected and initialized.
>  	 */
> -	local_daif_restore(DAIF_PROCCTX_NOIRQ);
> +	local_exceptions_cpu_init_mask(NOIRQ_CONTEXT);
>  
>  	/*
>  	 * TTBR0 is only used for the identity mapping at this stage. Make it
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index 17e4557404f2..f29094ec3843 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -43,7 +43,7 @@
>  #include <asm/cpu.h>
>  #include <asm/cputype.h>
>  #include <asm/cpu_ops.h>
> -#include <asm/daifflags.h>
> +#include <asm/interrupts/masking.h>
>  #include <asm/kvm_mmu.h>
>  #include <asm/mmu_context.h>
>  #include <asm/nmi.h>
> @@ -265,7 +265,7 @@ asmlinkage notrace void secondary_start_kernel(void)
>  	 * as the root irqchip has already been detected and initialized we can
>  	 * unmask IRQ and FIQ at the same time.
>  	 */
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exceptions_cpu_init_mask(PROCESS_CONTEXT);
>  
>  	/*
>  	 * OK, it's off to the idle thread for us
> @@ -372,7 +372,7 @@ void __noreturn cpu_die(void)
>  
>  	idle_task_exit();
>  
> -	local_daif_mask();
> +	local_exceptions_final_mask();
>  
>  	/* Tell cpuhp_bp_sync_dead() that this CPU is now safe to dispose of */
>  	cpuhp_ap_report_dead();
> @@ -898,14 +898,14 @@ void __noreturn arm64_nmi_cpu_stop(struct pt_regs *regs, bool die_on_crash)
>  	bool crash = IS_ENABLED(CONFIG_KEXEC_CORE) && crash_stop;
>  
>  	/*
> -	 * Use local_daif_mask() instead of local_irq_disable() to make sure
> +	 * Use local_exceptions_final_mask() instead of local_irq_disable() to make sure
>  	 * that pseudo-NMIs are disabled. The "stop" code starts with an IRQ
>  	 * and falls back to NMI (which might be pseudo). If the IRQ finally
>  	 * goes through right as we're timing out then the NMI could interrupt
>  	 * us. It's better to prevent the NMI and let the IRQ finish since the
>  	 * pt_regs will be better.
>  	 */
> -	local_daif_mask();
> +	local_exceptions_final_mask();
>  
>  #ifdef CONFIG_KEXEC_CORE
>  	if (crash && regs)
> diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c
> index c41724a40b75..61012b783279 100644
> --- a/arch/arm64/kernel/suspend.c
> +++ b/arch/arm64/kernel/suspend.c
> @@ -9,7 +9,7 @@
>  #include <asm/cacheflush.h>
>  #include <asm/cpufeature.h>
>  #include <asm/cpuidle.h>
> -#include <asm/daifflags.h>
> +#include <asm/interrupts/masking.h>
>  #include <asm/debug-monitors.h>
>  #include <asm/exec.h>
>  #include <asm/fpsimd.h>
> @@ -69,7 +69,7 @@ void notrace __cpu_suspend_exit(void)
>  	/*
>  	 * Restore HW breakpoint registers to sane values
>  	 * before debug exceptions are possibly reenabled
> -	 * by cpu_suspend()s local_daif_restore() call.
> +	 * by cpu_suspend()s local_exceptions_restore() call.
>  	 */
>  	if (hw_breakpoint_restore)
>  		hw_breakpoint_restore(cpu);
> @@ -97,7 +97,7 @@ void notrace __cpu_suspend_exit(void)
>  int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  {
>  	int ret = 0;
> -	unsigned long flags;
> +	arm64_exc_hwstates_t hwstate;
>  	struct sleep_stack_data state;
>  
>  	/*
> @@ -119,12 +119,12 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	 *
>  	 * Strictly speaking the trace_hardirqs_off() here is superfluous,
>  	 * hardirqs should be firmly off by now. This really ought to use
> -	 * something like raw_local_daif_save().
> +	 * something like raw_local_exceptions_save_mask().
>  	 *
>  	 * This also unmasks interrupts in PMR in order to reliably
>  	 * resume if we're using pseudo-NMIs.
>  	 */
> -	flags = local_daif_save();
> +	hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
>  
>  	/*
>  	 * Function graph tracer state gets inconsistent when the kernel
> @@ -158,11 +158,11 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	unpause_graph_tracing();
>  
>  	/*
> -	 * Restore pstate flags. OS lock and mdscr have been already
> +	 * Restore interrupt masks. OS lock and mdscr have been already
>  	 * restored, so from this point onwards, debugging is fully
>  	 * reenabled if it was enabled when core started shutdown.
>  	 */
> -	local_daif_restore(flags);
> +	local_exceptions_restore(hwstate);
>  
>  	return ret;
>  }
> diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
> index 7318e3e6a5f3..50566b455c80 100644
> --- a/arch/arm64/kvm/hyp/nvhe/switch.c
> +++ b/arch/arm64/kvm/hyp/nvhe/switch.c
> @@ -270,7 +270,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
>  	 * Naturally, we want to avoid this.
>  	 */
>  	if (system_uses_irq_prio_masking()) {
> -		gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
> +		gic_write_pmr(GIC_PRIO_IRQON);
>  		pmr_sync();
>  	}
>  
> diff --git a/arch/arm64/kvm/hyp/vgic-v3-sr.c b/arch/arm64/kvm/hyp/vgic-v3-sr.c
> index 74d4a509f1de..ee0d829a0642 100644
> --- a/arch/arm64/kvm/hyp/vgic-v3-sr.c
> +++ b/arch/arm64/kvm/hyp/vgic-v3-sr.c
> @@ -10,6 +10,7 @@
>  #include <linux/irqchip/arm-gic-v3.h>
>  #include <linux/kvm_host.h>
>  
> +#include <asm/interrupts/masking.h>
>  #include <asm/kvm_emulate.h>
>  #include <asm/kvm_hyp.h>
>  #include <asm/kvm_mmu.h>
> @@ -440,8 +441,8 @@ void __vgic_v3_init_lrs(void)
>  /* Return true if GICv3 is MMIO (GICv2) capable, false otherwise */
>  bool __vgic_v3_get_gic_config(void)
>  {
> +	arm64_exc_hwstates_t hwstate;
>  	u64 val, sre;
> -	unsigned long flags = 0;
>  
>  	sre = read_gicreg(ICC_SRE_EL1);
>  	/*
> @@ -462,7 +463,7 @@ bool __vgic_v3_get_gic_config(void)
>  	 * of the exception entry to EL2.
>  	 */
>  	if (has_vhe()) {
> -		flags = local_daif_save();
> +		hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
>  	} else {
>  		sysreg_clear_set_hcr(0, HCR_AMO | HCR_FMO | HCR_IMO);
>  		isb();
> @@ -477,7 +478,7 @@ bool __vgic_v3_get_gic_config(void)
>  	isb();
>  
>  	if (has_vhe()) {
> -		local_daif_restore(flags);
> +		local_exceptions_restore(hwstate);
>  	} else {
>  		sysreg_clear_set_hcr(HCR_AMO | HCR_FMO | HCR_IMO, 0);
>  		isb();
> diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
> index 7875911c0506..11d9feea7d20 100644
> --- a/arch/arm64/kvm/hyp/vhe/switch.c
> +++ b/arch/arm64/kvm/hyp/vhe/switch.c
> @@ -17,6 +17,7 @@
>  
>  #include <asm/barrier.h>
>  #include <asm/cpufeature.h>
> +#include <asm/interrupts/masking.h>
>  #include <asm/kprobes.h>
>  #include <asm/kvm_asm.h>
>  #include <asm/kvm_emulate.h>
> @@ -661,9 +662,10 @@ NOKPROBE_SYMBOL(__kvm_vcpu_run_vhe);
>  
>  int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
>  {
> +	arm64_exc_hwstates_t hwstate;
>  	int ret;
>  
> -	local_daif_mask();
> +	hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
>  
>  	/*
>  	 * Having IRQs masked via PMR when entering the guest means the GIC
> @@ -671,18 +673,18 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
>  	 * only way to get out will be via guest exceptions.
>  	 * Naturally, we want to avoid this.
>  	 *
> -	 * local_daif_mask() already sets GIC_PRIO_PSR_I_SET, we just need a
> -	 * dsb to ensure the redistributor is forwards EL2 IRQs to the CPU.
> +	 * local_exceptions_save_mask(), via helpers, sets
> +	 * GIC_PRIO_IRQON along with pmr_sync(), so we can be shure
> +	 * that redistributor forwards EL2 IRQs to the CPU.
>  	 */
> -	pmr_sync();
>  
>  	ret = __kvm_vcpu_run_vhe(vcpu);
>  
>  	/*
> -	 * local_daif_restore() takes care to properly restore PSTATE.DAIF
> +	 * local_exceptions_restore() takes care to properly restore PSTATE.DAIF
>  	 * and the GIC PMR if the host is using IRQ priorities.
>  	 */
> -	local_daif_restore(DAIF_PROCCTX_NOIRQ);
> +	local_exceptions_restore(hwstate);
>  
>  	return ret;
>  }
> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> index 79d90226fd5d..abe89b60a64a 100644
> --- a/arch/arm64/mm/mmu.c
> +++ b/arch/arm64/mm/mmu.c
> @@ -33,6 +33,7 @@
>  
>  #include <asm/barrier.h>
>  #include <asm/cputype.h>
> +#include <asm/interrupts/masking.h>
>  #include <asm/fixmap.h>
>  #include <asm/kasan.h>
>  #include <asm/kernel-pgtable.h>
> @@ -2329,7 +2330,7 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
>  	typedef void (ttbr_replace_func)(phys_addr_t);
>  	extern ttbr_replace_func idmap_cpu_replace_ttbr1;
>  	ttbr_replace_func *replace_phys;
> -	unsigned long daif;
> +	arm64_exc_hwstates_t hwstate;
>  
>  	/* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */
>  	phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp));
> @@ -2345,9 +2346,9 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
>  	 * We really don't want to take *any* exceptions while TTBR1 is
>  	 * in the process of being replaced so mask everything.
>  	 */
> -	daif = local_daif_save();
> +	hwstate = local_exceptions_save_mask(CRITICAL_CONTEXT);
>  	replace_phys(ttbr1);
> -	local_daif_restore(daif);
> +	local_exceptions_restore(hwstate);
>  
>  	cpu_uninstall_idmap();
>  }

Re: [PATCH 03/38] arm64: interrupts: Add common exception state helpers

[Jinjie Ruan]

在 2026/9/14 18:20, Vladimir Murzin 写道:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> The entry code and the rest of the kernel require different interrupt
> masking APIs, but all need a common representation of the hardware
> exception state.
> 
> Introduce exception contexts that map DAIF and PMR state to the
> corresponding masking context. Provide helpers to translate between
> exception contexts and hardware state and update the hardware state.
> 
> Updating PMR is unnecessary when a caller knows that its value has not
> changed. Allow such callers to avoid the update, while providing a
> force option for callers that cannot rely on the previous PMR state.
> 
> Entry specific and general purpose masking APIs will be built on these
> helpers in subsequent patches.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  .../include/asm/interrupts/common_flags.h     | 174 ++++++++++++++++++
>  1 file changed, 174 insertions(+)
>  create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
> 
> diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
> new file mode 100644
> index 000000000000..0de5ba1ae155
> --- /dev/null
> +++ b/arch/arm64/include/asm/interrupts/common_flags.h
> @@ -0,0 +1,174 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2025 Arm Ltd.
> + */
> +#ifndef __ASM_INTERRUPTS_COMMON_FLAGS_H
> +#define __ASM_INTERRUPTS_COMMON_FLAGS_H
> +
> +#include <asm/arch_gicv3.h>
> +#include <asm/bug.h>
> +#include <asm/cpufeature.h>
> +#include <asm/ptrace.h>
> +#include <asm/sysreg.h>
> +#include <asm/irqflags.h>
> +
> +#define DAIF_PROCCTX		0
> +#define DAIF_PROCCTX_NOIRQ	(PSR_I_BIT | PSR_F_BIT)
> +#define DAIF_ERRCTX		(PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
> +#define DAIF_MASK		(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
> +
> +/*
> + * Exception context mapping
> + *
> + * pseudo-NMI
> + *
> + * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
> + * ERROR    ->  AIF + IRQON
> + * NONMI    ->   IF + IRQON
> + * NOIRQ    ->    0 + IRQOFF
> + * PROCESS  ->    0 + IRQON
> + *
> + * Otherwise
> + *
> + * CRITICAL -> DAIF           (corresponds to the state on exception entry)
> + * ERROR    ->  AIF
> + * NONMI    ->   IF
> + * NOIRQ    ->   IF
> + * PROCESS  ->    0
> + */
> +typedef enum arm64_exc_context {
> +	PROCESS_CONTEXT,
> +	NOIRQ_CONTEXT,
> +	NONMI_CONTEXT,
> +	ERROR_CONTEXT,
> +	CRITICAL_CONTEXT,
> +} arm64_exc_context_t;
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_process_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_exc_hwstate_of_context(arm64_exc_context_t context) {
> +	switch (context) {
> +	case PROCESS_CONTEXT:
> +		return __arm64_exc_hwstate_of_process_context();
> +	case NOIRQ_CONTEXT:
> +		return __arm64_exc_hwstate_of_noirq_context();
> +	case NONMI_CONTEXT:
> +		return __arm64_exc_hwstate_of_nonmi_context();
> +	case ERROR_CONTEXT:
> +		return __arm64_exc_hwstate_of_error_context();
> +	case CRITICAL_CONTEXT:
> +		return __arm64_exc_hwstate_of_critical_context();
> +	default:
> +		BUG();
> +	}
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
> +{
> +	arm64_exc_hwstate_t state = {.daif=regs->pstate & DAIF_MASK};
> +
> +	if (system_uses_irq_prio_masking())
> +		state.pmr = regs->pmr;
> +
> +	return state;
> +}
> +
> +static __always_inline
> +void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
> +{
> +	bool pseudo_nmi = system_uses_irq_prio_masking();
> +
> +	barrier();
> +
> +	if (pseudo_nmi && hwstate.pmr == GIC_PRIO_IRQOFF && force) {
> +		/*
> +		 * There has been concern that the write to daif
> +		 * might be reordered before this write to PMR.
> +		 * From the ARM ARM DDI 0487D.a, section D1.7.1
> +		 * "Accessing PSTATE fields":
> +		 *   Writes to the PSTATE fields have side-effects on
> +		 *   various aspects of the PE operation. All of these
> +		 *   side-effects are guaranteed:
> +		 *     - Not to be visible to earlier instructions in
> +		 *       the execution stream.
> +		 *     - To be visible to later instructions in the
> +		 *       execution stream
> +		 *
> +		 * Also, writes to PMR are self-synchronizing, so no
> +		 * interrupts with a lower priority than PMR is signaled
> +		 * to the PE after the write.
> +		 *
> +		 * So we don't need additional synchronization here.
> +		 */
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +	}
> +
> +	write_sysreg(hwstate.daif, daif);
> +
> +	if (pseudo_nmi && hwstate.pmr == GIC_PRIO_IRQON && force) {
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +		pmr_sync();
> +	}
> +
> +	barrier();

LGTM
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

> +}
> +
> +static __always_inline
> +void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate)
> +{
> +	__arm64_update_exc_hwstate(hwstate, true);
> +}
> +
> +static __always_inline
> +void arm64_update_exc_context(arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(context);
> +
> +	arm64_update_exc_hwstate(hwstate);
> +}
> +#endif /* __ASM_INTERRUPTS_COMMON_FLAGS_H */

Re: [PATCH 03/38] arm64: interrupts: Add common exception state helpers

[Jinjie Ruan]

在 2026/9/14 18:20, Vladimir Murzin 写道:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> The entry code and the rest of the kernel require different interrupt
> masking APIs, but all need a common representation of the hardware
> exception state.
> 
> Introduce exception contexts that map DAIF and PMR state to the
> corresponding masking context. Provide helpers to translate between
> exception contexts and hardware state and update the hardware state.
> 
> Updating PMR is unnecessary when a caller knows that its value has not
> changed. Allow such callers to avoid the update, while providing a
> force option for callers that cannot rely on the previous PMR state.
> 
> Entry specific and general purpose masking APIs will be built on these
> helpers in subsequent patches.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  .../include/asm/interrupts/common_flags.h     | 174 ++++++++++++++++++
>  1 file changed, 174 insertions(+)
>  create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
> 
> diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
> new file mode 100644
> index 000000000000..0de5ba1ae155
> --- /dev/null
> +++ b/arch/arm64/include/asm/interrupts/common_flags.h
> @@ -0,0 +1,174 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2025 Arm Ltd.
> + */
> +#ifndef __ASM_INTERRUPTS_COMMON_FLAGS_H
> +#define __ASM_INTERRUPTS_COMMON_FLAGS_H
> +
> +#include <asm/arch_gicv3.h>
> +#include <asm/bug.h>
> +#include <asm/cpufeature.h>
> +#include <asm/ptrace.h>
> +#include <asm/sysreg.h>
> +#include <asm/irqflags.h>
> +
> +#define DAIF_PROCCTX		0
> +#define DAIF_PROCCTX_NOIRQ	(PSR_I_BIT | PSR_F_BIT)
> +#define DAIF_ERRCTX		(PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
> +#define DAIF_MASK		(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
> +
> +/*
> + * Exception context mapping
> + *
> + * pseudo-NMI
> + *
> + * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
> + * ERROR    ->  AIF + IRQON
> + * NONMI    ->   IF + IRQON
> + * NOIRQ    ->    0 + IRQOFF
> + * PROCESS  ->    0 + IRQON
> + *
> + * Otherwise
> + *
> + * CRITICAL -> DAIF           (corresponds to the state on exception entry)
> + * ERROR    ->  AIF
> + * NONMI    ->   IF
> + * NOIRQ    ->   IF
> + * PROCESS  ->    0
> + */
> +typedef enum arm64_exc_context {
> +	PROCESS_CONTEXT,
> +	NOIRQ_CONTEXT,
> +	NONMI_CONTEXT,
> +	ERROR_CONTEXT,
> +	CRITICAL_CONTEXT,
> +} arm64_exc_context_t;
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_process_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_exc_hwstate_of_context(arm64_exc_context_t context) {
> +	switch (context) {
> +	case PROCESS_CONTEXT:
> +		return __arm64_exc_hwstate_of_process_context();
> +	case NOIRQ_CONTEXT:
> +		return __arm64_exc_hwstate_of_noirq_context();
> +	case NONMI_CONTEXT:
> +		return __arm64_exc_hwstate_of_nonmi_context();
> +	case ERROR_CONTEXT:
> +		return __arm64_exc_hwstate_of_error_context();
> +	case CRITICAL_CONTEXT:
> +		return __arm64_exc_hwstate_of_critical_context();
> +	default:
> +		BUG();
> +	}
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
> +{
> +	arm64_exc_hwstate_t state = {.daif=regs->pstate & DAIF_MASK};
> +
> +	if (system_uses_irq_prio_masking())
> +		state.pmr = regs->pmr;
> +
> +	return state;
> +}
Hi Vladimir,

This helper looks more related to the 5th patch "arm64: entry: Introduce
entry specific exception masking helpers", as it is entry-related.
Moving it to 5th patch may make it easier to review.

Best regards,
Jinjie

> +
> +static __always_inline
> +void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
> +{
> +	bool pseudo_nmi = system_uses_irq_prio_masking();
> +
> +	barrier();
> +
> +	if (pseudo_nmi && hwstate.pmr == GIC_PRIO_IRQOFF && force) {
> +		/*
> +		 * There has been concern that the write to daif
> +		 * might be reordered before this write to PMR.
> +		 * From the ARM ARM DDI 0487D.a, section D1.7.1
> +		 * "Accessing PSTATE fields":
> +		 *   Writes to the PSTATE fields have side-effects on
> +		 *   various aspects of the PE operation. All of these
> +		 *   side-effects are guaranteed:
> +		 *     - Not to be visible to earlier instructions in
> +		 *       the execution stream.
> +		 *     - To be visible to later instructions in the
> +		 *       execution stream
> +		 *
> +		 * Also, writes to PMR are self-synchronizing, so no
> +		 * interrupts with a lower priority than PMR is signaled
> +		 * to the PE after the write.
> +		 *
> +		 * So we don't need additional synchronization here.
> +		 */
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +	}
> +
> +	write_sysreg(hwstate.daif, daif);
> +
> +	if (pseudo_nmi && hwstate.pmr == GIC_PRIO_IRQON && force) {
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +		pmr_sync();
> +	}
> +
> +	barrier();
> +}
> +
> +static __always_inline
> +void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate)
> +{
> +	__arm64_update_exc_hwstate(hwstate, true);
> +}
> +
> +static __always_inline
> +void arm64_update_exc_context(arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(context);
> +
> +	arm64_update_exc_hwstate(hwstate);
> +}
> +#endif /* __ASM_INTERRUPTS_COMMON_FLAGS_H */

Re: [PATCH 05/38] arm64: entry: Introduce entry specific exception masking helpers

[Jinjie Ruan]

在 2026/9/14 18:20, Vladimir Murzin 写道:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> The entry code handles interrupt masking differently from the rest of
> the kernel. Exception handlers enter and exit with all exceptions
> masked, but they must temporarily unmask the appropriate set of
> exceptions so that the rest of the handler executes with the expected
> exception state.
> 
> For EL0 handlers, this means dropping to masking context appropriate
> for the work to be performed. For EL1 handlers, this means restoring
> the masking context of the interrupted task. In both cases, all
> exceptions must be masked again before returning from the exception
> handler.
> 
> The rest of the kernel typically follows the opposite pattern: it
> raises the masking context to protect a critical section and later
> restores the previous context.
> 
> Given these different usage patterns, introduce a dedicated set of
> exception masking helpers for the entry code. Keeping these helpers
> separate from the generic interrupt masking APIs makes the intended
> usage explicit and helps avoid mixing the two masking models.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
> ---
>  arch/arm64/include/asm/interrupts/entry.h | 100 ++++++++++++++++++++++
>  1 file changed, 100 insertions(+)
>  create mode 100644 arch/arm64/include/asm/interrupts/entry.h
> 
> diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
> new file mode 100644
> index 000000000000..0cfca62aa25b
> --- /dev/null
> +++ b/arch/arm64/include/asm/interrupts/entry.h
> @@ -0,0 +1,100 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2025 Arm Ltd.
> + */
> +#ifndef __ASM_INTERRUPTS_ENTRY_H
> +#define __ASM_INTERRUPTS_ENTRY_H
> +
> +#include <asm/arch_gicv3.h>
> +#include <asm/bug.h>
> +#include <asm/cpufeature.h>
> +#include <asm/interrupts/common_flags.h>
> +
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
> +						  arm64_exc_hwstate_t next)
> +{
> +	bool irqs_disabled = arch_irqs_disabled_flags(next.flags);
> +	bool force;
> +
> +	arm64_debug_exc_hwstate(prev);
> +
> +	if (prev.flags == next.flags)
> +		return next;
> +
> +	if (!irqs_disabled)
> +		trace_hardirqs_on();
> +
> +	force = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
> +
> +	__arm64_update_exc_hwstate(next, force);
> +
> +	if (irqs_disabled)
> +		trace_hardirqs_off();
> +
> +	return next;
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_inherit_exc_context(struct pt_regs *regs)
> +{
> +	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
> +	arm64_exc_hwstate_t next = arm64_inherit_exc_hwstate(regs);

Could it be renamed to a more intuitive name, such as
arm64_exc_hwstate_of_regs()?

> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
> +
> +	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
> +		WARN_ON_ONCE(prev.daif < next.daif);
> +
> +		if (prev.daif == next.daif) {
> +			/*
> +			 * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so larger PMR value is weaker
> +			 */
> +			WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr > next.pmr);
> +		}

The if branch is a dead code now, we can remove it.

arm64_drop_exc_context is only used directly in the context of
el0t_64_error_handler below. It switches from ERROR_CONTEXT to
PROCESS_CONTEXT.

And now arm64_unmask_exc_context always switch from CRITICAL_CONTEXT
to ERROR_CONTEXT/NONMI_CONTEXT/NOIRQ_CONTEXT/PROCESS_CONTEXT.

So there is no case where prev and next DAIF are the same.

1009 static void noinstr __el0_error_handler_common(struct pt_regs *regs)
1010 {
1011 >-------unsigned long esr = read_sysreg(esr_el1);
1012 >-------arm64_exc_hwstate_t hwstate;
1013 >-------irqentry_state_t state;
1014
1015 >-------arm64_enter_from_user_mode(regs);
1016 >-------hwstate = arm64_unmask_exc_context(ERROR_CONTEXT);
1017 >-------state = irqentry_nmi_enter(regs);
1018 >-------do_serror(regs, esr);
1019 >-------irqentry_nmi_exit(regs, state);
1020 >-------hwstate = arm64_drop_exc_context(hwstate, PROCESS_CONTEXT);
1021 >-------arm64_exit_to_user_mode(regs, hwstate);
1022 >-------arm64_debug_exc_context(CRITICAL_CONTEXT);
1023 }
1024
1025 asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
1026 {
1027 >-------__el0_error_handler_common(regs);
1028 }


> +	}
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
> +
> +	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
> +		WARN_ON_ONCE(prev.daif > next.daif);
> +
> +		if (prev.daif == next.daif) {
> +			/*
> +			 * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so smaller PMR value is stronger
> +			 */
> +			WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr < next.pmr);
> +		}
> +	}
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_unmask_exc_context(arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
> +
> +	return arm64_drop_exc_context(prev, context);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_mask_exc_context(arm64_exc_hwstate_t prev)
> +{
> +	return arm64_lift_exc_context(prev, CRITICAL_CONTEXT);
> +}
> +
> +#endif /* __ASM_INTERRUPTS_ENTRY_H */

-- 
Best regards,
Jinjie

Re: [PATCH 26/38] arm64: nmi: Add handling of superpriority interrupts as NMIs

[Jinjie Ruan]

在 2026/9/14 18:20, Vladimir Murzin 写道:
> From: Mark Brown <broonie@kernel.org>
> 
> Our goal with superpriority interrupts is to use them as NMIs, taking
> advantage of the much smaller regions where they are masked to allow
> prompt handling of the most time-critical interrupts.
> 
> When an interrupt is configured with superpriority, we enter EL1 as we
> do for any other interrupt. The presence of a superpriority interrupt
> is indicated by a status bit in ISR_EL1. We check this bit before
> unmasking interrupts in elX_interrupt(), and if a superpriority
> interrupt is pending, we handle it as an NMI. Otherwise, normal
> interrupts are handled as usual.
> 
> Since superpriority interrupts are always handled as NMIs, the
> interrupt controller can rely on in_nmi() to distinguish them from
> ordinary interrupts.
> 
> Enable IPIs to use superpriority interrupts as NMIs, matching the
> existing pseudo-NMI behaviour.
> 
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
> ---
>  arch/arm64/include/asm/entry-common.h |  7 +++
>  arch/arm64/kernel/entry-common.c      | 83 +++++++++++++++++++++++----
>  arch/arm64/kernel/smp.c               |  2 +-
>  3 files changed, 79 insertions(+), 13 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
> index 73d82a8d8e95..0681ba91ac3b 100644
> --- a/arch/arm64/include/asm/entry-common.h
> +++ b/arch/arm64/include/asm/entry-common.h
> @@ -37,6 +37,13 @@ static inline bool arch_irqentry_exit_need_resched(void)
>  	if (system_uses_irq_prio_masking() && read_sysreg(daif))
>  		return false;
>  
> +	/*
> +	 * If AllInt is set then we must have handled an NMI, so skip
> +	 * preemption
> +	 */
> +	if (system_uses_nmi() && read_sysreg_s(SYS_ALLINT))

if (system_uses_nmi() && (read_sysreg_s(SYS_ALLINT) & ALLINT_ALLINT))

> +		return false;
> +
>  	/*
>  	 * Preempting a task from an IRQ means we leave copies of PSTATE
>  	 * on the stack. cpufeature's enable calls may modify PSTATE, but
> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> index fd0bfead6e70..31dae3b48f23 100644
> --- a/arch/arm64/kernel/entry-common.c
> +++ b/arch/arm64/kernel/entry-common.c
> @@ -31,6 +31,14 @@
>  #include <asm/sysreg.h>
>  #include <asm/system_misc.h>
>  
> +static __always_inline bool is_nmi(void)
> +{
> +	if (!system_uses_nmi())
> +		return false;
> +
> +	return !!(read_sysreg(isr_el1) & ISR_EL1_IS);
> +}
> +
>  /*
>   * Handle IRQ/context state management when entering from kernel mode.
>   * Before this function is called it is not safe to call regular kernel code,
> @@ -525,8 +533,8 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
>  	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
> -static __always_inline void __el1_pnmi(struct pt_regs *regs,
> -				       void (*handler)(struct pt_regs *))
> +static __always_inline void __el1_nmi(struct pt_regs *regs,
> +				      void (*handler)(struct pt_regs *))
>  {
>  	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
> @@ -545,7 +553,23 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
>  
>  	state = arm64_enter_from_kernel_mode(regs);
>  
> -	arm64_unmask_exc_context(NONMI_CONTEXT);
> +	/*
> +	 * For non-NMI systems, we can switch to NOIRQ context since
> +	 * NMIs cannot occur.
> +	 *
> +	 * For pseudo-NMI, we must keep NONMI context until we can
> +	 * determine the interrupt priority, which is done by the IRQ
> +	 * handler.
> +	 *
> +	 * For FEAT_NMI, we already know the interrupt is an IRQ, so we
> +	 * can switch to NOIRQ context immediately. NMIs do not
> +	 * interfere because they follow a separate handling path and do
> +	 * not share the IRQ handling state.
> +	 */
> +	if (system_uses_irq_prio_masking())
> +		arm64_unmask_exc_context(NONMI_CONTEXT);
> +	else
> +		arm64_unmask_exc_context(NOIRQ_CONTEXT);
>  
>  	irq_enter_rcu();
>  	do_interrupt_handler(regs, handler);
> @@ -565,8 +589,9 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
>  static void noinstr el1_interrupt(struct pt_regs *regs,
>  				  void (*handler)(struct pt_regs *))
>  {
> -	if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && regs_irqs_disabled(regs))
> -		__el1_pnmi(regs, handler);
> +	/* Is there a NMI to handle? */
> +	if (regs_irqs_disabled(regs) || is_nmi())
> +		__el1_nmi(regs, handler);
>  	else
>  		__el1_irq(regs, handler);
>  
> @@ -580,6 +605,8 @@ asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
>  
>  asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
>  {
> +	WARN_ON_ONCE(system_uses_nmi() && (read_sysreg(isr_el1) & ISR_EL1_FS));
> +
>  	el1_interrupt(regs, handle_arch_fiq);
>  }
>  
> @@ -903,20 +930,36 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
>  	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
> -static void noinstr el0_interrupt(struct pt_regs *regs,
> -				  void (*handler)(struct pt_regs *))
> +static __always_inline void __el0_nmi(struct pt_regs *regs,
> +				      void (*handler)(struct pt_regs *))
>  {
> -	arm64_enter_from_user_mode(regs);
> +	irqentry_state_t state;
> +	arm64_exc_hwstate_t hwstate;
>  
> -	arm64_unmask_exc_context(NONMI_CONTEXT);
> +	state = irqentry_nmi_enter(regs);
> +	hwstate = arm64_unmask_exc_context(NONMI_CONTEXT);
> +	do_interrupt_handler(regs, handler);
> +	arm64_mask_exc_context(hwstate);
> +	irqentry_nmi_exit(regs, state);
> +}
>  
> -	if (regs->pc & BIT(55))
> -		arm64_apply_bp_hardening();
> +static __always_inline void __el0_irq(struct pt_regs *regs,
> +				      void (*handler)(struct pt_regs *))
> +{
> +	arm64_enter_from_user_mode(regs);
> +	/*
> +	 * For the same reason as in el1_irq() keep NONMI
> +	 * context for pesudo-NMI, and switch to NOIRQ
> +	 * otherwise
> +	 */
> +	if (system_uses_irq_prio_masking())
> +		arm64_unmask_exc_context(NONMI_CONTEXT);
> +	else
> +		arm64_unmask_exc_context(NOIRQ_CONTEXT);
>  
>  	irq_enter_rcu();
>  	do_interrupt_handler(regs, handler);
>  	irq_exit_rcu();
> -
>  	/*
>  	 * For the same reason as in el1_irq() we effectivly
>  	 * have NOIRQ_CONTEXT on return from handler - keep
> @@ -924,6 +967,20 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
>  	 */
>  	arm64_debug_exc_context(NOIRQ_CONTEXT);
>  	arm64_exit_to_user_mode(regs, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
> +}
> +
> +static void noinstr el0_interrupt(struct pt_regs *regs,
> +				  void (*handler)(struct pt_regs *))
> +{
> +	if (regs->pc & BIT(55))
> +		arm64_apply_bp_hardening();
> +
> +	/* Is there a NMI to handle? */
> +	if (is_nmi())
> +		__el0_nmi(regs, handler);
> +	else
> +		__el0_irq(regs, handler);
> +
>  	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
> @@ -939,6 +996,8 @@ asmlinkage void noinstr el0t_64_irq_handler(struct pt_regs *regs)
>  
>  static void noinstr __el0_fiq_handler_common(struct pt_regs *regs)
>  {
> +	WARN_ON_ONCE(system_uses_nmi() && (read_sysreg(isr_el1) & ISR_EL1_FS));
> +
>  	el0_interrupt(regs, handle_arch_fiq);
>  }
>  
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index f29094ec3843..3d7060aedd66 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -1061,7 +1061,7 @@ static void smp_cross_call(const struct cpumask *target, unsigned int ipinr)
>  
>  static bool ipi_should_be_nmi(enum ipi_msg_type ipi)
>  {
> -	if (!system_uses_irq_prio_masking())
> +	if (!system_uses_nmi() && !system_uses_irq_prio_masking())
>  		return false;
>  
>  	switch (ipi) {

-- 
Best regards,
Jinjie

Re: [PATCH 24/38] arm64: nmi: Manage masking for superpriority interrupts

[Jinjie Ruan]

在 2026/9/14 18:20, Vladimir Murzin 写道:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Extend logic to handle and debug exception context/state with knowlage
> of FEAT_NMI.
> 
> Take care to order writes to ALLINT relative to DAIF, as clearing
> ALLINT before DAIF could result in taking an NMI while DAIF is fully
> masked, as could setting it after DAIF.
> 
> Since superpriority interrupts are not masked through DAIF like pseduo
> NMIs are, we also need to modify the assembler macros for managing
> DAIF to ensure that the masking is done in the assembly code.
> 
> Note that save_and_disable_irq/restore_irq and
> save_and_disable_daif/restore_irq pairs are used in distinct
> contextes:
> 
> - former is used in context of SW PAN to quickly disable/enable
>   preemption
> 
> - latter is used to completely mask all exceptions.
> 
> For that reason split save_and_disable_daif/restore_irq into more
> generic exception save restore pair and plumb with FEAT_NMI logic.
> 
> Co-developed-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/assembler.h            | 20 +++++++++--
>  .../include/asm/interrupts/common_flags.h     | 36 +++++++++++++++++++
>  arch/arm64/include/asm/interrupts/entry.h     |  6 ++--
>  arch/arm64/include/asm/interrupts/masking.h   |  3 +-
>  arch/arm64/include/asm/irqflags.h             |  3 +-
>  arch/arm64/kernel/entry.S                     | 12 +++----
>  6 files changed, 68 insertions(+), 12 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 0b58b550e8dc..bcdbc308afba 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -37,11 +37,27 @@
>  /*
>   * Save/restore interrupts.
>   */
> -	.macro save_and_disable_daif, flags
> -	mrs	\flags, daif
> +	.macro save_and_disable_exceptions, flags, tmp
> +	mrs     \flags, daif		// updates flags[9:6] with DAIF
> +#ifdef CONFIG_ARM64_NMI
> +alternative_if ARM64_NMI
> +	mrs_s	\tmp, SYS_ALLINT	// updates tmp[13] with AllInt
> +	msr_s	SYS_ALLINT_SET, xzr
> +	orr	\flags, \flags, \tmp	// now flags[13,9:6] carry pair of AllInt,DAIF
> +alternative_else_nop_endif
> +#endif
>  	msr	daifset, #0xf
>  	.endm
>  
> +	.macro  restore_exceptions, flags
> +	msr     daif, \flags		// bits other than flags[9:6] are ignored
> +#ifdef CONFIG_ARM64_NMI
> +alternative_if ARM64_NMI
> +	msr_s	SYS_ALLINT, \flags	// bits other than flags[13] are ignored
> +alternative_else_nop_endif
> +#endif
> +	.endm
> +
>  	.macro	save_and_disable_irq, flags
>  	mrs	\flags, daif
>  	msr	daifset, #3
> diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
> index c50c6e6473ec..18c9a392028a 100644
> --- a/arch/arm64/include/asm/interrupts/common_flags.h
> +++ b/arch/arm64/include/asm/interrupts/common_flags.h
> @@ -20,6 +20,14 @@
>  /*
>   * Exception context mapping
>   *
> + * FEAT_NMI
> + *
> + * CRITICAL -> DAIF + AllInt (corresponds to the state on exception entry)
> + * ERROR    ->  AIF + AllInt
> + * NONMI    ->   IF + AllInt
> + * NOIRQ    ->   IF
> + * PROCESS  ->    0
> + *
>   * pseudo-NMI
>   *
>   * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
> @@ -76,6 +84,7 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
>  		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
>  
>  	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +
>  }

An extra blank line.

>  
>  static __always_inline
> @@ -84,6 +93,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
>  	if (system_uses_irq_prio_masking())
>  		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
>  
> +	if (system_uses_nmi())
> +               return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .allint=ALLINT_ALLINT};
> +
>  	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
>  }
>  
> @@ -93,6 +105,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
>  	if (system_uses_irq_prio_masking())
>  		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
>  
> +	if (system_uses_nmi())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .allint=ALLINT_ALLINT};
> +
>  	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
>  }
>  
> @@ -102,6 +117,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
>  	if (system_uses_irq_prio_masking())
>  		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
>  
> +	if (system_uses_nmi())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .allint=ALLINT_ALLINT};
> +
>  	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
>  }
>  
> @@ -131,6 +149,9 @@ arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
>  	if (system_uses_irq_prio_masking())
>  		state.pmr = regs->pmr;
>  
> +	if (system_uses_nmi())
> +		state.allint = regs->pstate & PSR_ALLINT_BIT;
> +
>  	return state;
>  }
>  
> @@ -150,6 +171,9 @@ void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
>  	if (system_uses_irq_prio_masking()) {
>  		WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
>  			  actual.daif, actual.pmr, expected.daif, expected.pmr);
> +	} else if (system_uses_nmi()) {
> +		WARN_ONCE(1, "Unexpected DAIF+ALLINT: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
> +			  actual.daif, actual.allint, expected.daif, expected.allint);
>  	} else {
>  		WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
>  			  actual.daif, expected.daif);
> @@ -168,6 +192,7 @@ static __always_inline
>  void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
>  {
>  	bool pseudo_nmi = system_uses_irq_prio_masking();
> +	bool nmi = system_uses_nmi();
>  
>  	barrier();
>  
> @@ -194,8 +219,19 @@ void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
>  		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
>  	}
>  
> +	/*
> +	 * Try to order ALLINT writes to be consistent with the DAIF state :
> +	 * we don't want to take an NMI with DAIF masked or when it should
> +	 * be masked but isn't yet.
> +	 */
> +	if (nmi && (hwstate.allint & ALLINT_ALLINT) && force)
> +		_allint_set();
> +
>  	write_sysreg(hwstate.daif, daif);
>  
> +	if (nmi && !(hwstate.allint & ALLINT_ALLINT) && force)
> +		_allint_clear();
> +
>  	if (pseudo_nmi && hwstate.pmr == GIC_PRIO_IRQON && force) {
>  		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
>  		pmr_sync();
> diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
> index 0cfca62aa25b..2a522dd15556 100644
> --- a/arch/arm64/include/asm/interrupts/entry.h
> +++ b/arch/arm64/include/asm/interrupts/entry.h
> @@ -10,7 +10,6 @@
>  #include <asm/cpufeature.h>
>  #include <asm/interrupts/common_flags.h>
>  
> -
>  static __always_inline
>  arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
>  						  arm64_exc_hwstate_t next)
> @@ -26,7 +25,8 @@ arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
>  	if (!irqs_disabled)
>  		trace_hardirqs_on();
>  
> -	force = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
> +	force = (system_uses_irq_prio_masking() && prev.pmr != next.pmr) ||
> +		(system_uses_nmi() && prev.allint != next.allint);
>  
>  	__arm64_update_exc_hwstate(next, force);
>  
> @@ -58,6 +58,7 @@ arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
>  			 * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so larger PMR value is weaker
>  			 */
>  			WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr > next.pmr);
> +			WARN_ON_ONCE(system_uses_nmi() && prev.allint < next.allint);

"prev.allint < next.allint" check should not be constrained by whether
DAIF is equal, because it is possible that both DAIF and ALLINT change
when the context is dropped, And when the context is dropped, ALLINT
will only remain unchanged ( 0 -> 0 or 1 -> 1 )or be cleared, such as:

       ERROR    -> PROCESS

daif   AIF     ->   0
allint ALLINT  ->   0

>  		}
>  	}
>  
> @@ -77,6 +78,7 @@ arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
>  			 * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so smaller PMR value is stronger
>  			 */
>  			WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr < next.pmr);
> +			WARN_ON_ONCE(system_uses_nmi() && prev.allint > next.allint);

It is the same here.

Otherwise,
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

>  		}
>  	}
>  
> diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
> index 8729922974b0..6c5ca5befbc1 100644
> --- a/arch/arm64/include/asm/interrupts/masking.h
> +++ b/arch/arm64/include/asm/interrupts/masking.h
> @@ -30,7 +30,8 @@ arm64_exc_hwstates_t local_exceptions_save_mask(arm64_exc_context_t new)
>  	 * We've just got actual HW state so we can rely on that to
>  	 * optimize some unnecessary updates.
>  	 */
> -	force = system_uses_irq_prio_masking() && actual.pmr != state.pmr;
> +	force = (system_uses_irq_prio_masking() && actual.pmr != state.pmr) ||
> +		(system_uses_nmi() && actual.allint != state.allint);
>  
>  	if (!irqs_disabled)
>  		trace_hardirqs_on();
> diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
> index eedbdf7358e6..31b8f4e99532 100644
> --- a/arch/arm64/include/asm/irqflags.h
> +++ b/arch/arm64/include/asm/irqflags.h
> @@ -31,9 +31,10 @@
>  typedef union arm64_exc_hwstate {
>  	struct {
>  		u16 daif;
> +		u16 allint;
>  		u8 pmr;
>  		u8 __pad0;
> -		u32 __pad1;
> +		u16 __pad1;
>  	};
>  	unsigned long flags;
>  } arm64_exc_hwstate_t;
> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> index cb3be770f2d0..39ea3fdeb03a 100644
> --- a/arch/arm64/kernel/entry.S
> +++ b/arch/arm64/kernel/entry.S
> @@ -815,7 +815,7 @@ SYM_CODE_END(__bp_harden_el1_vectors)
>   *
>   */
>  SYM_FUNC_START(cpu_switch_to)
> -	save_and_disable_daif x11
> +	save_and_disable_exceptions x11, x12
>  	mov	x10, #THREAD_CPU_CONTEXT
>  	add	x8, x0, x10
>  	mov	x9, sp
> @@ -839,7 +839,7 @@ SYM_FUNC_START(cpu_switch_to)
>  	ptrauth_keys_install_kernel x1, x8, x9, x10
>  	scs_save x0
>  	scs_load_current
> -	restore_irq x11
> +	restore_exceptions x11
>  	ret
>  SYM_FUNC_END(cpu_switch_to)
>  NOKPROBE(cpu_switch_to)
> @@ -866,7 +866,7 @@ NOKPROBE(ret_from_fork)
>   * Calls func(regs) using this CPU's irq stack and shadow irq stack.
>   */
>  SYM_FUNC_START(call_on_irq_stack)
> -	save_and_disable_daif x9
> +	save_and_disable_exceptions x9, x10
>  #ifdef CONFIG_SHADOW_CALL_STACK
>  	get_current_task x16
>  	scs_save x16
> @@ -881,10 +881,10 @@ SYM_FUNC_START(call_on_irq_stack)
>  
>  	/* Move to the new stack and call the function there */
>  	add	sp, x16, #IRQ_STACK_SIZE
> -	restore_irq x9
> +	restore_exceptions x9
>  	blr	x1
>  
> -	save_and_disable_daif x9
> +	save_and_disable_exceptions x9, x10
>  	/*
>  	 * Restore the SP from the FP, and restore the FP and LR from the frame
>  	 * record.
> @@ -892,7 +892,7 @@ SYM_FUNC_START(call_on_irq_stack)
>  	mov	sp, x29
>  	ldp	x29, x30, [sp], #16
>  	scs_load_current
> -	restore_irq x9
> +	restore_exceptions x9
>  	ret
>  SYM_FUNC_END(call_on_irq_stack)
>  NOKPROBE(call_on_irq_stack)

-- 
Best regards,
Jinjie

Re: [PATCH 37/38] genirq: Do not call add_interrupt_randomness() in NMI context

[Jinjie Ruan]

在 2026/9/14 18:21, Vladimir Murzin 写道:
> handle_irq_event_percpu() calls add_interrupt_randomness(). Similarly
> to 512718bbc51b ("genirq/chip: Don't call add_interrupt_randomness()
> for NMIs"), calling it while handling an NMI is not safe.
> 
> Stop calling add_interrupt_randomness() when handle_irq_event_percpu()
> runs in NMI context.
> 
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  kernel/irq/handle.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
> index b7d52821837b..114fd63b0210 100644
> --- a/kernel/irq/handle.c
> +++ b/kernel/irq/handle.c
> @@ -245,7 +245,8 @@ irqreturn_t handle_irq_event_percpu(struct irq_desc *desc)
>  
>  	retval = __handle_irq_event_percpu(desc);
>  
> -	add_interrupt_randomness(desc->irq_data.irq);
> +	if (!in_nmi())
> +		add_interrupt_randomness(desc->irq_data.irq);


Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

>  
>  	if (!irq_settings_no_debug(desc))
>  		note_interrupt(desc, retval);

-- 
Best regards,
Jinjie

Re: [PATCH 36/38] genirq: Trust local irqchip when considering IRQ for NMI

[Jinjie Ruan]

在 2026/9/14 18:21, Vladimir Murzin 写道:
> Generic IRQ code sanitises whether an IRQ can be considered for NMI
> use. Currently, it only considers IRQs managed by the root irqchip.
> 
> GICv5 IPIs belong to a logical domain built on top of the LPI domain.
> Consequently, generic IRQ code rejects these IPIs for NMI use, even
> though their hardware state is managed by the root LPI domain.
> 
> One way to allow GICv5 IPIs to be used as NMIs would be to adjust the
> sanitisation logic to account for this domain hierarchy. However, the
> generic IRQ code can instead trust the local irqchip.
> 
> Remove the root irqchip check and use the local irqchip when
> considering an IRQ for NMI use.
> 
> Suggested-by: Marc Zyngier <maz@kernel.org>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  include/linux/irq.h | 2 +-
>  kernel/irq/manage.c | 5 -----
>  2 files changed, 1 insertion(+), 6 deletions(-)
> 
> diff --git a/include/linux/irq.h b/include/linux/irq.h
> index f485369b1b4f..92d3ad639ae8 100644
> --- a/include/linux/irq.h
> +++ b/include/linux/irq.h
> @@ -563,7 +563,7 @@ struct irq_chip {
>   * IRQCHIP_ONESHOT_SAFE:              One shot does not require mask/unmask
>   * IRQCHIP_EOI_THREADED:              Chip requires eoi() on unmask in threaded mode
>   * IRQCHIP_SUPPORTS_LEVEL_MSI:        Chip can provide two doorbells for Level MSIs
> - * IRQCHIP_SUPPORTS_NMI:              Chip can deliver NMIs, only for root irqchips
> + * IRQCHIP_SUPPORTS_NMI:              Chip can deliver NMIs
>   * IRQCHIP_ENABLE_WAKEUP_ON_SUSPEND:  Invokes __enable_irq()/__disable_irq() for wake irqs
>   *                                    in the suspend path if they are in disabled state
>   * IRQCHIP_AFFINITY_PRE_STARTUP:      Default affinity update before startup
> diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
> index 57eff26fa646..c9ad79743ef6 100644
> --- a/kernel/irq/manage.c
> +++ b/kernel/irq/manage.c
> @@ -1370,11 +1370,6 @@ static bool irq_supports_nmi(struct irq_desc *desc)
>  {
>  	struct irq_data *d = irq_desc_get_irq_data(desc);
>  
> -#ifdef CONFIG_IRQ_DOMAIN_HIERARCHY
> -	/* Only IRQs directly managed by the root irqchip can be set as NMI */
> -	if (d->parent_data)
> -		return false;
> -#endif

Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

>  	/* Don't support NMIs for chips behind a slow bus */
>  	if (d->chip->irq_bus_lock || d->chip->irq_bus_sync_unlock)
>  		return false;

-- 
Best regards,
Jinjie

Re: [PATCH 30/38] arm64: smp: Abstract SGI and LPI operations

[Vladimir Murzin]

On 9/16/26 10:01, Jinjie Ruan wrote:
> 
> 在 2026/9/14 18:20, Vladimir Murzin 写道:

[snip]

>> +static void ipi_lpi_setup(int ipi, int ncpus)
>>  {
>>  	for (int cpu = 0; cpu < ncpus; cpu++) {
>>  		int err, irq;
>> @@ -1158,6 +1188,20 @@ static void ipi_setup_lpi(int ipi, int ncpus)
>>  	}
>>  }
>>  
>> +static void ipi_lpi_send(const cpumask_t *mask, unsigned int nr) {
> Sashiko pointed out a typo in the formatting.
> 
> static void ipi_lpi_send(const cpumask_t *mask, unsigned int nr)
> {
> 

Ack.

Cheers
Vladimir

Re: [PATCH 32/38] arm64: nmi: Add Kconfig for NMI

[Vladimir Murzin]

On 9/16/26 10:06, Jinjie Ruan wrote:
> 
> 在 2026/9/14 18:20, Vladimir Murzin 写道:
>> From: Mark Brown <broonie@kernel.org>
>>
>> Since NMI handling is in some fairly hot paths we provide a Kconfig
>> option which allows support to be compiled out when not needed.
>>
>> Signed-off-by: Mark Brown <broonie@kernel.org>
>> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
>> ---
>>  arch/arm64/Kconfig | 18 ++++++++++++++++++
>>  1 file changed, 18 insertions(+)
>>
>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>> index 2738637ccb92..df5ff59715d1 100644
>> --- a/arch/arm64/Kconfig
>> +++ b/arch/arm64/Kconfig
>> @@ -2251,6 +2251,24 @@ endmenu # "ARMv8.7 architectural features"
>>  config AS_HAS_MOPS
>>  	def_bool $(as-instr,.arch_extension mops)
>>  
>> +menu "ARMv8.8 architectural features"
>> +
>> +config ARM64_NMI
>> +	bool "Enable support for Non-maskable Interrupts (NMI)"
>> +	default y
> We should select HAVE_PERF_EVENTS_NMI to enable perf-NMI-based
> hardlockup detector as pseudo NMI do.
> 

It seems there are ongoing discussions [1,2] about moving away from it
towards the buddy detector. So I can add it, but I'm not sure whether
it will still be necessary in the future.

[1] https://lore.kernel.org/linux-arm-kernel/amtyQH7lzLDKE3WD@thinkstation/
[2] https://lore.kernel.org/lkml/20260902-qcom-wdt-nmi-series-v3-0-f3999362a9ea@google.com/

Thanks
Vladimir

> arch/arm64/Kconfig:     select HAVE_PERF_EVENTS_NMI if ARM64_PSEUDO_NMI
> || ARM64_NMI
> 
>> +	help
>> +	  Non-maskable interrupts are an architecture and GIC feature
>> +	  which allow the system to configure some interrupts to be
>> +	  configured to have superpriority, allowing them to be handled
>> +	  before other interrupts and masked for shorter periods of time.
>> +
>> +	  The feature is detected at runtime, and will remain disabled
>> +	  if the cpu does not implement the feature. It will also be
>> +	  disabled if pseudo NMIs are enabled at runtime.
>> +
>> +endmenu # "ARMv8.8 architectural features"
>> +
>> +
>>  menu "ARMv8.9 architectural features"
>>  
>>  config ARM64_POE
> 

Re: [PATCH 23/38] arm64: cpufeature: Detect PE support for FEAT_NMI

[Vladimir Murzin]

On 9/16/26 10:22, Jinjie Ruan wrote:
> 
> 在 2026/9/14 18:20, Vladimir Murzin 写道:
>> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>>
>> Use of FEAT_NMI requires that all the PEs in the system and the GIC
>> have NMI support. This patch implements the PE part of that detection.
>>
>> In order to avoid problematic interactions between real and pseudo NMIs
>> we disable the architected feature if the user has enabled pseudo NMIs
>> on the command line. If this is done on a system where support for the
>> architected feature is detected then a warning is printed during boot in
>> order to help users spot what is likely to be a misconfiguration.
>>
>> As KVM does not care about the host kernel supporting FEAT_NMI or not
>> to allow guests to use it, split the CPU cap in two : ARM64_HAS_NMI is
>> the hardware support, ARM64_NMI is the kernel making use of it.
>>
>> Co-developed-by: Mark Brown <broonie@kernel.org>
>> Signed-off-by: Mark Brown <broonie@kernel.org>
>> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
>> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
>> ---
>>  arch/arm64/include/asm/cpucaps.h    |  2 +
>>  arch/arm64/include/asm/cpufeature.h | 10 +++++
>>  arch/arm64/kernel/cpufeature.c      | 68 ++++++++++++++++++++++++++++-
>>  arch/arm64/tools/cpucaps            |  2 +
>>  4 files changed, 81 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
>> index 76350b38f0d7..1a4ef4e94997 100644
>> --- a/arch/arm64/include/asm/cpucaps.h
>> +++ b/arch/arm64/include/asm/cpucaps.h
>> @@ -75,6 +75,8 @@ cpucap_is_possible(const unsigned int cap)
>>  		return IS_ENABLED(CONFIG_HW_PERF_EVENTS);
>>  	case ARM64_HAS_LSUI:
>>  		return IS_ENABLED(CONFIG_ARM64_LSUI);
>> +	case ARM64_NMI:
>> +		return IS_ENABLED(CONFIG_ARM64_NMI);
>>  	}
>>  
>>  	return true;
>> diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
>> index 09aa80df602a..c8b1bacdd161 100644
>> --- a/arch/arm64/include/asm/cpufeature.h
>> +++ b/arch/arm64/include/asm/cpufeature.h
>> @@ -809,6 +809,16 @@ static __always_inline bool system_uses_irq_prio_masking(void)
>>  	return alternative_has_cap_unlikely(ARM64_HAS_GIC_PRIO_MASKING);
>>  }
>>  
>> +static __always_inline bool system_supports_nmi(void)
>> +{
>> +	return alternative_has_cap_unlikely(ARM64_HAS_NMI);
>> +}
>> +
>> +static __always_inline bool system_uses_nmi(void)
>> +{
>> +	return alternative_has_cap_unlikely(ARM64_NMI);
>> +}
>> +
>>  static inline bool system_supports_mte(void)
>>  {
>>  	return alternative_has_cap_unlikely(ARM64_MTE);
>> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
>> index 32102c3912fa..f9fe7a875f85 100644
>> --- a/arch/arm64/kernel/cpufeature.c
>> +++ b/arch/arm64/kernel/cpufeature.c
>> @@ -83,13 +83,14 @@
>>  #include <asm/cpu_ops.h>
>>  #include <asm/fpsimd.h>
>>  #include <asm/hwcap.h>
>> +#include <asm/hypervisor.h>
>>  #include <asm/insn.h>
>> +#include <asm/interrupts/common_flags.h>
>>  #include <asm/kvm_host.h>
>>  #include <asm/mmu.h>
>>  #include <asm/mmu_context.h>
>>  #include <asm/mpam.h>
>>  #include <asm/mte.h>
>> -#include <asm/hypervisor.h>
>>  #include <asm/processor.h>
>>  #include <asm/smp.h>
>>  #include <asm/sysreg.h>
>> @@ -315,6 +316,7 @@ static const struct arm64_ftr_bits ftr_id_aa64pfr1[] = {
>>  	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_GCS),
>>  		       FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_GCS_SHIFT, 4, 0),
>>  	S_ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MTE_frac_SHIFT, 4, 0),
>> +	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_NMI_SHIFT, 4, 0),
>>  	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SME),
>>  		       FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_SME_SHIFT, 4, 0),
>>  	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MPAM_frac_SHIFT, 4, 0),
>> @@ -2341,6 +2343,51 @@ static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry
>>  }
>>  #endif
>>  
>> +#ifdef CONFIG_ARM64_NMI
>> +static bool can_use_nmi(const struct arm64_cpu_capabilities *entry, int scope)
>> +{
>> +	/*
>> +	 * ARM64_HAS_NMI has a lower index, and is a boot CPU
>> +	 * feature, so will be detected earlier.
>> +	 */
>> +	BUILD_BUG_ON(ARM64_NMI <= ARM64_HAS_NMI);
>> +	if (!cpus_have_cap(ARM64_HAS_NMI))
>> +		return false;
>> +
>> +	/*
>> +	 * Having both real and pseudo NMIs enabled simultaneously is
>> +	 * likely to cause confusion.  Since pseudo NMIs must be
>> +	 * enabled with an explicit command line option, if the user
>> +	 * has set that option on a system with real NMIs for some
>> +	 * reason assume they know what they're doing.
>> +	 *
>> +	 * ARM64_HAS_GIC_PRIO_MASKING has a lower index, and is a boot CPU
>> +	 * feature, so will be detected earlier.
>> +	 */
>> +	BUILD_BUG_ON(IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) &&
>> +			(ARM64_NMI <= ARM64_HAS_GIC_PRIO_MASKING));
>> +	if (cpus_have_cap(ARM64_HAS_GIC_PRIO_MASKING)) {
>> +		pr_info_once("Pseudo NMI enabled, not using architected NMI\n");
>> +		return false;
> Hi, Vladimir,
> 
> The user's actual requirement may be to use FEAT_NMI by default when it
> is available, and fall back to pseudo-NMI otherwise, since pseudo-NMI
> has inherent noise and is only a second choice when there is no hardware
> support. Can we design it this way?
> 

Hi Jinjie,

Hmm, currently users have to explicitly opt in to pseudo-NMI (enable
the config option and pass irqchip.gicv3_pseudo_nmi=1 on the command
line), and those users would continue to use pseudo-NMI even if
FEAT_NMI is available.

What you are asking is to prefer FEAT_NMI over pseudo-NMI when it is
available, effectively turning the explicit opt-in to pseudo-NMI into
a hint rather than a strict request, correct?

Cheers
Vladimir

> Best regards,
> Jinjie
> 
>> +	}
>> +
>> +	return true;
>> +}
>> +
>> +static void nmi_enable(const struct arm64_cpu_capabilities *__unused)
>> +{
>> +	/*
>> +	 * Enable use of NMIs controlled by ALLINT, SPINTMASK should
>> +	 * be clear by default but make it explicit that we are using
>> +	 * this mode.  Ensure that ALLINT is clear first in order to
>> +	 * avoid leaving things masked.
>> +	 */
>> +	_allint_clear();
>> +	sysreg_clear_set(sctlr_el1, SCTLR_EL1_SPINTMASK, SCTLR_EL1_NMI);
>> +	isb();
>> +}
>> +#endif
>> +
>>  static bool can_trap_icv_dir_el1(const struct arm64_cpu_capabilities *entry,
>>  				 int scope)
>>  {
>> @@ -3233,6 +3280,25 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
>>  		.matches = has_cpuid_feature,
>>  		ARM64_CPUID_FIELDS(ID_AA64ISAR3_EL1, LSUI, IMP)
>>  	},
>> +#endif
>> +	{
>> +		.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
>> +		.capability = ARM64_HAS_NMI,
>> +		.matches = has_cpuid_feature,
>> +		ARM64_CPUID_FIELDS(ID_AA64PFR1_EL1, NMI, IMP)
>> +	},
>> +#ifdef CONFIG_ARM64_NMI
>> +	/*
>> +	 * Depends on ARM64_HAS_NMI
>> +	 * Checks for conflict with pseudo-NMIs, giving them priority.
>> +	 */
>> +	{
>> +		.desc = "Non-maskable Interrupts",
>> +		.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
>> +		.capability = ARM64_NMI,
>> +		.matches = can_use_nmi,
>> +		.cpu_enable = nmi_enable,
>> +	},
>>  #endif
>>  	{},
>>  };
>> diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
>> index 2775ba3359cf..531583822ec2 100644
>> --- a/arch/arm64/tools/cpucaps
>> +++ b/arch/arm64/tools/cpucaps
>> @@ -52,6 +52,7 @@ HAS_LS64_V
>>  HAS_LSUI
>>  HAS_MOPS
>>  HAS_NESTED_VIRT
>> +HAS_NMI
>>  HAS_NV2P1
>>  HAS_NV3
>>  HAS_PAN
>> @@ -82,6 +83,7 @@ MTE
>>  MTE_ASYMM
>>  MTE_FAR
>>  MTE_STORE_ONLY
>> +NMI
>>  SME
>>  SME_FA64
>>  SME2
> 

Re: [PATCH 06/38] arm64: entry: replace DAIF helpers with entry helpers

[Vladimir Murzin]

On 9/16/26 10:34, Jinjie Ruan wrote:
> 
> 在 2026/9/14 18:20, Vladimir Murzin 写道:
>> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>>
>> Replace all uses of the `local_daif_...` helpers in `entry-common.c`
>> with the new entry-specific exception masking helpers.
>>
>> Also replace `local_irq_disable()` with a switch to NOIRQ context
>> using the new helpers. This provides:
>>
>> - consistency checks for hardware state
>> - a unified style for managing exception context
>>
>> Now that both the irqflags API and entry code handle DAIF and PMR
>> correctly, remove `GIC_PRIO_PSR_I_SET` from the PMR value set by
>> `kernel_entry` in `entry.S` and by `init_gic_priority_masking()` in
>> `kernel/smp.c`.
>>
>> The `local_daif_...` helpers and other specialized code still use
>> `GIC_PRIO_PSR_I_SET` for now; this commit does not change their
>> behavior.
>>
>> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
>> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
>> ---
>>  arch/arm64/kernel/entry-common.c | 258 ++++++++++++++++++++-----------
>>  arch/arm64/kernel/entry.S        |   2 +-
>>  arch/arm64/kernel/smp.c          |   2 +-
>>  3 files changed, 173 insertions(+), 89 deletions(-)
>>
>> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
>> index 72c03ccea59f..fd0bfead6e70 100644
>> --- a/arch/arm64/kernel/entry-common.c
>> +++ b/arch/arm64/kernel/entry-common.c
>> @@ -18,11 +18,11 @@
>>  #include <linux/thread_info.h>
>>  
>>  #include <asm/cpufeature.h>
>> -#include <asm/daifflags.h>
>>  #include <asm/esr.h>
>>  #include <asm/exception.h>
>>  #include <asm/fpsimd.h>
>>  #include <asm/irq_regs.h>
>> +#include <asm/interrupts/entry.h>
>>  #include <asm/kprobes.h>
>>  #include <asm/mmu.h>
>>  #include <asm/processor.h>
>> @@ -53,9 +53,10 @@ static noinstr irqentry_state_t arm64_enter_from_kernel_mode(struct pt_regs *reg
>>   * instrumentable code, or any code which may trigger an exception.
>>   */
>>  static void noinstr __arm64_exit_to_kernel_mode(struct pt_regs *regs,
>> -						irqentry_state_t state)
>> +						irqentry_state_t state,
>> +						arm64_exc_hwstate_t hwstate)
>>  {
>> -	local_daif_mask();
>> +	arm64_mask_exc_context(hwstate);
>>  	mte_check_tfsr_exit();
>>  	irqentry_exit_to_kernel_mode_after_preempt(regs, state);
>>  }
>> @@ -64,22 +65,24 @@ static void noinstr __arm64_exit_to_kernel_mode(struct pt_regs *regs,
>>   * We are returning from the context which allows involuntary kernel preemption
>>   */
>>  static void noinstr arm64_exit_to_kernel_mode_preempt(struct pt_regs *regs,
>> -						      irqentry_state_t state)
>> +						      irqentry_state_t state,
>> +						      arm64_exc_hwstate_t hwstate)
>>  {
>>  	irqentry_exit_to_kernel_mode_preempt(regs, state);
>> -	__arm64_exit_to_kernel_mode(regs, state);
>> +	__arm64_exit_to_kernel_mode(regs, state, hwstate);
>>  }
>>  
>>  static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
>> -					      irqentry_state_t state)
>> +					      irqentry_state_t state,
>> +					      arm64_exc_hwstate_t hwstate)
>>  {
>>  	if (!regs_irqs_disabled(regs)) {
>> -		local_irq_disable();
>> -		arm64_exit_to_kernel_mode_preempt(regs, state);
>> +		hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
> Hi Vladimir,
> 
> As we discussed earlier in the link below, I think it is neither
> necessary nor correct to switch to NOIRQ_CONTEXT here; the default
> local_irq_disable() is fine.
> 

Hi Jinjie,

Switching to NOIRQ_CONTEXT *unconditionally* is indeed incorrect, but
here we do it only if !regs_irqs_disabled(regs), which implies that
DAIF is 0.  Thus, switching to NOIRQ_CONTEXT is equivalent to
local_irq_disable().

I'd prefer not to mix different APIs and to keep the switch to
NOIRQ_CONTEXT, since this provides:
- consistency checks for the hardware state
- a unified style for managing exception context

If that makes this clearer we can replace regs_irqs_disabled(regs)
with arch_irqs_disabled_flags(hwstate.flags).

Thanks
Vladimir


> Link:
> https://lore.kernel.org/all/77080aa8-7df1-43f0-8c82-1096db05f745@huawei.com/
> 
> Best regards,
> Jinjie
> 

Re: [PATCH 09/38] arm64: replace local_daif helpers

[Vladimir Murzin]

On 9/17/26 09:14, Jinjie Ruan wrote:
> 
> 在 2026/9/14 18:20, Vladimir Murzin 写道:
>> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>>
>> Replace existing `local_daif_...` usage with the new `local_all_irqs...`
>> helpers, adding the necessary save-state struct where required.
>>
>> Slightly rework `apei_claim_sea()`'s handling of interrupt masking to
>> fit with the new APIs.  Currently, it saves the DAIF flags to restore
>> later and the `local_irq` ones to check if interrupts are disabled.
>> It then masks AIF to signal the SEA (forcing DAIF masking), and will
>> unmask to IF (re-enabling PMR masking if pseudo-NMIs are in use) if it
>> interrupted a task with interrupts unmaksed, restoring with the DAIF
>> flags from the beginning afterwards.
>>
>> As we want to preserve a "mask/unmask" process, change the logic from
>> "mask AIF/unmask A/unmask IF" to "mask IF/mask A/unmask A/unmask IF" :
>> first mask with `local_irq_save()`, mask/unmask further around the
>> SEA signaling, and restore at the end with `local_irq_restore()`.
>> Now that both APIs properly work with DAIF and PMR, this works
>> as expected and also removes the need for a "save flags without masking"
>> function for the new helpers, as this was the only user.
>>
>> Remove the use of GIC_PRIO_PSR_I_SET in `kvm/hyp/nvhe/switch.c`,
>> as we now check both PMR and DAIF everywhere and we don't need to signal
>> interrupts being masked in DAIF in the PMR anymore.
>>
>> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
>> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
>> ---
>>  arch/arm64/kernel/acpi.c          | 15 +++++++--------
>>  arch/arm64/kernel/hibernate.c     | 14 +++++++-------
>>  arch/arm64/kernel/machine_kexec.c |  4 ++--
>>  arch/arm64/kernel/setup.c         |  4 ++--
>>  arch/arm64/kernel/smp.c           | 10 +++++-----
>>  arch/arm64/kernel/suspend.c       | 14 +++++++-------
>>  arch/arm64/kvm/hyp/nvhe/switch.c  |  2 +-
>>  arch/arm64/kvm/hyp/vgic-v3-sr.c   |  7 ++++---
>>  arch/arm64/kvm/hyp/vhe/switch.c   | 14 ++++++++------
>>  arch/arm64/mm/mmu.c               |  7 ++++---
>>  10 files changed, 47 insertions(+), 44 deletions(-)
>>
>> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
>> index 681aa2bbc399..ea9a9da7e412 100644
>> --- a/arch/arm64/kernel/acpi.c
>> +++ b/arch/arm64/kernel/acpi.c
>> @@ -33,7 +33,7 @@
>>  #include <acpi/processor.h>
>>  #include <asm/cputype.h>
>>  #include <asm/cpu_ops.h>
>> -#include <asm/daifflags.h>
>> +#include <asm/interrupts/masking.h>
>>  #include <asm/smp_plat.h>
>>  
>>  int acpi_noirq = 1;		/* skip ACPI IRQ initialization */
>> @@ -391,15 +391,14 @@ int apei_claim_sea(struct pt_regs *regs)
>>  	int err = -ENOENT;
>>  	bool return_to_irqs_enabled;
>>  	unsigned long current_flags;
>> +	arm64_exc_hwstates_t hwstate;
>>  
>>  	if (!IS_ENABLED(CONFIG_ACPI_APEI_GHES))
>>  		return err;
>>  
>> -	current_flags = local_daif_save_flags();
>> -
>> -	/* current_flags isn't useful here as daif doesn't tell us about pNMI */
>> -	return_to_irqs_enabled = !irqs_disabled_flags(arch_local_save_flags());
>> +	local_irq_save(current_flags);
>>  
>> +	return_to_irqs_enabled = !irqs_disabled_flags(current_flags);
>>  	if (regs)
>>  		return_to_irqs_enabled = !regs_irqs_disabled(regs);
>>  
>> @@ -407,10 +406,11 @@ int apei_claim_sea(struct pt_regs *regs)
>>  	 * SEA can interrupt SError, mask it and describe this as an NMI so
>>  	 * that APEI defers the handling.
>>  	 */
>> -	local_daif_restore(DAIF_ERRCTX);
>> +	hwstate = local_exceptions_save_mask(ERROR_CONTEXT);
>>  	nmi_enter();
>>  	err = ghes_notify_sea();
>>  	nmi_exit();
>> +	local_exceptions_restore(hwstate);
> Hi Vladimir,
> 
> I think mixing local_irq_save()/local_irq_restore() and
> local_exceptions_save_mask()/local_exceptions_restore here may be
> problematic, because arch_local_irq_save() does not take into full
> account the state of DAIF and the pseudo-NMI. For example, when the
> pseudo-NMI is enabled, arch_local_irq_save() only sets PMR to
> GIC_PRIO_IRQOFF without taking DAIF into consideration, whereas
> local_exceptions_save_mask(NOIRQ_CONTEXT) additionally clears all the
> DAIF bits.
> 
> But whereas local_exceptions_save_mask() returns exactly the current
> hwstate arch_local_irq_save() sets, and local_exceptions_restore() also
> needs to restore a unified state.
> 

Hi Jinjie,

Yeah, that nesting makes my head spin. Would it be cleaner to apply
the masking sequentially, something like:

int apei_claim_sea(struct pt_regs *regs)
{
	arm64_exc_hwstates_t hwstate;
	bool irqs_disabled;
	int err = -ENOENT;

	if (!IS_ENABLED(CONFIG_ACPI_APEI_GHES))
		return err;

	/*
	 * SEA can interrupt SError, mask it and describe this as an NMI so
	 * that APEI defers the handling.
	 */
	hwstate = local_exceptions_save_mask(ERROR_CONTEXT);
	nmi_enter();
	err = ghes_notify_sea();
	nmi_exit();
	local_exceptions_restore(hwstate);

	if (err)
		return err;

	/*
	 * APEI NMI-like notifications are deferred to irq_work. Unless
	 * we interrupted irqs-masked code, we can do that now.
	 */
	irqs_disabled = regs ? regs_irqs_disabled(regs) : arch_irqs_disabled_flags(hwstate.saved.flags);

	if (!irqs_disabled) {
		hwstate = local_exceptions_save_mask(NOIRQ_CONTEXT);
		__irq_enter();
		irq_work_run();
		__irq_exit();
		local_exceptions_restore(hwstate);
	} else {
		pr_warn_ratelimited("APEI work queued but not completed\n");
		err = -EINPROGRESS;
	}

	return err;
}


Cheers
Vladimir

> Best regards,
> Jinjie
> 
>>  
>>  	/*
>>  	 * APEI NMI-like notifications are deferred to irq_work. Unless
>> @@ -418,7 +418,6 @@ int apei_claim_sea(struct pt_regs *regs)
>>  	 */
>>  	if (!err) {
>>  		if (return_to_irqs_enabled) {
>> -			local_daif_restore(DAIF_PROCCTX_NOIRQ);
>>  			__irq_enter();
>>  			irq_work_run();
>>  			__irq_exit();
>> @@ -428,7 +427,7 @@ int apei_claim_sea(struct pt_regs *regs)
>>  		}
>>  	}
>>  
>> -	local_daif_restore(current_flags);
>> +	local_irq_restore(current_flags);
>>  
>>  	return err;
>>  }

Re: [PATCH 03/38] arm64: interrupts: Add common exception state helpers

[Vladimir Murzin]

On 9/17/26 10:10, Jinjie Ruan wrote:
> 
> 在 2026/9/14 18:20, Vladimir Murzin 写道:
>> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>>

[snip]

>> +
>> +static __always_inline
>> +arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
>> +{
>> +	arm64_exc_hwstate_t state = {.daif=regs->pstate & DAIF_MASK};
>> +
>> +	if (system_uses_irq_prio_masking())
>> +		state.pmr = regs->pmr;
>> +
>> +	return state;
>> +}
> Hi Vladimir,
> 
> This helper looks more related to the 5th patch "arm64: entry: Introduce
> entry specific exception masking helpers", as it is entry-related.
> Moving it to 5th patch may make it easier to review.
> 
> Best regards,
> Jinjie
> 

Hi Jinjie,

Sure, I'll move it there :)

Thanks
Vladimir

Re: [PATCH 05/38] arm64: entry: Introduce entry specific exception masking helpers

[Vladimir Murzin]

On 9/17/26 10:53, Jinjie Ruan wrote:
> 
> 在 2026/9/14 18:20, Vladimir Murzin 写道:
>> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>>
>> The entry code handles interrupt masking differently from the rest of
>> the kernel. Exception handlers enter and exit with all exceptions
>> masked, but they must temporarily unmask the appropriate set of
>> exceptions so that the rest of the handler executes with the expected
>> exception state.
>>
>> For EL0 handlers, this means dropping to masking context appropriate
>> for the work to be performed. For EL1 handlers, this means restoring
>> the masking context of the interrupted task. In both cases, all
>> exceptions must be masked again before returning from the exception
>> handler.
>>
>> The rest of the kernel typically follows the opposite pattern: it
>> raises the masking context to protect a critical section and later
>> restores the previous context.
>>
>> Given these different usage patterns, introduce a dedicated set of
>> exception masking helpers for the entry code. Keeping these helpers
>> separate from the generic interrupt masking APIs makes the intended
>> usage explicit and helps avoid mixing the two masking models.
>>
>> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
>> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
>> ---
>>  arch/arm64/include/asm/interrupts/entry.h | 100 ++++++++++++++++++++++
>>  1 file changed, 100 insertions(+)
>>  create mode 100644 arch/arm64/include/asm/interrupts/entry.h
>>
>> diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
>> new file mode 100644
>> index 000000000000..0cfca62aa25b
>> --- /dev/null
>> +++ b/arch/arm64/include/asm/interrupts/entry.h
>> @@ -0,0 +1,100 @@
>> +/* SPDX-License-Identifier: GPL-2.0-only */
>> +/*
>> + * Copyright (C) 2025 Arm Ltd.
>> + */
>> +#ifndef __ASM_INTERRUPTS_ENTRY_H
>> +#define __ASM_INTERRUPTS_ENTRY_H
>> +
>> +#include <asm/arch_gicv3.h>
>> +#include <asm/bug.h>
>> +#include <asm/cpufeature.h>
>> +#include <asm/interrupts/common_flags.h>
>> +
>> +
>> +static __always_inline
>> +arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
>> +						  arm64_exc_hwstate_t next)
>> +{
>> +	bool irqs_disabled = arch_irqs_disabled_flags(next.flags);
>> +	bool force;
>> +
>> +	arm64_debug_exc_hwstate(prev);
>> +
>> +	if (prev.flags == next.flags)
>> +		return next;
>> +
>> +	if (!irqs_disabled)
>> +		trace_hardirqs_on();
>> +
>> +	force = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
>> +
>> +	__arm64_update_exc_hwstate(next, force);
>> +
>> +	if (irqs_disabled)
>> +		trace_hardirqs_off();
>> +
>> +	return next;
>> +}
>> +
>> +static __always_inline
>> +arm64_exc_hwstate_t arm64_inherit_exc_context(struct pt_regs *regs)
>> +{
>> +	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
>> +	arm64_exc_hwstate_t next = arm64_inherit_exc_hwstate(regs);
> Could it be renamed to a more intuitive name, such as
> arm64_exc_hwstate_of_regs()?
> 

Ack.

>> +
>> +	return __arm64_switch_exc_hwstate_to(prev, next);
>> +}
>> +
>> +static __always_inline
>> +arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
>> +{
>> +	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
>> +
>> +	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
>> +		WARN_ON_ONCE(prev.daif < next.daif);
>> +
>> +		if (prev.daif == next.daif) {
>> +			/*
>> +			 * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so larger PMR value is weaker
>> +			 */
>> +			WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr > next.pmr);
>> +		}
> The if branch is a dead code now, we can remove it.
> 

Let compiler do the job :)

> arm64_drop_exc_context is only used directly in the context of
> el0t_64_error_handler below. It switches from ERROR_CONTEXT to
> PROCESS_CONTEXT.
> 

Such specialization goes against the idea of generalization. We don't
want to touch the helpers every time we decide to change a caller -
it's too much cognitive load to keep all these details in mind, and
they're easy to miss.

Thanks
Vladimir

> And now arm64_unmask_exc_context always switch from CRITICAL_CONTEXT
> to ERROR_CONTEXT/NONMI_CONTEXT/NOIRQ_CONTEXT/PROCESS_CONTEXT.
> 
> So there is no case where prev and next DAIF are the same.
> 
> 1009 static void noinstr __el0_error_handler_common(struct pt_regs *regs)
> 1010 {
> 1011 >-------unsigned long esr = read_sysreg(esr_el1);
> 1012 >-------arm64_exc_hwstate_t hwstate;
> 1013 >-------irqentry_state_t state;
> 1014
> 1015 >-------arm64_enter_from_user_mode(regs);
> 1016 >-------hwstate = arm64_unmask_exc_context(ERROR_CONTEXT);
> 1017 >-------state = irqentry_nmi_enter(regs);
> 1018 >-------do_serror(regs, esr);
> 1019 >-------irqentry_nmi_exit(regs, state);
> 1020 >-------hwstate = arm64_drop_exc_context(hwstate, PROCESS_CONTEXT);
> 1021 >-------arm64_exit_to_user_mode(regs, hwstate);
> 1022 >-------arm64_debug_exc_context(CRITICAL_CONTEXT);
> 1023 }
> 1024
> 1025 asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
> 1026 {
> 1027 >-------__el0_error_handler_common(regs);
> 1028 }
> 
> 
>> +	}
>> +
>> +	return __arm64_switch_exc_hwstate_to(prev, next);
>> +}
>> +
>> +static __always_inline
>> +arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
>> +{
>> +	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
>> +
>> +	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
>> +		WARN_ON_ONCE(prev.daif > next.daif);
>> +
>> +		if (prev.daif == next.daif) {
>> +			/*
>> +			 * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so smaller PMR value is stronger
>> +			 */
>> +			WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr < next.pmr);
>> +		}
>> +	}
>> +
>> +	return __arm64_switch_exc_hwstate_to(prev, next);
>> +}
>> +
>> +static __always_inline
>> +arm64_exc_hwstate_t arm64_unmask_exc_context(arm64_exc_context_t context)
>> +{
>> +	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
>> +
>> +	return arm64_drop_exc_context(prev, context);
>> +}
>> +
>> +static __always_inline
>> +arm64_exc_hwstate_t arm64_mask_exc_context(arm64_exc_hwstate_t prev)
>> +{
>> +	return arm64_lift_exc_context(prev, CRITICAL_CONTEXT);
>> +}
>> +
>> +#endif /* __ASM_INTERRUPTS_ENTRY_H */
> -- Best regards, Jinjie
> 

Re: [PATCH 26/38] arm64: nmi: Add handling of superpriority interrupts as NMIs

[Vladimir Murzin]

On 9/17/26 12:37, Jinjie Ruan wrote:
> 
> 在 2026/9/14 18:20, Vladimir Murzin 写道:
>> From: Mark Brown <broonie@kernel.org>
>>
>> Our goal with superpriority interrupts is to use them as NMIs, taking
>> advantage of the much smaller regions where they are masked to allow
>> prompt handling of the most time-critical interrupts.
>>
>> When an interrupt is configured with superpriority, we enter EL1 as we
>> do for any other interrupt. The presence of a superpriority interrupt
>> is indicated by a status bit in ISR_EL1. We check this bit before
>> unmasking interrupts in elX_interrupt(), and if a superpriority
>> interrupt is pending, we handle it as an NMI. Otherwise, normal
>> interrupts are handled as usual.
>>
>> Since superpriority interrupts are always handled as NMIs, the
>> interrupt controller can rely on in_nmi() to distinguish them from
>> ordinary interrupts.
>>
>> Enable IPIs to use superpriority interrupts as NMIs, matching the
>> existing pseudo-NMI behaviour.
>>
>> Signed-off-by: Mark Brown <broonie@kernel.org>
>> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
>> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
>> ---
>>  arch/arm64/include/asm/entry-common.h |  7 +++
>>  arch/arm64/kernel/entry-common.c      | 83 +++++++++++++++++++++++----
>>  arch/arm64/kernel/smp.c               |  2 +-
>>  3 files changed, 79 insertions(+), 13 deletions(-)
>>
>> diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
>> index 73d82a8d8e95..0681ba91ac3b 100644
>> --- a/arch/arm64/include/asm/entry-common.h
>> +++ b/arch/arm64/include/asm/entry-common.h
>> @@ -37,6 +37,13 @@ static inline bool arch_irqentry_exit_need_resched(void)
>>  	if (system_uses_irq_prio_masking() && read_sysreg(daif))
>>  		return false;
>>  
>> +	/*
>> +	 * If AllInt is set then we must have handled an NMI, so skip
>> +	 * preemption
>> +	 */
>> +	if (system_uses_nmi() && read_sysreg_s(SYS_ALLINT))
> if (system_uses_nmi() && (read_sysreg_s(SYS_ALLINT) & ALLINT_ALLINT))
> 

Ack.

Thanks
Vladimir

Re: [PATCH 24/38] arm64: nmi: Manage masking for superpriority interrupts

[Vladimir Murzin]

On 9/17/26 12:41, Jinjie Ruan wrote:
> 
> 在 2026/9/14 18:20, Vladimir Murzin 写道:
>> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>>
>> Extend logic to handle and debug exception context/state with knowlage
>> of FEAT_NMI.
>>
>> Take care to order writes to ALLINT relative to DAIF, as clearing
>> ALLINT before DAIF could result in taking an NMI while DAIF is fully
>> masked, as could setting it after DAIF.
>>
>> Since superpriority interrupts are not masked through DAIF like pseduo
>> NMIs are, we also need to modify the assembler macros for managing
>> DAIF to ensure that the masking is done in the assembly code.
>>
>> Note that save_and_disable_irq/restore_irq and
>> save_and_disable_daif/restore_irq pairs are used in distinct
>> contextes:
>>
>> - former is used in context of SW PAN to quickly disable/enable
>>   preemption
>>
>> - latter is used to completely mask all exceptions.
>>
>> For that reason split save_and_disable_daif/restore_irq into more
>> generic exception save restore pair and plumb with FEAT_NMI logic.
>>
>> Co-developed-by: Mark Brown <broonie@kernel.org>
>> Signed-off-by: Mark Brown <broonie@kernel.org>
>> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
>> ---
>>  arch/arm64/include/asm/assembler.h            | 20 +++++++++--
>>  .../include/asm/interrupts/common_flags.h     | 36 +++++++++++++++++++
>>  arch/arm64/include/asm/interrupts/entry.h     |  6 ++--
>>  arch/arm64/include/asm/interrupts/masking.h   |  3 +-
>>  arch/arm64/include/asm/irqflags.h             |  3 +-
>>  arch/arm64/kernel/entry.S                     | 12 +++----
>>  6 files changed, 68 insertions(+), 12 deletions(-)
>>
>> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
>> index 0b58b550e8dc..bcdbc308afba 100644
>> --- a/arch/arm64/include/asm/assembler.h
>> +++ b/arch/arm64/include/asm/assembler.h
>> @@ -37,11 +37,27 @@
>>  /*
>>   * Save/restore interrupts.
>>   */
>> -	.macro save_and_disable_daif, flags
>> -	mrs	\flags, daif
>> +	.macro save_and_disable_exceptions, flags, tmp
>> +	mrs     \flags, daif		// updates flags[9:6] with DAIF
>> +#ifdef CONFIG_ARM64_NMI
>> +alternative_if ARM64_NMI
>> +	mrs_s	\tmp, SYS_ALLINT	// updates tmp[13] with AllInt
>> +	msr_s	SYS_ALLINT_SET, xzr
>> +	orr	\flags, \flags, \tmp	// now flags[13,9:6] carry pair of AllInt,DAIF
>> +alternative_else_nop_endif
>> +#endif
>>  	msr	daifset, #0xf
>>  	.endm
>>  
>> +	.macro  restore_exceptions, flags
>> +	msr     daif, \flags		// bits other than flags[9:6] are ignored
>> +#ifdef CONFIG_ARM64_NMI
>> +alternative_if ARM64_NMI
>> +	msr_s	SYS_ALLINT, \flags	// bits other than flags[13] are ignored
>> +alternative_else_nop_endif
>> +#endif
>> +	.endm
>> +
>>  	.macro	save_and_disable_irq, flags
>>  	mrs	\flags, daif
>>  	msr	daifset, #3
>> diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
>> index c50c6e6473ec..18c9a392028a 100644
>> --- a/arch/arm64/include/asm/interrupts/common_flags.h
>> +++ b/arch/arm64/include/asm/interrupts/common_flags.h
>> @@ -20,6 +20,14 @@
>>  /*
>>   * Exception context mapping
>>   *
>> + * FEAT_NMI
>> + *
>> + * CRITICAL -> DAIF + AllInt (corresponds to the state on exception entry)
>> + * ERROR    ->  AIF + AllInt
>> + * NONMI    ->   IF + AllInt
>> + * NOIRQ    ->   IF
>> + * PROCESS  ->    0
>> + *
>>   * pseudo-NMI
>>   *
>>   * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
>> @@ -76,6 +84,7 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
>>  		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
>>  
>>  	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
>> +
>>  }
> An extra blank line.
> 

Ack.

>>  
>>  static __always_inline
>> @@ -84,6 +93,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
>>  	if (system_uses_irq_prio_masking())
>>  		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
>>  
>> +	if (system_uses_nmi())
>> +               return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .allint=ALLINT_ALLINT};
>> +
>>  	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
>>  }
>>  
>> @@ -93,6 +105,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
>>  	if (system_uses_irq_prio_masking())
>>  		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
>>  
>> +	if (system_uses_nmi())
>> +		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .allint=ALLINT_ALLINT};
>> +
>>  	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
>>  }
>>  
>> @@ -102,6 +117,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
>>  	if (system_uses_irq_prio_masking())
>>  		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
>>  
>> +	if (system_uses_nmi())
>> +		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .allint=ALLINT_ALLINT};
>> +
>>  	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
>>  }
>>  
>> @@ -131,6 +149,9 @@ arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
>>  	if (system_uses_irq_prio_masking())
>>  		state.pmr = regs->pmr;
>>  
>> +	if (system_uses_nmi())
>> +		state.allint = regs->pstate & PSR_ALLINT_BIT;
>> +
>>  	return state;
>>  }
>>  
>> @@ -150,6 +171,9 @@ void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
>>  	if (system_uses_irq_prio_masking()) {
>>  		WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
>>  			  actual.daif, actual.pmr, expected.daif, expected.pmr);
>> +	} else if (system_uses_nmi()) {
>> +		WARN_ONCE(1, "Unexpected DAIF+ALLINT: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
>> +			  actual.daif, actual.allint, expected.daif, expected.allint);
>>  	} else {
>>  		WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
>>  			  actual.daif, expected.daif);
>> @@ -168,6 +192,7 @@ static __always_inline
>>  void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
>>  {
>>  	bool pseudo_nmi = system_uses_irq_prio_masking();
>> +	bool nmi = system_uses_nmi();
>>  
>>  	barrier();
>>  
>> @@ -194,8 +219,19 @@ void __arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool force)
>>  		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
>>  	}
>>  
>> +	/*
>> +	 * Try to order ALLINT writes to be consistent with the DAIF state :
>> +	 * we don't want to take an NMI with DAIF masked or when it should
>> +	 * be masked but isn't yet.
>> +	 */
>> +	if (nmi && (hwstate.allint & ALLINT_ALLINT) && force)
>> +		_allint_set();
>> +
>>  	write_sysreg(hwstate.daif, daif);
>>  
>> +	if (nmi && !(hwstate.allint & ALLINT_ALLINT) && force)
>> +		_allint_clear();
>> +
>>  	if (pseudo_nmi && hwstate.pmr == GIC_PRIO_IRQON && force) {
>>  		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
>>  		pmr_sync();
>> diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
>> index 0cfca62aa25b..2a522dd15556 100644
>> --- a/arch/arm64/include/asm/interrupts/entry.h
>> +++ b/arch/arm64/include/asm/interrupts/entry.h
>> @@ -10,7 +10,6 @@
>>  #include <asm/cpufeature.h>
>>  #include <asm/interrupts/common_flags.h>
>>  
>> -
>>  static __always_inline
>>  arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
>>  						  arm64_exc_hwstate_t next)
>> @@ -26,7 +25,8 @@ arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
>>  	if (!irqs_disabled)
>>  		trace_hardirqs_on();
>>  
>> -	force = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
>> +	force = (system_uses_irq_prio_masking() && prev.pmr != next.pmr) ||
>> +		(system_uses_nmi() && prev.allint != next.allint);
>>  
>>  	__arm64_update_exc_hwstate(next, force);
>>  
>> @@ -58,6 +58,7 @@ arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
>>  			 * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so larger PMR value is weaker
>>  			 */
>>  			WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr > next.pmr);
>> +			WARN_ON_ONCE(system_uses_nmi() && prev.allint < next.allint);
> "prev.allint < next.allint" check should not be constrained by whether
> DAIF is equal, because it is possible that both DAIF and ALLINT change
> when the context is dropped, And when the context is dropped, ALLINT
> will only remain unchanged ( 0 -> 0 or 1 -> 1 )or be cleared, such as:
> 
>        ERROR    -> PROCESS
> 
> daif   AIF     ->   0
> allint ALLINT  ->   0
> 

It is correct that we can check ALLINT independently, but I'm not sure
what value that brings. Most invalid transitions would be covered by
the DAIF check alone, except for transitions between NOIRQ and NONMI,
where we have to rely on ALLINT.

Cheers
Vladimir

>>  		}
>>  	}
>>  
>> @@ -77,6 +78,7 @@ arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
>>  			 * GIC_PRIO_IRQON is larger that GIC_PRIO_IRQOFF so smaller PMR value is stronger
>>  			 */
>>  			WARN_ON_ONCE(system_uses_irq_prio_masking() && prev.pmr < next.pmr);
>> +			WARN_ON_ONCE(system_uses_nmi() && prev.allint > next.allint);
> It is the same here.
> 
> Otherwise,
> Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
> 
>>  		}
>>  	}
>>  
>> diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
>> index 8729922974b0..6c5ca5befbc1 100644
>> --- a/arch/arm64/include/asm/interrupts/masking.h
>> +++ b/arch/arm64/include/asm/interrupts/masking.h
>> @@ -30,7 +30,8 @@ arm64_exc_hwstates_t local_exceptions_save_mask(arm64_exc_context_t new)
>>  	 * We've just got actual HW state so we can rely on that to
>>  	 * optimize some unnecessary updates.
>>  	 */
>> -	force = system_uses_irq_prio_masking() && actual.pmr != state.pmr;
>> +	force = (system_uses_irq_prio_masking() && actual.pmr != state.pmr) ||
>> +		(system_uses_nmi() && actual.allint != state.allint);
>>  
>>  	if (!irqs_disabled)
>>  		trace_hardirqs_on();
>> diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
>> index eedbdf7358e6..31b8f4e99532 100644
>> --- a/arch/arm64/include/asm/irqflags.h
>> +++ b/arch/arm64/include/asm/irqflags.h
>> @@ -31,9 +31,10 @@
>>  typedef union arm64_exc_hwstate {
>>  	struct {
>>  		u16 daif;
>> +		u16 allint;
>>  		u8 pmr;
>>  		u8 __pad0;
>> -		u32 __pad1;
>> +		u16 __pad1;
>>  	};
>>  	unsigned long flags;
>>  } arm64_exc_hwstate_t;
>> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
>> index cb3be770f2d0..39ea3fdeb03a 100644
>> --- a/arch/arm64/kernel/entry.S
>> +++ b/arch/arm64/kernel/entry.S
>> @@ -815,7 +815,7 @@ SYM_CODE_END(__bp_harden_el1_vectors)
>>   *
>>   */
>>  SYM_FUNC_START(cpu_switch_to)
>> -	save_and_disable_daif x11
>> +	save_and_disable_exceptions x11, x12
>>  	mov	x10, #THREAD_CPU_CONTEXT
>>  	add	x8, x0, x10
>>  	mov	x9, sp
>> @@ -839,7 +839,7 @@ SYM_FUNC_START(cpu_switch_to)
>>  	ptrauth_keys_install_kernel x1, x8, x9, x10
>>  	scs_save x0
>>  	scs_load_current
>> -	restore_irq x11
>> +	restore_exceptions x11
>>  	ret
>>  SYM_FUNC_END(cpu_switch_to)
>>  NOKPROBE(cpu_switch_to)
>> @@ -866,7 +866,7 @@ NOKPROBE(ret_from_fork)
>>   * Calls func(regs) using this CPU's irq stack and shadow irq stack.
>>   */
>>  SYM_FUNC_START(call_on_irq_stack)
>> -	save_and_disable_daif x9
>> +	save_and_disable_exceptions x9, x10
>>  #ifdef CONFIG_SHADOW_CALL_STACK
>>  	get_current_task x16
>>  	scs_save x16
>> @@ -881,10 +881,10 @@ SYM_FUNC_START(call_on_irq_stack)
>>  
>>  	/* Move to the new stack and call the function there */
>>  	add	sp, x16, #IRQ_STACK_SIZE
>> -	restore_irq x9
>> +	restore_exceptions x9
>>  	blr	x1
>>  
>> -	save_and_disable_daif x9
>> +	save_and_disable_exceptions x9, x10
>>  	/*
>>  	 * Restore the SP from the FP, and restore the FP and LR from the frame
>>  	 * record.
>> @@ -892,7 +892,7 @@ SYM_FUNC_START(call_on_irq_stack)
>>  	mov	sp, x29
>>  	ldp	x29, x30, [sp], #16
>>  	scs_load_current
>> -	restore_irq x9
>> +	restore_exceptions x9
>>  	ret
>>  SYM_FUNC_END(call_on_irq_stack)
>>  NOKPROBE(call_on_irq_stack)
> -- Best regards, Jinjie
> 

Re: [PATCH 03/38] arm64: interrupts: Add common exception state helpers

[Ashok Raj]

On Mon, Sep 21, 2026 at 03:10:32PM +0100, Vladimir Murzin wrote:
> On 9/17/26 10:10, Jinjie Ruan wrote:
> > 
> > 在 2026/9/14 18:20, Vladimir Murzin 写道:
> >> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> >>
> 
> [snip]
> 
> >> +
> >> +static __always_inline
> >> +arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)

Minor, this isn't inheriting but rather reads a hwstate out? 
maybe arm64_exc_hwstate_of_reg() or something like that..
> >> +{
> >> +	arm64_exc_hwstate_t state = {.daif=regs->pstate & DAIF_MASK};
> >> +
> >> +	if (system_uses_irq_prio_masking())
> >> +		state.pmr = regs->pmr;
> >> +
> >> +	return state;
> >> +}

Otherwise 

Reviewed-by: Ashok Raj <ashok.raj@oss.qualcomm.com>