[RFC PATCH 00/36] arm64: Add support for FEAT_NMI

[RFC PATCH 00/36] arm64: Add support for FEAT_NMI

[Vladimir Murzin]

FEAT_NMI provides an architected mechanism for supporting non-maskable
interrupts (NMIs) and less-masked interrupts (LMIs).

Since we already support pseudo-NMIs via priority masking, introducing
another flavour of NMI on top of the existing infrastructure could
easily become messy, making the code harder to follow and reason
about.

To avoid that, this series first makes room for the new NMI "tenant"
by restructuring the existing exception masking logic.

The main idea is to separate the logical view of exception state from
its hardware representation. To achieve this, we introduce logical
exception contexts that can be mapped onto the corresponding hardware
state. This naturally consolidates the hardware-specific handling into
a small number of places, while allowing the rest of the code to
operate purely in terms of logical exception contexts.

Since this restructuring is non-trivial and carries a risk of subtle
behavioural changes, the series adds extensive debug checks to verify
that the hardware state always matches the expected logical state.

With this restructuring in place, wiring FEAT_NMI into the new
framework becomes much more manageable.

This work would not have been possible without the contributions of
Ada Couprie Diaz, Mark Brown, and Lorenzo Pieralisi - please credit
them for everything that works well. Any remaining bugs or issues are
entirely my own.

I'd especially appreciate feedback on the overall approach. Please
don't hesitate to bikeshed the naming or other details - improving
clarity is one of the main goals of this series. Of course, technical
review is more than welcome as well.

The series would also benefit from extensive testing on real hardware
(without NMI, with pseudo-NMI, and with FEAT_NMI), as most of the
testing so far has been done on QEMU and FVP.

P.S.
I'm aware that Jinjie Ruan has a similar series on the mailing
list. I'm very open to collaborating and aligning our efforts if that
makes sense.

Thanks
Vladimir

Ada Couprie Diaz (19):
  arm64: debug: don't mask DAIF for mdscr_write()
  arm64: hibernate: mask DAIF before restoring hibernated kernel
  arm64: suspend: rely on daif helpers to handle PMR
  arm64: irq: introduce a helper for GIC priority initialization
  arm64: entry: mask DAIF before returning from C EL1 handlers
  irqchip/gic-v3: make the unmasking of pseudo-NMIs explicit when
    handling IRQs
  arm64: irqflags: introduce arm64-specific irqflags type
  arm64: irqflags: save and use both DAIF and PMR
  arm64: interrupts: introduce interrupt masking helpers for entry code
  arm64: entry: replace DAIF helpers with entry helpers
  arm64: interrupts: introduce generic interrupt masking helpers
  arm64: replace local_daif helpers
  arm64: cpuidle: use new helpers to bypass interrupt priority masking
  arm64: remove daifflags.h
  arm64: gicv3: remove GIC_PRIO_PSR_I_SET
  arm64: ptrace: Add PSR_ALLINT_BIT
  arm64: cpufeature: Detect PE support for FEAT_NMI
  arm64: nmi: Manage masking for superpriority interrupts
  arm64: irq: Report FEAT_NMI masking local IRQs

Lorenzo Pieralisi (1):
  irqchip/gic-v3: Implement FEAT_GICv3_NMI support

Mark Brown (5):
  arm64: booting: Document boot requirements for FEAT_NMI
  arm64: sysreg: Add definitions for immediate versions of MSR ALLINT
  arm64: idreg: Add an override for FEAT_NMI
  arm64: nmi: Add handling of superpriority interrupts as NMIs
  arm64: nmi: Add Kconfig for NMI

Vladimir Murzin (11):
  arm64: ptrace: Remove INIT_PSTATE_EL2
  arm64: suspend: Initialize PMR on resume
  arm64: process: Use helper to check exception state
  arm64: cpufeature: Remove system_has_prio_mask_debugging()
  arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS
  arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING
  efi/runtime-wrappers: Permit architectures to override IRQ flags
    checks
  arm64/efi: Implement override for IRQ flags checks
  arm64: suspend: Always initialise PSTATE.ALLINT
  arm64/efi: Add ALLINT to IRQ flags checks
  arm64: kprobes: Disable NMIs

 Documentation/arch/arm64/booting.rst          |   6 +
 arch/arm/include/asm/arch_gicv3.h             |   6 +-
 arch/arm64/Kconfig                            |  29 +-
 arch/arm64/include/asm/arch_gicv3.h           |   7 +-
 arch/arm64/include/asm/assembler.h            |  24 +-
 arch/arm64/include/asm/cpucaps.h              |   2 +
 arch/arm64/include/asm/cpufeature.h           |  14 +-
 arch/arm64/include/asm/cpuidle.h              |  14 +-
 arch/arm64/include/asm/daifflags.h            | 144 --------
 arch/arm64/include/asm/efi.h                  |  42 ++-
 arch/arm64/include/asm/entry-common.h         |  10 +-
 .../include/asm/interrupts/common_flags.h     | 248 ++++++++++++++
 arch/arm64/include/asm/interrupts/entry.h     | 116 +++++++
 arch/arm64/include/asm/interrupts/masking.h   | 176 ++++++++++
 arch/arm64/include/asm/irqflags.h             | 148 ++++-----
 arch/arm64/include/asm/kvm_host.h             |   1 -
 arch/arm64/include/asm/mmu_context.h          |   1 -
 arch/arm64/include/asm/ptrace.h               |  17 +-
 arch/arm64/include/asm/sysreg.h               |   2 +
 arch/arm64/include/uapi/asm/ptrace.h          |   1 +
 arch/arm64/kernel/acpi.c                      |  15 +-
 arch/arm64/kernel/cpufeature.c                |  68 +++-
 arch/arm64/kernel/debug-monitors.c            |   4 -
 arch/arm64/kernel/entry-common.c              | 311 ++++++++++++------
 arch/arm64/kernel/entry.S                     |  18 +-
 arch/arm64/kernel/hibernate.c                 |  23 +-
 arch/arm64/kernel/irq.c                       |   9 +-
 arch/arm64/kernel/machine_kexec.c             |   4 +-
 arch/arm64/kernel/pi/idreg-override.c         |   1 +
 arch/arm64/kernel/probes/kprobes.c            |  24 +-
 arch/arm64/kernel/process.c                   |  24 +-
 arch/arm64/kernel/setup.c                     |   4 +-
 arch/arm64/kernel/signal.c                    |   1 -
 arch/arm64/kernel/smp.c                       |  18 +-
 arch/arm64/kernel/suspend.c                   |  26 +-
 arch/arm64/kernel/traps.c                     |   1 -
 arch/arm64/kvm/hyp/nvhe/switch.c              |   2 +-
 arch/arm64/kvm/hyp/vgic-v3-sr.c               |   7 +-
 arch/arm64/kvm/hyp/vhe/switch.c               |  12 +-
 arch/arm64/mm/fault.c                         |   1 -
 arch/arm64/mm/mmu.c                           |   7 +-
 arch/arm64/mm/proc.S                          |   7 +
 arch/arm64/tools/cpucaps                      |   2 +
 drivers/firmware/efi/runtime-wrappers.c       |  32 +-
 drivers/irqchip/irq-gic-v3.c                  | 127 +++++--
 include/linux/irqchip/arm-gic-v3-prio.h       |   8 -
 include/linux/irqchip/arm-gic-v3.h            |   4 +
 47 files changed, 1247 insertions(+), 521 deletions(-)
 delete mode 100644 arch/arm64/include/asm/daifflags.h
 create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
 create mode 100644 arch/arm64/include/asm/interrupts/entry.h
 create mode 100644 arch/arm64/include/asm/interrupts/masking.h

-- 
2.34.1

[RFC PATCH 01/36] arm64: ptrace: Remove INIT_PSTATE_EL2

[Vladimir Murzin]

Last user of INIT_PSTATE_EL2 has gone with ae4b7e38e9a9 ("arm64: Allow
sticky E2H when entering EL1"), so remove it.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/ptrace.h | 2 --
 1 file changed, 2 deletions(-)

diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index 39582511ad72..f7dc5fb9427d 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -18,8 +18,6 @@
 
 #define INIT_PSTATE_EL1 \
 	(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL1h)
-#define INIT_PSTATE_EL2 \
-	(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL2h)
 
 #include <linux/irqchip/arm-gic-v3-prio.h>
 
-- 
2.34.1

[RFC PATCH 02/36] arm64: debug: don't mask DAIF for mdscr_write()

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Masking DAIF around the write to MDSCR_EL1 doesn't do anything: we can
write to sysregs with interrupts unmasked, and writing to PSTATE is
not a context synchronization event so it does not synchronize it.

This is done in the context of a general interrupt handling cleanup,
so it does not address the missing context synchronization for the
MDSCR_EL1 write, staying consistent with the current state.  This
should be addressed in a future patch.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/kernel/debug-monitors.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/arch/arm64/kernel/debug-monitors.c b/arch/arm64/kernel/debug-monitors.c
index 29307642f4c9..e271fbac5f82 100644
--- a/arch/arm64/kernel/debug-monitors.c
+++ b/arch/arm64/kernel/debug-monitors.c
@@ -40,10 +40,7 @@ u8 debug_monitors_arch(void)
  */
 static void mdscr_write(u64 mdscr)
 {
-	unsigned long flags;
-	flags = local_daif_save();
 	write_sysreg(mdscr, mdscr_el1);
-	local_daif_restore(flags);
 }
 NOKPROBE_SYMBOL(mdscr_write);
 
-- 
2.34.1

[RFC PATCH 03/36] arm64: hibernate: mask DAIF before restoring hibernated kernel

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

The arm64 hibernate code manages the exception masking in an unsound
way, leading to potential crashes and/or warnings during resume.

When a hibernation image is saved in `swsusp_arch_suspend()`, all DAIF
exceptions are masked (by virtue of `local_daif_save()`), and the
suspended image is saved assuming that all DAIF exceptions will remain
masked when the image is restored.

When a hibernation image is resumed by `swsusp_arch_resume()`, only
interrupts are masked (by virtue of `local_irq_save()` in
`resume_target_kernel()`). When pseudo-NMI is enabled the DAIF.IF bits
will be clear, and regardless of pseudo-NMI the DAIF.DA bits will be
clear.

This means that there are two problems:

(1) It is possible to take Debug, SError, or pseudo-NMI exceptions
    during the resume process. This is unsafe, as during the resume
    process both the old ane new kernels will tranisently be in an
    inconsistent state, and swsusp_arch_suspend_exit() won't retain
    an executable mapping of any exception vectors.

    Any exception taken here will be fatal and silent.

(2) When re-entering the resumed kernel, some DAIF bits will be clear
    unexpectedly. This permits Debug, SError, or pseudo-NMI exceptions
    to be taken for a short period while the resumed kernel is not yet
    in a consistent state.

    This is detected by CONFIG_ARM64_DEBUG_PRIORITY_MASKING.

Avoid these issues by masking all DAIF exceptions during resume.

Cc: stable@vger.kernel.org
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/kernel/hibernate.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 9717568518ba..d0d9bd91e639 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -405,6 +405,7 @@ int swsusp_arch_suspend(void)
 int __nocfi swsusp_arch_resume(void)
 {
 	int rc;
+	unsigned long flags;
 	void *zero_page;
 	size_t exit_size;
 	pgd_t *tmp_pg_dir;
@@ -465,9 +466,21 @@ int __nocfi swsusp_arch_resume(void)
 	if (el2_reset_needed())
 		__hyp_set_vectors(el2_vectors);
 
+	/*
+	 * It is necessary to mask all DAIF exceptions here as:
+	 *
+	 * - The copy of swsusp_arch_suspend_exit() in the hibernation
+	 *   text cannot handle taking any exceptions.
+	 *
+	 * - The suspended kernel masked all DAIF exceptions in
+	 *   swsusp_arch_resume(), and expects to be re-entered in the
+	 *   same state : with all DAIF exceptions masked.
+	 */
+	flags = local_daif_save();
 	hibernate_exit(virt_to_phys(tmp_pg_dir), resume_hdr.ttbr1_el1,
 		       resume_hdr.reenter_kernel, restore_pblist,
 		       resume_hdr.__hyp_stub_vectors, virt_to_phys(zero_page));
+	local_daif_restore(flags);
 
 	return 0;
 }
-- 
2.34.1

[RFC PATCH 04/36] arm64: suspend: rely on daif helpers to handle PMR

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Commit 77345ef70445 ("arm64: suspend: Use cpuidle context helpers
in cpu_suspend()") added cpuidle helpers to handle PMR manipulation
and restoration to ensure that the CPU receives interrupts when suspended
and pseudo-NMIs are enabled.

However, those helpers are called in between a pair of `local_daif_save()`
and `local_daif_restore()`, which already configure the PMR as expected.
Effectively, `arm_cpuidle_save_irq_context()` is a no-op here,
even when using pseudo-NMIs, and `arm_cpuidle_restore_irq_context()`
would not restore proper interrupt masking configuration early enough
if there were unexpected changes during suspend or resume.
(This can be observed with Trusted Firmware A (TF-A) at EL3 handling
suspend through PSCI. Even though it should not be the case, TF-A can
reset `ICC_PMR_EL1` during CPU_SUSPEND, thus resuming the kernel
with an inconsistent priority mask value on hardware implementing
more than the minimum number of priority levels, such as Morello.)

Thus : remove the cpuidle context helpers as they do not do anything,
but keep the comment mentioning the need for interrupts to reach the CPU
if we are using pseudo-NMIs.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/kernel/suspend.c | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)

diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c
index eaaff94329cd..c41724a40b75 100644
--- a/arch/arm64/kernel/suspend.c
+++ b/arch/arm64/kernel/suspend.c
@@ -99,7 +99,6 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 	int ret = 0;
 	unsigned long flags;
 	struct sleep_stack_data state;
-	struct arm_cpuidle_irq_context context;
 
 	/*
 	 * Some portions of CPU state (e.g. PSTATE.{PAN,DIT}) are initialized
@@ -121,6 +120,9 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 	 * Strictly speaking the trace_hardirqs_off() here is superfluous,
 	 * hardirqs should be firmly off by now. This really ought to use
 	 * something like raw_local_daif_save().
+	 *
+	 * This also unmasks interrupts in PMR in order to reliably
+	 * resume if we're using pseudo-NMIs.
 	 */
 	flags = local_daif_save();
 
@@ -131,12 +133,6 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 	 */
 	pause_graph_tracing();
 
-	/*
-	 * Switch to using DAIF.IF instead of PMR in order to reliably
-	 * resume if we're using pseudo-NMIs.
-	 */
-	arm_cpuidle_save_irq_context(&context);
-
 	ct_cpuidle_enter();
 
 	if (__cpu_suspend_enter(&state)) {
@@ -159,8 +155,6 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 		__cpu_suspend_exit();
 	}
 
-	arm_cpuidle_restore_irq_context(&context);
-
 	unpause_graph_tracing();
 
 	/*
-- 
2.34.1

[RFC PATCH 05/36] arm64: suspend: Initialize PMR on resume

[Vladimir Murzin]

When we resume from cpu_suspend() context tracking, specially,
ct_idle_exit() performs IRQ save/restore sequence. It doesn't cause
any functional issues since we have masked all exceptions prior
suspend and have not restored them. However, in case of pseudo-NMI PMR
can be set by firmware to arbitrary value, thus IRQ save/restore
routines manipulates this arbitrary value. Again, it doesn't cause any
issues since PMR variant of IRQ save helper carries a
__pmr_irqs_disabled_flags() guard.

Going forward __pmr_irqs_disabled_flags() guard will be gone and we
will call __pmr_local_irq_disable() unconditionally - that would cause
warning in case CONFIG_ARM64_DEBUG_PRIORITY_MASKING is set.

Initialize PMR to a value known to Linux on resume until the normal
exception restore path restores the saved DAIF and PMR state.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/mm/proc.S | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 22866b49be37..06c8bc9a85d6 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -169,6 +169,13 @@ alternative_if ARM64_HAS_RAS_EXTN
 	msr_s	SYS_DISR_EL1, xzr
 alternative_else_nop_endif
 
+#ifdef CONFIG_ARM64_PSEUDO_NMI
+alternative_if ARM64_HAS_GIC_PRIO_MASKING
+	mov	x1, #GIC_PRIO_IRQON
+	msr_s	SYS_ICC_PMR_EL1, x1
+alternative_else_nop_endif
+#endif
+
 	ptrauth_keys_install_kernel_nosync x14, x1, x2, x3
 	isb
 	ret
-- 
2.34.1

[RFC PATCH 06/36] arm64: irq: introduce a helper for GIC priority initialization

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Arm64's `init_IRQ()` calls `local_daif_restore()` to synchronize
interrupt masking via DAIF and switch to masking via PMR.  This
depends on a very specific behaviour of `local_daif_restore()` which
will clear DAIF if masking interrupts via PMR, which will get removed
in the future.

As `setup_arch()` cleared DA only earlier, introduce a dedicated
helper that explicitly initializes the PMR to mask interrupts and
clears DAIF, switching to IRQ priority masking.

Given it is a dedicated helper, add a lockdep assertion as
`setup_arch()` should always have called `trace_hardirqs_off()` when
clearing DA, otherwise something bad happened.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/daifflags.h | 15 +++++++++++++++
 arch/arm64/kernel/irq.c            |  7 +++----
 2 files changed, 18 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
index 795b35128467..56341578e7e3 100644
--- a/arch/arm64/include/asm/daifflags.h
+++ b/arch/arm64/include/asm/daifflags.h
@@ -141,4 +141,19 @@ static __always_inline void local_daif_inherit(struct pt_regs *regs)
 	 */
 	write_sysreg(flags, daif);
 }
+
+/*
+ * During early boot, we unmask PSR.DA before the GIC has been set up.
+ * If we use IRQ priority masking, the PMR and PSR will be out of sync
+ * after the GIC is enabled : sync them up.
+ */
+static inline void local_interrupt_priority_init(void)
+{
+	WARN_ON(read_sysreg(daif) & PSR_A_BIT);
+	lockdep_assert_irqs_disabled();
+
+	gic_write_pmr(GIC_PRIO_IRQOFF);
+	write_sysreg(DAIF_PROCCTX, daif);
+}
+
 #endif
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index 9fafd826002b..c73faa30268d 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -126,10 +126,9 @@ void __init init_IRQ(void)
 
 	if (system_uses_irq_prio_masking()) {
 		/*
-		 * Now that we have a stack for our IRQ handler, set
-		 * the PMR/PSR pair to a consistent state.
+		 * Now that we have a stack for our IRQ handler,
+		 * let's mask interrupts via the PMR.
 		 */
-		WARN_ON(read_sysreg(daif) & PSR_A_BIT);
-		local_daif_restore(DAIF_PROCCTX_NOIRQ);
+		local_interrupt_priority_init();
 	}
 }
-- 
2.34.1

[RFC PATCH 07/36] arm64: entry: mask DAIF before returning from C EL1 handlers

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Most EL1 exceptions already call `local_daif_mask()` before returning,
with the exception of debug exception handlers which do not change
DAIF, and the IRQ/FIQ/Error handlers.

However, DAIF get masked in `kernel_exit()` in all cases when
returning from EL1 C handlers anyway.

Move this masking from assembly to C by adding `local_daif_mask()`
before `irqentry_nmi_exit()` when it is called by EL1 handlers.

Remove the `disable_daif` assembly macro, as this was its only use.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/assembler.h | 4 ----
 arch/arm64/kernel/entry-common.c   | 2 ++
 arch/arm64/kernel/entry.S          | 4 ----
 3 files changed, 2 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index effae53e9739..0b58b550e8dc 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -34,10 +34,6 @@
 	wx\n	.req	w\n
 	.endr
 
-	.macro disable_daif
-	msr	daifset, #0xf
-	.endm
-
 /*
  * Save/restore interrupts.
  */
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index ceb4eb11232a..2be42d7f4eaa 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -495,6 +495,7 @@ static __always_inline void __el1_pnmi(struct pt_regs *regs,
 
 	state = irqentry_nmi_enter(regs);
 	do_interrupt_handler(regs, handler);
+	local_daif_mask();
 	irqentry_nmi_exit(regs, state);
 }
 
@@ -540,6 +541,7 @@ asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
 	local_daif_restore(DAIF_ERRCTX);
 	state = irqentry_nmi_enter(regs);
 	do_serror(regs, esr);
+	local_daif_mask();
 	irqentry_nmi_exit(regs, state);
 }
 
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index e0db14e9c843..f63049ac32dc 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -333,10 +333,6 @@ alternative_else_nop_endif
 	.endm
 
 	.macro	kernel_exit, el
-	.if	\el != 0
-	disable_daif
-	.endif
-
 #ifdef CONFIG_ARM64_PSEUDO_NMI
 alternative_if_not ARM64_HAS_GIC_PRIO_MASKING
 	b	.Lskip_pmr_restore\@
-- 
2.34.1

[RFC PATCH 08/36] irqchip/gic-v3: make the unmasking of pseudo-NMIs explicit when handling IRQs

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

`gic_arch_enable_irqs()` is only used when handling IRQs (which could
be pseudo-NMIs) and unmasking pseudo-NMIs.

The chain of `gic_pmr_mask_irqs()` and `gic_arch_enable_irqs()` for it
is slightly confusing without further explanation.

Remove `gic_arch_enable_irqs()` and instead do the whole pseudo-NMI
umasking in `gic_unmask_pnmis()`, making the operation explicit.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm/include/asm/arch_gicv3.h     | 6 +-----
 arch/arm64/include/asm/arch_gicv3.h   | 7 +++++--
 arch/arm64/include/asm/entry-common.h | 2 +-
 drivers/irqchip/irq-gic-v3.c          | 5 +----
 4 files changed, 8 insertions(+), 12 deletions(-)

diff --git a/arch/arm/include/asm/arch_gicv3.h b/arch/arm/include/asm/arch_gicv3.h
index 847590df7551..d4ac8d3271b1 100644
--- a/arch/arm/include/asm/arch_gicv3.h
+++ b/arch/arm/include/asm/arch_gicv3.h
@@ -246,11 +246,7 @@ static inline void gic_pmr_mask_irqs(void)
 	WARN_ON_ONCE(true);
 }
 
-static inline void gic_arch_enable_irqs(void)
-{
-	/* Should not get called. */
-	WARN_ON_ONCE(true);
-}
+static inline void gic_unmask_pnmis(void) {}
 
 static inline bool gic_has_relaxed_pmr_sync(void)
 {
diff --git a/arch/arm64/include/asm/arch_gicv3.h b/arch/arm64/include/asm/arch_gicv3.h
index d20b03931a8d..3dcb7b8309d9 100644
--- a/arch/arm64/include/asm/arch_gicv3.h
+++ b/arch/arm64/include/asm/arch_gicv3.h
@@ -178,9 +178,12 @@ static inline void gic_pmr_mask_irqs(void)
 	gic_write_pmr(GIC_PRIO_IRQOFF);
 }
 
-static inline void gic_arch_enable_irqs(void)
+static inline void gic_unmask_pnmis(void)
 {
-	asm volatile ("msr daifclr, #3" : : : "memory");
+	if (gic_prio_masking_enabled()) {
+		gic_pmr_mask_irqs();
+		asm volatile ("msr daifclr, #3" : : : "memory");
+	}
 }
 
 static inline bool gic_has_relaxed_pmr_sync(void)
diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
index cab8cd78f693..1905765159aa 100644
--- a/arch/arm64/include/asm/entry-common.h
+++ b/arch/arm64/include/asm/entry-common.h
@@ -32,7 +32,7 @@ static inline bool arch_irqentry_exit_need_resched(void)
 	/*
 	 * DAIF.DA are cleared at the start of IRQ/FIQ handling, and when GIC
 	 * priority masking is used the GIC irqchip driver will clear DAIF.IF
-	 * using gic_arch_enable_irqs() for normal IRQs. If anything is set in
+	 * in gic_unmask_pnmis() for normal IRQs. If anything is set in
 	 * DAIF we must have handled an NMI, so skip preemption.
 	 */
 	if (system_uses_irq_prio_masking() && read_sysreg(daif))
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 99444a1b2ffa..94c6a3f2b009 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -867,10 +867,7 @@ static void __gic_handle_irq_from_irqson(struct pt_regs *regs)
 		nmi_exit();
 	}
 
-	if (gic_prio_masking_enabled()) {
-		gic_pmr_mask_irqs();
-		gic_arch_enable_irqs();
-	}
+	gic_unmask_pnmis();
 
 	if (!is_nmi)
 		__gic_handle_irq(irqnr, regs);
-- 
2.34.1

[RFC PATCH 09/36] arm64: irqflags: introduce arm64-specific irqflags type

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

With pseudo-NMIs enabled, we have two mechanisms that control
interrupt masking in parallel :
- The DAIF flags, masking at the CPU
- The GIC PMR, masking before the CPU

However, our irqflags implementation currently assumes that only one
of the two is used at a time, so both DAIF and PMR masking use the same
`unsigned long flags` in their own way.
This is incorrect, as some parts of the kernel will mask interrupts
with DAIF directly or bypass the local_irq masking via the PMR,
and makes tracking the state and changes of both in parallel impossible.

The irqflags API expects `unsigned long`s to be passed around, but
they should not be manipulated outside of the arch-specific code.
So, we can encode the information we need however we want as long as
we return and accept `unsigned long`s.

Introduce a union type for arm64 irqflags whose first member is
a struct allowing us to track DAIF and PMR in parallel, and the second
is the `unsigned long` expected by the irqflags API.

DAIF is a two byte value, to maintain compatibility with existing defines.
PMR is a one byte value, which is the maximum amount of priority bits
allowed by the GICv3 architecture.

Update the internal irqflags functions to use this new union and convert
back and forth with the irqflags unsigned long.
There should be no functional changes.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/irqflags.h | 80 ++++++++++++++++++++-----------
 1 file changed, 53 insertions(+), 27 deletions(-)

diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index a8cb5a5c93b7..7775904ba6a9 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -9,6 +9,8 @@
 #include <asm/ptrace.h>
 #include <asm/sysreg.h>
 
+#include <linux/compiler.h>
+
 /*
  * Aarch64 has flags for masking: Debug, Asynchronous (serror), Interrupts and
  * FIQ exceptions, in the 'daif' register. We mask and unmask them in 'daif'
@@ -20,6 +22,22 @@
  * exceptions should be unmasked.
  */
 
+ /*
+  * Internally, we want to independently manipulate and track the different
+  * interrupt masking mechanisms.
+  * Externally, the generic irqflags API expects unsgined longs to represent
+  * the state of interrupts, which are treated as obscure arch-specific data.
+  */
+typedef union arm64_exc_hwstate {
+	struct {
+		u16 daif;
+		u8 pmr;
+	};
+	unsigned long flags;
+} arm64_exc_hwstate_t;
+
+static_assert(sizeof(arm64_exc_hwstate_t) == sizeof(unsigned long));
+
 static __always_inline void __daif_local_irq_enable(void)
 {
 	barrier();
@@ -77,14 +95,14 @@ static __always_inline void arch_local_irq_disable(void)
 	}
 }
 
-static __always_inline unsigned long __daif_local_save_flags(void)
+static __always_inline arm64_exc_hwstate_t __daif_local_save_flags(void)
 {
-	return read_sysreg(daif);
+	return (arm64_exc_hwstate_t){ .daif = read_sysreg(daif) };
 }
 
-static __always_inline unsigned long __pmr_local_save_flags(void)
+static __always_inline arm64_exc_hwstate_t __pmr_local_save_flags(void)
 {
-	return read_sysreg_s(SYS_ICC_PMR_EL1);
+	return (arm64_exc_hwstate_t){ .pmr = read_sysreg_s(SYS_ICC_PMR_EL1) };
 }
 
 /*
@@ -93,28 +111,32 @@ static __always_inline unsigned long __pmr_local_save_flags(void)
 static __always_inline unsigned long arch_local_save_flags(void)
 {
 	if (system_uses_irq_prio_masking()) {
-		return __pmr_local_save_flags();
+		return __pmr_local_save_flags().flags;
 	} else {
-		return __daif_local_save_flags();
+		return __daif_local_save_flags().flags;
 	}
 }
 
-static __always_inline bool __daif_irqs_disabled_flags(unsigned long flags)
+static __always_inline
+bool __daif_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
 {
-	return flags & PSR_I_BIT;
+	return hwstate.daif & PSR_I_BIT;
 }
 
-static __always_inline bool __pmr_irqs_disabled_flags(unsigned long flags)
+static __always_inline
+bool __pmr_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
 {
-	return flags != GIC_PRIO_IRQON;
+	return hwstate.pmr != GIC_PRIO_IRQON;
 }
 
 static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
 {
+	arm64_exc_hwstate_t hwstate = { .flags = flags };
+
 	if (system_uses_irq_prio_masking()) {
-		return __pmr_irqs_disabled_flags(flags);
+		return __pmr_irqs_disabled_flags(hwstate);
 	} else {
-		return __daif_irqs_disabled_flags(flags);
+		return __daif_irqs_disabled_flags(hwstate);
 	}
 }
 
@@ -137,49 +159,51 @@ static __always_inline bool arch_irqs_disabled(void)
 	}
 }
 
-static __always_inline unsigned long __daif_local_irq_save(void)
+static __always_inline arm64_exc_hwstate_t __daif_local_irq_save(void)
 {
-	unsigned long flags = __daif_local_save_flags();
+	arm64_exc_hwstate_t hwstate = __daif_local_save_flags();
 
 	__daif_local_irq_disable();
 
-	return flags;
+	return hwstate;
 }
 
-static __always_inline unsigned long __pmr_local_irq_save(void)
+static __always_inline arm64_exc_hwstate_t __pmr_local_irq_save(void)
 {
-	unsigned long flags = __pmr_local_save_flags();
+	arm64_exc_hwstate_t hwstate = __pmr_local_save_flags();
 
 	/*
 	 * There are too many states with IRQs disabled, just keep the current
 	 * state if interrupts are already disabled/masked.
 	 */
-	if (!__pmr_irqs_disabled_flags(flags))
+	if (!__pmr_irqs_disabled_flags(hwstate))
 		__pmr_local_irq_disable();
 
-	return flags;
+	return hwstate;
 }
 
 static __always_inline unsigned long arch_local_irq_save(void)
 {
 	if (system_uses_irq_prio_masking()) {
-		return __pmr_local_irq_save();
+		return __pmr_local_irq_save().flags;
 	} else {
-		return __daif_local_irq_save();
+		return __daif_local_irq_save().flags;
 	}
 }
 
-static __always_inline void __daif_local_irq_restore(unsigned long flags)
+static __always_inline
+void __daif_local_irq_restore(arm64_exc_hwstate_t hwstate)
 {
 	barrier();
-	write_sysreg(flags, daif);
+	write_sysreg(hwstate.daif, daif);
 	barrier();
 }
 
-static __always_inline void __pmr_local_irq_restore(unsigned long flags)
+static __always_inline
+void __pmr_local_irq_restore(arm64_exc_hwstate_t hwstate)
 {
 	barrier();
-	write_sysreg_s(flags, SYS_ICC_PMR_EL1);
+	write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
 	pmr_sync();
 	barrier();
 }
@@ -189,10 +213,12 @@ static __always_inline void __pmr_local_irq_restore(unsigned long flags)
  */
 static __always_inline void arch_local_irq_restore(unsigned long flags)
 {
+	arm64_exc_hwstate_t hwstate = { .flags = flags };
+
 	if (system_uses_irq_prio_masking()) {
-		__pmr_local_irq_restore(flags);
+		__pmr_local_irq_restore(hwstate);
 	} else {
-		__daif_local_irq_restore(flags);
+		__daif_local_irq_restore(hwstate);
 	}
 }
 
-- 
2.34.1

[RFC PATCH 10/36] arm64: irqflags: save and use both DAIF and PMR

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

With pseudo-NMIs enabled, both DAIF and the PMR affect interrupt masking.
Now that we have a type which can track both of them at the same time,
update our irqflags implementation to use it.

Save DAIF flags in all cases, as they can be manipulated directly by other
code, and the PMR if it is in use.

When checking if IRQs are disabled, now that we always save DAIF we can
check that the I flag is set and bypass checking the PMR if it is.
We can also properly check if PMR masks interrupts (PMR < GIC_PRIO_IRQON),
now that we don't need to rely on the GIC_PRIO_PSR_I_SET bit being set in
the PMR to know if DAIF is already masking interrupts.
Update `irqs_priority_unmasked()` to align with this change.

This allows us to remove the `__daif_...` and `__pmr_...` versions
of the save and check functions, as they are now unified.

We can reasonably merge the two `__{daif,pmr}_irq_restore()` functions
in the main one, as the DAIF and PMR values are properly split now.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/irqflags.h | 110 ++++++------------------------
 arch/arm64/include/asm/ptrace.h   |   2 +-
 2 files changed, 23 insertions(+), 89 deletions(-)

diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index 7775904ba6a9..62f047702493 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -95,117 +95,48 @@ static __always_inline void arch_local_irq_disable(void)
 	}
 }
 
-static __always_inline arm64_exc_hwstate_t __daif_local_save_flags(void)
-{
-	return (arm64_exc_hwstate_t){ .daif = read_sysreg(daif) };
-}
-
-static __always_inline arm64_exc_hwstate_t __pmr_local_save_flags(void)
-{
-	return (arm64_exc_hwstate_t){ .pmr = read_sysreg_s(SYS_ICC_PMR_EL1) };
-}
-
 /*
  * Save the current interrupt enable state.
  */
 static __always_inline unsigned long arch_local_save_flags(void)
 {
-	if (system_uses_irq_prio_masking()) {
-		return __pmr_local_save_flags().flags;
-	} else {
-		return __daif_local_save_flags().flags;
-	}
-}
+	arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
 
-static __always_inline
-bool __daif_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
-{
-	return hwstate.daif & PSR_I_BIT;
-}
+	if (system_uses_irq_prio_masking())
+		hwstate.pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
 
-static __always_inline
-bool __pmr_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
-{
-	return hwstate.pmr != GIC_PRIO_IRQON;
+	return hwstate.flags;
 }
 
 static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
 {
 	arm64_exc_hwstate_t hwstate = { .flags = flags };
 
-	if (system_uses_irq_prio_masking()) {
-		return __pmr_irqs_disabled_flags(hwstate);
-	} else {
-		return __daif_irqs_disabled_flags(hwstate);
-	}
-}
+	/* If I is set, the PMR doesn't matter: interrupts will not be taken. */
+	if (hwstate.daif & PSR_I_BIT)
+		return true;
 
-static __always_inline bool __daif_irqs_disabled(void)
-{
-	return __daif_irqs_disabled_flags(__daif_local_save_flags());
-}
+	if (system_uses_irq_prio_masking() && hwstate.pmr < GIC_PRIO_IRQON)
+		return true;
 
-static __always_inline bool __pmr_irqs_disabled(void)
-{
-	return __pmr_irqs_disabled_flags(__pmr_local_save_flags());
+	return false;
 }
 
 static __always_inline bool arch_irqs_disabled(void)
 {
-	if (system_uses_irq_prio_masking()) {
-		return __pmr_irqs_disabled();
-	} else {
-		return __daif_irqs_disabled();
-	}
-}
-
-static __always_inline arm64_exc_hwstate_t __daif_local_irq_save(void)
-{
-	arm64_exc_hwstate_t hwstate = __daif_local_save_flags();
-
-	__daif_local_irq_disable();
-
-	return hwstate;
-}
-
-static __always_inline arm64_exc_hwstate_t __pmr_local_irq_save(void)
-{
-	arm64_exc_hwstate_t hwstate = __pmr_local_save_flags();
-
-	/*
-	 * There are too many states with IRQs disabled, just keep the current
-	 * state if interrupts are already disabled/masked.
-	 */
-	if (!__pmr_irqs_disabled_flags(hwstate))
-		__pmr_local_irq_disable();
-
-	return hwstate;
+	return arch_irqs_disabled_flags(arch_local_save_flags());
 }
 
 static __always_inline unsigned long arch_local_irq_save(void)
 {
-	if (system_uses_irq_prio_masking()) {
-		return __pmr_local_irq_save().flags;
-	} else {
-		return __daif_local_irq_save().flags;
-	}
-}
+	unsigned long flags = arch_local_save_flags();
 
-static __always_inline
-void __daif_local_irq_restore(arm64_exc_hwstate_t hwstate)
-{
-	barrier();
-	write_sysreg(hwstate.daif, daif);
-	barrier();
-}
+	if (system_uses_irq_prio_masking())
+		__pmr_local_irq_disable();
+	else
+		__daif_local_irq_disable();
 
-static __always_inline
-void __pmr_local_irq_restore(arm64_exc_hwstate_t hwstate)
-{
-	barrier();
-	write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
-	pmr_sync();
-	barrier();
+	return flags;
 }
 
 /*
@@ -215,11 +146,14 @@ static __always_inline void arch_local_irq_restore(unsigned long flags)
 {
 	arm64_exc_hwstate_t hwstate = { .flags = flags };
 
+	barrier();
 	if (system_uses_irq_prio_masking()) {
-		__pmr_local_irq_restore(hwstate);
+		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
+		pmr_sync();
 	} else {
-		__daif_local_irq_restore(hwstate);
+		write_sysreg(hwstate.daif, daif);
 	}
+	barrier();
 }
 
 #endif /* __ASM_IRQFLAGS_H */
diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index f7dc5fb9427d..192eb97cd50b 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -205,7 +205,7 @@ static inline void forget_syscall(struct pt_regs *regs)
 
 #define irqs_priority_unmasked(regs)					\
 	(system_uses_irq_prio_masking() ?				\
-		(regs)->pmr == GIC_PRIO_IRQON :				\
+		(regs)->pmr >= GIC_PRIO_IRQON :				\
 		true)
 
 static __always_inline bool regs_irqs_disabled(const struct pt_regs *regs)
-- 
2.34.1

[RFC PATCH 11/36] arm64: interrupts: introduce interrupt masking helpers for entry code

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

The entry code handles interrupt masking differently from the rest of
the kernel. Exception handlers enter and exit with all exceptions
masked, but they must temporarily unmask the appropriate set of
exceptions so that the rest of the handler executes with the expected
exception state.

For EL0 handlers, this means dropping to masking context appropriate
for the work to be performed. For EL1 handlers, this means restoring
the masking context of the interrupted task. In both cases, all
exceptions must be masked again before returning from the exception
handler.

The rest of the kernel typically follows the opposite pattern: it
raises the masking context to protect a critical section and later
restores the previous context.

Given these different usage patterns, introduce a dedicated set of
exception masking helpers for the entry code. Keeping these helpers
separate from the generic interrupt masking APIs makes the intended
usage explicit and helps avoid mixing the two masking models.

To make the masking logic easier to reason about, introduce exception
contexts that map directly to the corresponding hardware exception
state. Along with these contexts, provide helpers to:

- translate an exception context into the corresponding hardware
  state,

- verify that the current hardware exception state matches the
  expected exception context,

- raise or lower the current exception context, and

- perform the common mask/unmask operations when the starting or
  target exception context is already known.

Tracking the current exception context also provides two additional
benefits:

- improved debugging by verifying that the hardware exception state
  matches the expected exception context, and

- avoiding unnecessary writes to the hardware exception state.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 .../include/asm/interrupts/common_flags.h     | 197 ++++++++++++++++++
 arch/arm64/include/asm/interrupts/entry.h     | 110 ++++++++++
 2 files changed, 307 insertions(+)
 create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
 create mode 100644 arch/arm64/include/asm/interrupts/entry.h

diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
new file mode 100644
index 000000000000..6ce60d1519e8
--- /dev/null
+++ b/arch/arm64/include/asm/interrupts/common_flags.h
@@ -0,0 +1,197 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2025 Arm Ltd.
+ */
+#ifndef __ASM_INTERRUPTS_COMMON_FLAGS_H
+#define __ASM_INTERRUPTS_COMMON_FLAGS_H
+
+#include <asm/arch_gicv3.h>
+#include <asm/bug.h>
+#include <asm/cpufeature.h>
+#include <asm/ptrace.h>
+#include <asm/sysreg.h>
+#include <asm/irqflags.h>
+
+#define DAIF_PROCCTX		0
+#define DAIF_PROCCTX_NOIRQ	(PSR_I_BIT | PSR_F_BIT)
+#define DAIF_ERRCTX		(PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
+#define DAIF_MASK		(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
+
+/*
+ * Exception context mapping
+ *
+ * pseudo-NMI
+ *
+ * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
+ * ERROR    ->  AIF + IRQON
+ * NONMI    ->   IF + IRQON
+ * NOIRQ    ->    0 + IRQOFF
+ * PROCESS  ->    0 + IRQON
+ *
+ * Otherwise
+ *
+ * CRITICAL -> DAIF           (corresponds to the state on exception entry)
+ * ERROR    ->  AIF
+ * NONMI    ->   IF
+ * NOIRQ    ->   IF
+ * PROCESS  ->    0
+ */
+typedef enum arm64_exc_context {
+	PROCESS_CONTEXT,
+	NOIRQ_CONTEXT,
+	NONMI_CONTEXT,
+	ERROR_CONTEXT,
+	CRITICAL_CONTEXT,
+} arm64_exc_context_t;
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_process_context(void)
+{
+	if (system_uses_irq_prio_masking())
+		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQON};
+
+	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX};
+}
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
+{
+	if (system_uses_irq_prio_masking())
+		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
+
+	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
+}
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
+{
+	if (system_uses_irq_prio_masking())
+		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
+
+	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
+}
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
+{
+	if (system_uses_irq_prio_masking())
+		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
+
+	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
+}
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
+{
+	if (system_uses_irq_prio_masking())
+		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
+
+	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_exc_hwstate_of_context(arm64_exc_context_t context) {
+	switch (context) {
+	case PROCESS_CONTEXT:
+		return __arm64_exc_hwstate_of_process_context();
+	case NOIRQ_CONTEXT:
+		return __arm64_exc_hwstate_of_noirq_context();
+	case NONMI_CONTEXT:
+		return __arm64_exc_hwstate_of_nonmi_context();
+	case ERROR_CONTEXT:
+		return __arm64_exc_hwstate_of_error_context();
+	case CRITICAL_CONTEXT:
+		return __arm64_exc_hwstate_of_critical_context();
+	default:
+		BUG();
+	}
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
+{
+	arm64_exc_hwstate_t state = {.daif=regs->pstate & DAIF_MASK};
+
+	if (system_uses_irq_prio_masking())
+		state.pmr = regs->pmr;
+
+	return state;
+}
+
+static __always_inline
+void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
+{
+	arm64_exc_hwstate_t actual;
+
+	if (!IS_ENABLED(CONFIG_DEBUG_IRQFLAGS))
+		return;
+
+	actual.flags = arch_local_save_flags();
+
+	if (expected.flags == actual.flags)
+		return;
+
+	if (system_uses_irq_prio_masking()) {
+		WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
+			  actual.daif, actual.pmr, expected.daif, expected.pmr);
+	} else {
+		WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
+			  actual.daif, expected.daif);
+	}
+}
+
+static __always_inline
+void arm64_debug_exc_context(arm64_exc_context_t context)
+{
+	arm64_exc_hwstate_t expected = arm64_exc_hwstate_of_context(context);
+
+	arm64_debug_exc_hwstate(expected);
+}
+
+static __always_inline
+void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
+{
+	if (system_uses_irq_prio_masking() &&
+	    update_pmr &&
+	    hwstate.pmr == GIC_PRIO_IRQOFF) {
+		/*
+		 * There has been concern that the write to daif
+		 * might be reordered before this write to PMR.
+		 * From the ARM ARM DDI 0487D.a, section D1.7.1
+		 * "Accessing PSTATE fields":
+		 *   Writes to the PSTATE fields have side-effects on
+		 *   various aspects of the PE operation. All of these
+		 *   side-effects are guaranteed:
+		 *     - Not to be visible to earlier instructions in
+		 *       the execution stream.
+		 *     - To be visible to later instructions in the
+		 *       execution stream
+		 *
+		 * Also, writes to PMR are self-synchronizing, so no
+		 * interrupts with a lower priority than PMR is signaled
+		 * to the PE after the write.
+		 *
+		 * So we don't need additional synchronization here.
+		 */
+		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
+		pmr_sync();
+	}
+
+	write_sysreg(hwstate.daif, daif);
+
+	if (system_uses_irq_prio_masking() &&
+	    update_pmr &&
+	    hwstate.pmr == GIC_PRIO_IRQON) {
+		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
+		pmr_sync();
+	}
+}
+
+static __always_inline
+void arm64_update_exc_context(arm64_exc_context_t context, bool update_pmr)
+{
+	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(context);
+
+	arm64_update_exc_hwstate(hwstate, update_pmr);
+}
+#endif /* __ASM_INTERRUPTS_COMMON_FLAGS_H */
diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
new file mode 100644
index 000000000000..3034c490ed66
--- /dev/null
+++ b/arch/arm64/include/asm/interrupts/entry.h
@@ -0,0 +1,110 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2025 Arm Ltd.
+ */
+#ifndef __ASM_INTERRUPTS_ENTRY_H
+#define __ASM_INTERRUPTS_ENTRY_H
+
+#include <asm/arch_gicv3.h>
+#include <asm/bug.h>
+#include <asm/cpufeature.h>
+#include <asm/interrupts/common_flags.h>
+
+
+static __always_inline
+arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
+						  arm64_exc_hwstate_t next)
+{
+	bool update_pmr = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
+
+	arm64_debug_exc_hwstate(prev);
+
+	if (prev.flags == next.flags)
+		return next;
+
+	if (!arch_irqs_disabled_flags(next.flags))
+		trace_hardirqs_on();
+
+	arm64_update_exc_hwstate(next, update_pmr);
+
+	if (arch_irqs_disabled_flags(next.flags))
+		trace_hardirqs_off();
+
+	return next;
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_inherit_exc_context(struct pt_regs *regs)
+{
+	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
+	arm64_exc_hwstate_t next = arm64_inherit_exc_hwstate(regs);
+
+	return __arm64_switch_exc_hwstate_to(prev, next);
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
+{
+	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
+
+	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
+		bool pnmi = system_uses_irq_prio_masking();
+
+		WARN_ON_ONCE(context > ERROR_CONTEXT &&
+			     prev.daif == DAIF_ERRCTX);
+
+		WARN_ON_ONCE(context > NONMI_CONTEXT &&
+			     prev.daif == DAIF_PROCCTX_NOIRQ);
+
+		WARN_ON_ONCE(context > NOIRQ_CONTEXT &&
+			     pnmi && prev.pmr == GIC_PRIO_IRQOFF);
+
+		WARN_ON_ONCE(context > PROCESS_CONTEXT &&
+			     ((pnmi && prev.daif == DAIF_PROCCTX && prev.pmr == GIC_PRIO_IRQON) ||
+			      (!pnmi && prev.daif == DAIF_PROCCTX)));
+	}
+
+	return __arm64_switch_exc_hwstate_to(prev, next);
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
+{
+	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
+
+	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
+		bool pnmi = system_uses_irq_prio_masking();
+
+		WARN_ON_ONCE(context < CRITICAL_CONTEXT &&
+			     prev.daif == DAIF_MASK);
+
+		WARN_ON_ONCE(context < ERROR_CONTEXT &&
+			     prev.daif == DAIF_ERRCTX);
+
+		WARN_ON_ONCE(context < NONMI_CONTEXT &&
+			     pnmi && prev.daif == DAIF_PROCCTX_NOIRQ);
+
+		WARN_ON_ONCE(context < NOIRQ_CONTEXT &&
+			     ((pnmi && prev.pmr == GIC_PRIO_IRQOFF) ||
+			      (!pnmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
+	}
+
+	return __arm64_switch_exc_hwstate_to(prev, next);
+}
+
+
+static __always_inline
+arm64_exc_hwstate_t arm64_unmask_exc_context(arm64_exc_context_t context)
+{
+	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
+
+	return arm64_drop_exc_context(prev, context);
+}
+
+static __always_inline
+arm64_exc_hwstate_t arm64_mask_exc_context(arm64_exc_hwstate_t prev)
+{
+	return arm64_lift_exc_context(prev, CRITICAL_CONTEXT);
+}
+
+#endif /* __ASM_INTERRUPTS_ENTRY_H */
-- 
2.34.1

[RFC PATCH 12/36] arm64: entry: replace DAIF helpers with entry helpers

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Replace all uses of the `local_daif_...` helpers in `entry-common.c`
with the new entry-specific exception masking helpers.

Also replace `local_irq_disable()` with a switch to NOIRQ context
using the new helpers. This provides:

- consistency checks for hardware state

- a unified style for managing exception context

Now that exception context is tracked explicitly, involuntary kernel
preemption can be optimized. We either preempt:

- from PROCESS exception context, or

- when returning from an interrupt handler, where the GIC code
  switches from NONMI to NOIRQ exception context

To support this split, divide `arm64_exit_to_kernel_mode()` into
preempt, non-preempt, and dispatch parts.

Remove `local_daif_inherit()`, since it is no longer used.

Now that both the irqflags API and entry code handle DAIF and PMR
correctly, remove `GIC_PRIO_PSR_I_SET` from the PMR value set by
`kernel_entry` in `entry.S` and by `init_gic_priority_masking()` in
`kernel/smp.c`.

The `local_daif_...` helpers and other specialized code still use
`GIC_PRIO_PSR_I_SET` for now; this commit does not change their
behavior.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/daifflags.h |  22 ---
 arch/arm64/kernel/entry-common.c   | 272 ++++++++++++++++++++---------
 arch/arm64/kernel/entry.S          |   2 +-
 arch/arm64/kernel/smp.c            |   2 +-
 4 files changed, 190 insertions(+), 108 deletions(-)

diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
index 56341578e7e3..6b14ec4d4dbe 100644
--- a/arch/arm64/include/asm/daifflags.h
+++ b/arch/arm64/include/asm/daifflags.h
@@ -120,28 +120,6 @@ static __always_inline void local_daif_restore(unsigned long flags)
 		trace_hardirqs_off();
 }
 
-/*
- * Called by synchronous exception handlers to restore the DAIF bits that were
- * modified by taking an exception.
- */
-static __always_inline void local_daif_inherit(struct pt_regs *regs)
-{
-	unsigned long flags = regs->pstate & DAIF_MASK;
-
-	if (!regs_irqs_disabled(regs))
-		trace_hardirqs_on();
-
-	if (system_uses_irq_prio_masking())
-		gic_write_pmr(regs->pmr);
-
-	/*
-	 * We can't use local_daif_restore(regs->pstate) here as
-	 * system_has_prio_mask_debugging() won't restore the I bit if it can
-	 * use the pmr instead.
-	 */
-	write_sysreg(flags, daif);
-}
-
 /*
  * During early boot, we unmask PSR.DA before the GIC has been set up.
  * If we use IRQ priority masking, the PMR and PSR will be out of sync
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 2be42d7f4eaa..a13653b228b7 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -18,11 +18,11 @@
 #include <linux/thread_info.h>
 
 #include <asm/cpufeature.h>
-#include <asm/daifflags.h>
 #include <asm/esr.h>
 #include <asm/exception.h>
 #include <asm/fpsimd.h>
 #include <asm/irq_regs.h>
+#include <asm/interrupts/entry.h>
 #include <asm/kprobes.h>
 #include <asm/mmu.h>
 #include <asm/processor.h>
@@ -52,12 +52,11 @@ static noinstr irqentry_state_t arm64_enter_from_kernel_mode(struct pt_regs *reg
  * After this function returns it is not safe to call regular kernel code,
  * instrumentable code, or any code which may trigger an exception.
  */
-static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
-					      irqentry_state_t state)
+static void noinstr __arm64_exit_to_kernel_mode(struct pt_regs *regs,
+						irqentry_state_t state,
+						arm64_exc_hwstate_t hwstate)
 {
-	local_irq_disable();
-	irqentry_exit_to_kernel_mode_preempt(regs, state);
-	local_daif_mask();
+	arm64_mask_exc_context(hwstate);
 	mte_check_tfsr_exit();
 	irqentry_exit_to_kernel_mode_after_preempt(regs, state);
 }
@@ -69,6 +68,30 @@ static __always_inline void arm64_syscall_enter_from_user_mode(struct pt_regs *r
 	sme_enter_from_user_mode();
 }
 
+/*
+ * We are returning from the context which allows involuntary kernel preemption
+ */
+static void noinstr arm64_exit_to_kernel_mode_preempt(struct pt_regs *regs,
+						      irqentry_state_t state,
+						      arm64_exc_hwstate_t hwstate)
+{
+	irqentry_exit_to_kernel_mode_preempt(regs, state);
+	__arm64_exit_to_kernel_mode(regs, state, hwstate);
+}
+
+static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
+					      irqentry_state_t state,
+					      arm64_exc_hwstate_t hwstate)
+{
+	if (!arch_irqs_disabled_flags(hwstate.flags)) {
+		hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
+		arm64_exit_to_kernel_mode_preempt(regs, state, hwstate);
+		return;
+	}
+
+	__arm64_exit_to_kernel_mode(regs, state, hwstate);
+}
+
 /*
  * Handle IRQ/context state management when entering from user mode.
  * Before this function is called it is not safe to call regular kernel code,
@@ -82,11 +105,12 @@ static __always_inline void arm64_enter_from_user_mode(struct pt_regs *regs)
 	sme_enter_from_user_mode();
 }
 
-static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs)
+static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs,
+							    arm64_exc_hwstate_t hwstate)
 {
-	local_irq_disable();
+	hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
 	syscall_exit_to_user_mode_prepare(regs);
-	local_daif_mask();
+	arm64_mask_exc_context(hwstate);
 	sme_exit_to_user_mode();
 	mte_check_tfsr_exit();
 	exit_to_user_mode();
@@ -97,11 +121,12 @@ static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs
  * After this function returns it is not safe to call regular kernel code,
  * instrumentable code, or any code which may trigger an exception.
  */
-static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
+static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs,
+						    arm64_exc_hwstate_t hwstate)
 {
-	local_irq_disable();
+	hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
 	irqentry_exit_to_user_mode_prepare(regs);
-	local_daif_mask();
+	arm64_mask_exc_context(hwstate);
 	sme_exit_to_user_mode();
 	mte_check_tfsr_exit();
 	exit_to_user_mode();
@@ -109,7 +134,10 @@ static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
 
 asmlinkage void noinstr asm_exit_to_user_mode(struct pt_regs *regs)
 {
-	arm64_syscall_exit_to_user_mode(regs);
+	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(PROCESS_CONTEXT);
+
+	arm64_syscall_exit_to_user_mode(regs, hwstate);
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 /*
@@ -315,63 +343,69 @@ UNHANDLED(el1t, 64, error)
 static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
 {
 	unsigned long far = read_sysreg(far_el1);
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_mem_abort(far, esr, regs);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static void noinstr el1_pc(struct pt_regs *regs, unsigned long esr)
 {
 	unsigned long far = read_sysreg(far_el1);
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_sp_pc_abort(far, esr, regs);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static void noinstr el1_undef(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_el1_undef(regs, esr);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static void noinstr el1_bti(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_el1_bti(regs, esr);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static void noinstr el1_gcs(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_el1_gcs(regs, esr);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static void noinstr el1_mops(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_el1_mops(regs, esr);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 static void noinstr el1_breakpt(struct pt_regs *regs, unsigned long esr)
@@ -431,12 +465,13 @@ static void noinstr el1_brk64(struct pt_regs *regs, unsigned long esr)
 
 static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	hwstate = arm64_inherit_exc_context(regs);
 	do_el1_fpac(regs, esr);
-	arm64_exit_to_kernel_mode(regs, state);
+	arm64_exit_to_kernel_mode(regs, state, hwstate);
 }
 
 asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
@@ -486,16 +521,20 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
 	default:
 		__panic_unhandled(regs, "64-bit el1h sync", esr);
 	}
+
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 static __always_inline void __el1_pnmi(struct pt_regs *regs,
 				       void (*handler)(struct pt_regs *))
 {
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	state = irqentry_nmi_enter(regs);
+	hwstate = arm64_unmask_exc_context(NONMI_CONTEXT);
 	do_interrupt_handler(regs, handler);
-	local_daif_mask();
+	arm64_mask_exc_context(hwstate);
 	irqentry_nmi_exit(regs, state);
 }
 
@@ -506,21 +545,32 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
 
 	state = arm64_enter_from_kernel_mode(regs);
 
+	arm64_unmask_exc_context(NONMI_CONTEXT);
+
 	irq_enter_rcu();
 	do_interrupt_handler(regs, handler);
 	irq_exit_rcu();
 
-	arm64_exit_to_kernel_mode(regs, state);
+	/*
+	 * If pseudo-NMIs are enabled and the interrupted context had
+	 * IRQs unmasked, the interrupt handler will have cleared DAIF
+	 * and switched to PMR masking in order to handle
+	 * NMIs. Otherwise it would keep IF.  In both cases on return
+	 * we effectivly have NOIRQ_CONTEXT - keep track of it
+	 */
+	arm64_debug_exc_context(NOIRQ_CONTEXT);
+	arm64_exit_to_kernel_mode_preempt(regs, state, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
 }
+
 static void noinstr el1_interrupt(struct pt_regs *regs,
 				  void (*handler)(struct pt_regs *))
 {
-	write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
-
 	if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && regs_irqs_disabled(regs))
 		__el1_pnmi(regs, handler);
 	else
 		__el1_irq(regs, handler);
+
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
@@ -536,28 +586,31 @@ asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
 asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
 {
 	unsigned long esr = read_sysreg(esr_el1);
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
-	local_daif_restore(DAIF_ERRCTX);
+	hwstate = arm64_unmask_exc_context(ERROR_CONTEXT);
 	state = irqentry_nmi_enter(regs);
 	do_serror(regs, esr);
-	local_daif_mask();
+	arm64_mask_exc_context(hwstate);
 	irqentry_nmi_exit(regs, state);
 }
 
 static void noinstr el0_da(struct pt_regs *regs, unsigned long esr)
 {
 	unsigned long far = read_sysreg(far_el1);
+	arm64_exc_hwstate_t hwstate;
 
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_mem_abort(far, esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
 {
 	unsigned long far = read_sysreg(far_el1);
+	arm64_exc_hwstate_t hwstate;
 
 	/*
 	 * We've taken an instruction abort from userspace and not yet
@@ -568,114 +621,139 @@ static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
 		arm64_apply_bp_hardening();
 
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_mem_abort(far, esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_fpsimd_acc(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_fpsimd_acc(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_sve_acc(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_sve_acc(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_sme_acc(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_sme_acc(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_fpsimd_exc(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_fpsimd_exc(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_sys(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_sys(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_pc(struct pt_regs *regs, unsigned long esr)
 {
 	unsigned long far = read_sysreg(far_el1);
+	arm64_exc_hwstate_t hwstate;
 
 	if (!is_ttbr0_addr(instruction_pointer(regs)))
 		arm64_apply_bp_hardening();
 
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_sp_pc_abort(far, esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_sp(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_sp_pc_abort(regs->sp, esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_undef(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_undef(regs, esr);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_bti(struct pt_regs *regs)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_bti(regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_mops(regs, esr);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_gcs(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_gcs(regs, esr);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_inv(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	bad_el0_sync(regs, 0, esr);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	if (!is_ttbr0_addr(regs->pc))
 		arm64_apply_bp_hardening();
 
@@ -683,12 +761,13 @@ static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
 	debug_exception_enter(regs);
 	do_breakpoint(esr, regs);
 	debug_exception_exit(regs);
-	local_daif_restore(DAIF_PROCCTX);
-	arm64_exit_to_user_mode(regs);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
 	bool step_done;
 
 	if (!is_ttbr0_addr(regs->pc))
@@ -702,50 +781,56 @@ static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
 	 * the single-step is complete.
 	 */
 	step_done = try_step_suspended_breakpoints(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	if (!step_done)
 		do_el0_softstep(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_watchpt(struct pt_regs *regs, unsigned long esr)
 {
 	/* Watchpoints are the only debug exception to write FAR_EL1 */
 	unsigned long far = read_sysreg(far_el1);
+	arm64_exc_hwstate_t hwstate;
 
 	arm64_enter_from_user_mode(regs);
 	debug_exception_enter(regs);
 	do_watchpoint(far, esr, regs);
 	debug_exception_exit(regs);
-	local_daif_restore(DAIF_PROCCTX);
-	arm64_exit_to_user_mode(regs);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_brk64(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_brk64(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_svc(struct pt_regs *regs)
 {
+	arm64_exc_hwstate_t hwstate;
 	arm64_syscall_enter_from_user_mode(regs);
 	cortex_a76_erratum_1463225_svc_handler();
 	fpsimd_syscall_enter();
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_svc(regs);
-	arm64_syscall_exit_to_user_mode(regs);
+	arm64_syscall_exit_to_user_mode(regs, hwstate);
 	fpsimd_syscall_exit();
 }
 
 static void noinstr el0_fpac(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_fpac(regs, esr);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
@@ -814,6 +899,8 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
 	default:
 		el0_inv(regs, esr);
 	}
+
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 static void noinstr el0_interrupt(struct pt_regs *regs,
@@ -821,7 +908,7 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
 {
 	arm64_enter_from_user_mode(regs);
 
-	write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
+	arm64_unmask_exc_context(NONMI_CONTEXT);
 
 	if (regs->pc & BIT(55))
 		arm64_apply_bp_hardening();
@@ -830,7 +917,14 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
 	do_interrupt_handler(regs, handler);
 	irq_exit_rcu();
 
-	arm64_exit_to_user_mode(regs);
+	/*
+	 * For the same reason as in el1_irq() we effectivly
+	 * have NOIRQ_CONTEXT on return from handler - keep
+	 * track of it
+	 */
+	arm64_debug_exc_context(NOIRQ_CONTEXT);
+	arm64_exit_to_user_mode(regs, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 static void noinstr __el0_irq_handler_common(struct pt_regs *regs)
@@ -856,15 +950,17 @@ asmlinkage void noinstr el0t_64_fiq_handler(struct pt_regs *regs)
 static void noinstr __el0_error_handler_common(struct pt_regs *regs)
 {
 	unsigned long esr = read_sysreg(esr_el1);
+	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
 
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_ERRCTX);
+	hwstate = arm64_unmask_exc_context(ERROR_CONTEXT);
 	state = irqentry_nmi_enter(regs);
 	do_serror(regs, esr);
 	irqentry_nmi_exit(regs, state);
-	local_daif_restore(DAIF_PROCCTX);
-	arm64_exit_to_user_mode(regs);
+	hwstate = arm64_drop_exc_context(hwstate, PROCESS_CONTEXT);
+	arm64_exit_to_user_mode(regs, hwstate);
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
@@ -875,27 +971,33 @@ asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
 #ifdef CONFIG_COMPAT
 static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_cp15(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_svc_compat(struct pt_regs *regs)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_syscall_enter_from_user_mode(regs);
 	cortex_a76_erratum_1463225_svc_handler();
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_el0_svc_compat(regs);
-	arm64_syscall_exit_to_user_mode(regs);
+	arm64_syscall_exit_to_user_mode(regs, hwstate);
 }
 
 static void noinstr el0_bkpt32(struct pt_regs *regs, unsigned long esr)
 {
+	arm64_exc_hwstate_t hwstate;
+
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
 	do_bkpt32(esr, regs);
-	arm64_exit_to_user_mode(regs);
+	arm64_exit_to_user_mode(regs, hwstate);
 }
 
 asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
@@ -946,6 +1048,8 @@ asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
 	default:
 		el0_inv(regs, esr);
 	}
+
+	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
 asmlinkage void noinstr el0t_32_irq_handler(struct pt_regs *regs)
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index f63049ac32dc..cb3be770f2d0 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -316,7 +316,7 @@ alternative_else_nop_endif
 
 	mrs_s	x20, SYS_ICC_PMR_EL1
 	str	w20, [sp, #S_PMR]
-	mov	x20, #GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET
+	mov	x20, #GIC_PRIO_IRQON
 	msr_s	SYS_ICC_PMR_EL1, x20
 
 .Lskip_pmr_save\@:
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index d46022f72075..b91cf163aac7 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -185,7 +185,7 @@ static void init_gic_priority_masking(void)
 	WARN_ON(!(cpuflags & PSR_I_BIT));
 	WARN_ON(!(cpuflags & PSR_F_BIT));
 
-	gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
+	gic_write_pmr(GIC_PRIO_IRQON);
 }
 
 /*
-- 
2.34.1

[RFC PATCH 13/36] arm64: process: Use helper to check exception state

[Vladimir Murzin]

debug_switch_state() explicitly checks the hardware exception state
for both the pseudo-NMI and non-NMI paths. The expected state matches
the NOIRQ_CONTEXT exception context introduced alongside the exception
masking helpers and state checkers.

Use the helper instead of open-coding these checks.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/kernel/process.c | 21 ++-------------------
 1 file changed, 2 insertions(+), 19 deletions(-)

diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 033643cd4e5e..ddda8d7aee24 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -51,6 +51,7 @@
 #include <asm/exec.h>
 #include <asm/fpsimd.h>
 #include <asm/gcs.h>
+#include <asm/interrupts/common_flags.h>
 #include <asm/mmu_context.h>
 #include <asm/mpam.h>
 #include <asm/mte.h>
@@ -738,25 +739,7 @@ void update_sctlr_el1(u64 sctlr)
 
 static inline void debug_switch_state(void)
 {
-	if (system_uses_irq_prio_masking()) {
-		unsigned long daif_expected = 0;
-		unsigned long daif_actual = read_sysreg(daif);
-		unsigned long pmr_expected = GIC_PRIO_IRQOFF;
-		unsigned long pmr_actual = read_sysreg_s(SYS_ICC_PMR_EL1);
-
-		WARN_ONCE(daif_actual != daif_expected ||
-			  pmr_actual != pmr_expected,
-			  "Unexpected DAIF + PMR: 0x%lx + 0x%lx (expected 0x%lx + 0x%lx)\n",
-			  daif_actual, pmr_actual,
-			  daif_expected, pmr_expected);
-	} else {
-		unsigned long daif_expected = DAIF_PROCCTX_NOIRQ;
-		unsigned long daif_actual = read_sysreg(daif);
-
-		WARN_ONCE(daif_actual != daif_expected,
-			  "Unexpected DAIF value: 0x%lx (expected 0x%lx)\n",
-			  daif_actual, daif_expected);
-	}
+	arm64_debug_exc_context(NOIRQ_CONTEXT);
 }
 
 /*
-- 
2.34.1

[RFC PATCH 14/36] arm64: interrupts: introduce generic interrupt masking helpers

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

As for the entry code, we want to replace `local_daif_...` helpers
so that they can properly handle both DAIF and PMR, as well controlling
their use more strongly.

Introduce new `local_all_irqs_...` helpers to replace them, which should
only be called in save/restore pairs.

Save the requested interrupt state as well, so we can check for
inconsistent interrupt masking in between save and restore.

There are two exceptions where it does not make sense to force
save/restore pairs for modifying the interrupt masks:
 - when initializing a CPU or
 - preparing to turn it off.

As we otherwise want to force save/restore pairs, those cases are
handled with specific helpers, making clear that they should not be
used outside of those cases, enforced with `CONFIG_DEBUG_IRQFLAGS`
enabled.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/interrupts/masking.h | 101 ++++++++++++++++++++
 1 file changed, 101 insertions(+)
 create mode 100644 arch/arm64/include/asm/interrupts/masking.h

diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
new file mode 100644
index 000000000000..66ee03f7ab68
--- /dev/null
+++ b/arch/arm64/include/asm/interrupts/masking.h
@@ -0,0 +1,101 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2025 Arm Ltd.
+ */
+#ifndef __ASM_INTERRUPTS_MASKING_H
+#define __ASM_INTERRUPTS_MASKING_H
+
+#include <asm/arch_gicv3.h>
+#include <asm/bug.h>
+#include <asm/cpufeature.h>
+#include <asm/interrupts/common_flags.h>
+#include <asm/ptrace.h>
+
+typedef struct arm64_exc_hwstates {
+	arm64_exc_hwstate_t saved;
+	arm64_exc_hwstate_t expected;
+} arm64_exc_hwstates_t;
+
+#ifdef CONFIG_DEBUG_IRQFLAGS
+/* Make sure the CPU init/tear down masking functions are only used once. */
+static DEFINE_PER_CPU(bool, irqs_masks_cpu_init_done);
+static DEFINE_PER_CPU(bool, irqs_masks_cpu_final_done);
+#endif
+
+static inline
+arm64_exc_hwstates_t local_all_irqs_save_mask(arm64_exc_context_t new)
+{
+	arm64_exc_hwstate_t state = arm64_exc_hwstate_of_context(new);
+	arm64_exc_hwstate_t actual = {.flags = arch_local_save_flags()};
+
+	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
+		bool pnmi = system_uses_irq_prio_masking();
+
+		WARN_ON_ONCE(new < CRITICAL_CONTEXT &&
+			     actual.daif == DAIF_MASK);
+
+		WARN_ON_ONCE(new < ERROR_CONTEXT &&
+			     actual.daif == DAIF_ERRCTX);
+
+		WARN_ON_ONCE(new < NONMI_CONTEXT &&
+			     pnmi && actual.daif == DAIF_PROCCTX_NOIRQ);
+
+		WARN_ON_ONCE(new < NOIRQ_CONTEXT &&
+			     ((pnmi && actual.pmr == GIC_PRIO_IRQOFF) ||
+			      (!pnmi && actual.daif == DAIF_PROCCTX_NOIRQ)));
+	}
+
+	arm64_update_exc_hwstate(state, actual.pmr != state.pmr);
+
+	if (!arch_irqs_disabled_flags(actual.flags))
+		trace_hardirqs_off();
+
+	return (arm64_exc_hwstates_t){.saved = actual, .expected = state};
+}
+
+static inline void local_all_irqs_restore(arm64_exc_hwstates_t states)
+{
+	arm64_debug_exc_hwstate(states.expected);
+
+	if (!arch_irqs_disabled_flags(states.saved.flags))
+		trace_hardirqs_on();
+
+	arm64_update_exc_hwstate(states.saved, true);
+}
+
+#ifdef CONFIG_DEBUG_IRQFLAGS
+static inline
+void local_all_irqs_cpu_init_mask(arm64_exc_context_t context)
+{
+	WARN_ON(__this_cpu_read(irqs_masks_cpu_init_done));
+	if (context == PROCESS_CONTEXT)
+		trace_hardirqs_on();
+	arm64_update_exc_context(context, true);
+	__this_cpu_write(irqs_masks_cpu_init_done, true);
+	__this_cpu_write(irqs_masks_cpu_final_done, false);
+}
+
+static inline void local_all_irqs_final_mask(void)
+{
+	WARN_ON(__this_cpu_read(irqs_masks_cpu_final_done));
+	arm64_update_exc_context(CRITICAL_CONTEXT, true);
+	trace_hardirqs_off();
+	__this_cpu_write(irqs_masks_cpu_final_done, true);
+	__this_cpu_write(irqs_masks_cpu_init_done, false);
+}
+#else /* CONFIG_DEBUG_IRQFLAGS */
+static inline
+void local_all_irqs_cpu_init_mask(arm64_exc_context_t context)
+{
+	if (context == PROCESS_CONTEXT)
+		trace_hardirqs_on();
+	arm64_update_exc_context(context, true);
+}
+
+static inline void local_all_irqs_final_mask(void)
+{
+	arm64_update_exc_context(CRITICAL_CONTEXT, true);
+	trace_hardirqs_off();
+}
+#endif /* CONFIG_DEBUG_IRQFLAGS */
+#endif /* __ASM_INTERRUPTS_MASKING_H */
-- 
2.34.1

[RFC PATCH 15/36] arm64: replace local_daif helpers

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Replace existing `local_daif_...` usage with the new `local_all_irqs...`
helpers, adding the necessary save-state struct where required.

Slightly rework `apei_claim_sea()`'s handling of interrupt masking to
fit with the new APIs.  Currently, it saves the DAIF flags to restore
later and the `local_irq` ones to check if interrupts are disabled.
It then masks AIF to signal the SEA (forcing DAIF masking), and will
unmask to IF (re-enabling PMR masking if pseudo-NMIs are in use) if it
interrupted a task with interrupts unmaksed, restoring with the DAIF
flags from the beginning afterwards.

As we want to preserve a "mask/unmask" process, change the logic from
"mask AIF/unmask A/unmask IF" to "mask IF/mask A/unmask A/unmask IF" :
first mask with `local_irq_save()`, mask/unmask further around the
SEA signaling, and restore at the end with `local_irq_restore()`.
Now that both APIs properly work with DAIF and PMR, this works
as expected and also removes the need for a "save flags without masking"
function for the new helpers, as this was the only user.

Remove the use of GIC_PRIO_PSR_I_SET in `kvm/hyp/nvhe/switch.c`,
as we now check both PMR and DAIF everywhere and we don't need to signal
interrupts being masked in DAIF in the PMR anymore.

As we removed the `daifflags.h` include in preparation for its removal,
move `local_interrupt_priority_init()` to `asm/interrupts/masking.h`.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/daifflags.h          | 14 --------------
 arch/arm64/include/asm/interrupts/masking.h | 14 ++++++++++++++
 arch/arm64/kernel/acpi.c                    | 15 +++++++--------
 arch/arm64/kernel/hibernate.c               | 16 ++++++++--------
 arch/arm64/kernel/irq.c                     |  2 +-
 arch/arm64/kernel/machine_kexec.c           |  4 ++--
 arch/arm64/kernel/setup.c                   |  4 ++--
 arch/arm64/kernel/smp.c                     | 14 +++++++-------
 arch/arm64/kernel/suspend.c                 | 14 +++++++-------
 arch/arm64/kvm/hyp/nvhe/switch.c            |  2 +-
 arch/arm64/kvm/hyp/vgic-v3-sr.c             |  7 ++++---
 arch/arm64/kvm/hyp/vhe/switch.c             | 12 +++++++-----
 arch/arm64/mm/mmu.c                         |  7 ++++---
 13 files changed, 64 insertions(+), 61 deletions(-)

diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
index 6b14ec4d4dbe..8f097a2d9099 100644
--- a/arch/arm64/include/asm/daifflags.h
+++ b/arch/arm64/include/asm/daifflags.h
@@ -120,18 +120,4 @@ static __always_inline void local_daif_restore(unsigned long flags)
 		trace_hardirqs_off();
 }
 
-/*
- * During early boot, we unmask PSR.DA before the GIC has been set up.
- * If we use IRQ priority masking, the PMR and PSR will be out of sync
- * after the GIC is enabled : sync them up.
- */
-static inline void local_interrupt_priority_init(void)
-{
-	WARN_ON(read_sysreg(daif) & PSR_A_BIT);
-	lockdep_assert_irqs_disabled();
-
-	gic_write_pmr(GIC_PRIO_IRQOFF);
-	write_sysreg(DAIF_PROCCTX, daif);
-}
-
 #endif
diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
index 66ee03f7ab68..19d32618b2b9 100644
--- a/arch/arm64/include/asm/interrupts/masking.h
+++ b/arch/arm64/include/asm/interrupts/masking.h
@@ -98,4 +98,18 @@ static inline void local_all_irqs_final_mask(void)
 	trace_hardirqs_off();
 }
 #endif /* CONFIG_DEBUG_IRQFLAGS */
+
+/*
+ * During early boot, we unmask PSR.DA before the GIC has been set up.
+ * If we use IRQ priority masking, the PMR and PSR will be out of sync
+ * after the GIC is enabled : sync them up.
+ */
+static inline void local_interrupt_priority_init(void)
+{
+	WARN_ON(read_sysreg(daif) & PSR_A_BIT);
+	lockdep_assert_irqs_disabled();
+
+	arm64_update_exc_context(NOIRQ_CONTEXT, true);
+}
+
 #endif /* __ASM_INTERRUPTS_MASKING_H */
diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
index 5891f92c2035..8ec784f96693 100644
--- a/arch/arm64/kernel/acpi.c
+++ b/arch/arm64/kernel/acpi.c
@@ -33,7 +33,7 @@
 #include <acpi/processor.h>
 #include <asm/cputype.h>
 #include <asm/cpu_ops.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/smp_plat.h>
 
 int acpi_noirq = 1;		/* skip ACPI IRQ initialization */
@@ -391,15 +391,14 @@ int apei_claim_sea(struct pt_regs *regs)
 	int err = -ENOENT;
 	bool return_to_irqs_enabled;
 	unsigned long current_flags;
+	arm64_exc_hwstates_t all_irqs_state;
 
 	if (!IS_ENABLED(CONFIG_ACPI_APEI_GHES))
 		return err;
 
-	current_flags = local_daif_save_flags();
-
-	/* current_flags isn't useful here as daif doesn't tell us about pNMI */
-	return_to_irqs_enabled = !irqs_disabled_flags(arch_local_save_flags());
+	local_irq_save(current_flags);
 
+	return_to_irqs_enabled = !irqs_disabled_flags(current_flags);
 	if (regs)
 		return_to_irqs_enabled = !regs_irqs_disabled(regs);
 
@@ -407,10 +406,11 @@ int apei_claim_sea(struct pt_regs *regs)
 	 * SEA can interrupt SError, mask it and describe this as an NMI so
 	 * that APEI defers the handling.
 	 */
-	local_daif_restore(DAIF_ERRCTX);
+	all_irqs_state = local_all_irqs_save_mask(ERROR_CONTEXT);
 	nmi_enter();
 	err = ghes_notify_sea();
 	nmi_exit();
+	local_all_irqs_restore(all_irqs_state);
 
 	/*
 	 * APEI NMI-like notifications are deferred to irq_work. Unless
@@ -418,7 +418,6 @@ int apei_claim_sea(struct pt_regs *regs)
 	 */
 	if (!err) {
 		if (return_to_irqs_enabled) {
-			local_daif_restore(DAIF_PROCCTX_NOIRQ);
 			__irq_enter();
 			irq_work_run();
 			__irq_exit();
@@ -428,7 +427,7 @@ int apei_claim_sea(struct pt_regs *regs)
 		}
 	}
 
-	local_daif_restore(current_flags);
+	local_irq_restore(current_flags);
 
 	return err;
 }
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index d0d9bd91e639..4dd40593f736 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -20,7 +20,7 @@
 #include <asm/barrier.h>
 #include <asm/cacheflush.h>
 #include <asm/cputype.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/irqflags.h>
 #include <asm/kexec.h>
 #include <asm/memory.h>
@@ -332,16 +332,16 @@ static void swsusp_mte_restore_tags(void)
 
 int swsusp_arch_suspend(void)
 {
-	int ret = 0;
-	unsigned long flags;
 	struct sleep_stack_data state;
+	arm64_exc_hwstates_t flags;
+	int ret = 0;
 
 	if (cpus_are_stuck_in_kernel()) {
 		pr_err("Can't hibernate: no mechanism to offline secondary CPUs.\n");
 		return -EBUSY;
 	}
 
-	flags = local_daif_save();
+	flags = local_all_irqs_save_mask(CRITICAL_CONTEXT);
 
 	if (__cpu_suspend_enter(&state)) {
 		/* make the crash dump kernel image visible/saveable */
@@ -391,7 +391,7 @@ int swsusp_arch_suspend(void)
 		spectre_v4_enable_mitigation(NULL);
 	}
 
-	local_daif_restore(flags);
+	local_all_irqs_restore(flags);
 
 	return ret;
 }
@@ -405,11 +405,11 @@ int swsusp_arch_suspend(void)
 int __nocfi swsusp_arch_resume(void)
 {
 	int rc;
-	unsigned long flags;
 	void *zero_page;
 	size_t exit_size;
 	pgd_t *tmp_pg_dir;
 	phys_addr_t el2_vectors;
+	arm64_exc_hwstates_t flags;
 	void __noreturn (*hibernate_exit)(phys_addr_t, phys_addr_t, void *,
 					  void *, phys_addr_t, phys_addr_t);
 	struct trans_pgd_info trans_info = {
@@ -476,11 +476,11 @@ int __nocfi swsusp_arch_resume(void)
 	 *   swsusp_arch_resume(), and expects to be re-entered in the
 	 *   same state : with all DAIF exceptions masked.
 	 */
-	flags = local_daif_save();
+	flags = local_all_irqs_save_mask(CRITICAL_CONTEXT);
 	hibernate_exit(virt_to_phys(tmp_pg_dir), resume_hdr.ttbr1_el1,
 		       resume_hdr.reenter_kernel, restore_pblist,
 		       resume_hdr.__hyp_stub_vectors, virt_to_phys(zero_page));
-	local_daif_restore(flags);
+	local_all_irqs_restore(flags);
 
 	return 0;
 }
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index c73faa30268d..36da80169aae 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -21,7 +21,7 @@
 #include <linux/seq_file.h>
 #include <linux/smp.h>
 #include <linux/vmalloc.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/exception.h>
 #include <asm/numa.h>
 #include <asm/softirq_stack.h>
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index c5693a32e49b..0ff45f918488 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -17,7 +17,7 @@
 
 #include <asm/cacheflush.h>
 #include <asm/cpu_ops.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/memory.h>
 #include <asm/mmu.h>
 #include <asm/mmu_context.h>
@@ -173,7 +173,7 @@ void machine_kexec(struct kimage *kimage)
 
 	pr_info("Bye!\n");
 
-	local_daif_mask();
+	local_all_irqs_final_mask();
 
 	/*
 	 * Both restart and kernel_reloc will shutdown the MMU, disable data
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 23c05dc7a8f2..14f283f7ac3a 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -37,7 +37,7 @@
 #include <asm/fixmap.h>
 #include <asm/cpu.h>
 #include <asm/cputype.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/elf.h>
 #include <asm/cpufeature.h>
 #include <asm/cpu_ops.h>
@@ -311,7 +311,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 	 * IRQ and FIQ will be unmasked after the root irqchip has been
 	 * detected and initialized.
 	 */
-	local_daif_restore(DAIF_PROCCTX_NOIRQ);
+	local_all_irqs_cpu_init_mask(NOIRQ_CONTEXT);
 
 	/*
 	 * TTBR0 is only used for the identity mapping at this stage. Make it
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index b91cf163aac7..257d50529d14 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -42,7 +42,7 @@
 #include <asm/cpu.h>
 #include <asm/cputype.h>
 #include <asm/cpu_ops.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/kvm_mmu.h>
 #include <asm/mmu_context.h>
 #include <asm/numa.h>
@@ -263,7 +263,7 @@ asmlinkage notrace void secondary_start_kernel(void)
 	 * as the root irqchip has already been detected and initialized we can
 	 * unmask IRQ and FIQ at the same time.
 	 */
-	local_daif_restore(DAIF_PROCCTX);
+	local_all_irqs_cpu_init_mask(PROCESS_CONTEXT);
 
 	/*
 	 * OK, it's off to the idle thread for us
@@ -370,7 +370,7 @@ void __noreturn cpu_die(void)
 
 	idle_task_exit();
 
-	local_daif_mask();
+	local_all_irqs_final_mask();
 
 	/* Tell cpuhp_bp_sync_dead() that this CPU is now safe to dispose of */
 	cpuhp_ap_report_dead();
@@ -870,7 +870,7 @@ static void __noreturn local_cpu_stop(unsigned int cpu)
 {
 	set_cpu_online(cpu, false);
 
-	local_daif_mask();
+	local_all_irqs_final_mask();
 	sdei_mask_local_cpu();
 	cpu_park_loop();
 }
@@ -889,14 +889,14 @@ static void __noreturn ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs
 {
 #ifdef CONFIG_KEXEC_CORE
 	/*
-	 * Use local_daif_mask() instead of local_irq_disable() to make sure
-	 * that pseudo-NMIs are disabled. The "crash stop" code starts with
+	 * Use local_all_irqs_final_mask() instead of local_irq_disable() to make
+	 * sure that pseudo-NMIs are disabled. The "crash stop" code starts with
 	 * an IRQ and falls back to NMI (which might be pseudo). If the IRQ
 	 * finally goes through right as we're timing out then the NMI could
 	 * interrupt us. It's better to prevent the NMI and let the IRQ
 	 * finish since the pt_regs will be better.
 	 */
-	local_daif_mask();
+	local_all_irqs_final_mask();
 
 	crash_save_cpu(regs, cpu);
 
diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c
index c41724a40b75..d8c62d7e8922 100644
--- a/arch/arm64/kernel/suspend.c
+++ b/arch/arm64/kernel/suspend.c
@@ -9,7 +9,7 @@
 #include <asm/cacheflush.h>
 #include <asm/cpufeature.h>
 #include <asm/cpuidle.h>
-#include <asm/daifflags.h>
+#include <asm/interrupts/masking.h>
 #include <asm/debug-monitors.h>
 #include <asm/exec.h>
 #include <asm/fpsimd.h>
@@ -69,7 +69,7 @@ void notrace __cpu_suspend_exit(void)
 	/*
 	 * Restore HW breakpoint registers to sane values
 	 * before debug exceptions are possibly reenabled
-	 * by cpu_suspend()s local_daif_restore() call.
+	 * by cpu_suspend()s local_all_irqs_save_mask() call.
 	 */
 	if (hw_breakpoint_restore)
 		hw_breakpoint_restore(cpu);
@@ -97,7 +97,7 @@ void notrace __cpu_suspend_exit(void)
 int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 {
 	int ret = 0;
-	unsigned long flags;
+	arm64_exc_hwstates_t flags;
 	struct sleep_stack_data state;
 
 	/*
@@ -119,12 +119,12 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 	 *
 	 * Strictly speaking the trace_hardirqs_off() here is superfluous,
 	 * hardirqs should be firmly off by now. This really ought to use
-	 * something like raw_local_daif_save().
+	 * something like raw_local_all_irqs_save_mask().
 	 *
 	 * This also unmasks interrupts in PMR in order to reliably
 	 * resume if we're using pseudo-NMIs.
 	 */
-	flags = local_daif_save();
+	flags = local_all_irqs_save_mask(CRITICAL_CONTEXT);
 
 	/*
 	 * Function graph tracer state gets inconsistent when the kernel
@@ -158,11 +158,11 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 	unpause_graph_tracing();
 
 	/*
-	 * Restore pstate flags. OS lock and mdscr have been already
+	 * Restore interrupt masks. OS lock and mdscr have been already
 	 * restored, so from this point onwards, debugging is fully
 	 * reenabled if it was enabled when core started shutdown.
 	 */
-	local_daif_restore(flags);
+	local_all_irqs_restore(flags);
 
 	return ret;
 }
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index 7318e3e6a5f3..50566b455c80 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -270,7 +270,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
 	 * Naturally, we want to avoid this.
 	 */
 	if (system_uses_irq_prio_masking()) {
-		gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
+		gic_write_pmr(GIC_PRIO_IRQON);
 		pmr_sync();
 	}
 
diff --git a/arch/arm64/kvm/hyp/vgic-v3-sr.c b/arch/arm64/kvm/hyp/vgic-v3-sr.c
index c4d2f1feea8b..402d51430599 100644
--- a/arch/arm64/kvm/hyp/vgic-v3-sr.c
+++ b/arch/arm64/kvm/hyp/vgic-v3-sr.c
@@ -10,6 +10,7 @@
 #include <linux/irqchip/arm-gic-v3.h>
 #include <linux/kvm_host.h>
 
+#include <asm/interrupts/masking.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_hyp.h>
 #include <asm/kvm_mmu.h>
@@ -446,7 +447,7 @@ void __vgic_v3_init_lrs(void)
 u64 __vgic_v3_get_gic_config(void)
 {
 	u64 val, sre;
-	unsigned long flags = 0;
+	arm64_exc_hwstates_t flags;
 
 	/*
 	 * In compat mode, we cannot access ICC_SRE_EL1 at any EL
@@ -476,7 +477,7 @@ u64 __vgic_v3_get_gic_config(void)
 	 * of the exception entry to EL2.
 	 */
 	if (has_vhe()) {
-		flags = local_daif_save();
+		flags = local_all_irqs_save_mask(CRITICAL_CONTEXT);
 	} else {
 		sysreg_clear_set_hcr(0, HCR_AMO | HCR_FMO | HCR_IMO);
 		isb();
@@ -491,7 +492,7 @@ u64 __vgic_v3_get_gic_config(void)
 	isb();
 
 	if (has_vhe()) {
-		local_daif_restore(flags);
+		local_all_irqs_restore(flags);
 	} else {
 		sysreg_clear_set_hcr(HCR_AMO | HCR_FMO | HCR_IMO, 0);
 		isb();
diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
index bbe9cebd3d9d..e7475d796f0a 100644
--- a/arch/arm64/kvm/hyp/vhe/switch.c
+++ b/arch/arm64/kvm/hyp/vhe/switch.c
@@ -17,6 +17,7 @@
 
 #include <asm/barrier.h>
 #include <asm/cpufeature.h>
+#include <asm/interrupts/masking.h>
 #include <asm/kprobes.h>
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
@@ -630,8 +631,9 @@ NOKPROBE_SYMBOL(__kvm_vcpu_run_vhe);
 int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
 {
 	int ret;
+	arm64_exc_hwstates_t irqs_state;
 
-	local_daif_mask();
+	irqs_state = local_all_irqs_save_mask(CRITICAL_CONTEXT);
 
 	/*
 	 * Having IRQs masked via PMR when entering the guest means the GIC
@@ -639,18 +641,18 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
 	 * only way to get out will be via guest exceptions.
 	 * Naturally, we want to avoid this.
 	 *
-	 * local_daif_mask() already sets GIC_PRIO_PSR_I_SET, we just need a
-	 * dsb to ensure the redistributor is forwards EL2 IRQs to the CPU.
+	 * local_all_irqs_save_mask() already sets GIC_PRIO_IRQON, we just need a
+	 * dsb to ensure the redistributor forwards EL2 IRQs to the CPU.
 	 */
 	pmr_sync();
 
 	ret = __kvm_vcpu_run_vhe(vcpu);
 
 	/*
-	 * local_daif_restore() takes care to properly restore PSTATE.DAIF
+	 * local_all_irqs_restore() takes care to properly restore PSTATE.DAIF
 	 * and the GIC PMR if the host is using IRQ priorities.
 	 */
-	local_daif_restore(DAIF_PROCCTX_NOIRQ);
+	local_all_irqs_restore(irqs_state);
 
 	return ret;
 }
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index f2be501468ce..f2c74911f5d9 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -32,6 +32,7 @@
 
 #include <asm/barrier.h>
 #include <asm/cputype.h>
+#include <asm/interrupts/masking.h>
 #include <asm/fixmap.h>
 #include <asm/kasan.h>
 #include <asm/kernel-pgtable.h>
@@ -2307,7 +2308,7 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
 	typedef void (ttbr_replace_func)(phys_addr_t);
 	extern ttbr_replace_func idmap_cpu_replace_ttbr1;
 	ttbr_replace_func *replace_phys;
-	unsigned long daif;
+	arm64_exc_hwstates_t all_irqs;
 
 	/* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */
 	phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp));
@@ -2323,9 +2324,9 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
 	 * We really don't want to take *any* exceptions while TTBR1 is
 	 * in the process of being replaced so mask everything.
 	 */
-	daif = local_daif_save();
+	all_irqs = local_all_irqs_save_mask(CRITICAL_CONTEXT);
 	replace_phys(ttbr1);
-	local_daif_restore(daif);
+	local_all_irqs_restore(all_irqs);
 
 	cpu_uninstall_idmap();
 }
-- 
2.34.1

[RFC PATCH 16/36] arm64: cpuidle: use new helpers to bypass interrupt priority masking

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

When the CPU goes into WFI and pseudo-NMIs are in use, we need to make
sure that interrupts are not masked by the PMR otherwise they would not
reach the CPU and wake it up.

Now that we have a proper API that handles both DAIF and PMR, extend it
to provide helpers allowing to temporarily switch to masking interrupts
via DAIF only when interrupt priorities are in use.
This could allow other parts of the code to make use of it and makes it
easier to check for proper use.

Replace the custom cpuidle helper with those new helpers.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/cpuidle.h            | 14 ++---
 arch/arm64/include/asm/interrupts/masking.h | 58 +++++++++++++++++++++
 2 files changed, 62 insertions(+), 10 deletions(-)

diff --git a/arch/arm64/include/asm/cpuidle.h b/arch/arm64/include/asm/cpuidle.h
index 2047713e097d..0c7ccdaf1832 100644
--- a/arch/arm64/include/asm/cpuidle.h
+++ b/arch/arm64/include/asm/cpuidle.h
@@ -5,22 +5,17 @@
 #include <asm/proc-fns.h>
 
 #ifdef CONFIG_ARM64_PSEUDO_NMI
-#include <asm/arch_gicv3.h>
+#include <asm/interrupts/masking.h>
 
 struct arm_cpuidle_irq_context {
-	unsigned long pmr;
-	unsigned long daif_bits;
+	arm64_exc_hwstates_t arm64_context;
 };
 
 #define arm_cpuidle_save_irq_context(__c)				\
 	do {								\
 		struct arm_cpuidle_irq_context *c = __c;		\
 		if (system_uses_irq_prio_masking()) {			\
-			c->daif_bits = read_sysreg(daif);		\
-			write_sysreg(c->daif_bits | PSR_I_BIT | PSR_F_BIT, \
-				     daif);				\
-			c->pmr = gic_read_pmr();			\
-			gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); \
+			c->arm64_context = local_all_irqs_force_daif_save(); \
 		}							\
 	} while (0)
 
@@ -28,8 +23,7 @@ struct arm_cpuidle_irq_context {
 	do {								\
 		struct arm_cpuidle_irq_context *c = __c;		\
 		if (system_uses_irq_prio_masking()) {			\
-			gic_write_pmr(c->pmr);				\
-			write_sysreg(c->daif_bits, daif);		\
+			local_all_irqs_force_daif_restore(c->arm64_context); \
 		}							\
 	} while (0)
 #else
diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
index 19d32618b2b9..9dc37a571094 100644
--- a/arch/arm64/include/asm/interrupts/masking.h
+++ b/arch/arm64/include/asm/interrupts/masking.h
@@ -99,6 +99,64 @@ static inline void local_all_irqs_final_mask(void)
 }
 #endif /* CONFIG_DEBUG_IRQFLAGS */
 
+/*
+ * In some cases, WFI or guest entry for example, we always want interrupts
+ * to reach the CPU even if masked. Masking via the PMR prevents them from
+ * reaching the CPU and waking it up.
+ * Force IRQ masking using DAIF by raising the priority mask
+ * and setting the IF flags.
+ *
+ * Should only be called when IRQs are already masked.
+ */
+static inline arm64_exc_hwstates_t local_all_irqs_force_daif_save(void)
+{
+	arm64_exc_hwstates_t states = {};
+	/*
+	 * Cannot use lockdep_assert here as idle entry enables hardirqs
+	 * while keeping interrupts masked.
+	 */
+	WARN_ON_ONCE(!irqs_disabled());
+
+	if (system_uses_irq_prio_masking()) {
+	        states.saved.daif = read_sysreg(daif);
+	        states.saved.pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
+		/*
+		 * We might have IF set or unset. In case IF already set orring with IF
+		 * won't change anything, combined with PMR set to IRQON we might become
+		 * one of contexts: CRITICAL, ERROR or NONMI. In case IF is unset (which
+		 * also implies that DA is unset) orring with IF and combining with PMR
+		 * set to IRQON would effectively gives us NONMI context.
+		 */
+		states.expected.daif = states.saved.daif | DAIF_PROCCTX_NOIRQ;
+		states.expected.pmr = GIC_PRIO_IRQON;
+
+		arm64_update_exc_hwstate(states.expected, true);
+	}
+
+	return states;
+}
+
+/*
+ * Return to masking with the PMR, restoring previously saved DAIF and PMR.
+ *
+ * IRQs or interrupt priority masking should not have been re-enabled in between
+ * the save and restore.
+ */
+static inline
+void local_all_irqs_force_daif_restore(arm64_exc_hwstates_t states)
+{
+	/*
+	 * Cannot use lockdep_assert here as idle entry enables hardirqs
+	 * while keeping interrupts masked.
+	 */
+	WARN_ON_ONCE(!irqs_disabled());
+
+	if (system_uses_irq_prio_masking()) {
+		arm64_debug_exc_hwstate(states.expected);
+		arm64_update_exc_hwstate(states.saved, true);
+	}
+}
+
 /*
  * During early boot, we unmask PSR.DA before the GIC has been set up.
  * If we use IRQ priority masking, the PMR and PSR will be out of sync
-- 
2.34.1

[RFC PATCH 17/36] arm64: remove daifflags.h

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Interrupt masking is now split in different APIs and `local_daif_...`
functions are not in use anymore.
The `DAIF_...` masks defines are now in `asm/interrupts/common_flags.h`,
so we can safely remove `daifflags.h` and all `#include`s referring to it.

Only kprobes.c and process.c use `DAIF_...` masks defines, so add the
new header there.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/daifflags.h    | 123 --------------------------
 arch/arm64/include/asm/entry-common.h |   1 -
 arch/arm64/include/asm/kvm_host.h     |   1 -
 arch/arm64/include/asm/mmu_context.h  |   1 -
 arch/arm64/kernel/debug-monitors.c    |   1 -
 arch/arm64/kernel/probes/kprobes.c    |   2 +-
 arch/arm64/kernel/signal.c            |   1 -
 arch/arm64/kernel/traps.c             |   1 -
 arch/arm64/mm/fault.c                 |   1 -
 9 files changed, 1 insertion(+), 131 deletions(-)
 delete mode 100644 arch/arm64/include/asm/daifflags.h

diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
deleted file mode 100644
index 8f097a2d9099..000000000000
--- a/arch/arm64/include/asm/daifflags.h
+++ /dev/null
@@ -1,123 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (C) 2017 ARM Ltd.
- */
-#ifndef __ASM_DAIFFLAGS_H
-#define __ASM_DAIFFLAGS_H
-
-#include <linux/irqflags.h>
-
-#include <asm/arch_gicv3.h>
-#include <asm/barrier.h>
-#include <asm/cpufeature.h>
-#include <asm/ptrace.h>
-
-#define DAIF_PROCCTX		0
-#define DAIF_PROCCTX_NOIRQ	(PSR_I_BIT | PSR_F_BIT)
-#define DAIF_ERRCTX		(PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
-#define DAIF_MASK		(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
-
-
-/* mask/save/unmask/restore all exceptions, including interrupts. */
-static __always_inline void local_daif_mask(void)
-{
-	WARN_ON(system_has_prio_mask_debugging() &&
-		(read_sysreg_s(SYS_ICC_PMR_EL1) == (GIC_PRIO_IRQOFF |
-						    GIC_PRIO_PSR_I_SET)));
-
-	asm volatile(
-		"msr	daifset, #0xf		// local_daif_mask\n"
-		:
-		:
-		: "memory");
-
-	/* Don't really care for a dsb here, we don't intend to enable IRQs */
-	if (system_uses_irq_prio_masking())
-		gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
-
-	trace_hardirqs_off();
-}
-
-static __always_inline unsigned long local_daif_save_flags(void)
-{
-	unsigned long flags;
-
-	flags = read_sysreg(daif);
-
-	if (system_uses_irq_prio_masking()) {
-		/* If IRQs are masked with PMR, reflect it in the flags */
-		if (read_sysreg_s(SYS_ICC_PMR_EL1) != GIC_PRIO_IRQON)
-			flags |= PSR_I_BIT | PSR_F_BIT;
-	}
-
-	return flags;
-}
-
-static __always_inline unsigned long local_daif_save(void)
-{
-	unsigned long flags;
-
-	flags = local_daif_save_flags();
-
-	local_daif_mask();
-
-	return flags;
-}
-
-static __always_inline void local_daif_restore(unsigned long flags)
-{
-	bool irq_disabled = flags & PSR_I_BIT;
-
-	WARN_ON(system_has_prio_mask_debugging() &&
-		(read_sysreg(daif) & (PSR_I_BIT | PSR_F_BIT)) != (PSR_I_BIT | PSR_F_BIT));
-
-	if (!irq_disabled) {
-		trace_hardirqs_on();
-
-		if (system_uses_irq_prio_masking()) {
-			gic_write_pmr(GIC_PRIO_IRQON);
-			pmr_sync();
-		}
-	} else if (system_uses_irq_prio_masking()) {
-		u64 pmr;
-
-		if (!(flags & PSR_A_BIT)) {
-			/*
-			 * If interrupts are disabled but we can take
-			 * asynchronous errors, we can take NMIs
-			 */
-			flags &= ~(PSR_I_BIT | PSR_F_BIT);
-			pmr = GIC_PRIO_IRQOFF;
-		} else {
-			pmr = GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET;
-		}
-
-		/*
-		 * There has been concern that the write to daif
-		 * might be reordered before this write to PMR.
-		 * From the ARM ARM DDI 0487D.a, section D1.7.1
-		 * "Accessing PSTATE fields":
-		 *   Writes to the PSTATE fields have side-effects on
-		 *   various aspects of the PE operation. All of these
-		 *   side-effects are guaranteed:
-		 *     - Not to be visible to earlier instructions in
-		 *       the execution stream.
-		 *     - To be visible to later instructions in the
-		 *       execution stream
-		 *
-		 * Also, writes to PMR are self-synchronizing, so no
-		 * interrupts with a lower priority than PMR is signaled
-		 * to the PE after the write.
-		 *
-		 * So we don't need additional synchronization here.
-		 */
-		gic_write_pmr(pmr);
-	}
-
-	write_sysreg(flags, daif);
-
-	if (irq_disabled)
-		trace_hardirqs_off();
-}
-
-#endif
diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
index 1905765159aa..73d82a8d8e95 100644
--- a/arch/arm64/include/asm/entry-common.h
+++ b/arch/arm64/include/asm/entry-common.h
@@ -6,7 +6,6 @@
 #include <linux/thread_info.h>
 
 #include <asm/cpufeature.h>
-#include <asm/daifflags.h>
 #include <asm/fpsimd.h>
 #include <asm/mte.h>
 #include <asm/stacktrace.h>
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index bae2c4f92ef5..74358e93f1bb 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -23,7 +23,6 @@
 #include <asm/barrier.h>
 #include <asm/cpufeature.h>
 #include <asm/cputype.h>
-#include <asm/daifflags.h>
 #include <asm/fpsimd.h>
 #include <asm/kvm.h>
 #include <asm/kvm_asm.h>
diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index 803b68758152..b3354a1350cf 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -19,7 +19,6 @@
 
 #include <asm/cacheflush.h>
 #include <asm/cpufeature.h>
-#include <asm/daifflags.h>
 #include <asm/gcs.h>
 #include <asm/proc-fns.h>
 #include <asm/cputype.h>
diff --git a/arch/arm64/kernel/debug-monitors.c b/arch/arm64/kernel/debug-monitors.c
index e271fbac5f82..5db27c7e86bb 100644
--- a/arch/arm64/kernel/debug-monitors.c
+++ b/arch/arm64/kernel/debug-monitors.c
@@ -19,7 +19,6 @@
 
 #include <asm/cpufeature.h>
 #include <asm/cputype.h>
-#include <asm/daifflags.h>
 #include <asm/debug-monitors.h>
 #include <asm/exception.h>
 #include <asm/kgdb.h>
diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c
index 43a0361a8bf0..d9d25d184309 100644
--- a/arch/arm64/kernel/probes/kprobes.c
+++ b/arch/arm64/kernel/probes/kprobes.c
@@ -24,9 +24,9 @@
 #include <linux/vmalloc.h>
 
 #include <asm/cacheflush.h>
-#include <asm/daifflags.h>
 #include <asm/debug-monitors.h>
 #include <asm/insn.h>
+#include <asm/interrupts/common_flags.h>
 #include <asm/irq.h>
 #include <asm/text-patching.h>
 #include <asm/ptrace.h>
diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
index 38e6fa204c17..2276d316cc9e 100644
--- a/arch/arm64/kernel/signal.c
+++ b/arch/arm64/kernel/signal.c
@@ -22,7 +22,6 @@
 #include <linux/syscalls.h>
 #include <linux/pkeys.h>
 
-#include <asm/daifflags.h>
 #include <asm/debug-monitors.h>
 #include <asm/elf.h>
 #include <asm/exception.h>
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index 914282016069..77b6be6fc4c1 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -33,7 +33,6 @@
 #include <asm/atomic.h>
 #include <asm/bug.h>
 #include <asm/cpufeature.h>
-#include <asm/daifflags.h>
 #include <asm/debug-monitors.h>
 #include <asm/efi.h>
 #include <asm/esr.h>
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 85e23388f9bb..83aca64745f0 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -34,7 +34,6 @@
 #include <asm/cpufeature.h>
 #include <asm/efi.h>
 #include <asm/exception.h>
-#include <asm/daifflags.h>
 #include <asm/debug-monitors.h>
 #include <asm/esr.h>
 #include <asm/kprobes.h>
-- 
2.34.1

[RFC PATCH 18/36] arm64: gicv3: remove GIC_PRIO_PSR_I_SET

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

`GIC_PRIO_PSR_I_SET` was used to keep track of sections of the kernel
where interrupts always need to reach the CPU and could not be masked
via interrupt priorities, or DAIF bypassed the status of the PMR.

This flag unmasked IRQs when OR'd to GIC_PRIO_IRQOFF and ensured that
that IRQs were seen as disabled by the `local_irqs_...` functions,
as they would only check the PMR if pseudo-NMIs were enabled.

This is not needed anymore as our IRQ masking APIs now know how to handle
interrupt priorities properly and in parallel with DAIF : remove it.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/ptrace.h         | 2 --
 include/linux/irqchip/arm-gic-v3-prio.h | 8 --------
 2 files changed, 10 deletions(-)

diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index 192eb97cd50b..c630165882e8 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -24,8 +24,6 @@
 #define GIC_PRIO_IRQON		GICV3_PRIO_UNMASKED
 #define GIC_PRIO_IRQOFF		GICV3_PRIO_IRQ
 
-#define GIC_PRIO_PSR_I_SET	GICV3_PRIO_PSR_I_SET
-
 /* Additional SPSR bits not exposed in the UABI */
 #define PSR_MODE_THREAD_BIT	(1 << 0)
 #define PSR_IL_BIT		(1 << 20)
diff --git a/include/linux/irqchip/arm-gic-v3-prio.h b/include/linux/irqchip/arm-gic-v3-prio.h
index 44157c9abb78..464bf1d29453 100644
--- a/include/linux/irqchip/arm-gic-v3-prio.h
+++ b/include/linux/irqchip/arm-gic-v3-prio.h
@@ -20,17 +20,11 @@
  * GICV3_PRIO_NMI is the PMR view of the priority of pseudo-NMIs. This can be
  * written to the PMR to mask pseudo-NMIs.
  *
- * On arm64 some code sections either automatically switch back to PSR.I or
- * explicitly require to not use priority masking. If bit GICV3_PRIO_PSR_I_SET
- * is included in the priority mask, it indicates that PSR.I should be set and
- * interrupt disabling temporarily does not rely on IRQ priorities.
  */
 #define GICV3_PRIO_UNMASKED	0xe0
 #define GICV3_PRIO_IRQ		0xc0
 #define GICV3_PRIO_NMI		0x80
 
-#define GICV3_PRIO_PSR_I_SET	(1 << 4)
-
 #ifndef __ASSEMBLER__
 
 #define __gicv3_prio_to_ns(p)	(0xff & ((p) << 1))
@@ -45,8 +39,6 @@ static_assert(__gicv3_prio_valid_ns(GICV3_PRIO_IRQ));
 static_assert(GICV3_PRIO_NMI < GICV3_PRIO_IRQ);
 static_assert(GICV3_PRIO_IRQ < GICV3_PRIO_UNMASKED);
 
-static_assert(GICV3_PRIO_IRQ < (GICV3_PRIO_IRQ | GICV3_PRIO_PSR_I_SET));
-
 #endif /* __ASSEMBLER */
 
 #endif /* __LINUX_IRQCHIP_ARM_GIC_V3_PRIO_H */
-- 
2.34.1

[RFC PATCH 19/36] arm64: cpufeature: Remove system_has_prio_mask_debugging()

[Vladimir Murzin]

Users of system_has_prio_mask_debugging() have gone, so remove it.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/cpufeature.h | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index a57870fa96db..95cd8d4acd26 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -821,12 +821,6 @@ static inline bool system_supports_mte(void)
 	return alternative_has_cap_unlikely(ARM64_MTE);
 }
 
-static inline bool system_has_prio_mask_debugging(void)
-{
-	return IS_ENABLED(CONFIG_ARM64_DEBUG_PRIORITY_MASKING) &&
-	       system_uses_irq_prio_masking();
-}
-
 static inline bool system_supports_bti(void)
 {
 	return cpus_have_final_cap(ARM64_BTI);
-- 
2.34.1

[RFC PATCH 20/36] arm64: irqflags: Switch to CONFIG_DEBUG_IRQFLAGS

[Vladimir Murzin]

All exception state debug logic now lives under the generic
CONFIG_DEBUG_IRQFLAGS. Move the two remaining cases from the dedicated
config option to CONFIG_DEBUG_IRQFLAGS as well, keeping all related
debug logic under a single configuration option.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/irqflags.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index 62f047702493..585d3ce15d80 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -47,7 +47,7 @@ static __always_inline void __daif_local_irq_enable(void)
 
 static __always_inline void __pmr_local_irq_enable(void)
 {
-	if (IS_ENABLED(CONFIG_ARM64_DEBUG_PRIORITY_MASKING)) {
+	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
 		u32 pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
 		WARN_ON_ONCE(pmr != GIC_PRIO_IRQON && pmr != GIC_PRIO_IRQOFF);
 	}
@@ -76,7 +76,7 @@ static __always_inline void __daif_local_irq_disable(void)
 
 static __always_inline void __pmr_local_irq_disable(void)
 {
-	if (IS_ENABLED(CONFIG_ARM64_DEBUG_PRIORITY_MASKING)) {
+	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
 		u32 pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
 		WARN_ON_ONCE(pmr != GIC_PRIO_IRQON && pmr != GIC_PRIO_IRQOFF);
 	}
-- 
2.34.1

[RFC PATCH 21/36] arm64: Kconfig: Remove CONFIG_ARM64_DEBUG_PRIORITY_MASKING

[Vladimir Murzin]

Last users of CONFIG_ARM64_DEBUG_PRIORITY_MASKING have gone, so remove
it.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/Kconfig | 11 -----------
 1 file changed, 11 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b3afe0688919..e36dc4831200 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2334,17 +2334,6 @@ config ARM64_PSEUDO_NMI
 
 	  If unsure, say N
 
-if ARM64_PSEUDO_NMI
-config ARM64_DEBUG_PRIORITY_MASKING
-	bool "Debug interrupt priority masking"
-	help
-	  This adds runtime checks to functions enabling/disabling
-	  interrupts when using priority masking. The additional checks verify
-	  the validity of ICC_PMR_EL1 when calling concerned functions.
-
-	  If unsure, say N
-endif # ARM64_PSEUDO_NMI
-
 config RELOCATABLE
 	bool "Build a relocatable kernel image" if EXPERT
 	select ARCH_HAS_RELR
-- 
2.34.1

[RFC PATCH 22/36] efi/runtime-wrappers: Permit architectures to override IRQ flags checks

[Vladimir Murzin]

Representation of IRQ flags on arm64 has recently changed. They now
track both DAIF and PMR state while remaining compatible with the
irqflags API by encoding the state in an unsigned long.

While the generic check can still detect corruption of the IRQ flags
caused by EFI runtime services, the resulting error message is cryptic
because it does not understand the arm64-specific representation.

Permit architectures to override the IRQ flags checks and associated
error messages, while keeping the existing implementation as the
default.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 drivers/firmware/efi/runtime-wrappers.c | 32 +++++++++++++++----------
 1 file changed, 20 insertions(+), 12 deletions(-)

diff --git a/drivers/firmware/efi/runtime-wrappers.c b/drivers/firmware/efi/runtime-wrappers.c
index da8d29621644..498fb3612643 100644
--- a/drivers/firmware/efi/runtime-wrappers.c
+++ b/drivers/firmware/efi/runtime-wrappers.c
@@ -137,6 +137,25 @@ struct efi_runtime_work efi_rts_work;
 #define arch_efi_restore_flags(state_flags)	local_irq_restore(state_flags)
 #endif
 
+#ifndef arch_efi_check_flags
+#define arch_efi_check_flags(state_flags, caller) 							\
+do {													\
+	unsigned long cur_flags, mismatch;								\
+													\
+	cur_flags = efi_call_virt_save_flags();								\
+													\
+	mismatch = state_flags ^ cur_flags;								\
+	if (!WARN_ON_ONCE(mismatch & ARCH_EFI_IRQ_FLAGS_MASK))						\
+		break;											\
+													\
+	add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_NOW_UNRELIABLE); 					\
+	pr_err_ratelimited(FW_BUG "IRQ flags corrupted (0x%08lx=>0x%08lx) by EFI call from %pS\n",	\
+			   state_flags, cur_flags,							\
+			   caller ?: __builtin_return_address(0));					\
+	arch_efi_restore_flags(state_flags);								\
+} while(0);
+#endif
+
 unsigned long efi_call_virt_save_flags(void)
 {
 	unsigned long flags;
@@ -147,18 +166,7 @@ unsigned long efi_call_virt_save_flags(void)
 
 void efi_call_virt_check_flags(unsigned long flags, const void *caller)
 {
-	unsigned long cur_flags, mismatch;
-
-	cur_flags = efi_call_virt_save_flags();
-
-	mismatch = flags ^ cur_flags;
-	if (!WARN_ON_ONCE(mismatch & ARCH_EFI_IRQ_FLAGS_MASK))
-		return;
-
-	add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_NOW_UNRELIABLE);
-	pr_err_ratelimited(FW_BUG "IRQ flags corrupted (0x%08lx=>0x%08lx) by EFI call from %pS\n",
-			   flags, cur_flags, caller ?: __builtin_return_address(0));
-	arch_efi_restore_flags(flags);
+	arch_efi_check_flags(flags, caller);
 }
 
 /*
-- 
2.34.1

[RFC PATCH 23/36] arm64/efi: Implement override for IRQ flags checks

[Vladimir Murzin]

Take advantage of the recent changes to the way arm64 represents IRQ
flags and implement an architecture-specific override for the EFI IRQ
flags check.

IRQ flags now carry both DAIF and PMR state. EFI is not expected to
modify either of them, so check both when verifying that IRQ state is
preserved across EFI runtime calls.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/efi.h | 38 ++++++++++++++++++++++++++++++------
 1 file changed, 32 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h
index e8a9783235cb..479741d52c90 100644
--- a/arch/arm64/include/asm/efi.h
+++ b/arch/arm64/include/asm/efi.h
@@ -5,6 +5,7 @@
 #include <asm/boot.h>
 #include <asm/cpufeature.h>
 #include <asm/fpsimd.h>
+#include <asm/interrupts/common_flags.h>
 #include <asm/io.h>
 #include <asm/memory.h>
 #include <asm/mmu_context.h>
@@ -48,18 +49,43 @@ void arch_efi_call_virt_teardown(void);
 	(efi_rt_stack_top != NULL &&					\
 	 on_task_stack(current, READ_ONCE(efi_rt_stack_top[-1]), 1))
 
-#define ARCH_EFI_IRQ_FLAGS_MASK (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
-
 /*
  * Even when Linux uses IRQ priorities for IRQ disabling, EFI does not.
  * And EFI shouldn't really play around with priority masking as it is not aware
  * which priorities the OS has assigned to its interrupts.
  */
 #define arch_efi_save_flags(state_flags)		\
-	((void)((state_flags) = read_sysreg(daif)))
-
-#define arch_efi_restore_flags(state_flags)	write_sysreg(state_flags, daif)
-
+	((void)((state_flags) = arch_local_save_flags()))
+
+#define arch_efi_restore_flags(state_flags)			\
+do {								\
+	arm64_exc_hwstate_t __st = {.flags=state_flags};	\
+								\
+	arm64_update_exc_hwstate(__st, true, true);		\
+ } while(0)
+
+#define arch_efi_check_flags(state_flags, caller)									\
+do {															\
+	arm64_exc_hwstate_t __actual = {.flags =  arch_local_save_flags() };						\
+	arm64_exc_hwstate_t __expected = {.flags=state_flags};								\
+															\
+	if (__expected.flags == __actual.flags)										\
+		break;													\
+															\
+	add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_NOW_UNRELIABLE);							\
+															\
+	if (system_uses_irq_prio_masking()) {										\
+		pr_err_ratelimited(FW_BUG "DAIF+PMR corrupted (0x%x + 0x%x=>0x%x + 0x%x) by EFI call from %pS\n",	\
+				   __expected.daif, __expected.pmr, __actual.daif, __actual.pmr,			\
+				   caller ?: __builtin_return_address(0));						\
+	} else {													\
+		pr_err_ratelimited(FW_BUG "DAIF corrupted (0x%x=>0x%x) by EFI call from %pS\n",				\
+				   __expected.daif, __actual.daif,							\
+				   caller ?: __builtin_return_address(0));						\
+	}														\
+															\
+	arch_efi_restore_flags(state_flags);										\
+} while(0)
 
 /* arch specific definitions used by the stub code */
 
-- 
2.34.1

[RFC PATCH 24/36] arm64: booting: Document boot requirements for FEAT_NMI

[Vladimir Murzin]

From: Mark Brown <broonie@kernel.org>

In order to use FEAT_NMI we must be able to use ALLINT, require that
it behave as though not trapped when it is present.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 Documentation/arch/arm64/booting.rst | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Documentation/arch/arm64/booting.rst b/Documentation/arch/arm64/booting.rst
index 13ef311dace8..3170b26ae981 100644
--- a/Documentation/arch/arm64/booting.rst
+++ b/Documentation/arch/arm64/booting.rst
@@ -568,6 +568,12 @@ Before jumping into the kernel, the following conditions must be met:
 
     - HCRX_EL2.EnASR (bit 2) must be initialised to 0b1.
 
+  For CPUs with support for Non-maskable Interrupts (FEAT_NMI):
+
+  - If the kernel is entered at EL1 and EL2 is present:
+
+    - HCRX_EL2.TALLINT (bit 6) must be initialised to 0b0.
+
 The requirements described above for CPU mode, caches, MMUs, architected
 timers, coherency and system registers apply to all CPUs.  All CPUs must
 enter the kernel in the same exception level.  Where the values documented
-- 
2.34.1

[RFC PATCH 25/36] arm64: sysreg: Add definitions for immediate versions of MSR ALLINT

[Vladimir Murzin]

From: Mark Brown <broonie@kernel.org>

Encodings are provided for ALLINT which allow setting of ALLINT.ALLINT
using an immediate rather than requiring that a register be loaded with
the value to write. Since these don't currently fit within the scheme we
have for sysreg generation add manual encodings like we currently do for
other similar registers such as SVCR.

Since it is required that these immediate versions be encoded with xzr
as the source register, provide asm wrappers which ensure this is the
case.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/interrupts/common_flags.h | 16 ++++++++++++++++
 arch/arm64/include/asm/sysreg.h                  |  2 ++
 2 files changed, 18 insertions(+)

diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
index 6ce60d1519e8..72ed6e75d146 100644
--- a/arch/arm64/include/asm/interrupts/common_flags.h
+++ b/arch/arm64/include/asm/interrupts/common_flags.h
@@ -44,6 +44,22 @@ typedef enum arm64_exc_context {
 	CRITICAL_CONTEXT,
 } arm64_exc_context_t;
 
+
+#ifdef CONFIG_ARM64_NMI
+static __always_inline void _allint_clear(void)
+{
+	asm volatile(__msr_s(SYS_ALLINT_CLR, "xzr"));
+}
+
+static __always_inline void _allint_set(void)
+{
+	asm volatile(__msr_s(SYS_ALLINT_SET, "xzr"));
+}
+#else
+static __always_inline void _allint_clear(void) {}
+static __always_inline void _allint_set(void) {}
+#endif /* CONFIG_ARM64_NMI */
+
 static __always_inline
 arm64_exc_hwstate_t __arm64_exc_hwstate_of_process_context(void)
 {
diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index 7aa08d59d494..6d2c6bc1b985 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -179,6 +179,8 @@
  * System registers, organised loosely by encoding but grouped together
  * where the architected name contains an index. e.g. ID_MMFR<n>_EL1.
  */
+#define SYS_ALLINT_CLR			sys_reg(0, 1, 4, 0, 0)
+#define SYS_ALLINT_SET			sys_reg(0, 1, 4, 1, 0)
 #define SYS_SVCR_SMSTOP_SM_EL0		sys_reg(0, 3, 4, 2, 3)
 #define SYS_SVCR_SMSTART_SM_EL0		sys_reg(0, 3, 4, 3, 3)
 #define SYS_SVCR_SMSTOP_SMZA_EL0	sys_reg(0, 3, 4, 6, 3)
-- 
2.34.1

[RFC PATCH 26/36] arm64: ptrace: Add PSR_ALLINT_BIT

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

When FEAT_NMI is implemented, ALLINT is part of PSTATE and will be saved
to SPSR on exception entry.
As it can mask interrupts, depending on configuration, we will need to
check its value in interruted tasks, so add the corresponding bit
to ptrace.h.

Given its impact on interrupt and NMI masking, use it to update
`print_pstate()` so that it displays ALLINT state.
Display it after DAIF so that interrupt masking fields are together.

Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/uapi/asm/ptrace.h | 1 +
 arch/arm64/kernel/process.c          | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/uapi/asm/ptrace.h b/arch/arm64/include/uapi/asm/ptrace.h
index 6fed93fb2536..99352dd823cc 100644
--- a/arch/arm64/include/uapi/asm/ptrace.h
+++ b/arch/arm64/include/uapi/asm/ptrace.h
@@ -48,6 +48,7 @@
 #define PSR_D_BIT	0x00000200
 #define PSR_BTYPE_MASK	0x00000c00
 #define PSR_SSBS_BIT	0x00001000
+#define PSR_ALLINT_BIT	0x00002000
 #define PSR_PAN_BIT	0x00400000
 #define PSR_UAO_BIT	0x00800000
 #define PSR_DIT_BIT	0x01000000
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index ddda8d7aee24..8d4ceef4f67f 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -183,7 +183,7 @@ static void print_pstate(struct pt_regs *regs)
 		const char *btype_str = btypes[(pstate & PSR_BTYPE_MASK) >>
 					       PSR_BTYPE_SHIFT];
 
-		printk("pstate: %08llx (%c%c%c%c %c%c%c%c %cPAN %cUAO %cTCO %cDIT %cSSBS BTYPE=%s)\n",
+		printk("pstate: %08llx (%c%c%c%c %c%c%c%c %cALLINT %cPAN %cUAO %cTCO %cDIT %cSSBS BTYPE=%s)\n",
 			pstate,
 			pstate & PSR_N_BIT ? 'N' : 'n',
 			pstate & PSR_Z_BIT ? 'Z' : 'z',
@@ -193,6 +193,7 @@ static void print_pstate(struct pt_regs *regs)
 			pstate & PSR_A_BIT ? 'A' : 'a',
 			pstate & PSR_I_BIT ? 'I' : 'i',
 			pstate & PSR_F_BIT ? 'F' : 'f',
+			pstate & PSR_ALLINT_BIT ? '+' : '-',
 			pstate & PSR_PAN_BIT ? '+' : '-',
 			pstate & PSR_UAO_BIT ? '+' : '-',
 			pstate & PSR_TCO_BIT ? '+' : '-',
-- 
2.34.1

[RFC PATCH 27/36] arm64: idreg: Add an override for FEAT_NMI

[Vladimir Murzin]

From: Mark Brown <broonie@kernel.org>

Add a named override for FEAT_NMI, allowing it to be explicitly disabled
in case of problems.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/kernel/pi/idreg-override.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c
index bc57b290e5e7..e5ea280452c3 100644
--- a/arch/arm64/kernel/pi/idreg-override.c
+++ b/arch/arm64/kernel/pi/idreg-override.c
@@ -154,6 +154,7 @@ static const struct ftr_set_desc pfr1 __prel64_initconst = {
 		FIELD("bt", ID_AA64PFR1_EL1_BT_SHIFT, NULL ),
 		FIELD("gcs", ID_AA64PFR1_EL1_GCS_SHIFT, NULL),
 		FIELD("mte", ID_AA64PFR1_EL1_MTE_SHIFT, NULL),
+		FIELD("nmi", ID_AA64PFR1_EL1_NMI_SHIFT, NULL),
 		FIELD("sme", ID_AA64PFR1_EL1_SME_SHIFT, pfr1_sme_filter),
 		FIELD("mpam_frac", ID_AA64PFR1_EL1_MPAM_frac_SHIFT, NULL),
 		{}
-- 
2.34.1

[RFC PATCH 28/36] arm64: cpufeature: Detect PE support for FEAT_NMI

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Use of FEAT_NMI requires that all the PEs in the system and the GIC
have NMI support. This patch implements the PE part of that detection.

In order to avoid problematic interactions between real and pseudo NMIs
we disable the architected feature if the user has enabled pseudo NMIs
on the command line. If this is done on a system where support for the
architected feature is detected then a warning is printed during boot in
order to help users spot what is likely to be a misconfiguration.

As KVM does not care about the host kernel supporting FEAT_NMI or not
to allow guests to use it, split the CPU cap in two : ARM64_HAS_NMI is
the hardware support, ARM64_NMI is the kernel making use of it.

Co-developed-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/cpucaps.h    |  2 +
 arch/arm64/include/asm/cpufeature.h | 10 +++++
 arch/arm64/kernel/cpufeature.c      | 68 ++++++++++++++++++++++++++++-
 arch/arm64/tools/cpucaps            |  2 +
 4 files changed, 81 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 25c61cda901c..176a63f1cc1b 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -75,6 +75,8 @@ cpucap_is_possible(const unsigned int cap)
 		return IS_ENABLED(CONFIG_HW_PERF_EVENTS);
 	case ARM64_HAS_LSUI:
 		return IS_ENABLED(CONFIG_ARM64_LSUI);
+	case ARM64_NMI:
+		return IS_ENABLED(CONFIG_ARM64_NMI);
 	}
 
 	return true;
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index 95cd8d4acd26..84b817b29155 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -816,6 +816,16 @@ static __always_inline bool system_uses_irq_prio_masking(void)
 	return alternative_has_cap_unlikely(ARM64_HAS_GIC_PRIO_MASKING);
 }
 
+static __always_inline bool system_supports_nmi(void)
+{
+	return alternative_has_cap_unlikely(ARM64_HAS_NMI);
+}
+
+static __always_inline bool system_uses_nmi(void)
+{
+	return alternative_has_cap_unlikely(ARM64_NMI);
+}
+
 static inline bool system_supports_mte(void)
 {
 	return alternative_has_cap_unlikely(ARM64_MTE);
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 9a22df0c5120..bb22292ebabc 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -83,13 +83,14 @@
 #include <asm/cpu_ops.h>
 #include <asm/fpsimd.h>
 #include <asm/hwcap.h>
+#include <asm/hypervisor.h>
 #include <asm/insn.h>
+#include <asm/interrupts/common_flags.h>
 #include <asm/kvm_host.h>
 #include <asm/mmu.h>
 #include <asm/mmu_context.h>
 #include <asm/mpam.h>
 #include <asm/mte.h>
-#include <asm/hypervisor.h>
 #include <asm/processor.h>
 #include <asm/smp.h>
 #include <asm/sysreg.h>
@@ -315,6 +316,7 @@ static const struct arm64_ftr_bits ftr_id_aa64pfr1[] = {
 	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_GCS),
 		       FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_GCS_SHIFT, 4, 0),
 	S_ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MTE_frac_SHIFT, 4, 0),
+	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_NMI_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SME),
 		       FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_SME_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MPAM_frac_SHIFT, 4, 0),
@@ -2320,6 +2322,51 @@ static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry
 }
 #endif
 
+#ifdef CONFIG_ARM64_NMI
+static bool can_use_nmi(const struct arm64_cpu_capabilities *entry, int scope)
+{
+	/*
+	 * ARM64_HAS_NMI has a lower index, and is a boot CPU
+	 * feature, so will be detected earlier.
+	 */
+	BUILD_BUG_ON(ARM64_NMI <= ARM64_HAS_NMI);
+	if (!cpus_have_cap(ARM64_HAS_NMI))
+		return false;
+
+	/*
+	 * Having both real and pseudo NMIs enabled simultaneously is
+	 * likely to cause confusion.  Since pseudo NMIs must be
+	 * enabled with an explicit command line option, if the user
+	 * has set that option on a system with real NMIs for some
+	 * reason assume they know what they're doing.
+	 *
+	 * ARM64_HAS_GIC_PRIO_MASKING has a lower index, and is a boot CPU
+	 * feature, so will be detected earlier.
+	 */
+	BUILD_BUG_ON(IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) &&
+			(ARM64_NMI <= ARM64_HAS_GIC_PRIO_MASKING));
+	if (cpus_have_cap(ARM64_HAS_GIC_PRIO_MASKING)) {
+		pr_info("Pseudo NMI enabled, not using architected NMI\n");
+		return false;
+	}
+
+	return true;
+}
+
+static void nmi_enable(const struct arm64_cpu_capabilities *__unused)
+{
+	/*
+	 * Enable use of NMIs controlled by ALLINT, SPINTMASK should
+	 * be clear by default but make it explicit that we are using
+	 * this mode.  Ensure that ALLINT is clear first in order to
+	 * avoid leaving things masked.
+	 */
+	_allint_clear();
+	sysreg_clear_set(sctlr_el1, SCTLR_EL1_SPINTMASK, SCTLR_EL1_NMI);
+	isb();
+}
+#endif
+
 static bool can_trap_icv_dir_el1(const struct arm64_cpu_capabilities *entry,
 				 int scope)
 {
@@ -3198,6 +3245,25 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.matches = has_cpuid_feature,
 		ARM64_CPUID_FIELDS(ID_AA64ISAR3_EL1, LSUI, IMP)
 	},
+#endif
+	{
+		.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
+		.capability = ARM64_HAS_NMI,
+		.matches = has_cpuid_feature,
+		ARM64_CPUID_FIELDS(ID_AA64PFR1_EL1, NMI, IMP)
+	},
+#ifdef CONFIG_ARM64_NMI
+	/*
+	 * Depends on ARM64_HAS_NMI
+	 * Checks for conflict with pseudo-NMIs, giving them priority.
+	 */
+	{
+		.desc = "Non-maskable Interrupts",
+		.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
+		.capability = ARM64_NMI,
+		.matches = can_use_nmi,
+		.cpu_enable = nmi_enable,
+	},
 #endif
 	{},
 };
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index 9b85a84f6fd4..2117b3ef0b82 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -51,6 +51,7 @@ HAS_LS64_V
 HAS_LSUI
 HAS_MOPS
 HAS_NESTED_VIRT
+HAS_NMI
 HAS_BBML2_NOABORT
 HAS_PAN
 HAS_PMUV3
@@ -80,6 +81,7 @@ MTE
 MTE_ASYMM
 MTE_FAR
 MTE_STORE_ONLY
+NMI
 SME
 SME_FA64
 SME2
-- 
2.34.1

[RFC PATCH 29/36] arm64: nmi: Manage masking for superpriority interrupts

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

Extend logic to handle and debug exception context/state with knowlage
of FEAT_NMI.

Take care to order writes to ALLINT relative to DAIF, as clearing
ALLINT before DAIF could result in taking an NMI while DAIF is fully
masked, as could setting it after DAIF.

Since superpriority interrupts are not masked through DAIF like pseduo
NMIs are, we also need to modify the assembler macros for managing
DAIF to ensure that the masking is done in the assembly code.

Note that save_and_disable_irq/restore_irq and
save_and_disable_daif/restore_irq pairs are used in distinct
contextes:

- former is used in context of SW PAN to quickly disable/enable
  preemption

- latter is used to completely mask all exceptions.

For that reason split save_and_disable_daif/restore_irq into more
generic exception save restore pair and plumb with FEAT_NMI logic.

Co-developed-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
---
 arch/arm64/include/asm/assembler.h            | 20 ++++++++-
 .../include/asm/interrupts/common_flags.h     | 41 +++++++++++++++++--
 arch/arm64/include/asm/interrupts/entry.h     | 18 +++++---
 arch/arm64/include/asm/interrupts/masking.h   | 25 ++++++-----
 arch/arm64/include/asm/irqflags.h             |  1 +
 arch/arm64/kernel/entry.S                     | 12 +++---
 6 files changed, 89 insertions(+), 28 deletions(-)

diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 0b58b550e8dc..bcdbc308afba 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -37,11 +37,27 @@
 /*
  * Save/restore interrupts.
  */
-	.macro save_and_disable_daif, flags
-	mrs	\flags, daif
+	.macro save_and_disable_exceptions, flags, tmp
+	mrs     \flags, daif		// updates flags[9:6] with DAIF
+#ifdef CONFIG_ARM64_NMI
+alternative_if ARM64_NMI
+	mrs_s	\tmp, SYS_ALLINT	// updates tmp[13] with AllInt
+	msr_s	SYS_ALLINT_SET, xzr
+	orr	\flags, \flags, \tmp	// now flags[13,9:6] carry pair of AllInt,DAIF
+alternative_else_nop_endif
+#endif
 	msr	daifset, #0xf
 	.endm
 
+	.macro  restore_exceptions, flags
+	msr     daif, \flags		// bits other than flags[9:6] are ignored
+#ifdef CONFIG_ARM64_NMI
+alternative_if ARM64_NMI
+	msr_s	SYS_ALLINT, \flags	// bits other than flags[13] are ignored
+alternative_else_nop_endif
+#endif
+	.endm
+
 	.macro	save_and_disable_irq, flags
 	mrs	\flags, daif
 	msr	daifset, #3
diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
index 72ed6e75d146..315aaec3ea34 100644
--- a/arch/arm64/include/asm/interrupts/common_flags.h
+++ b/arch/arm64/include/asm/interrupts/common_flags.h
@@ -20,6 +20,14 @@
 /*
  * Exception context mapping
  *
+ * FEAT_NMI
+ *
+ * CRITICAL -> DAIF + AllInt (corresponds to the state on exception entry)
+ * ERROR    ->  AIF + AllInt
+ * NONMI    ->   IF + AllInt
+ * NOIRQ    ->   IF
+ * PROCESS  ->    0
+ *
  * pseudo-NMI
  *
  * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
@@ -76,6 +84,7 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
 		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
 
 	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
+
 }
 
 static __always_inline
@@ -84,6 +93,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
 	if (system_uses_irq_prio_masking())
 		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
 
+	if (system_uses_nmi())
+               return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .allint=ALLINT_ALLINT};
+
 	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
 }
 
@@ -93,6 +105,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
 	if (system_uses_irq_prio_masking())
 		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
 
+	if (system_uses_nmi())
+		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .allint=ALLINT_ALLINT};
+
 	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
 }
 
@@ -102,6 +117,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
 	if (system_uses_irq_prio_masking())
 		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
 
+	if (system_uses_nmi())
+		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .allint=ALLINT_ALLINT};
+
 	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
 }
 
@@ -131,6 +149,9 @@ arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
 	if (system_uses_irq_prio_masking())
 		state.pmr = regs->pmr;
 
+	if (system_uses_nmi())
+		state.allint = regs->pstate & PSR_ALLINT_BIT;
+
 	return state;
 }
 
@@ -150,6 +171,9 @@ void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
 	if (system_uses_irq_prio_masking()) {
 		WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
 			  actual.daif, actual.pmr, expected.daif, expected.pmr);
+	} else if (system_uses_nmi()) {
+		WARN_ONCE(1, "Unexpected DAIF+ALLINT: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
+			  actual.daif, actual.allint, expected.daif, expected.allint);
 	} else {
 		WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
 			  actual.daif, expected.daif);
@@ -165,7 +189,7 @@ void arm64_debug_exc_context(arm64_exc_context_t context)
 }
 
 static __always_inline
-void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
+void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr, bool update_allint)
 {
 	if (system_uses_irq_prio_masking() &&
 	    update_pmr &&
@@ -193,8 +217,19 @@ void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
 		pmr_sync();
 	}
 
+	/*
+	 * Try to order ALLINT writes to be consistent with the DAIF state :
+	 * we don't want to take an NMI with DAIF masked or when it should
+	 * be masked but isn't yet.
+	 */
+	if (system_uses_nmi() && update_allint && hwstate.allint)
+		_allint_set();
+
 	write_sysreg(hwstate.daif, daif);
 
+	if (system_uses_nmi() && update_allint && !hwstate.allint)
+		_allint_clear();
+
 	if (system_uses_irq_prio_masking() &&
 	    update_pmr &&
 	    hwstate.pmr == GIC_PRIO_IRQON) {
@@ -204,10 +239,10 @@ void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
 }
 
 static __always_inline
-void arm64_update_exc_context(arm64_exc_context_t context, bool update_pmr)
+void arm64_update_exc_context(arm64_exc_context_t context, bool update_pmr,  bool update_allint)
 {
 	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(context);
 
-	arm64_update_exc_hwstate(hwstate, update_pmr);
+	arm64_update_exc_hwstate(hwstate, update_pmr, update_allint);
 }
 #endif /* __ASM_INTERRUPTS_COMMON_FLAGS_H */
diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
index 3034c490ed66..986536ecd058 100644
--- a/arch/arm64/include/asm/interrupts/entry.h
+++ b/arch/arm64/include/asm/interrupts/entry.h
@@ -10,12 +10,12 @@
 #include <asm/cpufeature.h>
 #include <asm/interrupts/common_flags.h>
 
-
 static __always_inline
 arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
 						  arm64_exc_hwstate_t next)
 {
 	bool update_pmr = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
+	bool update_allint = system_uses_nmi() && prev.allint != next.allint;
 
 	arm64_debug_exc_hwstate(prev);
 
@@ -25,7 +25,7 @@ arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
 	if (!arch_irqs_disabled_flags(next.flags))
 		trace_hardirqs_on();
 
-	arm64_update_exc_hwstate(next, update_pmr);
+	arm64_update_exc_hwstate(next, update_pmr, update_allint);
 
 	if (arch_irqs_disabled_flags(next.flags))
 		trace_hardirqs_off();
@@ -49,15 +49,18 @@ arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
 
 	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
 		bool pnmi = system_uses_irq_prio_masking();
+		bool nmi = system_uses_nmi();
 
 		WARN_ON_ONCE(context > ERROR_CONTEXT &&
 			     prev.daif == DAIF_ERRCTX);
 
 		WARN_ON_ONCE(context > NONMI_CONTEXT &&
-			     prev.daif == DAIF_PROCCTX_NOIRQ);
+			     ((nmi && prev.daif == DAIF_PROCCTX_NOIRQ && prev.allint == ALLINT_ALLINT) ||
+			      (!nmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
 
 		WARN_ON_ONCE(context > NOIRQ_CONTEXT &&
-			     pnmi && prev.pmr == GIC_PRIO_IRQOFF);
+			     ((pnmi && prev.pmr == GIC_PRIO_IRQOFF) ||
+			      (nmi && prev.daif == DAIF_PROCCTX_NOIRQ && prev.allint != ALLINT_ALLINT)));
 
 		WARN_ON_ONCE(context > PROCESS_CONTEXT &&
 			     ((pnmi && prev.daif == DAIF_PROCCTX && prev.pmr == GIC_PRIO_IRQON) ||
@@ -74,6 +77,7 @@ arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
 
 	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
 		bool pnmi = system_uses_irq_prio_masking();
+		bool nmi = system_uses_nmi();
 
 		WARN_ON_ONCE(context < CRITICAL_CONTEXT &&
 			     prev.daif == DAIF_MASK);
@@ -82,11 +86,13 @@ arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
 			     prev.daif == DAIF_ERRCTX);
 
 		WARN_ON_ONCE(context < NONMI_CONTEXT &&
-			     pnmi && prev.daif == DAIF_PROCCTX_NOIRQ);
+			     ((pnmi && prev.daif == DAIF_PROCCTX_NOIRQ) ||
+			      (nmi &&  prev.daif == DAIF_PROCCTX_NOIRQ && prev.allint == ALLINT_ALLINT)));
 
 		WARN_ON_ONCE(context < NOIRQ_CONTEXT &&
 			     ((pnmi && prev.pmr == GIC_PRIO_IRQOFF) ||
-			      (!pnmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
+			      (nmi && prev.daif == DAIF_PROCCTX_NOIRQ && prev.allint != ALLINT_ALLINT) ||
+			      (!pnmi && !nmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
 	}
 
 	return __arm64_switch_exc_hwstate_to(prev, next);
diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
index 9dc37a571094..178e915f43b9 100644
--- a/arch/arm64/include/asm/interrupts/masking.h
+++ b/arch/arm64/include/asm/interrupts/masking.h
@@ -30,6 +30,7 @@ arm64_exc_hwstates_t local_all_irqs_save_mask(arm64_exc_context_t new)
 
 	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
 		bool pnmi = system_uses_irq_prio_masking();
+		bool nmi = system_uses_nmi();
 
 		WARN_ON_ONCE(new < CRITICAL_CONTEXT &&
 			     actual.daif == DAIF_MASK);
@@ -38,14 +39,16 @@ arm64_exc_hwstates_t local_all_irqs_save_mask(arm64_exc_context_t new)
 			     actual.daif == DAIF_ERRCTX);
 
 		WARN_ON_ONCE(new < NONMI_CONTEXT &&
-			     pnmi && actual.daif == DAIF_PROCCTX_NOIRQ);
+			     ((pnmi && actual.daif == DAIF_PROCCTX_NOIRQ) ||
+			      (nmi &&  actual.daif == DAIF_PROCCTX_NOIRQ && actual.allint == ALLINT_ALLINT)));
 
 		WARN_ON_ONCE(new < NOIRQ_CONTEXT &&
 			     ((pnmi && actual.pmr == GIC_PRIO_IRQOFF) ||
-			      (!pnmi && actual.daif == DAIF_PROCCTX_NOIRQ)));
+			      (nmi && actual.daif == DAIF_PROCCTX_NOIRQ && actual.allint != ALLINT_ALLINT) ||
+			      (!pnmi && !nmi && actual.daif == DAIF_PROCCTX_NOIRQ)));
 	}
 
-	arm64_update_exc_hwstate(state, actual.pmr != state.pmr);
+	arm64_update_exc_hwstate(state, actual.pmr != state.pmr, actual.allint != state.allint);
 
 	if (!arch_irqs_disabled_flags(actual.flags))
 		trace_hardirqs_off();
@@ -60,7 +63,7 @@ static inline void local_all_irqs_restore(arm64_exc_hwstates_t states)
 	if (!arch_irqs_disabled_flags(states.saved.flags))
 		trace_hardirqs_on();
 
-	arm64_update_exc_hwstate(states.saved, true);
+	arm64_update_exc_hwstate(states.saved, true, true);
 }
 
 #ifdef CONFIG_DEBUG_IRQFLAGS
@@ -70,7 +73,7 @@ void local_all_irqs_cpu_init_mask(arm64_exc_context_t context)
 	WARN_ON(__this_cpu_read(irqs_masks_cpu_init_done));
 	if (context == PROCESS_CONTEXT)
 		trace_hardirqs_on();
-	arm64_update_exc_context(context, true);
+	arm64_update_exc_context(context, true, true);
 	__this_cpu_write(irqs_masks_cpu_init_done, true);
 	__this_cpu_write(irqs_masks_cpu_final_done, false);
 }
@@ -78,7 +81,7 @@ void local_all_irqs_cpu_init_mask(arm64_exc_context_t context)
 static inline void local_all_irqs_final_mask(void)
 {
 	WARN_ON(__this_cpu_read(irqs_masks_cpu_final_done));
-	arm64_update_exc_context(CRITICAL_CONTEXT, true);
+	arm64_update_exc_context(CRITICAL_CONTEXT, true, true);
 	trace_hardirqs_off();
 	__this_cpu_write(irqs_masks_cpu_final_done, true);
 	__this_cpu_write(irqs_masks_cpu_init_done, false);
@@ -89,12 +92,12 @@ void local_all_irqs_cpu_init_mask(arm64_exc_context_t context)
 {
 	if (context == PROCESS_CONTEXT)
 		trace_hardirqs_on();
-	arm64_update_exc_context(context, true);
+	arm64_update_exc_context(context, true, true);
 }
 
 static inline void local_all_irqs_final_mask(void)
 {
-	arm64_update_exc_context(CRITICAL_CONTEXT, true);
+	arm64_update_exc_context(CRITICAL_CONTEXT, true, true);
 	trace_hardirqs_off();
 }
 #endif /* CONFIG_DEBUG_IRQFLAGS */
@@ -130,7 +133,7 @@ static inline arm64_exc_hwstates_t local_all_irqs_force_daif_save(void)
 		states.expected.daif = states.saved.daif | DAIF_PROCCTX_NOIRQ;
 		states.expected.pmr = GIC_PRIO_IRQON;
 
-		arm64_update_exc_hwstate(states.expected, true);
+		arm64_update_exc_hwstate(states.expected, true, false);
 	}
 
 	return states;
@@ -153,7 +156,7 @@ void local_all_irqs_force_daif_restore(arm64_exc_hwstates_t states)
 
 	if (system_uses_irq_prio_masking()) {
 		arm64_debug_exc_hwstate(states.expected);
-		arm64_update_exc_hwstate(states.saved, true);
+		arm64_update_exc_hwstate(states.saved, true, false);
 	}
 }
 
@@ -167,7 +170,7 @@ static inline void local_interrupt_priority_init(void)
 	WARN_ON(read_sysreg(daif) & PSR_A_BIT);
 	lockdep_assert_irqs_disabled();
 
-	arm64_update_exc_context(NOIRQ_CONTEXT, true);
+	arm64_update_exc_context(NOIRQ_CONTEXT, true, true);
 }
 
 #endif /* __ASM_INTERRUPTS_MASKING_H */
diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index 585d3ce15d80..b571d34bf11d 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -31,6 +31,7 @@
 typedef union arm64_exc_hwstate {
 	struct {
 		u16 daif;
+		u16 allint;
 		u8 pmr;
 	};
 	unsigned long flags;
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index cb3be770f2d0..39ea3fdeb03a 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -815,7 +815,7 @@ SYM_CODE_END(__bp_harden_el1_vectors)
  *
  */
 SYM_FUNC_START(cpu_switch_to)
-	save_and_disable_daif x11
+	save_and_disable_exceptions x11, x12
 	mov	x10, #THREAD_CPU_CONTEXT
 	add	x8, x0, x10
 	mov	x9, sp
@@ -839,7 +839,7 @@ SYM_FUNC_START(cpu_switch_to)
 	ptrauth_keys_install_kernel x1, x8, x9, x10
 	scs_save x0
 	scs_load_current
-	restore_irq x11
+	restore_exceptions x11
 	ret
 SYM_FUNC_END(cpu_switch_to)
 NOKPROBE(cpu_switch_to)
@@ -866,7 +866,7 @@ NOKPROBE(ret_from_fork)
  * Calls func(regs) using this CPU's irq stack and shadow irq stack.
  */
 SYM_FUNC_START(call_on_irq_stack)
-	save_and_disable_daif x9
+	save_and_disable_exceptions x9, x10
 #ifdef CONFIG_SHADOW_CALL_STACK
 	get_current_task x16
 	scs_save x16
@@ -881,10 +881,10 @@ SYM_FUNC_START(call_on_irq_stack)
 
 	/* Move to the new stack and call the function there */
 	add	sp, x16, #IRQ_STACK_SIZE
-	restore_irq x9
+	restore_exceptions x9
 	blr	x1
 
-	save_and_disable_daif x9
+	save_and_disable_exceptions x9, x10
 	/*
 	 * Restore the SP from the FP, and restore the FP and LR from the frame
 	 * record.
@@ -892,7 +892,7 @@ SYM_FUNC_START(call_on_irq_stack)
 	mov	sp, x29
 	ldp	x29, x30, [sp], #16
 	scs_load_current
-	restore_irq x9
+	restore_exceptions x9
 	ret
 SYM_FUNC_END(call_on_irq_stack)
 NOKPROBE(call_on_irq_stack)
-- 
2.34.1

[RFC PATCH 30/36] arm64: irq: Report FEAT_NMI masking local IRQs

[Vladimir Murzin]

From: Ada Couprie Diaz <ada.coupriediaz@arm.com>

As we clear SCTLR_EL1.SPINTMASK when enabling FEAT_NMI, ALLINT masks
IRQs and FIQs regardless of superpriority.
Update irqflags.h and `regs_irqs_disabled()` to take it into account,
so we properly keep track of

We have documentation at the top of irqflags.h which explains the DAIF
masking. Since the additional masking with NMIs is related and also
covers the IF in DAIF, extend the comment to note what's going on
with NMIs.

Co-developed-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/irqflags.h | 17 +++++++++++++++++
 arch/arm64/include/asm/ptrace.h   |  9 ++++++++-
 2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index b571d34bf11d..a291a231d644 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -20,6 +20,16 @@
  * always masked and unmasked together, and have no side effects for other
  * flags. Keeping to this order makes it easier for entry.S to know which
  * exceptions should be unmasked.
+ *
+ * With the addition of the FEAT_NMI extension we gain an additional
+ * class of superpriority IRQ/FIQ which is separately masked with a
+ * choice of modes controlled by SCTLR_ELn.{SPINTMASK,NMI}.
+ * Linux sets SPINTMASK to 0 and NMI to 1 which results in ALLINT.ALLINT
+ * masking both superpriority interrupts and IRQ/FIQ regardless of the
+ * I and F settings. Since these superpriority interrupts are being
+ * used as NMIs we do not include them in the interrupt masking here,
+ * anything that requires that NMIs be masked needs to explicitly do so,
+ * but we do check for ALLINT masking IRQs/FIQs.
  */
 
  /*
@@ -103,6 +113,9 @@ static __always_inline unsigned long arch_local_save_flags(void)
 {
 	arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
 
+	if (system_uses_nmi())
+		hwstate.allint = read_sysreg_s(SYS_ALLINT);
+
 	if (system_uses_irq_prio_masking())
 		hwstate.pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
 
@@ -117,6 +130,10 @@ static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
 	if (hwstate.daif & PSR_I_BIT)
 		return true;
 
+	/* SCTLR_EL1.SPINTMASK is clear, so ALLINT masks *all* IRQs/FIQs. */
+	if (system_uses_nmi() && hwstate.allint > 0)
+		return true;
+
 	if (system_uses_irq_prio_masking() && hwstate.pmr < GIC_PRIO_IRQON)
 		return true;
 
diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index c630165882e8..096d85d75bd7 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -206,9 +206,16 @@ static inline void forget_syscall(struct pt_regs *regs)
 		(regs)->pmr >= GIC_PRIO_IRQON :				\
 		true)
 
+#define irqs_allint_clear(regs)				\
+	(system_uses_nmi() ?				\
+		((regs)->pstate & PSR_ALLINT_BIT) == 0:	\
+		true)
+
 static __always_inline bool regs_irqs_disabled(const struct pt_regs *regs)
 {
-	return (regs->pstate & PSR_I_BIT) || !irqs_priority_unmasked(regs);
+	return (regs->pstate & PSR_I_BIT) ||
+		!irqs_priority_unmasked(regs) ||
+		!irqs_allint_clear(regs);
 }
 
 #define interrupts_enabled(regs)	(!regs_irqs_disabled(regs))
-- 
2.34.1

[RFC PATCH 31/36] arm64: nmi: Add handling of superpriority interrupts as NMIs

[Vladimir Murzin]

From: Mark Brown <broonie@kernel.org>

Our goal with superpriority interrupts is to use them as NMIs, taking
advantage of the much smaller regions where they are masked to allow
prompt handling of the most time-critical interrupts.

When an interrupt is configured with superpriority, we enter EL1 as we
do for any other interrupt. The presence of a superpriority interrupt
is indicated by a status bit in ISR_EL1. We check this bit before
unmasking interrupts in elX_interrupt(), and if a superpriority
interrupt is pending, we handle it as an NMI. Otherwise, normal
interrupts are handled as usual.

Since superpriority interrupts are always handled as NMIs, the
interrupt controller can rely on in_nmi() to distinguish them from
ordinary interrupts.

Enable IPIs to use superpriority interrupts as NMIs, matching the
existing pseudo-NMI behaviour.
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/entry-common.h |  7 ++++
 arch/arm64/kernel/entry-common.c      | 59 ++++++++++++++++++---------
 arch/arm64/kernel/smp.c               |  2 +-
 3 files changed, 48 insertions(+), 20 deletions(-)

diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
index 73d82a8d8e95..0681ba91ac3b 100644
--- a/arch/arm64/include/asm/entry-common.h
+++ b/arch/arm64/include/asm/entry-common.h
@@ -37,6 +37,13 @@ static inline bool arch_irqentry_exit_need_resched(void)
 	if (system_uses_irq_prio_masking() && read_sysreg(daif))
 		return false;
 
+	/*
+	 * If AllInt is set then we must have handled an NMI, so skip
+	 * preemption
+	 */
+	if (system_uses_nmi() && read_sysreg_s(SYS_ALLINT))
+		return false;
+
 	/*
 	 * Preempting a task from an IRQ means we leave copies of PSTATE
 	 * on the stack. cpufeature's enable calls may modify PSTATE, but
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index a13653b228b7..de71d5a3a6a1 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -525,8 +525,8 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
 	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
-static __always_inline void __el1_pnmi(struct pt_regs *regs,
-				       void (*handler)(struct pt_regs *))
+static __always_inline void __el1_nmi(struct pt_regs *regs,
+				      void (*handler)(struct pt_regs *))
 {
 	arm64_exc_hwstate_t hwstate;
 	irqentry_state_t state;
@@ -545,7 +545,10 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
 
 	state = arm64_enter_from_kernel_mode(regs);
 
-	arm64_unmask_exc_context(NONMI_CONTEXT);
+	if (system_uses_nmi())
+		arm64_unmask_exc_context(NOIRQ_CONTEXT);
+	else
+		arm64_unmask_exc_context(NONMI_CONTEXT);
 
 	irq_enter_rcu();
 	do_interrupt_handler(regs, handler);
@@ -565,8 +568,11 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
 static void noinstr el1_interrupt(struct pt_regs *regs,
 				  void (*handler)(struct pt_regs *))
 {
-	if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && regs_irqs_disabled(regs))
-		__el1_pnmi(regs, handler);
+	/* Is there a NMI to handle? */
+	if (regs_irqs_disabled(regs))
+		__el1_nmi(regs, handler);
+	else if (system_uses_nmi() && (read_sysreg(isr_el1) & (ISR_EL1_IS | ISR_EL1_FS)))
+		__el1_nmi(regs, handler);
 	else
 		__el1_irq(regs, handler);
 
@@ -906,24 +912,39 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
 static void noinstr el0_interrupt(struct pt_regs *regs,
 				  void (*handler)(struct pt_regs *))
 {
-	arm64_enter_from_user_mode(regs);
-
-	arm64_unmask_exc_context(NONMI_CONTEXT);
-
 	if (regs->pc & BIT(55))
 		arm64_apply_bp_hardening();
 
-	irq_enter_rcu();
-	do_interrupt_handler(regs, handler);
-	irq_exit_rcu();
+	/* Is there a NMI to handle? */
+	if (system_uses_nmi() && (read_sysreg(isr_el1) & (ISR_EL1_IS | ISR_EL1_FS))) {
+		irqentry_state_t state;
+		arm64_exc_hwstate_t hwstate;
+
+		state = irqentry_nmi_enter(regs);
+		hwstate = arm64_unmask_exc_context(NONMI_CONTEXT);
+		do_interrupt_handler(regs, handler);
+		arm64_mask_exc_context(hwstate);
+		irqentry_nmi_exit(regs, state);
+	} else {
+		arm64_enter_from_user_mode(regs);
+
+		if (system_uses_nmi())
+			arm64_unmask_exc_context(NOIRQ_CONTEXT);
+		else
+			arm64_unmask_exc_context(NONMI_CONTEXT);
+
+		irq_enter_rcu();
+		do_interrupt_handler(regs, handler);
+		irq_exit_rcu();
+		/*
+		 * For the same reason as in el1_irq() we effectivly
+		 * have NOIRQ_CONTEXT on return from handler - keep
+		 * track of it
+		 */
+		arm64_debug_exc_context(NOIRQ_CONTEXT);
+		arm64_exit_to_user_mode(regs, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
+	}
 
-	/*
-	 * For the same reason as in el1_irq() we effectivly
-	 * have NOIRQ_CONTEXT on return from handler - keep
-	 * track of it
-	 */
-	arm64_debug_exc_context(NOIRQ_CONTEXT);
-	arm64_exit_to_user_mode(regs, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
 	arm64_debug_exc_context(CRITICAL_CONTEXT);
 }
 
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 257d50529d14..80e35a8e5c8f 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -1035,7 +1035,7 @@ static void smp_cross_call(const struct cpumask *target, unsigned int ipinr)
 
 static bool ipi_should_be_nmi(enum ipi_msg_type ipi)
 {
-	if (!system_uses_irq_prio_masking())
+	if (!system_uses_nmi() && !system_uses_irq_prio_masking())
 		return false;
 
 	switch (ipi) {
-- 
2.34.1

[RFC PATCH 32/36] arm64: suspend: Always initialise PSTATE.ALLINT

[Vladimir Murzin]

PSTATE.ALLINT is always set to the inverse of SCTLR_ELx.SPINTMASK,
regardless of the value of SCTLR_ELx.NMI. SCTLR_ELx.NMI is initialised
to 0 by default, so PSTATE.ALLINT does not cause any interrupt
masking. With upcoming FEAT_NMI support, SCTLR_ELx.NMI will be set as
part of the enable sequence.

However, during CPU suspend/resume, we reinitialise PSTATE to
INIT_PSTATE_EL1. INIT_PSTATE_EL1 currently does not set
PSR_ALLINT_BIT, but SCTLR_ELx.NMI is restored from the saved
value. Since all exceptions are masked during CPU suspend/resume,
avoid a state where SCTLR_ELx.NMI is restored to 1 but PSTATE.ALLINT
is clear, since that would permit an NMI during resume.

Include PSR_ALLINT_BIT in INIT_PSTATE_EL1 so PSTATE.ALLINT is always
set.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/ptrace.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index 096d85d75bd7..ab1cf7669140 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -17,7 +17,7 @@
 #define CurrentEL_EL2		(2 << 2)
 
 #define INIT_PSTATE_EL1 \
-	(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL1h)
+	(PSR_ALLINT_BIT | PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL1h)
 
 #include <linux/irqchip/arm-gic-v3-prio.h>
 
-- 
2.34.1

[RFC PATCH 33/36] arm64/efi: Add ALLINT to IRQ flags checks

[Vladimir Murzin]

EFI is not expected to modify PSTATE.ALLINT, so check that it remains
unchanged across EFI runtime calls.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/include/asm/efi.h | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h
index 479741d52c90..ea8403c7bdaf 100644
--- a/arch/arm64/include/asm/efi.h
+++ b/arch/arm64/include/asm/efi.h
@@ -78,6 +78,10 @@ do {															\
 		pr_err_ratelimited(FW_BUG "DAIF+PMR corrupted (0x%x + 0x%x=>0x%x + 0x%x) by EFI call from %pS\n",	\
 				   __expected.daif, __expected.pmr, __actual.daif, __actual.pmr,			\
 				   caller ?: __builtin_return_address(0));						\
+	} else if (system_uses_nmi()) {											\
+		pr_err_ratelimited(FW_BUG "DAIF+ALLINT corrupted (0x%x + 0x%x=>0x%x + 0x%x) by EFI call from %pS\n",	\
+				   __expected.daif, __expected.allint, __actual.daif, __actual.allint,			\
+				   caller ?: __builtin_return_address(0));						\
 	} else {													\
 		pr_err_ratelimited(FW_BUG "DAIF corrupted (0x%x=>0x%x) by EFI call from %pS\n",				\
 				   __expected.daif, __actual.daif,							\
-- 
2.34.1

[RFC PATCH 34/36] arm64: kprobes: Disable NMIs

[Vladimir Murzin]

Kprobes masks all DAIF exceptions to keep things simple and avoid
nested exceptions. With FEAT_NMI, that is no longer enough. Mask
ALLINT as well to ensure things stay simple.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/kernel/probes/kprobes.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c
index d9d25d184309..defbe6061546 100644
--- a/arch/arm64/kernel/probes/kprobes.c
+++ b/arch/arm64/kernel/probes/kprobes.c
@@ -193,17 +193,27 @@ static void __kprobes set_current_kprobe(struct kprobe *p)
  * the kprobe state is per-CPU and doesn't get migrated.
  */
 static void __kprobes kprobes_save_local_irqflag(struct kprobe_ctlblk *kcb,
-						struct pt_regs *regs)
+						 struct pt_regs *regs)
 {
-	kcb->saved_irqflag = regs->pstate & DAIF_MASK;
-	regs->pstate |= DAIF_MASK;
+	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
+
+	kcb->saved_irqflag = arm64_inherit_exc_hwstate(regs).flags;
+
+	regs->pstate &= ~(DAIF_MASK | PSR_ALLINT_BIT);
+
+	regs->pstate |= next.daif;
+	regs->pstate |= next.allint;
 }
 
 static void __kprobes kprobes_restore_local_irqflag(struct kprobe_ctlblk *kcb,
-						struct pt_regs *regs)
+						    struct pt_regs *regs)
 {
-	regs->pstate &= ~DAIF_MASK;
-	regs->pstate |= kcb->saved_irqflag;
+	arm64_exc_hwstate_t prev = {.flags = kcb->saved_irqflag};
+
+	regs->pstate &= ~(DAIF_MASK | PSR_ALLINT_BIT);
+
+	regs->pstate |= prev.daif;
+	regs->pstate |= prev.allint;
 }
 
 static void __kprobes setup_singlestep(struct kprobe *p,
-- 
2.34.1

[RFC PATCH 35/36] arm64: nmi: Add Kconfig for NMI

[Vladimir Murzin]

From: Mark Brown <broonie@kernel.org>

Since NMI handling is in some fairly hot paths we provide a Kconfig
option which allows support to be compiled out when not needed.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 arch/arm64/Kconfig | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index e36dc4831200..968a9d97549a 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2195,6 +2195,24 @@ endmenu # "ARMv8.7 architectural features"
 config AS_HAS_MOPS
 	def_bool $(as-instr,.arch_extension mops)
 
+menu "ARMv8.8 architectural features"
+
+config ARM64_NMI
+	bool "Enable support for Non-maskable Interrupts (NMI)"
+	default y
+	help
+	  Non-maskable interrupts are an architecture and GIC feature
+	  which allow the system to configure some interrupts to be
+	  configured to have superpriority, allowing them to be handled
+	  before other interrupts and masked for shorter periods of time.
+
+	  The feature is detected at runtime, and will remain disabled
+	  if the cpu does not implement the feature. It will also be
+	  disabled if pseudo NMIs are enabled at runtime.
+
+endmenu # "ARMv8.8 architectural features"
+
+
 menu "ARMv8.9 architectural features"
 
 config ARM64_POE
-- 
2.34.1

[RFC PATCH 36/36] irqchip/gic-v3: Implement FEAT_GICv3_NMI support

[Vladimir Murzin]

From: Lorenzo Pieralisi <lpieralisi@kernel.org>

The FEAT_GICv3_NMI GIC feature coupled with the CPU FEAT_NMI enables
handling NMI interrupts in HW on aarch64, by adding a superpriority
interrupt to the existing GIC priority scheme.

Implement GIC driver support for the FEAT_GICv3_NMI feature.

Rename gic_supports_nmi() helper function to gic_supports_pseudo_nmis()
to make the pseudo NMIs code path clearer and more explicit.

Check, through the ARM64 capabilitity infrastructure, if support
for FEAT_NMI was detected on the core and the system has not overridden
the detection and forced pseudo-NMIs enablement.

If FEAT_NMI is detected, it was not overridden (check embedded in the
system_uses_nmi() call) and the GIC supports the FEAT_GICv3_NMI
feature, initialize NMIs related HW GIC registers and route irq to NMI
handling logic.

Signed-off-by: Lorenzo Pieralisi <lpieralisi@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
---
 drivers/irqchip/irq-gic-v3.c       | 122 +++++++++++++++++++++++++----
 include/linux/irqchip/arm-gic-v3.h |   4 +
 2 files changed, 111 insertions(+), 15 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 94c6a3f2b009..625be4fc06b4 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -64,6 +64,7 @@ struct gic_chip_data {
 	u32			nr_redist_regions;
 	u64			flags;
 	bool			has_rss;
+	bool			has_nmi;
 	unsigned int		ppi_nr;
 	struct partition_affinity *parts;
 	unsigned int		nr_parts;
@@ -87,7 +88,7 @@ static DEFINE_STATIC_KEY_TRUE(supports_deactivate_key);
 #define GIC_LINE_NR	min(GICD_TYPER_SPIS(gic_data.rdists.gicd_typer), 1020U)
 #define GIC_ESPI_NR	GICD_TYPER_ESPIS(gic_data.rdists.gicd_typer)
 
-static bool nmi_support_forbidden;
+static bool pnmi_support_forbidden;
 
 /*
  * There are 16 SGIs, though we only actually use 8 in Linux. The other 8 SGIs
@@ -185,7 +186,7 @@ static void __init gic_prio_init(void)
 				pr_warn("Broken GIC integration, security disabled\n");
 		} else {
 			pr_warn("Broken GIC integration, pNMI forbidden\n");
-			nmi_support_forbidden = true;
+			pnmi_support_forbidden = true;
 		}
 	}
 
@@ -252,6 +253,37 @@ enum gic_intid_range {
 	__INVALID_RANGE__
 };
 
+#ifdef CONFIG_ARM64
+#include <asm/cpufeature.h>
+
+static inline bool gic_supports_v3_3_nmi(void)
+{
+	return gic_data.has_nmi && system_uses_nmi();
+}
+
+static inline u64 gic_read_nmiar(void)
+{
+	u64 irqstat;
+
+	irqstat = read_sysreg_s(SYS_ICC_NMIAR1_EL1);
+
+	dsb(sy);
+
+	return irqstat;
+}
+
+#else
+static inline bool gic_supports_v3_3_nmi(void)
+{
+	return false;
+}
+
+static inline u64 gic_read_nmiar(void)
+{
+	BUG();
+}
+#endif
+
 static enum gic_intid_range __get_intid_range(irq_hw_number_t hwirq)
 {
 	switch (hwirq) {
@@ -438,6 +470,8 @@ static u32 convert_offset_index(struct irq_data *d, u32 offset, u32 *index)
 			return GICD_ICFGRnE;
 		case GICD_IROUTER:
 			return GICD_IROUTERnE;
+		case GICD_INMIR:
+			return GICD_INMIRnE;
 		default:
 			break;
 		}
@@ -467,6 +501,42 @@ static int gic_peek_irq(struct irq_data *d, u32 offset)
 	return !!(readl_relaxed(base + offset + (index / 32) * 4) & mask);
 }
 
+static DEFINE_RAW_SPINLOCK(irq_controller_lock);
+
+static void gic_irq_configure_nmi(struct irq_data *d, bool enable)
+{
+	void __iomem *base, *addr;
+	u32 offset, index, mask, val;
+
+	offset = convert_offset_index(d, GICD_INMIR, &index);
+	mask = 1 << (index % 32);
+
+	if (gic_irq_in_rdist(d))
+		base = gic_data_rdist_sgi_base();
+	else
+		base = gic_dist_base_alias(d);
+
+	addr = base + offset + (index / 32) * 4;
+
+	raw_spin_lock(&irq_controller_lock);
+
+	val = readl_relaxed(addr);
+	val = enable ? (val | mask) : (val & ~mask);
+	writel_relaxed(val, addr);
+
+	raw_spin_unlock(&irq_controller_lock);
+}
+
+static void gic_irq_enable_nmi(struct irq_data *d)
+{
+	gic_irq_configure_nmi(d, true);
+}
+
+static void gic_irq_disable_nmi(struct irq_data *d)
+{
+	gic_irq_configure_nmi(d, false);
+}
+
 static void gic_poke_irq(struct irq_data *d, u32 offset)
 {
 	void __iomem *base;
@@ -512,7 +582,7 @@ static void gic_unmask_irq(struct irq_data *d)
 	gic_poke_irq(d, GICD_ISENABLER);
 }
 
-static inline bool gic_supports_nmi(void)
+static inline bool gic_supports_pseudo_nmis(void)
 {
 	return IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) &&
 	       static_branch_likely(&supports_pseudo_nmis);
@@ -598,7 +668,7 @@ static int gic_irq_nmi_setup(struct irq_data *d)
 {
 	struct irq_desc *desc = irq_to_desc(d->irq);
 
-	if (!gic_supports_nmi())
+	if (!gic_supports_pseudo_nmis() && !gic_supports_v3_3_nmi())
 		return -EINVAL;
 
 	if (gic_peek_irq(d, GICD_ISENABLER)) {
@@ -617,7 +687,10 @@ static int gic_irq_nmi_setup(struct irq_data *d)
 	if (!gic_irq_in_rdist(d))
 		desc->handle_irq = handle_fasteoi_nmi;
 
-	gic_irq_set_prio(d, dist_prio_nmi);
+	if (gic_supports_v3_3_nmi())
+		gic_irq_enable_nmi(d);
+	else
+		gic_irq_set_prio(d, dist_prio_nmi);
 
 	return 0;
 }
@@ -626,7 +699,7 @@ static void gic_irq_nmi_teardown(struct irq_data *d)
 {
 	struct irq_desc *desc = irq_to_desc(d->irq);
 
-	if (WARN_ON(!gic_supports_nmi()))
+	if (WARN_ON(!gic_supports_pseudo_nmis() && !gic_supports_v3_3_nmi()))
 		return;
 
 	if (gic_peek_irq(d, GICD_ISENABLER)) {
@@ -645,7 +718,10 @@ static void gic_irq_nmi_teardown(struct irq_data *d)
 	if (!gic_irq_in_rdist(d))
 		desc->handle_irq = handle_fasteoi_irq;
 
-	gic_irq_set_prio(d, dist_prio_irq);
+	if (gic_supports_v3_3_nmi())
+		gic_irq_disable_nmi(d);
+	else
+		gic_irq_set_prio(d, dist_prio_irq);
 }
 
 static bool gic_arm64_erratum_2941627_needed(struct irq_data *d)
@@ -804,7 +880,7 @@ static inline void gic_complete_ack(u32 irqnr)
 
 static bool gic_rpr_is_nmi_prio(void)
 {
-	if (!gic_supports_nmi())
+	if (!gic_supports_pseudo_nmis())
 		return false;
 
 	return unlikely(gic_read_rpr() == GICV3_PRIO_NMI);
@@ -836,7 +912,8 @@ static void __gic_handle_nmi(u32 irqnr, struct pt_regs *regs)
 	gic_complete_ack(irqnr);
 
 	if (generic_handle_domain_nmi(gic_data.domain, irqnr)) {
-		WARN_ONCE(true, "Unexpected pseudo-NMI (irqnr %u)\n", irqnr);
+		WARN_ONCE(true, "Unexpected %sNMI (irqnr %u)\n",
+			  gic_supports_pseudo_nmis() ? "pseudo-" : "", irqnr);
 		gic_deactivate_unhandled(irqnr);
 	}
 }
@@ -911,7 +988,11 @@ static void __gic_handle_irq_from_irqsoff(struct pt_regs *regs)
 
 static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
 {
-	if (unlikely(gic_supports_nmi() && !interrupts_enabled(regs)))
+	if (gic_supports_v3_3_nmi() && in_nmi()) {
+		u32 irqnr = gic_read_nmiar();
+
+		__gic_handle_nmi(irqnr, regs);
+	} else if (unlikely(gic_supports_pseudo_nmis() && !interrupts_enabled(regs)))
 		__gic_handle_irq_from_irqsoff(regs);
 	else
 		__gic_handle_irq_from_irqson(regs);
@@ -1162,7 +1243,7 @@ static void gic_cpu_sys_reg_init(void)
 	/* Set priority mask register */
 	if (!gic_prio_masking_enabled()) {
 		write_gicreg(DEFAULT_PMR_VALUE, ICC_PMR_EL1);
-	} else if (gic_supports_nmi()) {
+	} else if (gic_supports_pseudo_nmis()) {
 		/*
 		 * Check that all CPUs use the same priority space.
 		 *
@@ -1955,15 +2036,25 @@ static const struct gic_quirk gic_quirks[] = {
 	}
 };
 
-static void gic_enable_nmi_support(void)
+static void gic_enable_pseudo_nmis(void)
 {
-	if (!gic_prio_masking_enabled() || nmi_support_forbidden)
-		return;
-
 	pr_info("Pseudo-NMIs enabled using %s ICC_PMR_EL1 synchronisation\n",
 		gic_has_relaxed_pmr_sync() ? "relaxed" : "forced");
 
 	static_branch_enable(&supports_pseudo_nmis);
+}
+
+static void gic_enable_nmi_support(void)
+{
+	if ((!gic_prio_masking_enabled() || pnmi_support_forbidden) && !gic_supports_v3_3_nmi())
+		return;
+
+	/*
+	 * Initialize pseudo-NMIs only if GIC driver cannot take advantage
+	 * of core (FEAT_NMI) and GIC (FEAT_GICv3_NMI) in HW
+	 */
+	if (!gic_supports_v3_3_nmi())
+		gic_enable_pseudo_nmis();
 
 	if (static_branch_likely(&supports_deactivate_key))
 		gic_eoimode1_chip.flags |= IRQCHIP_SUPPORTS_NMI;
@@ -2032,6 +2123,7 @@ static int __init gic_init_bases(phys_addr_t dist_phys_base,
 	irq_domain_update_bus_token(gic_data.domain, DOMAIN_BUS_WIRED);
 
 	gic_data.has_rss = !!(typer & GICD_TYPER_RSS);
+	gic_data.has_nmi = !!(typer & GICD_TYPER_NMI);
 
 	if (typer & GICD_TYPER_MBIS) {
 		err = mbi_init(handle, gic_data.domain);
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index ea5fd2374ebe..ebccd76cadda 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -30,6 +30,7 @@
 #define GICD_ICFGR			0x0C00
 #define GICD_IGRPMODR			0x0D00
 #define GICD_NSACR			0x0E00
+#define GICD_INMIR			0x0F80
 #define GICD_IGROUPRnE			0x1000
 #define GICD_ISENABLERnE		0x1200
 #define GICD_ICENABLERnE		0x1400
@@ -39,6 +40,7 @@
 #define GICD_ICACTIVERnE		0x1C00
 #define GICD_IPRIORITYRnE		0x2000
 #define GICD_ICFGRnE			0x3000
+#define GICD_INMIRnE			0x3B00
 #define GICD_IROUTER			0x6000
 #define GICD_IROUTERnE			0x8000
 #define GICD_IDREGS			0xFFD0
@@ -83,6 +85,7 @@
 #define GICD_TYPER_LPIS			(1U << 17)
 #define GICD_TYPER_MBIS			(1U << 16)
 #define GICD_TYPER_ESPI			(1U << 8)
+#define GICD_TYPER_NMI			(1U << 9)
 
 #define GICD_TYPER_ID_BITS(typer)	((((typer) >> 19) & 0x1f) + 1)
 #define GICD_TYPER_NUM_LPIS(typer)	((((typer) >> 11) & 0x1f) + 1)
@@ -238,6 +241,7 @@
 #define GICR_ICFGR0			GICD_ICFGR
 #define GICR_IGRPMODR0			GICD_IGRPMODR
 #define GICR_NSACR			GICD_NSACR
+#define GICR_INMIR0			GICD_INMIR
 
 #define GICR_TYPER_PLPIS		(1U << 0)
 #define GICR_TYPER_VLPIS		(1U << 1)
-- 
2.34.1

Re: [RFC PATCH 01/36] arm64: ptrace: Remove INIT_PSTATE_EL2

[Jinjie Ruan]

On 7/9/2026 8:12 PM, Vladimir Murzin wrote:
> Last user of INIT_PSTATE_EL2 has gone with ae4b7e38e9a9 ("arm64: Allow
> sticky E2H when entering EL1"), so remove it.
> 
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/ptrace.h | 2 --
>  1 file changed, 2 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
> index 39582511ad72..f7dc5fb9427d 100644
> --- a/arch/arm64/include/asm/ptrace.h
> +++ b/arch/arm64/include/asm/ptrace.h
> @@ -18,8 +18,6 @@
>  
>  #define INIT_PSTATE_EL1 \
>  	(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL1h)
> -#define INIT_PSTATE_EL2 \
> -	(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL2h)

Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

>  
>  #include <linux/irqchip/arm-gic-v3-prio.h>
>  

Re: [RFC PATCH 02/36] arm64: debug: don't mask DAIF for mdscr_write()

[Jinjie Ruan]

On 7/9/2026 8:12 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Masking DAIF around the write to MDSCR_EL1 doesn't do anything: we can
> write to sysregs with interrupts unmasked, and writing to PSTATE is
> not a context synchronization event so it does not synchronize it.
> 
> This is done in the context of a general interrupt handling cleanup,
> so it does not address the missing context synchronization for the
> MDSCR_EL1 write, staying consistent with the current state.  This
> should be addressed in a future patch.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/kernel/debug-monitors.c | 3 ---
>  1 file changed, 3 deletions(-)
> 
> diff --git a/arch/arm64/kernel/debug-monitors.c b/arch/arm64/kernel/debug-monitors.c
> index 29307642f4c9..e271fbac5f82 100644
> --- a/arch/arm64/kernel/debug-monitors.c
> +++ b/arch/arm64/kernel/debug-monitors.c
> @@ -40,10 +40,7 @@ u8 debug_monitors_arch(void)
>   */
>  static void mdscr_write(u64 mdscr)
>  {
> -	unsigned long flags;
> -	flags = local_daif_save();
>  	write_sysreg(mdscr, mdscr_el1);
> -	local_daif_restore(flags);

commit 478fcb2cdb23 ("arm64: Debugging support") used the
local_dbg_save() to set the DAIF.D bit.

In my opinion, this was originally intended to avoid the debug exception
being triggered immediately when the single-step mode is enabled during
the write of MDSCR_EL1.SS?


+ * MDSCR access routines.
+ */
+static void mdscr_write(u32 mdscr)
+{
+       unsigned long flags;
+       local_dbg_save(flags);
+       asm volatile("msr mdscr_el1, %0" :: "r" (mdscr));
+       local_dbg_restore(flags);
+}


+#define local_dbg_save(flags)
       \
+       do {
       \
+               typecheck(unsigned long, flags);
       \
+               asm volatile(
       \
+               "mrs    %0, daif                        //
local_dbg_save\n"    \
+               "msr    daifset, #8"
       \
+               : "=r" (flags) : : "memory");
       \
+       } while (0)


>  }
>  NOKPROBE_SYMBOL(mdscr_write);
>  

Re: [RFC PATCH 03/36] arm64: hibernate: mask DAIF before restoring hibernated kernel

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> The arm64 hibernate code manages the exception masking in an unsound
> way, leading to potential crashes and/or warnings during resume.
> 
> When a hibernation image is saved in `swsusp_arch_suspend()`, all DAIF
> exceptions are masked (by virtue of `local_daif_save()`), and the
> suspended image is saved assuming that all DAIF exceptions will remain
> masked when the image is restored.
> 
> When a hibernation image is resumed by `swsusp_arch_resume()`, only
> interrupts are masked (by virtue of `local_irq_save()` in

local_irq_disable()?

> `resume_target_kernel()`). When pseudo-NMI is enabled the DAIF.IF bits
> will be clear, and regardless of pseudo-NMI the DAIF.DA bits will be
> clear.
> 
> This means that there are two problems:
> 
> (1) It is possible to take Debug, SError, or pseudo-NMI exceptions
>     during the resume process. This is unsafe, as during the resume
>     process both the old ane new kernels will tranisently be in an
>     inconsistent state, and swsusp_arch_suspend_exit() won't retain
>     an executable mapping of any exception vectors.
> 
>     Any exception taken here will be fatal and silent.
> 
> (2) When re-entering the resumed kernel, some DAIF bits will be clear
>     unexpectedly. This permits Debug, SError, or pseudo-NMI exceptions
>     to be taken for a short period while the resumed kernel is not yet
>     in a consistent state.
> 
>     This is detected by CONFIG_ARM64_DEBUG_PRIORITY_MASKING.
> 
> Avoid these issues by masking all DAIF exceptions during resume.
> 
> Cc: stable@vger.kernel.org
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/kernel/hibernate.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
> index 9717568518ba..d0d9bd91e639 100644
> --- a/arch/arm64/kernel/hibernate.c
> +++ b/arch/arm64/kernel/hibernate.c
> @@ -405,6 +405,7 @@ int swsusp_arch_suspend(void)
>  int __nocfi swsusp_arch_resume(void)
>  {
>  	int rc;
> +	unsigned long flags;
>  	void *zero_page;
>  	size_t exit_size;
>  	pgd_t *tmp_pg_dir;
> @@ -465,9 +466,21 @@ int __nocfi swsusp_arch_resume(void)
>  	if (el2_reset_needed())
>  		__hyp_set_vectors(el2_vectors);
>  
> +	/*
> +	 * It is necessary to mask all DAIF exceptions here as:
> +	 *
> +	 * - The copy of swsusp_arch_suspend_exit() in the hibernation
> +	 *   text cannot handle taking any exceptions.
> +	 *
> +	 * - The suspended kernel masked all DAIF exceptions in
> +	 *   swsusp_arch_resume(), and expects to be re-entered in the
> +	 *   same state : with all DAIF exceptions masked.
> +	 */
> +	flags = local_daif_save();
>  	hibernate_exit(virt_to_phys(tmp_pg_dir), resume_hdr.ttbr1_el1,
>  		       resume_hdr.reenter_kernel, restore_pblist,
>  		       resume_hdr.__hyp_stub_vectors, virt_to_phys(zero_page));
> +	local_daif_restore(flags);
>  
>  	return 0;
>  }

Re: [RFC PATCH 26/36] arm64: ptrace: Add PSR_ALLINT_BIT

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> When FEAT_NMI is implemented, ALLINT is part of PSTATE and will be saved
> to SPSR on exception entry.
> As it can mask interrupts, depending on configuration, we will need to
> check its value in interruted tasks, so add the corresponding bit
> to ptrace.h.
> 
> Given its impact on interrupt and NMI masking, use it to update
> `print_pstate()` so that it displays ALLINT state.
> Display it after DAIF so that interrupt masking fields are together.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/uapi/asm/ptrace.h | 1 +
>  arch/arm64/kernel/process.c          | 3 ++-
>  2 files changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/include/uapi/asm/ptrace.h b/arch/arm64/include/uapi/asm/ptrace.h
> index 6fed93fb2536..99352dd823cc 100644
> --- a/arch/arm64/include/uapi/asm/ptrace.h
> +++ b/arch/arm64/include/uapi/asm/ptrace.h
> @@ -48,6 +48,7 @@
>  #define PSR_D_BIT	0x00000200
>  #define PSR_BTYPE_MASK	0x00000c00
>  #define PSR_SSBS_BIT	0x00001000
> +#define PSR_ALLINT_BIT	0x00002000
>  #define PSR_PAN_BIT	0x00400000
>  #define PSR_UAO_BIT	0x00800000
>  #define PSR_DIT_BIT	0x01000000
> diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
> index ddda8d7aee24..8d4ceef4f67f 100644
> --- a/arch/arm64/kernel/process.c
> +++ b/arch/arm64/kernel/process.c
> @@ -183,7 +183,7 @@ static void print_pstate(struct pt_regs *regs)
>  		const char *btype_str = btypes[(pstate & PSR_BTYPE_MASK) >>
>  					       PSR_BTYPE_SHIFT];
>  
> -		printk("pstate: %08llx (%c%c%c%c %c%c%c%c %cPAN %cUAO %cTCO %cDIT %cSSBS BTYPE=%s)\n",
> +		printk("pstate: %08llx (%c%c%c%c %c%c%c%c %cALLINT %cPAN %cUAO %cTCO %cDIT %cSSBS BTYPE=%s)\n",
>  			pstate,
>  			pstate & PSR_N_BIT ? 'N' : 'n',
>  			pstate & PSR_Z_BIT ? 'Z' : 'z',
> @@ -193,6 +193,7 @@ static void print_pstate(struct pt_regs *regs)
>  			pstate & PSR_A_BIT ? 'A' : 'a',
>  			pstate & PSR_I_BIT ? 'I' : 'i',
>  			pstate & PSR_F_BIT ? 'F' : 'f',
> +			pstate & PSR_ALLINT_BIT ? '+' : '-',

Hi Vladimir,

This seems to be correct, but the status of the ALLINT bit
should not be displayed if FEAT_NMI is not enabled (system_uses_nmi()).

According to the ARM64 ALLINT manual,

	"On a Warm reset, this field resets to an architecturally UNKNOWN value."

If the hardware supports FEAT_NMI but the config is not enabled, the
value of the ALLINT bit is a random value, which is very misleading to
show it.

Best regards,
Jinjie


>  			pstate & PSR_PAN_BIT ? '+' : '-',
>  			pstate & PSR_UAO_BIT ? '+' : '-',
>  			pstate & PSR_TCO_BIT ? '+' : '-',

Re: [RFC PATCH 27/36] arm64: idreg: Add an override for FEAT_NMI

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Mark Brown <broonie@kernel.org>
> 
> Add a named override for FEAT_NMI, allowing it to be explicitly disabled
> in case of problems.
> 
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/kernel/pi/idreg-override.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c
> index bc57b290e5e7..e5ea280452c3 100644
> --- a/arch/arm64/kernel/pi/idreg-override.c
> +++ b/arch/arm64/kernel/pi/idreg-override.c
> @@ -154,6 +154,7 @@ static const struct ftr_set_desc pfr1 __prel64_initconst = {
>  		FIELD("bt", ID_AA64PFR1_EL1_BT_SHIFT, NULL ),
>  		FIELD("gcs", ID_AA64PFR1_EL1_GCS_SHIFT, NULL),
>  		FIELD("mte", ID_AA64PFR1_EL1_MTE_SHIFT, NULL),
> +		FIELD("nmi", ID_AA64PFR1_EL1_NMI_SHIFT, NULL),

Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

>  		FIELD("sme", ID_AA64PFR1_EL1_SME_SHIFT, pfr1_sme_filter),
>  		FIELD("mpam_frac", ID_AA64PFR1_EL1_MPAM_frac_SHIFT, NULL),
>  		{}

Re: [RFC PATCH 28/36] arm64: cpufeature: Detect PE support for FEAT_NMI

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Use of FEAT_NMI requires that all the PEs in the system and the GIC
> have NMI support. This patch implements the PE part of that detection.
> 
> In order to avoid problematic interactions between real and pseudo NMIs
> we disable the architected feature if the user has enabled pseudo NMIs
> on the command line. If this is done on a system where support for the
> architected feature is detected then a warning is printed during boot in
> order to help users spot what is likely to be a misconfiguration.
> 
> As KVM does not care about the host kernel supporting FEAT_NMI or not
> to allow guests to use it, split the CPU cap in two : ARM64_HAS_NMI is
> the hardware support, ARM64_NMI is the kernel making use of it.
> 
> Co-developed-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/cpucaps.h    |  2 +
>  arch/arm64/include/asm/cpufeature.h | 10 +++++
>  arch/arm64/kernel/cpufeature.c      | 68 ++++++++++++++++++++++++++++-
>  arch/arm64/tools/cpucaps            |  2 +
>  4 files changed, 81 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
> index 25c61cda901c..176a63f1cc1b 100644
> --- a/arch/arm64/include/asm/cpucaps.h
> +++ b/arch/arm64/include/asm/cpucaps.h
> @@ -75,6 +75,8 @@ cpucap_is_possible(const unsigned int cap)
>  		return IS_ENABLED(CONFIG_HW_PERF_EVENTS);
>  	case ARM64_HAS_LSUI:
>  		return IS_ENABLED(CONFIG_ARM64_LSUI);
> +	case ARM64_NMI:
> +		return IS_ENABLED(CONFIG_ARM64_NMI);
>  	}
>  
>  	return true;
> diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
> index 95cd8d4acd26..84b817b29155 100644
> --- a/arch/arm64/include/asm/cpufeature.h
> +++ b/arch/arm64/include/asm/cpufeature.h
> @@ -816,6 +816,16 @@ static __always_inline bool system_uses_irq_prio_masking(void)
>  	return alternative_has_cap_unlikely(ARM64_HAS_GIC_PRIO_MASKING);
>  }
>  
> +static __always_inline bool system_supports_nmi(void)
> +{
> +	return alternative_has_cap_unlikely(ARM64_HAS_NMI);
> +}
> +
> +static __always_inline bool system_uses_nmi(void)
> +{
> +	return alternative_has_cap_unlikely(ARM64_NMI);
> +}
> +
>  static inline bool system_supports_mte(void)
>  {
>  	return alternative_has_cap_unlikely(ARM64_MTE);
> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> index 9a22df0c5120..bb22292ebabc 100644
> --- a/arch/arm64/kernel/cpufeature.c
> +++ b/arch/arm64/kernel/cpufeature.c
> @@ -83,13 +83,14 @@
>  #include <asm/cpu_ops.h>
>  #include <asm/fpsimd.h>
>  #include <asm/hwcap.h>
> +#include <asm/hypervisor.h>
>  #include <asm/insn.h>
> +#include <asm/interrupts/common_flags.h>
>  #include <asm/kvm_host.h>
>  #include <asm/mmu.h>
>  #include <asm/mmu_context.h>
>  #include <asm/mpam.h>
>  #include <asm/mte.h>
> -#include <asm/hypervisor.h>
>  #include <asm/processor.h>
>  #include <asm/smp.h>
>  #include <asm/sysreg.h>
> @@ -315,6 +316,7 @@ static const struct arm64_ftr_bits ftr_id_aa64pfr1[] = {
>  	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_GCS),
>  		       FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_GCS_SHIFT, 4, 0),
>  	S_ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MTE_frac_SHIFT, 4, 0),
> +	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_NMI_SHIFT, 4, 0),
>  	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SME),
>  		       FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_SME_SHIFT, 4, 0),
>  	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MPAM_frac_SHIFT, 4, 0),
> @@ -2320,6 +2322,51 @@ static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry
>  }
>  #endif
>  
> +#ifdef CONFIG_ARM64_NMI
> +static bool can_use_nmi(const struct arm64_cpu_capabilities *entry, int scope)
> +{
> +	/*
> +	 * ARM64_HAS_NMI has a lower index, and is a boot CPU
> +	 * feature, so will be detected earlier.
> +	 */
> +	BUILD_BUG_ON(ARM64_NMI <= ARM64_HAS_NMI);
> +	if (!cpus_have_cap(ARM64_HAS_NMI))
> +		return false;
> +
> +	/*
> +	 * Having both real and pseudo NMIs enabled simultaneously is
> +	 * likely to cause confusion.  Since pseudo NMIs must be
> +	 * enabled with an explicit command line option, if the user
> +	 * has set that option on a system with real NMIs for some
> +	 * reason assume they know what they're doing.
> +	 *
> +	 * ARM64_HAS_GIC_PRIO_MASKING has a lower index, and is a boot CPU
> +	 * feature, so will be detected earlier.
> +	 */
> +	BUILD_BUG_ON(IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) &&
> +			(ARM64_NMI <= ARM64_HAS_GIC_PRIO_MASKING));
> +	if (cpus_have_cap(ARM64_HAS_GIC_PRIO_MASKING)) {
> +		pr_info("Pseudo NMI enabled, not using architected NMI\n");
> +		return false;

A small suggestion is to change it to pr_info_once. Otherwise, a large
number of print information will be displayed.

Otherwise, LGTM
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>



# dmesg | grep NMI
[    0.000000] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.000000] GICv3: Pseudo-NMIs enabled using relaxed ICC_PMR_EL1
synchronisation
[    0.434430] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.461345] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.475924] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.491364] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.507948] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.523741] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.543870] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.561246] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.579303] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.597354] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.611880] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.630754] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.649572] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.666047] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.681356] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.707386] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.727123] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.744232] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.769575] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.788145] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.811170] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.831739] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.855084] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.876659] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.897121] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.923611] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.943820] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.967216] CPU features: Pseudo NMI enabled, not using architected NMI
[    0.997958] CPU features: Pseudo NMI enabled, not using architected NMI
[    1.027661] CPU features: Pseudo NMI enabled, not using architected NMI
[    1.057660] CPU features: Pseudo NMI enabled, not using architected NMI
[    8.532896] hw perfevents: enabled with armv8_pmuv3 PMU driver, 7
(0,8000003f) counters available, using NMIs
[    8.536298] watchdog: NMI not fully supported


> +	}
> +
> +	return true;
> +}
> +
> +static void nmi_enable(const struct arm64_cpu_capabilities *__unused)
> +{
> +	/*
> +	 * Enable use of NMIs controlled by ALLINT, SPINTMASK should
> +	 * be clear by default but make it explicit that we are using
> +	 * this mode.  Ensure that ALLINT is clear first in order to
> +	 * avoid leaving things masked.
> +	 */
> +	_allint_clear();
> +	sysreg_clear_set(sctlr_el1, SCTLR_EL1_SPINTMASK, SCTLR_EL1_NMI);
> +	isb();
> +}
> +#endif
> +
>  static bool can_trap_icv_dir_el1(const struct arm64_cpu_capabilities *entry,
>  				 int scope)
>  {
> @@ -3198,6 +3245,25 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
>  		.matches = has_cpuid_feature,
>  		ARM64_CPUID_FIELDS(ID_AA64ISAR3_EL1, LSUI, IMP)
>  	},
> +#endif
> +	{
> +		.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
> +		.capability = ARM64_HAS_NMI,
> +		.matches = has_cpuid_feature,
> +		ARM64_CPUID_FIELDS(ID_AA64PFR1_EL1, NMI, IMP)
> +	},
> +#ifdef CONFIG_ARM64_NMI
> +	/*
> +	 * Depends on ARM64_HAS_NMI
> +	 * Checks for conflict with pseudo-NMIs, giving them priority.
> +	 */
> +	{
> +		.desc = "Non-maskable Interrupts",
> +		.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
> +		.capability = ARM64_NMI,
> +		.matches = can_use_nmi,
> +		.cpu_enable = nmi_enable,
> +	},
>  #endif
>  	{},
>  };
> diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
> index 9b85a84f6fd4..2117b3ef0b82 100644
> --- a/arch/arm64/tools/cpucaps
> +++ b/arch/arm64/tools/cpucaps
> @@ -51,6 +51,7 @@ HAS_LS64_V
>  HAS_LSUI
>  HAS_MOPS
>  HAS_NESTED_VIRT
> +HAS_NMI
>  HAS_BBML2_NOABORT
>  HAS_PAN
>  HAS_PMUV3
> @@ -80,6 +81,7 @@ MTE
>  MTE_ASYMM
>  MTE_FAR
>  MTE_STORE_ONLY
> +NMI
>  SME
>  SME_FA64
>  SME2

Re: [RFC PATCH 24/36] arm64: booting: Document boot requirements for FEAT_NMI

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Mark Brown <broonie@kernel.org>
> 
> In order to use FEAT_NMI we must be able to use ALLINT, require that
> it behave as though not trapped when it is present.
> 
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  Documentation/arch/arm64/booting.rst | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/Documentation/arch/arm64/booting.rst b/Documentation/arch/arm64/booting.rst
> index 13ef311dace8..3170b26ae981 100644
> --- a/Documentation/arch/arm64/booting.rst
> +++ b/Documentation/arch/arm64/booting.rst
> @@ -568,6 +568,12 @@ Before jumping into the kernel, the following conditions must be met:
>  
>      - HCRX_EL2.EnASR (bit 2) must be initialised to 0b1.
>  
> +  For CPUs with support for Non-maskable Interrupts (FEAT_NMI):
> +
> +  - If the kernel is entered at EL1 and EL2 is present:
> +
> +    - HCRX_EL2.TALLINT (bit 6) must be initialised to 0b0.
> +

Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

>  The requirements described above for CPU mode, caches, MMUs, architected
>  timers, coherency and system registers apply to all CPUs.  All CPUs must
>  enter the kernel in the same exception level.  Where the values documented

Re: [RFC PATCH 35/36] arm64: nmi: Add Kconfig for NMI

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Mark Brown <broonie@kernel.org>
> 
> Since NMI handling is in some fairly hot paths we provide a Kconfig
> option which allows support to be compiled out when not needed.
> 
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/Kconfig | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
> 
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index e36dc4831200..968a9d97549a 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -2195,6 +2195,24 @@ endmenu # "ARMv8.7 architectural features"
>  config AS_HAS_MOPS
>  	def_bool $(as-instr,.arch_extension mops)
>  
> +menu "ARMv8.8 architectural features"
> +
> +config ARM64_NMI
> +	bool "Enable support for Non-maskable Interrupts (NMI)"
> +	default y
> +	help
> +	  Non-maskable interrupts are an architecture and GIC feature
> +	  which allow the system to configure some interrupts to be
> +	  configured to have superpriority, allowing them to be handled
> +	  before other interrupts and masked for shorter periods of time.
> +
> +	  The feature is detected at runtime, and will remain disabled
> +	  if the cpu does not implement the feature. It will also be
> +	  disabled if pseudo NMIs are enabled at runtime.
> +
> +endmenu # "ARMv8.8 architectural features"

Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

> +
> +
>  menu "ARMv8.9 architectural features"
>  
>  config ARM64_POE

Re: [RFC PATCH 03/36] arm64: hibernate: mask DAIF before restoring hibernated kernel

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> The arm64 hibernate code manages the exception masking in an unsound
> way, leading to potential crashes and/or warnings during resume.
> 
> When a hibernation image is saved in `swsusp_arch_suspend()`, all DAIF
> exceptions are masked (by virtue of `local_daif_save()`), and the
> suspended image is saved assuming that all DAIF exceptions will remain
> masked when the image is restored.
> 
> When a hibernation image is resumed by `swsusp_arch_resume()`, only
> interrupts are masked (by virtue of `local_irq_save()` in
> `resume_target_kernel()`). When pseudo-NMI is enabled the DAIF.IF bits
> will be clear, and regardless of pseudo-NMI the DAIF.DA bits will be
> clear.
> 
> This means that there are two problems:
> 
> (1) It is possible to take Debug, SError, or pseudo-NMI exceptions
>     during the resume process. This is unsafe, as during the resume
>     process both the old ane new kernels will tranisently be in an
>     inconsistent state, and swsusp_arch_suspend_exit() won't retain
>     an executable mapping of any exception vectors.
> 
>     Any exception taken here will be fatal and silent.
> 
> (2) When re-entering the resumed kernel, some DAIF bits will be clear
>     unexpectedly. This permits Debug, SError, or pseudo-NMI exceptions
>     to be taken for a short period while the resumed kernel is not yet
>     in a consistent state.
> 
>     This is detected by CONFIG_ARM64_DEBUG_PRIORITY_MASKING.
> 
> Avoid these issues by masking all DAIF exceptions during resume.
> 
> Cc: stable@vger.kernel.org
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/kernel/hibernate.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
> index 9717568518ba..d0d9bd91e639 100644
> --- a/arch/arm64/kernel/hibernate.c
> +++ b/arch/arm64/kernel/hibernate.c
> @@ -405,6 +405,7 @@ int swsusp_arch_suspend(void)
>  int __nocfi swsusp_arch_resume(void)
>  {
>  	int rc;
> +	unsigned long flags;
>  	void *zero_page;
>  	size_t exit_size;
>  	pgd_t *tmp_pg_dir;
> @@ -465,9 +466,21 @@ int __nocfi swsusp_arch_resume(void)
>  	if (el2_reset_needed())
>  		__hyp_set_vectors(el2_vectors);
>  
> +	/*
> +	 * It is necessary to mask all DAIF exceptions here as:
> +	 *
> +	 * - The copy of swsusp_arch_suspend_exit() in the hibernation
> +	 *   text cannot handle taking any exceptions.
> +	 *
> +	 * - The suspended kernel masked all DAIF exceptions in
> +	 *   swsusp_arch_resume(), and expects to be re-entered in the
> +	 *   same state : with all DAIF exceptions masked.
> +	 */
> +	flags = local_daif_save();
>  	hibernate_exit(virt_to_phys(tmp_pg_dir), resume_hdr.ttbr1_el1,
>  		       resume_hdr.reenter_kernel, restore_pblist,
>  		       resume_hdr.__hyp_stub_vectors, virt_to_phys(zero_page));

hibernate_exit() is a noreturn function, the following
local_daif_restore() will not be called.

> +	local_daif_restore(flags);
>  
>  	return 0;
>  }

Re: [RFC PATCH 03/36] arm64: hibernate: mask DAIF before restoring hibernated kernel

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> The arm64 hibernate code manages the exception masking in an unsound
> way, leading to potential crashes and/or warnings during resume.
> 
> When a hibernation image is saved in `swsusp_arch_suspend()`, all DAIF
> exceptions are masked (by virtue of `local_daif_save()`), and the
> suspended image is saved assuming that all DAIF exceptions will remain
> masked when the image is restored.
> 
> When a hibernation image is resumed by `swsusp_arch_resume()`, only
> interrupts are masked (by virtue of `local_irq_save()` in
> `resume_target_kernel()`). When pseudo-NMI is enabled the DAIF.IF bits
> will be clear, and regardless of pseudo-NMI the DAIF.DA bits will be
> clear.
> 
> This means that there are two problems:
> 
> (1) It is possible to take Debug, SError, or pseudo-NMI exceptions
>     during the resume process. This is unsafe, as during the resume
>     process both the old ane new kernels will tranisently be in an
>     inconsistent state, and swsusp_arch_suspend_exit() won't retain
>     an executable mapping of any exception vectors.
> 
>     Any exception taken here will be fatal and silent.
> 
> (2) When re-entering the resumed kernel, some DAIF bits will be clear
>     unexpectedly. This permits Debug, SError, or pseudo-NMI exceptions
>     to be taken for a short period while the resumed kernel is not yet
>     in a consistent state.
> 
>     This is detected by CONFIG_ARM64_DEBUG_PRIORITY_MASKING.
> 
> Avoid these issues by masking all DAIF exceptions during resume.
> 
> Cc: stable@vger.kernel.org
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/kernel/hibernate.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
> index 9717568518ba..d0d9bd91e639 100644
> --- a/arch/arm64/kernel/hibernate.c
> +++ b/arch/arm64/kernel/hibernate.c
> @@ -405,6 +405,7 @@ int swsusp_arch_suspend(void)
>  int __nocfi swsusp_arch_resume(void)
>  {
>  	int rc;
> +	unsigned long flags;
>  	void *zero_page;
>  	size_t exit_size;
>  	pgd_t *tmp_pg_dir;
> @@ -465,9 +466,21 @@ int __nocfi swsusp_arch_resume(void)
>  	if (el2_reset_needed())
>  		__hyp_set_vectors(el2_vectors);
>  
> +	/*
> +	 * It is necessary to mask all DAIF exceptions here as:
> +	 *
> +	 * - The copy of swsusp_arch_suspend_exit() in the hibernation
> +	 *   text cannot handle taking any exceptions.
> +	 *
> +	 * - The suspended kernel masked all DAIF exceptions in
> +	 *   swsusp_arch_resume(), and expects to be re-entered in the
> +	 *   same state : with all DAIF exceptions masked.
> +	 */
> +	flags = local_daif_save();
>  	hibernate_exit(virt_to_phys(tmp_pg_dir), resume_hdr.ttbr1_el1,
>  		       resume_hdr.reenter_kernel, restore_pblist,
>  		       resume_hdr.__hyp_stub_vectors, virt_to_phys(zero_page));
> +	local_daif_restore(flags);

I believe that the local_daif_save() here is also unnecessary because
hibernate_exit() returns from the "return 0 branch" of
__cpu_suspend_enter() in swsusp_arch_suspend(), and before that,
local_daif_save() has already been called to mask all exceptions.

swsusp_arch_suspend(void)
    -> flags = local_daif_save();
    -> if (__cpu_suspend_enter(&state)) {
           ...
        } else {                // _cpu_resume <- hibernate_exit()

           ...
           in_suspend = 0;
           ...
        }
     -> local_daif_restore(flags);

>  
>  	return 0;
>  }

Re: [RFC PATCH 03/36] arm64: hibernate: mask DAIF before restoring hibernated kernel

[Liao, Chang]

在 2026/7/9 20:13, Vladimir Murzin 写道:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> The arm64 hibernate code manages the exception masking in an unsound
> way, leading to potential crashes and/or warnings during resume.
> 
> When a hibernation image is saved in `swsusp_arch_suspend()`, all DAIF
> exceptions are masked (by virtue of `local_daif_save()`), and the
> suspended image is saved assuming that all DAIF exceptions will remain
> masked when the image is restored.
> 
> When a hibernation image is resumed by `swsusp_arch_resume()`, only
> interrupts are masked (by virtue of `local_irq_save()` in
> `resume_target_kernel()`). When pseudo-NMI is enabled the DAIF.IF bits
> will be clear, and regardless of pseudo-NMI the DAIF.DA bits will be
> clear.
> 
> This means that there are two problems:
> 
> (1) It is possible to take Debug, SError, or pseudo-NMI exceptions
>     during the resume process. This is unsafe, as during the resume
>     process both the old ane new kernels will tranisently be in an
>     inconsistent state, and swsusp_arch_suspend_exit() won't retain
>     an executable mapping of any exception vectors.
> 
>     Any exception taken here will be fatal and silent.
> 
> (2) When re-entering the resumed kernel, some DAIF bits will be clear
>     unexpectedly. This permits Debug, SError, or pseudo-NMI exceptions
>     to be taken for a short period while the resumed kernel is not yet
>     in a consistent state.
> 
>     This is detected by CONFIG_ARM64_DEBUG_PRIORITY_MASKING.
> 
> Avoid these issues by masking all DAIF exceptions during resume.
> 
> Cc: stable@vger.kernel.org
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/kernel/hibernate.c | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
> index 9717568518ba..d0d9bd91e639 100644
> --- a/arch/arm64/kernel/hibernate.c
> +++ b/arch/arm64/kernel/hibernate.c
> @@ -405,6 +405,7 @@ int swsusp_arch_suspend(void)
>  int __nocfi swsusp_arch_resume(void)
>  {
>  	int rc;
> +	unsigned long flags;
>  	void *zero_page;
>  	size_t exit_size;
>  	pgd_t *tmp_pg_dir;
> @@ -465,9 +466,21 @@ int __nocfi swsusp_arch_resume(void)
>  	if (el2_reset_needed())
>  		__hyp_set_vectors(el2_vectors);
>  
> +	/*
> +	 * It is necessary to mask all DAIF exceptions here as:
> +	 *
> +	 * - The copy of swsusp_arch_suspend_exit() in the hibernation
> +	 *   text cannot handle taking any exceptions.
> +	 *
> +	 * - The suspended kernel masked all DAIF exceptions in
> +	 *   swsusp_arch_resume(), and expects to be re-entered in the
> +	 *   same state : with all DAIF exceptions masked.
> +	 */
> +	flags = local_daif_save();
>  	hibernate_exit(virt_to_phys(tmp_pg_dir), resume_hdr.ttbr1_el1,
>  		       resume_hdr.reenter_kernel, restore_pblist,
>  		       resume_hdr.__hyp_stub_vectors, virt_to_phys(zero_page));
> +	local_daif_restore(flags);

As Jinjie said, the resumed kernel never return from hibernate_exit(). If that's
the case, it seems better to place unreachable() rather than local_daif_restore(),
would you agree?

>  
>  	return 0;
>  }


-- 
BR
Liao, Chang

Re: [RFC PATCH 04/36] arm64: suspend: rely on daif helpers to handle PMR

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Commit 77345ef70445 ("arm64: suspend: Use cpuidle context helpers
> in cpu_suspend()") added cpuidle helpers to handle PMR manipulation
> and restoration to ensure that the CPU receives interrupts when suspended
> and pseudo-NMIs are enabled.
> 
> However, those helpers are called in between a pair of `local_daif_save()`
> and `local_daif_restore()`, which already configure the PMR as expected.
> Effectively, `arm_cpuidle_save_irq_context()` is a no-op here,
> even when using pseudo-NMIs, and `arm_cpuidle_restore_irq_context()`
> would not restore proper interrupt masking configuration early enough
> if there were unexpected changes during suspend or resume.
> (This can be observed with Trusted Firmware A (TF-A) at EL3 handling
> suspend through PSCI. Even though it should not be the case, TF-A can
> reset `ICC_PMR_EL1` during CPU_SUSPEND, thus resuming the kernel
> with an inconsistent priority mask value on hardware implementing
> more than the minimum number of priority levels, such as Morello.)
> 
> Thus : remove the cpuidle context helpers as they do not do anything,
> but keep the comment mentioning the need for interrupts to reach the CPU
> if we are using pseudo-NMIs.


local_daif_save()/local_daif_restore() has done all relevant PMR and
DAIF save and restore tasks.

LGTM
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/kernel/suspend.c | 12 +++---------
>  1 file changed, 3 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c
> index eaaff94329cd..c41724a40b75 100644
> --- a/arch/arm64/kernel/suspend.c
> +++ b/arch/arm64/kernel/suspend.c
> @@ -99,7 +99,6 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	int ret = 0;
>  	unsigned long flags;
>  	struct sleep_stack_data state;
> -	struct arm_cpuidle_irq_context context;
>  
>  	/*
>  	 * Some portions of CPU state (e.g. PSTATE.{PAN,DIT}) are initialized
> @@ -121,6 +120,9 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	 * Strictly speaking the trace_hardirqs_off() here is superfluous,
>  	 * hardirqs should be firmly off by now. This really ought to use
>  	 * something like raw_local_daif_save().
> +	 *
> +	 * This also unmasks interrupts in PMR in order to reliably
> +	 * resume if we're using pseudo-NMIs.
>  	 */
>  	flags = local_daif_save();
>  
> @@ -131,12 +133,6 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	 */
>  	pause_graph_tracing();
>  
> -	/*
> -	 * Switch to using DAIF.IF instead of PMR in order to reliably
> -	 * resume if we're using pseudo-NMIs.
> -	 */
> -	arm_cpuidle_save_irq_context(&context);
> -
>  	ct_cpuidle_enter();
>  
>  	if (__cpu_suspend_enter(&state)) {
> @@ -159,8 +155,6 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  		__cpu_suspend_exit();
>  	}
>  
> -	arm_cpuidle_restore_irq_context(&context);
> -
>  	unpause_graph_tracing();
>  
>  	/*

Re: [RFC PATCH 10/36] arm64: irqflags: save and use both DAIF and PMR

[Liao, Chang]

在 2026/7/9 20:13, Vladimir Murzin 写道:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> With pseudo-NMIs enabled, both DAIF and the PMR affect interrupt masking.
> Now that we have a type which can track both of them at the same time,
> update our irqflags implementation to use it.
> 
> Save DAIF flags in all cases, as they can be manipulated directly by other
> code, and the PMR if it is in use.
> 
> When checking if IRQs are disabled, now that we always save DAIF we can
> check that the I flag is set and bypass checking the PMR if it is.
> We can also properly check if PMR masks interrupts (PMR < GIC_PRIO_IRQON),
> now that we don't need to rely on the GIC_PRIO_PSR_I_SET bit being set in
> the PMR to know if DAIF is already masking interrupts.
> Update `irqs_priority_unmasked()` to align with this change.
> 
> This allows us to remove the `__daif_...` and `__pmr_...` versions
> of the save and check functions, as they are now unified.
> 
> We can reasonably merge the two `__{daif,pmr}_irq_restore()` functions
> in the main one, as the DAIF and PMR values are properly split now.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/irqflags.h | 110 ++++++------------------------
>  arch/arm64/include/asm/ptrace.h   |   2 +-
>  2 files changed, 23 insertions(+), 89 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
> index 7775904ba6a9..62f047702493 100644
> --- a/arch/arm64/include/asm/irqflags.h
> +++ b/arch/arm64/include/asm/irqflags.h
> @@ -95,117 +95,48 @@ static __always_inline void arch_local_irq_disable(void)
>  	}
>  }
>  
> -static __always_inline arm64_exc_hwstate_t __daif_local_save_flags(void)
> -{
> -	return (arm64_exc_hwstate_t){ .daif = read_sysreg(daif) };
> -}
> -
> -static __always_inline arm64_exc_hwstate_t __pmr_local_save_flags(void)
> -{
> -	return (arm64_exc_hwstate_t){ .pmr = read_sysreg_s(SYS_ICC_PMR_EL1) };
> -}
> -
>  /*
>   * Save the current interrupt enable state.
>   */
>  static __always_inline unsigned long arch_local_save_flags(void)
>  {
> -	if (system_uses_irq_prio_masking()) {
> -		return __pmr_local_save_flags().flags;
> -	} else {
> -		return __daif_local_save_flags().flags;
> -	}
> -}
> +	arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
>  
> -static __always_inline
> -bool __daif_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
> -{
> -	return hwstate.daif & PSR_I_BIT;
> -}
> +	if (system_uses_irq_prio_masking())
> +		hwstate.pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
>  
> -static __always_inline
> -bool __pmr_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
> -{
> -	return hwstate.pmr != GIC_PRIO_IRQON;
> +	return hwstate.flags;
>  }
>  
>  static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
>  {
>  	arm64_exc_hwstate_t hwstate = { .flags = flags };
>  
> -	if (system_uses_irq_prio_masking()) {
> -		return __pmr_irqs_disabled_flags(hwstate);
> -	} else {
> -		return __daif_irqs_disabled_flags(hwstate);
> -	}
> -}
> +	/* If I is set, the PMR doesn't matter: interrupts will not be taken. */
> +	if (hwstate.daif & PSR_I_BIT)
> +		return true;
>  
> -static __always_inline bool __daif_irqs_disabled(void)
> -{
> -	return __daif_irqs_disabled_flags(__daif_local_save_flags());
> -}
> +	if (system_uses_irq_prio_masking() && hwstate.pmr < GIC_PRIO_IRQON)
> +		return true;
>  
> -static __always_inline bool __pmr_irqs_disabled(void)
> -{
> -	return __pmr_irqs_disabled_flags(__pmr_local_save_flags());
> +	return false;
>  }
>  
>  static __always_inline bool arch_irqs_disabled(void)
>  {
> -	if (system_uses_irq_prio_masking()) {
> -		return __pmr_irqs_disabled();
> -	} else {
> -		return __daif_irqs_disabled();
> -	}
> -}
> -
> -static __always_inline arm64_exc_hwstate_t __daif_local_irq_save(void)
> -{
> -	arm64_exc_hwstate_t hwstate = __daif_local_save_flags();
> -
> -	__daif_local_irq_disable();
> -
> -	return hwstate;
> -}
> -
> -static __always_inline arm64_exc_hwstate_t __pmr_local_irq_save(void)
> -{
> -	arm64_exc_hwstate_t hwstate = __pmr_local_save_flags();
> -
> -	/*
> -	 * There are too many states with IRQs disabled, just keep the current
> -	 * state if interrupts are already disabled/masked.
> -	 */
> -	if (!__pmr_irqs_disabled_flags(hwstate))
> -		__pmr_local_irq_disable();
> -
> -	return hwstate;
> +	return arch_irqs_disabled_flags(arch_local_save_flags());
>  }
>  
>  static __always_inline unsigned long arch_local_irq_save(void)
>  {
> -	if (system_uses_irq_prio_masking()) {
> -		return __pmr_local_irq_save().flags;
> -	} else {
> -		return __daif_local_irq_save().flags;
> -	}
> -}
> +	unsigned long flags = arch_local_save_flags();
>  
> -static __always_inline
> -void __daif_local_irq_restore(arm64_exc_hwstate_t hwstate)
> -{
> -	barrier();
> -	write_sysreg(hwstate.daif, daif);
> -	barrier();
> -}
> +	if (system_uses_irq_prio_masking())
> +		__pmr_local_irq_disable();
> +	else
> +		__daif_local_irq_disable();
>  
> -static __always_inline
> -void __pmr_local_irq_restore(arm64_exc_hwstate_t hwstate)
> -{
> -	barrier();
> -	write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> -	pmr_sync();
> -	barrier();
> +	return flags;
>  }
>  
>  /*
> @@ -215,11 +146,14 @@ static __always_inline void arch_local_irq_restore(unsigned long flags)
>  {
>  	arm64_exc_hwstate_t hwstate = { .flags = flags };
>  
> +	barrier();
>  	if (system_uses_irq_prio_masking()) {
> -		__pmr_local_irq_restore(hwstate);
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +		pmr_sync();
>  	} else {
> -		__daif_local_irq_restore(hwstate);
> +		write_sysreg(hwstate.daif, daif);
>  	}
> +	barrier();
>  }
>  
>  #endif /* __ASM_IRQFLAGS_H */
> diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
> index f7dc5fb9427d..192eb97cd50b 100644
> --- a/arch/arm64/include/asm/ptrace.h
> +++ b/arch/arm64/include/asm/ptrace.h
> @@ -205,7 +205,7 @@ static inline void forget_syscall(struct pt_regs *regs)
>  
>  #define irqs_priority_unmasked(regs)					\
>  	(system_uses_irq_prio_masking() ?				\
> -		(regs)->pmr == GIC_PRIO_IRQON :				\
> +		(regs)->pmr >= GIC_PRIO_IRQON :				\
>  		true)
>  
>  static __always_inline bool regs_irqs_disabled(const struct pt_regs *regs)

Is there any reason to keep interrupt_enabled()? It appears to be rarely used
across interrupt and exception code. So far, i've only found a single use in
gic_handle_irq(), corret me if i'm wrong. So using regs_irqs_disable() directly
might reduce the number of small helper that people need to remember.

-- 
BR
Liao, Chang

Re: [RFC PATCH 04/36] arm64: suspend: rely on daif helpers to handle PMR

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Commit 77345ef70445 ("arm64: suspend: Use cpuidle context helpers
> in cpu_suspend()") added cpuidle helpers to handle PMR manipulation
> and restoration to ensure that the CPU receives interrupts when suspended
> and pseudo-NMIs are enabled.
> 
> However, those helpers are called in between a pair of `local_daif_save()`
> and `local_daif_restore()`, which already configure the PMR as expected.
> Effectively, `arm_cpuidle_save_irq_context()` is a no-op here,
> even when using pseudo-NMIs, and `arm_cpuidle_restore_irq_context()`
> would not restore proper interrupt masking configuration early enough
> if there were unexpected changes during suspend or resume.
> (This can be observed with Trusted Firmware A (TF-A) at EL3 handling
> suspend through PSCI. Even though it should not be the case, TF-A can
> reset `ICC_PMR_EL1` during CPU_SUSPEND, thus resuming the kernel
> with an inconsistent priority mask value on hardware implementing
> more than the minimum number of priority levels, such as Morello.)
> 
> Thus : remove the cpuidle context helpers as they do not do anything,
> but keep the comment mentioning the need for interrupts to reach the CPU
> if we are using pseudo-NMIs.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/kernel/suspend.c | 12 +++---------
>  1 file changed, 3 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c
> index eaaff94329cd..c41724a40b75 100644
> --- a/arch/arm64/kernel/suspend.c
> +++ b/arch/arm64/kernel/suspend.c
> @@ -99,7 +99,6 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	int ret = 0;
>  	unsigned long flags;
>  	struct sleep_stack_data state;
> -	struct arm_cpuidle_irq_context context;
>  
>  	/*
>  	 * Some portions of CPU state (e.g. PSTATE.{PAN,DIT}) are initialized
> @@ -121,6 +120,9 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	 * Strictly speaking the trace_hardirqs_off() here is superfluous,
>  	 * hardirqs should be firmly off by now. This really ought to use
>  	 * something like raw_local_daif_save().
> +	 *
> +	 * This also unmasks interrupts in PMR in order to reliably
> +	 * resume if we're using pseudo-NMIs.
>  	 */
>  	flags = local_daif_save();
>  
> @@ -131,12 +133,6 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	 */
>  	pause_graph_tracing();
>  
> -	/*
> -	 * Switch to using DAIF.IF instead of PMR in order to reliably
> -	 * resume if we're using pseudo-NMIs.
> -	 */
> -	arm_cpuidle_save_irq_context(&context);
> -
>  	ct_cpuidle_enter();
>  
>  	if (__cpu_suspend_enter(&state)) {
> @@ -159,8 +155,6 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  		__cpu_suspend_exit();
>  	}
>  
> -	arm_cpuidle_restore_irq_context(&context);

If we use pseudo-NMIs only, the behavior after simplification is also
consistent.

- initial PMR is GIC_PRIO_IRQOFF
- initial DAIF is 0

Then the original flow:

1、local_daif_save()：
   1)、As pmr != GIC_PRIO_IRQON, saved flags = “PSR_I_BIT | PSR_F_BIT”
   2)、set all DAIF bit and set PMR "GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET"

2、arm_cpuidle_save_irq_context()
   1)、save daif( all set) and pmr (GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET)
   2)、set daif to DAIF.IF set (daif all set) and pmr to "GIC_PRIO_IRQON
| GIC_PRIO_PSR_I_SET"  -> duplicate code

3、arm_cpuidle_restore_irq_context()
   1)、restore PMR to “GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET"
   2)、restore DAIF to all set.

4、local_daif_restore() restore saved flags(“PSR_I_BIT | PSR_F_BIT”)
   1)、set pmr to GIC_PRIO_IRQOFF
   2)、set daif to 0

After remove
arm_cpuidle_save_irq_context()/arm_cpuidle_restore_irq_context():

1、local_daif_save()：
   1)、As pmr != GIC_PRIO_IRQON, saved flags = “PSR_I_BIT | PSR_F_BIT”
   2)、set all DAIF bit and set PMR "GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET"

2、local_daif_restore() restore saved flags(“PSR_I_BIT | PSR_F_BIT”)
   1)、set pmr to GIC_PRIO_IRQOFF
   2）、set daif to 0

> -
>  	unpause_graph_tracing();
>  
>  	/*

Re: [RFC PATCH 06/36] arm64: irq: introduce a helper for GIC priority initialization

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Arm64's `init_IRQ()` calls `local_daif_restore()` to synchronize
> interrupt masking via DAIF and switch to masking via PMR.  This
> depends on a very specific behaviour of `local_daif_restore()` which
> will clear DAIF if masking interrupts via PMR, which will get removed
> in the future.
> 
> As `setup_arch()` cleared DA only earlier, introduce a dedicated

Did you mean local_daif_restore(DAIF_PROCCTX_NOIRQ) in setup_arch()?

> helper that explicitly initializes the PMR to mask interrupts and
> clears DAIF, switching to IRQ priority masking.
> 
> Given it is a dedicated helper, add a lockdep assertion as
> `setup_arch()` should always have called `trace_hardirqs_off()` when
> clearing DA, otherwise something bad happened.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/daifflags.h | 15 +++++++++++++++
>  arch/arm64/kernel/irq.c            |  7 +++----
>  2 files changed, 18 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
> index 795b35128467..56341578e7e3 100644
> --- a/arch/arm64/include/asm/daifflags.h
> +++ b/arch/arm64/include/asm/daifflags.h
> @@ -141,4 +141,19 @@ static __always_inline void local_daif_inherit(struct pt_regs *regs)
>  	 */
>  	write_sysreg(flags, daif);
>  }
> +
> +/*
> + * During early boot, we unmask PSR.DA before the GIC has been set up.
> + * If we use IRQ priority masking, the PMR and PSR will be out of sync
> + * after the GIC is enabled : sync them up.
> + */
> +static inline void local_interrupt_priority_init(void)
> +{
> +	WARN_ON(read_sysreg(daif) & PSR_A_BIT);
> +	lockdep_assert_irqs_disabled();
> +
> +	gic_write_pmr(GIC_PRIO_IRQOFF);
> +	write_sysreg(DAIF_PROCCTX, daif);
> +}
> +
>  #endif
> diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
> index 9fafd826002b..c73faa30268d 100644
> --- a/arch/arm64/kernel/irq.c
> +++ b/arch/arm64/kernel/irq.c
> @@ -126,10 +126,9 @@ void __init init_IRQ(void)
>  
>  	if (system_uses_irq_prio_masking()) {
>  		/*
> -		 * Now that we have a stack for our IRQ handler, set
> -		 * the PMR/PSR pair to a consistent state.
> +		 * Now that we have a stack for our IRQ handler,
> +		 * let's mask interrupts via the PMR.
>  		 */
> -		WARN_ON(read_sysreg(daif) & PSR_A_BIT);
> -		local_daif_restore(DAIF_PROCCTX_NOIRQ);
> +		local_interrupt_priority_init();
>  	}
>  }

Re: [RFC PATCH 06/36] arm64: irq: introduce a helper for GIC priority initialization

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Arm64's `init_IRQ()` calls `local_daif_restore()` to synchronize
> interrupt masking via DAIF and switch to masking via PMR.  This
> depends on a very specific behaviour of `local_daif_restore()` which
> will clear DAIF if masking interrupts via PMR, which will get removed
> in the future.
> 
> As `setup_arch()` cleared DA only earlier, introduce a dedicated
> helper that explicitly initializes the PMR to mask interrupts and
> clears DAIF, switching to IRQ priority masking.

I believe that the `local_daif_restore(DAIF_PROCCTX_NOIRQ)` inside
`setup_arch()` should be directly replaced with the following code.
Otherwise, since the pseudo NMI has not yet taken effect at this point,
it could lead to ambiguity.

diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 23c05dc7a8f2..35c5310783f4 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -311,7 +311,8 @@ void __init __no_sanitize_address setup_arch(char
**cmdline_p)
         * IRQ and FIQ will be unmasked after the root irqchip has been
         * detected and initialized.
         */
-       local_daif_restore(DAIF_PROCCTX_NOIRQ);
+       write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
+       trace_hardirqs_off();



> 
> Given it is a dedicated helper, add a lockdep assertion as
> `setup_arch()` should always have called `trace_hardirqs_off()` when
> clearing DA, otherwise something bad happened.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/daifflags.h | 15 +++++++++++++++
>  arch/arm64/kernel/irq.c            |  7 +++----
>  2 files changed, 18 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
> index 795b35128467..56341578e7e3 100644
> --- a/arch/arm64/include/asm/daifflags.h
> +++ b/arch/arm64/include/asm/daifflags.h
> @@ -141,4 +141,19 @@ static __always_inline void local_daif_inherit(struct pt_regs *regs)
>  	 */
>  	write_sysreg(flags, daif);
>  }
> +
> +/*
> + * During early boot, we unmask PSR.DA before the GIC has been set up.
> + * If we use IRQ priority masking, the PMR and PSR will be out of sync
> + * after the GIC is enabled : sync them up.
> + */
> +static inline void local_interrupt_priority_init(void)
> +{
> +	WARN_ON(read_sysreg(daif) & PSR_A_BIT);
> +	lockdep_assert_irqs_disabled();
> +
> +	gic_write_pmr(GIC_PRIO_IRQOFF);
> +	write_sysreg(DAIF_PROCCTX, daif);
> +}
> +
>  #endif
> diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
> index 9fafd826002b..c73faa30268d 100644
> --- a/arch/arm64/kernel/irq.c
> +++ b/arch/arm64/kernel/irq.c
> @@ -126,10 +126,9 @@ void __init init_IRQ(void)
>  
>  	if (system_uses_irq_prio_masking()) {
>  		/*
> -		 * Now that we have a stack for our IRQ handler, set
> -		 * the PMR/PSR pair to a consistent state.
> +		 * Now that we have a stack for our IRQ handler,
> +		 * let's mask interrupts via the PMR.
>  		 */
> -		WARN_ON(read_sysreg(daif) & PSR_A_BIT);
> -		local_daif_restore(DAIF_PROCCTX_NOIRQ);
> +		local_interrupt_priority_init();
>  	}
>  }

Re: [RFC PATCH 06/36] arm64: irq: introduce a helper for GIC priority initialization

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Arm64's `init_IRQ()` calls `local_daif_restore()` to synchronize
> interrupt masking via DAIF and switch to masking via PMR.  This
> depends on a very specific behaviour of `local_daif_restore()` which
> will clear DAIF if masking interrupts via PMR, which will get removed
> in the future.
> 
> As `setup_arch()` cleared DA only earlier, introduce a dedicated
> helper that explicitly initializes the PMR to mask interrupts and
> clears DAIF, switching to IRQ priority masking.
> 
> Given it is a dedicated helper, add a lockdep assertion as
> `setup_arch()` should always have called `trace_hardirqs_off()` when
> clearing DA, otherwise something bad happened.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/daifflags.h | 15 +++++++++++++++
>  arch/arm64/kernel/irq.c            |  7 +++----
>  2 files changed, 18 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
> index 795b35128467..56341578e7e3 100644
> --- a/arch/arm64/include/asm/daifflags.h
> +++ b/arch/arm64/include/asm/daifflags.h
> @@ -141,4 +141,19 @@ static __always_inline void local_daif_inherit(struct pt_regs *regs)
>  	 */
>  	write_sysreg(flags, daif);
>  }
> +
> +/*
> + * During early boot, we unmask PSR.DA before the GIC has been set up.
> + * If we use IRQ priority masking, the PMR and PSR will be out of sync
> + * after the GIC is enabled : sync them up.
> + */
> +static inline void local_interrupt_priority_init(void)

Maybe switch_to_pmr_masking()?

> +{
> +	WARN_ON(read_sysreg(daif) & PSR_A_BIT);
> +	lockdep_assert_irqs_disabled();
> +
> +	gic_write_pmr(GIC_PRIO_IRQOFF);
> +	write_sysreg(DAIF_PROCCTX, daif);

otherwise LGTM
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

> +}
> +
>  #endif
> diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
> index 9fafd826002b..c73faa30268d 100644
> --- a/arch/arm64/kernel/irq.c
> +++ b/arch/arm64/kernel/irq.c
> @@ -126,10 +126,9 @@ void __init init_IRQ(void)
>  
>  	if (system_uses_irq_prio_masking()) {
>  		/*
> -		 * Now that we have a stack for our IRQ handler, set
> -		 * the PMR/PSR pair to a consistent state.
> +		 * Now that we have a stack for our IRQ handler,
> +		 * let's mask interrupts via the PMR.
>  		 */
> -		WARN_ON(read_sysreg(daif) & PSR_A_BIT);
> -		local_daif_restore(DAIF_PROCCTX_NOIRQ);
> +		local_interrupt_priority_init();
>  	}
>  }

Re: [RFC PATCH 07/36] arm64: entry: mask DAIF before returning from C EL1 handlers

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Most EL1 exceptions already call `local_daif_mask()` before returning,
> with the exception of debug exception handlers which do not change
> DAIF, and the IRQ/FIQ/Error handlers.
> 
> However, DAIF get masked in `kernel_exit()` in all cases when
> returning from EL1 C handlers anyway.
> 
> Move this masking from assembly to C by adding `local_daif_mask()`
> before `irqentry_nmi_exit()` when it is called by EL1 handlers.
> 
> Remove the `disable_daif` assembly macro, as this was its only use.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/assembler.h | 4 ----
>  arch/arm64/kernel/entry-common.c   | 2 ++
>  arch/arm64/kernel/entry.S          | 4 ----
>  3 files changed, 2 insertions(+), 8 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index effae53e9739..0b58b550e8dc 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -34,10 +34,6 @@
>  	wx\n	.req	w\n
>  	.endr
>  
> -	.macro disable_daif
> -	msr	daifset, #0xf
> -	.endm
> -
>  /*
>   * Save/restore interrupts.
>   */
> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> index ceb4eb11232a..2be42d7f4eaa 100644
> --- a/arch/arm64/kernel/entry-common.c
> +++ b/arch/arm64/kernel/entry-common.c
> @@ -495,6 +495,7 @@ static __always_inline void __el1_pnmi(struct pt_regs *regs,
>  
>  	state = irqentry_nmi_enter(regs);
>  	do_interrupt_handler(regs, handler);
> +	local_daif_mask();
>  	irqentry_nmi_exit(regs, state);
>  }
>  
> @@ -540,6 +541,7 @@ asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
>  	local_daif_restore(DAIF_ERRCTX);
>  	state = irqentry_nmi_enter(regs);
>  	do_serror(regs, esr);
> +	local_daif_mask();
>  	irqentry_nmi_exit(regs, state);

This is similar to my patch below:

https://lore.kernel.org/all/20260703100135.2512312-4-ruanjinjie@huawei.com/

I think there are at least three problems：

1、disable_daif is missing for el1h_64_sync_handler() and __el1_irq()

2、The order after moving to the C code is incorrect. disable_daif should
be the exit point for C, that is, after irqentry_nmi_exit(), it is
unclear why this order change occurred ?

3、local_daif_mask () has added some pseudo NMI operations and
trace_hardirqs_off ().

>  }
>  
> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> index e0db14e9c843..f63049ac32dc 100644
> --- a/arch/arm64/kernel/entry.S
> +++ b/arch/arm64/kernel/entry.S
> @@ -333,10 +333,6 @@ alternative_else_nop_endif
>  	.endm
>  
>  	.macro	kernel_exit, el
> -	.if	\el != 0
> -	disable_daif
> -	.endif
> -
>  #ifdef CONFIG_ARM64_PSEUDO_NMI
>  alternative_if_not ARM64_HAS_GIC_PRIO_MASKING
>  	b	.Lskip_pmr_restore\@

Re: [RFC PATCH 08/36] irqchip/gic-v3: make the unmasking of pseudo-NMIs explicit when handling IRQs

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> `gic_arch_enable_irqs()` is only used when handling IRQs (which could
> be pseudo-NMIs) and unmasking pseudo-NMIs.
> 
> The chain of `gic_pmr_mask_irqs()` and `gic_arch_enable_irqs()` for it
> is slightly confusing without further explanation.
> 
> Remove `gic_arch_enable_irqs()` and instead do the whole pseudo-NMI
> umasking in `gic_unmask_pnmis()`, making the operation explicit.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm/include/asm/arch_gicv3.h     | 6 +-----
>  arch/arm64/include/asm/arch_gicv3.h   | 7 +++++--
>  arch/arm64/include/asm/entry-common.h | 2 +-
>  drivers/irqchip/irq-gic-v3.c          | 5 +----
>  4 files changed, 8 insertions(+), 12 deletions(-)
> 
> diff --git a/arch/arm/include/asm/arch_gicv3.h b/arch/arm/include/asm/arch_gicv3.h
> index 847590df7551..d4ac8d3271b1 100644
> --- a/arch/arm/include/asm/arch_gicv3.h
> +++ b/arch/arm/include/asm/arch_gicv3.h
> @@ -246,11 +246,7 @@ static inline void gic_pmr_mask_irqs(void)
>  	WARN_ON_ONCE(true);
>  }
>  
> -static inline void gic_arch_enable_irqs(void)
> -{
> -	/* Should not get called. */
> -	WARN_ON_ONCE(true);
> -}
> +static inline void gic_unmask_pnmis(void) {}
>  
>  static inline bool gic_has_relaxed_pmr_sync(void)
>  {
> diff --git a/arch/arm64/include/asm/arch_gicv3.h b/arch/arm64/include/asm/arch_gicv3.h
> index d20b03931a8d..3dcb7b8309d9 100644
> --- a/arch/arm64/include/asm/arch_gicv3.h
> +++ b/arch/arm64/include/asm/arch_gicv3.h
> @@ -178,9 +178,12 @@ static inline void gic_pmr_mask_irqs(void)
>  	gic_write_pmr(GIC_PRIO_IRQOFF);
>  }
>  
> -static inline void gic_arch_enable_irqs(void)
> +static inline void gic_unmask_pnmis(void)
>  {
> -	asm volatile ("msr daifclr, #3" : : : "memory");
> +	if (gic_prio_masking_enabled()) {
> +		gic_pmr_mask_irqs();
> +		asm volatile ("msr daifclr, #3" : : : "memory");
> +	}

LGTM
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

>  }
>  
>  static inline bool gic_has_relaxed_pmr_sync(void)
> diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
> index cab8cd78f693..1905765159aa 100644
> --- a/arch/arm64/include/asm/entry-common.h
> +++ b/arch/arm64/include/asm/entry-common.h
> @@ -32,7 +32,7 @@ static inline bool arch_irqentry_exit_need_resched(void)
>  	/*
>  	 * DAIF.DA are cleared at the start of IRQ/FIQ handling, and when GIC
>  	 * priority masking is used the GIC irqchip driver will clear DAIF.IF
> -	 * using gic_arch_enable_irqs() for normal IRQs. If anything is set in
> +	 * in gic_unmask_pnmis() for normal IRQs. If anything is set in
>  	 * DAIF we must have handled an NMI, so skip preemption.
>  	 */
>  	if (system_uses_irq_prio_masking() && read_sysreg(daif))
> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> index 99444a1b2ffa..94c6a3f2b009 100644
> --- a/drivers/irqchip/irq-gic-v3.c
> +++ b/drivers/irqchip/irq-gic-v3.c
> @@ -867,10 +867,7 @@ static void __gic_handle_irq_from_irqson(struct pt_regs *regs)
>  		nmi_exit();
>  	}
>  
> -	if (gic_prio_masking_enabled()) {
> -		gic_pmr_mask_irqs();
> -		gic_arch_enable_irqs();
> -	}
> +	gic_unmask_pnmis();
>  
>  	if (!is_nmi)
>  		__gic_handle_irq(irqnr, regs);

Re: [RFC PATCH 10/36] arm64: irqflags: save and use both DAIF and PMR

[Jinjie Ruan]

On 7/10/2026 11:53 AM, Liao, Chang wrote:
> 在 2026/7/9 20:13, Vladimir Murzin 写道:
>> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>>
>> With pseudo-NMIs enabled, both DAIF and the PMR affect interrupt masking.
>> Now that we have a type which can track both of them at the same time,
>> update our irqflags implementation to use it.
>>
>> Save DAIF flags in all cases, as they can be manipulated directly by other
>> code, and the PMR if it is in use.
>>
>> When checking if IRQs are disabled, now that we always save DAIF we can
>> check that the I flag is set and bypass checking the PMR if it is.
>> We can also properly check if PMR masks interrupts (PMR < GIC_PRIO_IRQON),
>> now that we don't need to rely on the GIC_PRIO_PSR_I_SET bit being set in
>> the PMR to know if DAIF is already masking interrupts.
>> Update `irqs_priority_unmasked()` to align with this change.
>>
>> This allows us to remove the `__daif_...` and `__pmr_...` versions
>> of the save and check functions, as they are now unified.
>>
>> We can reasonably merge the two `__{daif,pmr}_irq_restore()` functions
>> in the main one, as the DAIF and PMR values are properly split now.
>>
>> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
>> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
>> ---
>>  arch/arm64/include/asm/irqflags.h | 110 ++++++------------------------
>>  arch/arm64/include/asm/ptrace.h   |   2 +-
>>  2 files changed, 23 insertions(+), 89 deletions(-)
>>
>> diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
>> index 7775904ba6a9..62f047702493 100644
>> --- a/arch/arm64/include/asm/irqflags.h
>> +++ b/arch/arm64/include/asm/irqflags.h
>> @@ -95,117 +95,48 @@ static __always_inline void arch_local_irq_disable(void)
>>  	}
>>  }
>>  
>> -static __always_inline arm64_exc_hwstate_t __daif_local_save_flags(void)
>> -{
>> -	return (arm64_exc_hwstate_t){ .daif = read_sysreg(daif) };
>> -}
>> -
>> -static __always_inline arm64_exc_hwstate_t __pmr_local_save_flags(void)
>> -{
>> -	return (arm64_exc_hwstate_t){ .pmr = read_sysreg_s(SYS_ICC_PMR_EL1) };
>> -}
>> -
>>  /*
>>   * Save the current interrupt enable state.
>>   */
>>  static __always_inline unsigned long arch_local_save_flags(void)
>>  {
>> -	if (system_uses_irq_prio_masking()) {
>> -		return __pmr_local_save_flags().flags;
>> -	} else {
>> -		return __daif_local_save_flags().flags;
>> -	}
>> -}
>> +	arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
>>  
>> -static __always_inline
>> -bool __daif_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
>> -{
>> -	return hwstate.daif & PSR_I_BIT;
>> -}
>> +	if (system_uses_irq_prio_masking())
>> +		hwstate.pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
>>  
>> -static __always_inline
>> -bool __pmr_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
>> -{
>> -	return hwstate.pmr != GIC_PRIO_IRQON;
>> +	return hwstate.flags;
>>  }
>>  
>>  static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
>>  {
>>  	arm64_exc_hwstate_t hwstate = { .flags = flags };
>>  
>> -	if (system_uses_irq_prio_masking()) {
>> -		return __pmr_irqs_disabled_flags(hwstate);
>> -	} else {
>> -		return __daif_irqs_disabled_flags(hwstate);
>> -	}
>> -}
>> +	/* If I is set, the PMR doesn't matter: interrupts will not be taken. */
>> +	if (hwstate.daif & PSR_I_BIT)
>> +		return true;
>>  
>> -static __always_inline bool __daif_irqs_disabled(void)
>> -{
>> -	return __daif_irqs_disabled_flags(__daif_local_save_flags());
>> -}
>> +	if (system_uses_irq_prio_masking() && hwstate.pmr < GIC_PRIO_IRQON)
>> +		return true;
>>  
>> -static __always_inline bool __pmr_irqs_disabled(void)
>> -{
>> -	return __pmr_irqs_disabled_flags(__pmr_local_save_flags());
>> +	return false;
>>  }
>>  
>>  static __always_inline bool arch_irqs_disabled(void)
>>  {
>> -	if (system_uses_irq_prio_masking()) {
>> -		return __pmr_irqs_disabled();
>> -	} else {
>> -		return __daif_irqs_disabled();
>> -	}
>> -}
>> -
>> -static __always_inline arm64_exc_hwstate_t __daif_local_irq_save(void)
>> -{
>> -	arm64_exc_hwstate_t hwstate = __daif_local_save_flags();
>> -
>> -	__daif_local_irq_disable();
>> -
>> -	return hwstate;
>> -}
>> -
>> -static __always_inline arm64_exc_hwstate_t __pmr_local_irq_save(void)
>> -{
>> -	arm64_exc_hwstate_t hwstate = __pmr_local_save_flags();
>> -
>> -	/*
>> -	 * There are too many states with IRQs disabled, just keep the current
>> -	 * state if interrupts are already disabled/masked.
>> -	 */
>> -	if (!__pmr_irqs_disabled_flags(hwstate))
>> -		__pmr_local_irq_disable();
>> -
>> -	return hwstate;
>> +	return arch_irqs_disabled_flags(arch_local_save_flags());
>>  }
>>  
>>  static __always_inline unsigned long arch_local_irq_save(void)
>>  {
>> -	if (system_uses_irq_prio_masking()) {
>> -		return __pmr_local_irq_save().flags;
>> -	} else {
>> -		return __daif_local_irq_save().flags;
>> -	}
>> -}
>> +	unsigned long flags = arch_local_save_flags();
>>  
>> -static __always_inline
>> -void __daif_local_irq_restore(arm64_exc_hwstate_t hwstate)
>> -{
>> -	barrier();
>> -	write_sysreg(hwstate.daif, daif);
>> -	barrier();
>> -}
>> +	if (system_uses_irq_prio_masking())
>> +		__pmr_local_irq_disable();
>> +	else
>> +		__daif_local_irq_disable();
>>  
>> -static __always_inline
>> -void __pmr_local_irq_restore(arm64_exc_hwstate_t hwstate)
>> -{
>> -	barrier();
>> -	write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
>> -	pmr_sync();
>> -	barrier();
>> +	return flags;
>>  }
>>  
>>  /*
>> @@ -215,11 +146,14 @@ static __always_inline void arch_local_irq_restore(unsigned long flags)
>>  {
>>  	arm64_exc_hwstate_t hwstate = { .flags = flags };
>>  
>> +	barrier();
>>  	if (system_uses_irq_prio_masking()) {
>> -		__pmr_local_irq_restore(hwstate);
>> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
>> +		pmr_sync();
>>  	} else {
>> -		__daif_local_irq_restore(hwstate);
>> +		write_sysreg(hwstate.daif, daif);
>>  	}
>> +	barrier();
>>  }
>>  
>>  #endif /* __ASM_IRQFLAGS_H */
>> diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
>> index f7dc5fb9427d..192eb97cd50b 100644
>> --- a/arch/arm64/include/asm/ptrace.h
>> +++ b/arch/arm64/include/asm/ptrace.h
>> @@ -205,7 +205,7 @@ static inline void forget_syscall(struct pt_regs *regs)
>>  
>>  #define irqs_priority_unmasked(regs)					\
>>  	(system_uses_irq_prio_masking() ?				\
>> -		(regs)->pmr == GIC_PRIO_IRQON :				\
>> +		(regs)->pmr >= GIC_PRIO_IRQON :				\
>>  		true)
>>  
>>  static __always_inline bool regs_irqs_disabled(const struct pt_regs *regs)
> 
> Is there any reason to keep interrupt_enabled()? It appears to be rarely used
> across interrupt and exception code. So far, i've only found a single use in
> gic_handle_irq(), corret me if i'm wrong. So using regs_irqs_disable() directly
> might reduce the number of small helper that people need to remember.

For arm32.

> 

Re: [RFC PATCH 09/36] arm64: irqflags: introduce arm64-specific irqflags type

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> With pseudo-NMIs enabled, we have two mechanisms that control
> interrupt masking in parallel :
> - The DAIF flags, masking at the CPU
> - The GIC PMR, masking before the CPU
> 
> However, our irqflags implementation currently assumes that only one
> of the two is used at a time, so both DAIF and PMR masking use the same
> `unsigned long flags` in their own way.
> This is incorrect, as some parts of the kernel will mask interrupts
> with DAIF directly or bypass the local_irq masking via the PMR,
> and makes tracking the state and changes of both in parallel impossible.
> 
> The irqflags API expects `unsigned long`s to be passed around, but
> they should not be manipulated outside of the arch-specific code.
> So, we can encode the information we need however we want as long as
> we return and accept `unsigned long`s.
> 
> Introduce a union type for arm64 irqflags whose first member is
> a struct allowing us to track DAIF and PMR in parallel, and the second
> is the `unsigned long` expected by the irqflags API.
> 
> DAIF is a two byte value, to maintain compatibility with existing defines.
> PMR is a one byte value, which is the maximum amount of priority bits
> allowed by the GICv3 architecture.
> 
> Update the internal irqflags functions to use this new union and convert
> back and forth with the irqflags unsigned long.
> There should be no functional changes.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/irqflags.h | 80 ++++++++++++++++++++-----------
>  1 file changed, 53 insertions(+), 27 deletions(-)

I believe that using the newly introduced arm64_exc_hwstate_t only at
the lowest level results in the least changes and is the most readable.

otherwise, LGTM
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

arch/arm64/include/asm/irqflags.h | 42
++++++++++++++++++++++++++++++++++++------
 1 file changed, 36 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/irqflags.h
b/arch/arm64/include/asm/irqflags.h
index a8cb5a5c93b7..4b4521007183 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -9,6 +9,8 @@
 #include <asm/ptrace.h>
 #include <asm/sysreg.h>

+#include <linux/compiler.h>
+
 /*
  * Aarch64 has flags for masking: Debug, Asynchronous (serror),
Interrupts and
  * FIQ exceptions, in the 'daif' register. We mask and unmask them in
'daif'
@@ -20,6 +22,22 @@
  * exceptions should be unmasked.
  */

+ /*
+  * Internally, we want to independently manipulate and track the different
+  * interrupt masking mechanisms.
+  * Externally, the generic irqflags API expects unsgined longs to
represent
+  * the state of interrupts, which are treated as obscure arch-specific
data.
+  */
+typedef union arm64_exc_hwstate {
+       struct {
+               u16 daif;
+               u8 pmr;
+       };
+       unsigned long flags;
+} arm64_exc_hwstate_t;
+
+static_assert(sizeof(arm64_exc_hwstate_t) == sizeof(unsigned long));
+
 static __always_inline void __daif_local_irq_enable(void)
 {
        barrier();
@@ -79,12 +97,16 @@ static __always_inline void arch_local_irq_disable(void)

 static __always_inline unsigned long __daif_local_save_flags(void)
 {
-       return read_sysreg(daif);
+       arm64_exc_hwstate_t state = { .daif = read_sysreg(daif) };
+
+       return state.flags;
 }

 static __always_inline unsigned long __pmr_local_save_flags(void)
 {
-       return read_sysreg_s(SYS_ICC_PMR_EL1);
+       arm64_exc_hwstate_t state = { .pmr =
read_sysreg_s(SYS_ICC_PMR_EL1) };
+
+       return state.flags;
 }

 /*
@@ -101,12 +123,16 @@ static __always_inline unsigned long
arch_local_save_flags(void)

 static __always_inline bool __daif_irqs_disabled_flags(unsigned long flags)
 {
-       return flags & PSR_I_BIT;
+       arm64_exc_hwstate_t hwstate = { .flags = flags };
+
+       return hwstate.daif & PSR_I_BIT;
 }

 static __always_inline bool __pmr_irqs_disabled_flags(unsigned long flags)
 {
-       return flags != GIC_PRIO_IRQON;
+       arm64_exc_hwstate_t hwstate = { .flags = flags };
+
+       return hwstate.pmr != GIC_PRIO_IRQON;
 }

 static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
@@ -171,15 +197,19 @@ static __always_inline unsigned long
arch_local_irq_save(void)

 static __always_inline void __daif_local_irq_restore(unsigned long flags)
 {
+       arm64_exc_hwstate_t hwstate = { .flags = flags };
+
        barrier();
-       write_sysreg(flags, daif);
+       write_sysreg(hwstate.daif, daif);
        barrier();
 }

 static __always_inline void __pmr_local_irq_restore(unsigned long flags)
 {
+       arm64_exc_hwstate_t hwstate = { .flags = flags };
+
        barrier();
-       write_sysreg_s(flags, SYS_ICC_PMR_EL1);
+       write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
        pmr_sync();
        barrier();
 }


> 
> diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
> index a8cb5a5c93b7..7775904ba6a9 100644
> --- a/arch/arm64/include/asm/irqflags.h
> +++ b/arch/arm64/include/asm/irqflags.h
> @@ -9,6 +9,8 @@
>  #include <asm/ptrace.h>
>  #include <asm/sysreg.h>
>  
> +#include <linux/compiler.h>
> +
>  /*
>   * Aarch64 has flags for masking: Debug, Asynchronous (serror), Interrupts and
>   * FIQ exceptions, in the 'daif' register. We mask and unmask them in 'daif'
> @@ -20,6 +22,22 @@
>   * exceptions should be unmasked.
>   */
>  
> + /*
> +  * Internally, we want to independently manipulate and track the different
> +  * interrupt masking mechanisms.
> +  * Externally, the generic irqflags API expects unsgined longs to represent
> +  * the state of interrupts, which are treated as obscure arch-specific data.
> +  */
> +typedef union arm64_exc_hwstate {
> +	struct {
> +		u16 daif;
> +		u8 pmr;
> +	};
> +	unsigned long flags;
> +} arm64_exc_hwstate_t;
> +
> +static_assert(sizeof(arm64_exc_hwstate_t) == sizeof(unsigned long));
> +
>  static __always_inline void __daif_local_irq_enable(void)
>  {
>  	barrier();
> @@ -77,14 +95,14 @@ static __always_inline void arch_local_irq_disable(void)
>  	}
>  }
>  
> -static __always_inline unsigned long __daif_local_save_flags(void)
> +static __always_inline arm64_exc_hwstate_t __daif_local_save_flags(void)
>  {
> -	return read_sysreg(daif);
> +	return (arm64_exc_hwstate_t){ .daif = read_sysreg(daif) };
>  }
>  
> -static __always_inline unsigned long __pmr_local_save_flags(void)
> +static __always_inline arm64_exc_hwstate_t __pmr_local_save_flags(void)
>  {
> -	return read_sysreg_s(SYS_ICC_PMR_EL1);
> +	return (arm64_exc_hwstate_t){ .pmr = read_sysreg_s(SYS_ICC_PMR_EL1) };
>  }
>  
>  /*
> @@ -93,28 +111,32 @@ static __always_inline unsigned long __pmr_local_save_flags(void)
>  static __always_inline unsigned long arch_local_save_flags(void)
>  {
>  	if (system_uses_irq_prio_masking()) {
> -		return __pmr_local_save_flags();
> +		return __pmr_local_save_flags().flags;
>  	} else {
> -		return __daif_local_save_flags();
> +		return __daif_local_save_flags().flags;
>  	}
>  }
>  
> -static __always_inline bool __daif_irqs_disabled_flags(unsigned long flags)
> +static __always_inline
> +bool __daif_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
>  {
> -	return flags & PSR_I_BIT;
> +	return hwstate.daif & PSR_I_BIT;
>  }
>  
> -static __always_inline bool __pmr_irqs_disabled_flags(unsigned long flags)
> +static __always_inline
> +bool __pmr_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
>  {
> -	return flags != GIC_PRIO_IRQON;
> +	return hwstate.pmr != GIC_PRIO_IRQON;
>  }
>  
>  static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
>  {
> +	arm64_exc_hwstate_t hwstate = { .flags = flags };
> +
>  	if (system_uses_irq_prio_masking()) {
> -		return __pmr_irqs_disabled_flags(flags);
> +		return __pmr_irqs_disabled_flags(hwstate);
>  	} else {
> -		return __daif_irqs_disabled_flags(flags);
> +		return __daif_irqs_disabled_flags(hwstate);
>  	}
>  }
>  
> @@ -137,49 +159,51 @@ static __always_inline bool arch_irqs_disabled(void)
>  	}
>  }
>  
> -static __always_inline unsigned long __daif_local_irq_save(void)
> +static __always_inline arm64_exc_hwstate_t __daif_local_irq_save(void)
>  {
> -	unsigned long flags = __daif_local_save_flags();
> +	arm64_exc_hwstate_t hwstate = __daif_local_save_flags();
>  
>  	__daif_local_irq_disable();
>  
> -	return flags;
> +	return hwstate;
>  }
>  
> -static __always_inline unsigned long __pmr_local_irq_save(void)
> +static __always_inline arm64_exc_hwstate_t __pmr_local_irq_save(void)
>  {
> -	unsigned long flags = __pmr_local_save_flags();
> +	arm64_exc_hwstate_t hwstate = __pmr_local_save_flags();
>  
>  	/*
>  	 * There are too many states with IRQs disabled, just keep the current
>  	 * state if interrupts are already disabled/masked.
>  	 */
> -	if (!__pmr_irqs_disabled_flags(flags))
> +	if (!__pmr_irqs_disabled_flags(hwstate))
>  		__pmr_local_irq_disable();
>  
> -	return flags;
> +	return hwstate;
>  }
>  
>  static __always_inline unsigned long arch_local_irq_save(void)
>  {
>  	if (system_uses_irq_prio_masking()) {
> -		return __pmr_local_irq_save();
> +		return __pmr_local_irq_save().flags;
>  	} else {
> -		return __daif_local_irq_save();
> +		return __daif_local_irq_save().flags;
>  	}
>  }
>  
> -static __always_inline void __daif_local_irq_restore(unsigned long flags)
> +static __always_inline
> +void __daif_local_irq_restore(arm64_exc_hwstate_t hwstate)
>  {
>  	barrier();
> -	write_sysreg(flags, daif);
> +	write_sysreg(hwstate.daif, daif);
>  	barrier();
>  }
>  
> -static __always_inline void __pmr_local_irq_restore(unsigned long flags)
> +static __always_inline
> +void __pmr_local_irq_restore(arm64_exc_hwstate_t hwstate)
>  {
>  	barrier();
> -	write_sysreg_s(flags, SYS_ICC_PMR_EL1);
> +	write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
>  	pmr_sync();
>  	barrier();
>  }
> @@ -189,10 +213,12 @@ static __always_inline void __pmr_local_irq_restore(unsigned long flags)
>   */
>  static __always_inline void arch_local_irq_restore(unsigned long flags)
>  {
> +	arm64_exc_hwstate_t hwstate = { .flags = flags };
> +
>  	if (system_uses_irq_prio_masking()) {
> -		__pmr_local_irq_restore(flags);
> +		__pmr_local_irq_restore(hwstate);
>  	} else {
> -		__daif_local_irq_restore(flags);
> +		__daif_local_irq_restore(hwstate);
>  	}
>  }
>  

Re: [RFC PATCH 10/36] arm64: irqflags: save and use both DAIF and PMR

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> With pseudo-NMIs enabled, both DAIF and the PMR affect interrupt masking.
> Now that we have a type which can track both of them at the same time,
> update our irqflags implementation to use it.
> 
> Save DAIF flags in all cases, as they can be manipulated directly by other
> code, and the PMR if it is in use.
> 
> When checking if IRQs are disabled, now that we always save DAIF we can
> check that the I flag is set and bypass checking the PMR if it is.
> We can also properly check if PMR masks interrupts (PMR < GIC_PRIO_IRQON),
> now that we don't need to rely on the GIC_PRIO_PSR_I_SET bit being set in
> the PMR to know if DAIF is already masking interrupts.
> Update `irqs_priority_unmasked()` to align with this change.
> 
> This allows us to remove the `__daif_...` and `__pmr_...` versions
> of the save and check functions, as they are now unified.
> 
> We can reasonably merge the two `__{daif,pmr}_irq_restore()` functions
> in the main one, as the DAIF and PMR values are properly split now.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/irqflags.h | 110 ++++++------------------------
>  arch/arm64/include/asm/ptrace.h   |   2 +-
>  2 files changed, 23 insertions(+), 89 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
> index 7775904ba6a9..62f047702493 100644
> --- a/arch/arm64/include/asm/irqflags.h
> +++ b/arch/arm64/include/asm/irqflags.h
> @@ -95,117 +95,48 @@ static __always_inline void arch_local_irq_disable(void)
>  	}
>  }
>  
> -static __always_inline arm64_exc_hwstate_t __daif_local_save_flags(void)
> -{
> -	return (arm64_exc_hwstate_t){ .daif = read_sysreg(daif) };
> -}
> -
> -static __always_inline arm64_exc_hwstate_t __pmr_local_save_flags(void)
> -{
> -	return (arm64_exc_hwstate_t){ .pmr = read_sysreg_s(SYS_ICC_PMR_EL1) };
> -}
> -
>  /*
>   * Save the current interrupt enable state.
>   */
>  static __always_inline unsigned long arch_local_save_flags(void)
>  {
> -	if (system_uses_irq_prio_masking()) {
> -		return __pmr_local_save_flags().flags;
> -	} else {
> -		return __daif_local_save_flags().flags;
> -	}
> -}
> +	arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
>  
> -static __always_inline
> -bool __daif_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
> -{
> -	return hwstate.daif & PSR_I_BIT;
> -}
> +	if (system_uses_irq_prio_masking())
> +		hwstate.pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
>  
> -static __always_inline
> -bool __pmr_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
> -{
> -	return hwstate.pmr != GIC_PRIO_IRQON;
> +	return hwstate.flags;
>  }
>  
>  static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
>  {
>  	arm64_exc_hwstate_t hwstate = { .flags = flags };
>  
> -	if (system_uses_irq_prio_masking()) {
> -		return __pmr_irqs_disabled_flags(hwstate);
> -	} else {
> -		return __daif_irqs_disabled_flags(hwstate);
> -	}
> -}
> +	/* If I is set, the PMR doesn't matter: interrupts will not be taken. */
> +	if (hwstate.daif & PSR_I_BIT)
> +		return true;
>  
> -static __always_inline bool __daif_irqs_disabled(void)
> -{
> -	return __daif_irqs_disabled_flags(__daif_local_save_flags());
> -}
> +	if (system_uses_irq_prio_masking() && hwstate.pmr < GIC_PRIO_IRQON)

This concept of being less than or greater than is not easy to understand.

> +		return true;
>  
> -static __always_inline bool __pmr_irqs_disabled(void)
> -{
> -	return __pmr_irqs_disabled_flags(__pmr_local_save_flags());
> +	return false;
>  }
>  
>  static __always_inline bool arch_irqs_disabled(void)
>  {
> -	if (system_uses_irq_prio_masking()) {
> -		return __pmr_irqs_disabled();
> -	} else {
> -		return __daif_irqs_disabled();
> -	}
> -}
> -
> -static __always_inline arm64_exc_hwstate_t __daif_local_irq_save(void)
> -{
> -	arm64_exc_hwstate_t hwstate = __daif_local_save_flags();
> -
> -	__daif_local_irq_disable();
> -
> -	return hwstate;
> -}
> -
> -static __always_inline arm64_exc_hwstate_t __pmr_local_irq_save(void)
> -{
> -	arm64_exc_hwstate_t hwstate = __pmr_local_save_flags();
> -
> -	/*
> -	 * There are too many states with IRQs disabled, just keep the current
> -	 * state if interrupts are already disabled/masked.
> -	 */
> -	if (!__pmr_irqs_disabled_flags(hwstate))
> -		__pmr_local_irq_disable();
> -
> -	return hwstate;
> +	return arch_irqs_disabled_flags(arch_local_save_flags());
>  }
>  
>  static __always_inline unsigned long arch_local_irq_save(void)
>  {
> -	if (system_uses_irq_prio_masking()) {
> -		return __pmr_local_irq_save().flags;
> -	} else {
> -		return __daif_local_irq_save().flags;
> -	}
> -}
> +	unsigned long flags = arch_local_save_flags();
>  
> -static __always_inline
> -void __daif_local_irq_restore(arm64_exc_hwstate_t hwstate)
> -{
> -	barrier();
> -	write_sysreg(hwstate.daif, daif);
> -	barrier();
> -}
> +	if (system_uses_irq_prio_masking())
> +		__pmr_local_irq_disable();
> +	else
> +		__daif_local_irq_disable();
>  
> -static __always_inline
> -void __pmr_local_irq_restore(arm64_exc_hwstate_t hwstate)
> -{
> -	barrier();
> -	write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> -	pmr_sync();
> -	barrier();
> +	return flags;
>  }
>  
>  /*
> @@ -215,11 +146,14 @@ static __always_inline void arch_local_irq_restore(unsigned long flags)
>  {
>  	arm64_exc_hwstate_t hwstate = { .flags = flags };
>  
> +	barrier();
>  	if (system_uses_irq_prio_masking()) {
> -		__pmr_local_irq_restore(hwstate);
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +		pmr_sync();
>  	} else {
> -		__daif_local_irq_restore(hwstate);
> +		write_sysreg(hwstate.daif, daif);
>  	}
> +	barrier();
>  }
>  
>  #endif /* __ASM_IRQFLAGS_H */
> diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
> index f7dc5fb9427d..192eb97cd50b 100644
> --- a/arch/arm64/include/asm/ptrace.h
> +++ b/arch/arm64/include/asm/ptrace.h
> @@ -205,7 +205,7 @@ static inline void forget_syscall(struct pt_regs *regs)
>  
>  #define irqs_priority_unmasked(regs)					\
>  	(system_uses_irq_prio_masking() ?				\
> -		(regs)->pmr == GIC_PRIO_IRQON :				\
> +		(regs)->pmr >= GIC_PRIO_IRQON :				\
>  		true)
>  
>  static __always_inline bool regs_irqs_disabled(const struct pt_regs *regs)

Re: [RFC PATCH 10/36] arm64: irqflags: save and use both DAIF and PMR

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> With pseudo-NMIs enabled, both DAIF and the PMR affect interrupt masking.
> Now that we have a type which can track both of them at the same time,
> update our irqflags implementation to use it.
> 
> Save DAIF flags in all cases, as they can be manipulated directly by other
> code, and the PMR if it is in use.
> 
> When checking if IRQs are disabled, now that we always save DAIF we can
> check that the I flag is set and bypass checking the PMR if it is.
> We can also properly check if PMR masks interrupts (PMR < GIC_PRIO_IRQON),
> now that we don't need to rely on the GIC_PRIO_PSR_I_SET bit being set in
> the PMR to know if DAIF is already masking interrupts.
> Update `irqs_priority_unmasked()` to align with this change.
> 
> This allows us to remove the `__daif_...` and `__pmr_...` versions
> of the save and check functions, as they are now unified.
> 
> We can reasonably merge the two `__{daif,pmr}_irq_restore()` functions
> in the main one, as the DAIF and PMR values are properly split now.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/irqflags.h | 110 ++++++------------------------
>  arch/arm64/include/asm/ptrace.h   |   2 +-
>  2 files changed, 23 insertions(+), 89 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
> index 7775904ba6a9..62f047702493 100644
> --- a/arch/arm64/include/asm/irqflags.h
> +++ b/arch/arm64/include/asm/irqflags.h
> @@ -95,117 +95,48 @@ static __always_inline void arch_local_irq_disable(void)
>  	}
>  }
>  
> -static __always_inline arm64_exc_hwstate_t __daif_local_save_flags(void)
> -{
> -	return (arm64_exc_hwstate_t){ .daif = read_sysreg(daif) };
> -}
> -
> -static __always_inline arm64_exc_hwstate_t __pmr_local_save_flags(void)
> -{
> -	return (arm64_exc_hwstate_t){ .pmr = read_sysreg_s(SYS_ICC_PMR_EL1) };
> -}
> -
>  /*
>   * Save the current interrupt enable state.
>   */
>  static __always_inline unsigned long arch_local_save_flags(void)
>  {
> -	if (system_uses_irq_prio_masking()) {
> -		return __pmr_local_save_flags().flags;
> -	} else {
> -		return __daif_local_save_flags().flags;
> -	}
> -}
> +	arm64_exc_hwstate_t hwstate = { .daif = read_sysreg(daif) };
>  
> -static __always_inline
> -bool __daif_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
> -{
> -	return hwstate.daif & PSR_I_BIT;
> -}
> +	if (system_uses_irq_prio_masking())
> +		hwstate.pmr = read_sysreg_s(SYS_ICC_PMR_EL1);
>  
> -static __always_inline
> -bool __pmr_irqs_disabled_flags(arm64_exc_hwstate_t hwstate)
> -{
> -	return hwstate.pmr != GIC_PRIO_IRQON;
> +	return hwstate.flags;
>  }
>  
>  static __always_inline bool arch_irqs_disabled_flags(unsigned long flags)
>  {
>  	arm64_exc_hwstate_t hwstate = { .flags = flags };
>  
> -	if (system_uses_irq_prio_masking()) {
> -		return __pmr_irqs_disabled_flags(hwstate);
> -	} else {
> -		return __daif_irqs_disabled_flags(hwstate);
> -	}
> -}
> +	/* If I is set, the PMR doesn't matter: interrupts will not be taken. */
> +	if (hwstate.daif & PSR_I_BIT)
> +		return true;
>  
> -static __always_inline bool __daif_irqs_disabled(void)
> -{
> -	return __daif_irqs_disabled_flags(__daif_local_save_flags());
> -}
> +	if (system_uses_irq_prio_masking() && hwstate.pmr < GIC_PRIO_IRQON)

I think "hwstate.pmr == GIC_PRIO_IRQOFF" is fine, Besides DAIF.I bit set
and pmr is GIC_PRIO_IRQOFF, are there any other possible cases?

> +		return true;
>  
> -static __always_inline bool __pmr_irqs_disabled(void)
> -{
> -	return __pmr_irqs_disabled_flags(__pmr_local_save_flags());
> +	return false;
>  }
>  
>  static __always_inline bool arch_irqs_disabled(void)
>  {
> -	if (system_uses_irq_prio_masking()) {
> -		return __pmr_irqs_disabled();
> -	} else {
> -		return __daif_irqs_disabled();
> -	}
> -}
> -
> -static __always_inline arm64_exc_hwstate_t __daif_local_irq_save(void)
> -{
> -	arm64_exc_hwstate_t hwstate = __daif_local_save_flags();
> -
> -	__daif_local_irq_disable();
> -
> -	return hwstate;
> -}
> -
> -static __always_inline arm64_exc_hwstate_t __pmr_local_irq_save(void)
> -{
> -	arm64_exc_hwstate_t hwstate = __pmr_local_save_flags();
> -
> -	/*
> -	 * There are too many states with IRQs disabled, just keep the current
> -	 * state if interrupts are already disabled/masked.
> -	 */
> -	if (!__pmr_irqs_disabled_flags(hwstate))
> -		__pmr_local_irq_disable();
> -
> -	return hwstate;
> +	return arch_irqs_disabled_flags(arch_local_save_flags());
>  }
>  
>  static __always_inline unsigned long arch_local_irq_save(void)
>  {
> -	if (system_uses_irq_prio_masking()) {
> -		return __pmr_local_irq_save().flags;
> -	} else {
> -		return __daif_local_irq_save().flags;
> -	}
> -}
> +	unsigned long flags = arch_local_save_flags();
>  
> -static __always_inline
> -void __daif_local_irq_restore(arm64_exc_hwstate_t hwstate)
> -{
> -	barrier();
> -	write_sysreg(hwstate.daif, daif);
> -	barrier();
> -}
> +	if (system_uses_irq_prio_masking())
> +		__pmr_local_irq_disable();
> +	else
> +		__daif_local_irq_disable();
>  
> -static __always_inline
> -void __pmr_local_irq_restore(arm64_exc_hwstate_t hwstate)
> -{
> -	barrier();
> -	write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> -	pmr_sync();
> -	barrier();
> +	return flags;
>  }
>  
>  /*
> @@ -215,11 +146,14 @@ static __always_inline void arch_local_irq_restore(unsigned long flags)
>  {
>  	arm64_exc_hwstate_t hwstate = { .flags = flags };
>  
> +	barrier();
>  	if (system_uses_irq_prio_masking()) {
> -		__pmr_local_irq_restore(hwstate);
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +		pmr_sync();
>  	} else {
> -		__daif_local_irq_restore(hwstate);
> +		write_sysreg(hwstate.daif, daif);
>  	}
> +	barrier();
>  }
>  
>  #endif /* __ASM_IRQFLAGS_H */
> diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
> index f7dc5fb9427d..192eb97cd50b 100644
> --- a/arch/arm64/include/asm/ptrace.h
> +++ b/arch/arm64/include/asm/ptrace.h
> @@ -205,7 +205,7 @@ static inline void forget_syscall(struct pt_regs *regs)
>  
>  #define irqs_priority_unmasked(regs)					\
>  	(system_uses_irq_prio_masking() ?				\
> -		(regs)->pmr == GIC_PRIO_IRQON :				\
> +		(regs)->pmr >= GIC_PRIO_IRQON :				\

I think "=== GIC_PRIO_IRQON" is fine.

otherwise, LGTM
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

>  		true)
>  
>  static __always_inline bool regs_irqs_disabled(const struct pt_regs *regs)

Re: [RFC PATCH 11/36] arm64: interrupts: introduce interrupt masking helpers for entry code

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> The entry code handles interrupt masking differently from the rest of
> the kernel. Exception handlers enter and exit with all exceptions
> masked, but they must temporarily unmask the appropriate set of
> exceptions so that the rest of the handler executes with the expected
> exception state.
> 
> For EL0 handlers, this means dropping to masking context appropriate
> for the work to be performed. For EL1 handlers, this means restoring
> the masking context of the interrupted task. In both cases, all
> exceptions must be masked again before returning from the exception
> handler.
> 
> The rest of the kernel typically follows the opposite pattern: it
> raises the masking context to protect a critical section and later
> restores the previous context.
> 
> Given these different usage patterns, introduce a dedicated set of
> exception masking helpers for the entry code. Keeping these helpers
> separate from the generic interrupt masking APIs makes the intended
> usage explicit and helps avoid mixing the two masking models.
> 
> To make the masking logic easier to reason about, introduce exception
> contexts that map directly to the corresponding hardware exception
> state. Along with these contexts, provide helpers to:
> 
> - translate an exception context into the corresponding hardware
>   state,
> 
> - verify that the current hardware exception state matches the
>   expected exception context,
> 
> - raise or lower the current exception context, and
> 
> - perform the common mask/unmask operations when the starting or
>   target exception context is already known.
> 
> Tracking the current exception context also provides two additional
> benefits:
> 
> - improved debugging by verifying that the hardware exception state
>   matches the expected exception context, and
> 
> - avoiding unnecessary writes to the hardware exception state.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  .../include/asm/interrupts/common_flags.h     | 197 ++++++++++++++++++
>  arch/arm64/include/asm/interrupts/entry.h     | 110 ++++++++++
>  2 files changed, 307 insertions(+)
>  create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
>  create mode 100644 arch/arm64/include/asm/interrupts/entry.h
> 
> diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
> new file mode 100644
> index 000000000000..6ce60d1519e8
> --- /dev/null
> +++ b/arch/arm64/include/asm/interrupts/common_flags.h
> @@ -0,0 +1,197 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2025 Arm Ltd.
> + */
> +#ifndef __ASM_INTERRUPTS_COMMON_FLAGS_H
> +#define __ASM_INTERRUPTS_COMMON_FLAGS_H
> +
> +#include <asm/arch_gicv3.h>
> +#include <asm/bug.h>
> +#include <asm/cpufeature.h>
> +#include <asm/ptrace.h>
> +#include <asm/sysreg.h>
> +#include <asm/irqflags.h>
> +
> +#define DAIF_PROCCTX		0
> +#define DAIF_PROCCTX_NOIRQ	(PSR_I_BIT | PSR_F_BIT)
> +#define DAIF_ERRCTX		(PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
> +#define DAIF_MASK		(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
> +
> +/*
> + * Exception context mapping
> + *
> + * pseudo-NMI
> + *
> + * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
> + * ERROR    ->  AIF + IRQON
> + * NONMI    ->   IF + IRQON
> + * NOIRQ    ->    0 + IRQOFF
> + * PROCESS  ->    0 + IRQON
> + *
> + * Otherwise
> + *
> + * CRITICAL -> DAIF           (corresponds to the state on exception entry)
> + * ERROR    ->  AIF
> + * NONMI    ->   IF
> + * NOIRQ    ->   IF
> + * PROCESS  ->    0

without the allint of FEAT_NMI, the NONMI here is meaningless.

With allint, maybe:

+ * CRITICAL -> ALLINT + DAIF           (corresponds to the state on
exception entry)
+ * ERROR    ->  ALLINT + AIF
+ * NONMI    ->  ALLINT + IF
+ * NOIRQ    ->   IF
+ * PROCESS  ->    0


> + */
> +typedef enum arm64_exc_context {
> +	PROCESS_CONTEXT,
> +	NOIRQ_CONTEXT,
> +	NONMI_CONTEXT,
> +	ERROR_CONTEXT,
> +	CRITICAL_CONTEXT,
> +} arm64_exc_context_t;
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_process_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_exc_hwstate_of_context(arm64_exc_context_t context) {
> +	switch (context) {
> +	case PROCESS_CONTEXT:
> +		return __arm64_exc_hwstate_of_process_context();
> +	case NOIRQ_CONTEXT:
> +		return __arm64_exc_hwstate_of_noirq_context();
> +	case NONMI_CONTEXT:
> +		return __arm64_exc_hwstate_of_nonmi_context();
> +	case ERROR_CONTEXT:
> +		return __arm64_exc_hwstate_of_error_context();
> +	case CRITICAL_CONTEXT:
> +		return __arm64_exc_hwstate_of_critical_context();
> +	default:
> +		BUG();
> +	}
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
> +{
> +	arm64_exc_hwstate_t state = {.daif=regs->pstate & DAIF_MASK};
> +
> +	if (system_uses_irq_prio_masking())
> +		state.pmr = regs->pmr;
> +
> +	return state;
> +}
> +
> +static __always_inline
> +void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
> +{
> +	arm64_exc_hwstate_t actual;
> +
> +	if (!IS_ENABLED(CONFIG_DEBUG_IRQFLAGS))
> +		return;
> +
> +	actual.flags = arch_local_save_flags();
> +
> +	if (expected.flags == actual.flags)
> +		return;
> +
> +	if (system_uses_irq_prio_masking()) {
> +		WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
> +			  actual.daif, actual.pmr, expected.daif, expected.pmr);
> +	} else {
> +		WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
> +			  actual.daif, expected.daif);
> +	}
> +}
> +
> +static __always_inline
> +void arm64_debug_exc_context(arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t expected = arm64_exc_hwstate_of_context(context);
> +
> +	arm64_debug_exc_hwstate(expected);
> +}
> +
> +static __always_inline
> +void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
> +{
> +	if (system_uses_irq_prio_masking() &&
> +	    update_pmr &&
> +	    hwstate.pmr == GIC_PRIO_IRQOFF) {
> +		/*
> +		 * There has been concern that the write to daif
> +		 * might be reordered before this write to PMR.
> +		 * From the ARM ARM DDI 0487D.a, section D1.7.1
> +		 * "Accessing PSTATE fields":
> +		 *   Writes to the PSTATE fields have side-effects on
> +		 *   various aspects of the PE operation. All of these
> +		 *   side-effects are guaranteed:
> +		 *     - Not to be visible to earlier instructions in
> +		 *       the execution stream.
> +		 *     - To be visible to later instructions in the
> +		 *       execution stream
> +		 *
> +		 * Also, writes to PMR are self-synchronizing, so no
> +		 * interrupts with a lower priority than PMR is signaled
> +		 * to the PE after the write.
> +		 *
> +		 * So we don't need additional synchronization here.
> +		 */
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +		pmr_sync();
> +	}
> +
> +	write_sysreg(hwstate.daif, daif);
> +
> +	if (system_uses_irq_prio_masking() &&
> +	    update_pmr &&
> +	    hwstate.pmr == GIC_PRIO_IRQON) {
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +		pmr_sync();
> +	}
> +}
> +
> +static __always_inline
> +void arm64_update_exc_context(arm64_exc_context_t context, bool update_pmr)
> +{
> +	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(context);
> +
> +	arm64_update_exc_hwstate(hwstate, update_pmr);
> +}
> +#endif /* __ASM_INTERRUPTS_COMMON_FLAGS_H */
> diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
> new file mode 100644
> index 000000000000..3034c490ed66
> --- /dev/null
> +++ b/arch/arm64/include/asm/interrupts/entry.h
> @@ -0,0 +1,110 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2025 Arm Ltd.
> + */
> +#ifndef __ASM_INTERRUPTS_ENTRY_H
> +#define __ASM_INTERRUPTS_ENTRY_H
> +
> +#include <asm/arch_gicv3.h>
> +#include <asm/bug.h>
> +#include <asm/cpufeature.h>
> +#include <asm/interrupts/common_flags.h>
> +
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
> +						  arm64_exc_hwstate_t next)
> +{
> +	bool update_pmr = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
> +
> +	arm64_debug_exc_hwstate(prev);
> +
> +	if (prev.flags == next.flags)
> +		return next;
> +
> +	if (!arch_irqs_disabled_flags(next.flags))
> +		trace_hardirqs_on();
> +
> +	arm64_update_exc_hwstate(next, update_pmr);
> +
> +	if (arch_irqs_disabled_flags(next.flags))
> +		trace_hardirqs_off();
> +
> +	return next;
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_inherit_exc_context(struct pt_regs *regs)
> +{
> +	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
> +	arm64_exc_hwstate_t next = arm64_inherit_exc_hwstate(regs);
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
> +
> +	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
> +		bool pnmi = system_uses_irq_prio_masking();
> +
> +		WARN_ON_ONCE(context > ERROR_CONTEXT &&
> +			     prev.daif == DAIF_ERRCTX);
> +
> +		WARN_ON_ONCE(context > NONMI_CONTEXT &&
> +			     prev.daif == DAIF_PROCCTX_NOIRQ);
> +
> +		WARN_ON_ONCE(context > NOIRQ_CONTEXT &&
> +			     pnmi && prev.pmr == GIC_PRIO_IRQOFF);
> +
> +		WARN_ON_ONCE(context > PROCESS_CONTEXT &&
> +			     ((pnmi && prev.daif == DAIF_PROCCTX && prev.pmr == GIC_PRIO_IRQON) ||
> +			      (!pnmi && prev.daif == DAIF_PROCCTX)));
> +	}
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
> +
> +	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
> +		bool pnmi = system_uses_irq_prio_masking();
> +
> +		WARN_ON_ONCE(context < CRITICAL_CONTEXT &&
> +			     prev.daif == DAIF_MASK);
> +
> +		WARN_ON_ONCE(context < ERROR_CONTEXT &&
> +			     prev.daif == DAIF_ERRCTX);
> +
> +		WARN_ON_ONCE(context < NONMI_CONTEXT &&
> +			     pnmi && prev.daif == DAIF_PROCCTX_NOIRQ);
> +
> +		WARN_ON_ONCE(context < NOIRQ_CONTEXT &&
> +			     ((pnmi && prev.pmr == GIC_PRIO_IRQOFF) ||
> +			      (!pnmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
> +	}
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_unmask_exc_context(arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
> +
> +	return arm64_drop_exc_context(prev, context);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_mask_exc_context(arm64_exc_hwstate_t prev)
> +{
> +	return arm64_lift_exc_context(prev, CRITICAL_CONTEXT);
> +}
> +
> +#endif /* __ASM_INTERRUPTS_ENTRY_H */

Re: [RFC PATCH 12/36] arm64: entry: replace DAIF helpers with entry helpers

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Replace all uses of the `local_daif_...` helpers in `entry-common.c`
> with the new entry-specific exception masking helpers.
> 
> Also replace `local_irq_disable()` with a switch to NOIRQ context
> using the new helpers. This provides:
> 
> - consistency checks for hardware state
> 
> - a unified style for managing exception context
> 
> Now that exception context is tracked explicitly, involuntary kernel
> preemption can be optimized. We either preempt:
> 
> - from PROCESS exception context, or
> 
> - when returning from an interrupt handler, where the GIC code
>   switches from NONMI to NOIRQ exception context
> 
> To support this split, divide `arm64_exit_to_kernel_mode()` into
> preempt, non-preempt, and dispatch parts.
> 
> Remove `local_daif_inherit()`, since it is no longer used.
> 
> Now that both the irqflags API and entry code handle DAIF and PMR
> correctly, remove `GIC_PRIO_PSR_I_SET` from the PMR value set by
> `kernel_entry` in `entry.S` and by `init_gic_priority_masking()` in
> `kernel/smp.c`.
> 
> The `local_daif_...` helpers and other specialized code still use
> `GIC_PRIO_PSR_I_SET` for now; this commit does not change their
> behavior.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/daifflags.h |  22 ---
>  arch/arm64/kernel/entry-common.c   | 272 ++++++++++++++++++++---------
>  arch/arm64/kernel/entry.S          |   2 +-
>  arch/arm64/kernel/smp.c            |   2 +-
>  4 files changed, 190 insertions(+), 108 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
> index 56341578e7e3..6b14ec4d4dbe 100644
> --- a/arch/arm64/include/asm/daifflags.h
> +++ b/arch/arm64/include/asm/daifflags.h
> @@ -120,28 +120,6 @@ static __always_inline void local_daif_restore(unsigned long flags)
>  		trace_hardirqs_off();
>  }
>  
> -/*
> - * Called by synchronous exception handlers to restore the DAIF bits that were
> - * modified by taking an exception.
> - */
> -static __always_inline void local_daif_inherit(struct pt_regs *regs)
> -{
> -	unsigned long flags = regs->pstate & DAIF_MASK;
> -
> -	if (!regs_irqs_disabled(regs))
> -		trace_hardirqs_on();
> -
> -	if (system_uses_irq_prio_masking())
> -		gic_write_pmr(regs->pmr);
> -
> -	/*
> -	 * We can't use local_daif_restore(regs->pstate) here as
> -	 * system_has_prio_mask_debugging() won't restore the I bit if it can
> -	 * use the pmr instead.
> -	 */
> -	write_sysreg(flags, daif);
> -}
> -
>  /*
>   * During early boot, we unmask PSR.DA before the GIC has been set up.
>   * If we use IRQ priority masking, the PMR and PSR will be out of sync
> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> index 2be42d7f4eaa..a13653b228b7 100644
> --- a/arch/arm64/kernel/entry-common.c
> +++ b/arch/arm64/kernel/entry-common.c
> @@ -18,11 +18,11 @@
>  #include <linux/thread_info.h>
>  
>  #include <asm/cpufeature.h>
> -#include <asm/daifflags.h>
>  #include <asm/esr.h>
>  #include <asm/exception.h>
>  #include <asm/fpsimd.h>
>  #include <asm/irq_regs.h>
> +#include <asm/interrupts/entry.h>
>  #include <asm/kprobes.h>
>  #include <asm/mmu.h>
>  #include <asm/processor.h>
> @@ -52,12 +52,11 @@ static noinstr irqentry_state_t arm64_enter_from_kernel_mode(struct pt_regs *reg
>   * After this function returns it is not safe to call regular kernel code,
>   * instrumentable code, or any code which may trigger an exception.
>   */
> -static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
> -					      irqentry_state_t state)
> +static void noinstr __arm64_exit_to_kernel_mode(struct pt_regs *regs,
> +						irqentry_state_t state,
> +						arm64_exc_hwstate_t hwstate)
>  {
> -	local_irq_disable();
> -	irqentry_exit_to_kernel_mode_preempt(regs, state);
> -	local_daif_mask();
> +	arm64_mask_exc_context(hwstate);
>  	mte_check_tfsr_exit();
>  	irqentry_exit_to_kernel_mode_after_preempt(regs, state);
>  }
> @@ -69,6 +68,30 @@ static __always_inline void arm64_syscall_enter_from_user_mode(struct pt_regs *r
>  	sme_enter_from_user_mode();
>  }
>  
> +/*
> + * We are returning from the context which allows involuntary kernel preemption
> + */
> +static void noinstr arm64_exit_to_kernel_mode_preempt(struct pt_regs *regs,
> +						      irqentry_state_t state,
> +						      arm64_exc_hwstate_t hwstate)
> +{
> +	irqentry_exit_to_kernel_mode_preempt(regs, state);
> +	__arm64_exit_to_kernel_mode(regs, state, hwstate);
> +}
> +
> +static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
> +					      irqentry_state_t state,
> +					      arm64_exc_hwstate_t hwstate)
> +{
> +	if (!arch_irqs_disabled_flags(hwstate.flags)) {
> +		hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
> +		arm64_exit_to_kernel_mode_preempt(regs, state, hwstate);
> +		return;
> +	}
> +
> +	__arm64_exit_to_kernel_mode(regs, state, hwstate);
> +}
> +
>  /*
>   * Handle IRQ/context state management when entering from user mode.
>   * Before this function is called it is not safe to call regular kernel code,
> @@ -82,11 +105,12 @@ static __always_inline void arm64_enter_from_user_mode(struct pt_regs *regs)
>  	sme_enter_from_user_mode();
>  }
>  
> -static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs)
> +static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs,
> +							    arm64_exc_hwstate_t hwstate)
>  {
> -	local_irq_disable();
> +	hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
>  	syscall_exit_to_user_mode_prepare(regs);
> -	local_daif_mask();
> +	arm64_mask_exc_context(hwstate);
>  	sme_exit_to_user_mode();
>  	mte_check_tfsr_exit();
>  	exit_to_user_mode();
> @@ -97,11 +121,12 @@ static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs
>   * After this function returns it is not safe to call regular kernel code,
>   * instrumentable code, or any code which may trigger an exception.
>   */
> -static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
> +static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs,
> +						    arm64_exc_hwstate_t hwstate)
>  {
> -	local_irq_disable();
> +	hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
>  	irqentry_exit_to_user_mode_prepare(regs);
> -	local_daif_mask();
> +	arm64_mask_exc_context(hwstate);
>  	sme_exit_to_user_mode();
>  	mte_check_tfsr_exit();
>  	exit_to_user_mode();
> @@ -109,7 +134,10 @@ static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
>  
>  asmlinkage void noinstr asm_exit_to_user_mode(struct pt_regs *regs)
>  {
> -	arm64_syscall_exit_to_user_mode(regs);
> +	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(PROCESS_CONTEXT);
> +
> +	arm64_syscall_exit_to_user_mode(regs, hwstate);
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  /*
> @@ -315,63 +343,69 @@ UNHANDLED(el1t, 64, error)
>  static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
>  {
>  	unsigned long far = read_sysreg(far_el1);
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);
>  	do_mem_abort(far, esr, regs);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  static void noinstr el1_pc(struct pt_regs *regs, unsigned long esr)
>  {
>  	unsigned long far = read_sysreg(far_el1);
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);
>  	do_sp_pc_abort(far, esr, regs);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  static void noinstr el1_undef(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);

local_daif_inherit do not trace_hardirqs_off() but only
trace_hardirqs_off()，but now arm64_inherit_exc_context() do it, is this
a functional change?

>  	do_el1_undef(regs, esr);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  static void noinstr el1_bti(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);
>  	do_el1_bti(regs, esr);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  static void noinstr el1_gcs(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);
>  	do_el1_gcs(regs, esr);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  static void noinstr el1_mops(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);
>  	do_el1_mops(regs, esr);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  static void noinstr el1_breakpt(struct pt_regs *regs, unsigned long esr)
> @@ -431,12 +465,13 @@ static void noinstr el1_brk64(struct pt_regs *regs, unsigned long esr)
>  
>  static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);
>  	do_el1_fpac(regs, esr);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
> @@ -486,16 +521,20 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
>  	default:
>  		__panic_unhandled(regs, "64-bit el1h sync", esr);
>  	}
> +
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  static __always_inline void __el1_pnmi(struct pt_regs *regs,
>  				       void (*handler)(struct pt_regs *))
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = irqentry_nmi_enter(regs);
> +	hwstate = arm64_unmask_exc_context(NONMI_CONTEXT);
>  	do_interrupt_handler(regs, handler);
> -	local_daif_mask();
> +	arm64_mask_exc_context(hwstate);
>  	irqentry_nmi_exit(regs, state);
>  }
>  
> @@ -506,21 +545,32 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
>  
>  	state = arm64_enter_from_kernel_mode(regs);
>  
> +	arm64_unmask_exc_context(NONMI_CONTEXT);
> +
>  	irq_enter_rcu();
>  	do_interrupt_handler(regs, handler);
>  	irq_exit_rcu();
>  
> -	arm64_exit_to_kernel_mode(regs, state);
> +	/*
> +	 * If pseudo-NMIs are enabled and the interrupted context had
> +	 * IRQs unmasked, the interrupt handler will have cleared DAIF
> +	 * and switched to PMR masking in order to handle
> +	 * NMIs. Otherwise it would keep IF.  In both cases on return
> +	 * we effectivly have NOIRQ_CONTEXT - keep track of it
> +	 */
> +	arm64_debug_exc_context(NOIRQ_CONTEXT);
> +	arm64_exit_to_kernel_mode_preempt(regs, state, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
>  }
> +
>  static void noinstr el1_interrupt(struct pt_regs *regs,
>  				  void (*handler)(struct pt_regs *))
>  {
> -	write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
> -
>  	if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && regs_irqs_disabled(regs))
>  		__el1_pnmi(regs, handler);
>  	else
>  		__el1_irq(regs, handler);
> +
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
> @@ -536,28 +586,31 @@ asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
>  asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
>  {
>  	unsigned long esr = read_sysreg(esr_el1);
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
> -	local_daif_restore(DAIF_ERRCTX);
> +	hwstate = arm64_unmask_exc_context(ERROR_CONTEXT);
>  	state = irqentry_nmi_enter(regs);
>  	do_serror(regs, esr);
> -	local_daif_mask();
> +	arm64_mask_exc_context(hwstate);
>  	irqentry_nmi_exit(regs, state);
>  }
>  
>  static void noinstr el0_da(struct pt_regs *regs, unsigned long esr)
>  {
>  	unsigned long far = read_sysreg(far_el1);
> +	arm64_exc_hwstate_t hwstate;
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_mem_abort(far, esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
>  {
>  	unsigned long far = read_sysreg(far_el1);
> +	arm64_exc_hwstate_t hwstate;
>  
>  	/*
>  	 * We've taken an instruction abort from userspace and not yet
> @@ -568,114 +621,139 @@ static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
>  		arm64_apply_bp_hardening();
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_mem_abort(far, esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_fpsimd_acc(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_fpsimd_acc(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_sve_acc(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_sve_acc(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_sme_acc(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_sme_acc(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_fpsimd_exc(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_fpsimd_exc(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_sys(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_sys(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_pc(struct pt_regs *regs, unsigned long esr)
>  {
>  	unsigned long far = read_sysreg(far_el1);
> +	arm64_exc_hwstate_t hwstate;
>  
>  	if (!is_ttbr0_addr(instruction_pointer(regs)))
>  		arm64_apply_bp_hardening();
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_sp_pc_abort(far, esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_sp(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_sp_pc_abort(regs->sp, esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_undef(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_undef(regs, esr);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_bti(struct pt_regs *regs)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_bti(regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_mops(regs, esr);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_gcs(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_gcs(regs, esr);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_inv(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	bad_el0_sync(regs, 0, esr);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	if (!is_ttbr0_addr(regs->pc))
>  		arm64_apply_bp_hardening();
>  
> @@ -683,12 +761,13 @@ static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
>  	debug_exception_enter(regs);
>  	do_breakpoint(esr, regs);
>  	debug_exception_exit(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> -	arm64_exit_to_user_mode(regs);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	bool step_done;
>  
>  	if (!is_ttbr0_addr(regs->pc))
> @@ -702,50 +781,56 @@ static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
>  	 * the single-step is complete.
>  	 */
>  	step_done = try_step_suspended_breakpoints(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	if (!step_done)
>  		do_el0_softstep(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_watchpt(struct pt_regs *regs, unsigned long esr)
>  {
>  	/* Watchpoints are the only debug exception to write FAR_EL1 */
>  	unsigned long far = read_sysreg(far_el1);
> +	arm64_exc_hwstate_t hwstate;
>  
>  	arm64_enter_from_user_mode(regs);
>  	debug_exception_enter(regs);
>  	do_watchpoint(far, esr, regs);
>  	debug_exception_exit(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> -	arm64_exit_to_user_mode(regs);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_brk64(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_brk64(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_svc(struct pt_regs *regs)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	arm64_syscall_enter_from_user_mode(regs);
>  	cortex_a76_erratum_1463225_svc_handler();
>  	fpsimd_syscall_enter();
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_svc(regs);
> -	arm64_syscall_exit_to_user_mode(regs);
> +	arm64_syscall_exit_to_user_mode(regs, hwstate);
>  	fpsimd_syscall_exit();
>  }
>  
>  static void noinstr el0_fpac(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_fpac(regs, esr);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
> @@ -814,6 +899,8 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
>  	default:
>  		el0_inv(regs, esr);
>  	}
> +
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  static void noinstr el0_interrupt(struct pt_regs *regs,
> @@ -821,7 +908,7 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
>  {
>  	arm64_enter_from_user_mode(regs);
>  
> -	write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
> +	arm64_unmask_exc_context(NONMI_CONTEXT);
>  
>  	if (regs->pc & BIT(55))
>  		arm64_apply_bp_hardening();
> @@ -830,7 +917,14 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
>  	do_interrupt_handler(regs, handler);
>  	irq_exit_rcu();
>  
> -	arm64_exit_to_user_mode(regs);
> +	/*
> +	 * For the same reason as in el1_irq() we effectivly
> +	 * have NOIRQ_CONTEXT on return from handler - keep
> +	 * track of it
> +	 */
> +	arm64_debug_exc_context(NOIRQ_CONTEXT);
> +	arm64_exit_to_user_mode(regs, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  static void noinstr __el0_irq_handler_common(struct pt_regs *regs)
> @@ -856,15 +950,17 @@ asmlinkage void noinstr el0t_64_fiq_handler(struct pt_regs *regs)
>  static void noinstr __el0_error_handler_common(struct pt_regs *regs)
>  {
>  	unsigned long esr = read_sysreg(esr_el1);
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_ERRCTX);
> +	hwstate = arm64_unmask_exc_context(ERROR_CONTEXT);
>  	state = irqentry_nmi_enter(regs);
>  	do_serror(regs, esr);
>  	irqentry_nmi_exit(regs, state);
> -	local_daif_restore(DAIF_PROCCTX);
> -	arm64_exit_to_user_mode(regs);
> +	hwstate = arm64_drop_exc_context(hwstate, PROCESS_CONTEXT);
> +	arm64_exit_to_user_mode(regs, hwstate);
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
> @@ -875,27 +971,33 @@ asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
>  #ifdef CONFIG_COMPAT
>  static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_cp15(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_svc_compat(struct pt_regs *regs)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_syscall_enter_from_user_mode(regs);
>  	cortex_a76_erratum_1463225_svc_handler();
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_svc_compat(regs);
> -	arm64_syscall_exit_to_user_mode(regs);
> +	arm64_syscall_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_bkpt32(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_bkpt32(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
> @@ -946,6 +1048,8 @@ asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
>  	default:
>  		el0_inv(regs, esr);
>  	}
> +
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  asmlinkage void noinstr el0t_32_irq_handler(struct pt_regs *regs)
> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> index f63049ac32dc..cb3be770f2d0 100644
> --- a/arch/arm64/kernel/entry.S
> +++ b/arch/arm64/kernel/entry.S
> @@ -316,7 +316,7 @@ alternative_else_nop_endif
>  
>  	mrs_s	x20, SYS_ICC_PMR_EL1
>  	str	w20, [sp, #S_PMR]
> -	mov	x20, #GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET
> +	mov	x20, #GIC_PRIO_IRQON
>  	msr_s	SYS_ICC_PMR_EL1, x20
>  
>  .Lskip_pmr_save\@:
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index d46022f72075..b91cf163aac7 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -185,7 +185,7 @@ static void init_gic_priority_masking(void)
>  	WARN_ON(!(cpuflags & PSR_I_BIT));
>  	WARN_ON(!(cpuflags & PSR_F_BIT));
>  
> -	gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
> +	gic_write_pmr(GIC_PRIO_IRQON);
>  }
>  
>  /*

Re: [RFC PATCH 11/36] arm64: interrupts: introduce interrupt masking helpers for entry code

[Liao, Chang]

在 2026/7/9 20:13, Vladimir Murzin 写道:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> The entry code handles interrupt masking differently from the rest of
> the kernel. Exception handlers enter and exit with all exceptions
> masked, but they must temporarily unmask the appropriate set of
> exceptions so that the rest of the handler executes with the expected
> exception state.
> 
> For EL0 handlers, this means dropping to masking context appropriate
> for the work to be performed. For EL1 handlers, this means restoring
> the masking context of the interrupted task. In both cases, all
> exceptions must be masked again before returning from the exception
> handler.
> 
> The rest of the kernel typically follows the opposite pattern: it
> raises the masking context to protect a critical section and later
> restores the previous context.
> 
> Given these different usage patterns, introduce a dedicated set of
> exception masking helpers for the entry code. Keeping these helpers
> separate from the generic interrupt masking APIs makes the intended
> usage explicit and helps avoid mixing the two masking models.
> 
> To make the masking logic easier to reason about, introduce exception
> contexts that map directly to the corresponding hardware exception
> state. Along with these contexts, provide helpers to:
> 
> - translate an exception context into the corresponding hardware
>   state,
> 
> - verify that the current hardware exception state matches the
>   expected exception context,
> 
> - raise or lower the current exception context, and
> 
> - perform the common mask/unmask operations when the starting or
>   target exception context is already known.
> 
> Tracking the current exception context also provides two additional
> benefits:
> 
> - improved debugging by verifying that the hardware exception state
>   matches the expected exception context, and
> 
> - avoiding unnecessary writes to the hardware exception state.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  .../include/asm/interrupts/common_flags.h     | 197 ++++++++++++++++++
>  arch/arm64/include/asm/interrupts/entry.h     | 110 ++++++++++
>  2 files changed, 307 insertions(+)
>  create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
>  create mode 100644 arch/arm64/include/asm/interrupts/entry.h
> 
> diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
> new file mode 100644
> index 000000000000..6ce60d1519e8
> --- /dev/null
> +++ b/arch/arm64/include/asm/interrupts/common_flags.h
> @@ -0,0 +1,197 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2025 Arm Ltd.
> + */
> +#ifndef __ASM_INTERRUPTS_COMMON_FLAGS_H
> +#define __ASM_INTERRUPTS_COMMON_FLAGS_H
> +
> +#include <asm/arch_gicv3.h>
> +#include <asm/bug.h>
> +#include <asm/cpufeature.h>
> +#include <asm/ptrace.h>
> +#include <asm/sysreg.h>
> +#include <asm/irqflags.h>
> +
> +#define DAIF_PROCCTX		0
> +#define DAIF_PROCCTX_NOIRQ	(PSR_I_BIT | PSR_F_BIT)
> +#define DAIF_ERRCTX		(PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
> +#define DAIF_MASK		(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
> +
> +/*
> + * Exception context mapping
> + *
> + * pseudo-NMI
> + *
> + * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
> + * ERROR    ->  AIF + IRQON
> + * NONMI    ->   IF + IRQON
> + * NOIRQ    ->    0 + IRQOFF
> + * PROCESS  ->    0 + IRQON
> + *
> + * Otherwise
> + *
> + * CRITICAL -> DAIF           (corresponds to the state on exception entry)
> + * ERROR    ->  AIF
> + * NONMI    ->   IF
> + * NOIRQ    ->   IF
> + * PROCESS  ->    0
> + */
> +typedef enum arm64_exc_context {
> +	PROCESS_CONTEXT,
> +	NOIRQ_CONTEXT,
> +	NONMI_CONTEXT,
> +	ERROR_CONTEXT,
> +	CRITICAL_CONTEXT,
> +} arm64_exc_context_t;
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_process_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_exc_hwstate_of_context(arm64_exc_context_t context) {
> +	switch (context) {
> +	case PROCESS_CONTEXT:
> +		return __arm64_exc_hwstate_of_process_context();
> +	case NOIRQ_CONTEXT:
> +		return __arm64_exc_hwstate_of_noirq_context();
> +	case NONMI_CONTEXT:
> +		return __arm64_exc_hwstate_of_nonmi_context();
> +	case ERROR_CONTEXT:
> +		return __arm64_exc_hwstate_of_error_context();
> +	case CRITICAL_CONTEXT:
> +		return __arm64_exc_hwstate_of_critical_context();
> +	default:
> +		BUG();
> +	}
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
> +{
> +	arm64_exc_hwstate_t state = {.daif=regs->pstate & DAIF_MASK};
> +
> +	if (system_uses_irq_prio_masking())
> +		state.pmr = regs->pmr;
> +
> +	return state;
> +}
> +
> +static __always_inline
> +void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
> +{
> +	arm64_exc_hwstate_t actual;
> +
> +	if (!IS_ENABLED(CONFIG_DEBUG_IRQFLAGS))
> +		return;
> +
> +	actual.flags = arch_local_save_flags();
> +
> +	if (expected.flags == actual.flags)
> +		return;
> +
> +	if (system_uses_irq_prio_masking()) {
> +		WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
> +			  actual.daif, actual.pmr, expected.daif, expected.pmr);
> +	} else {
> +		WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
> +			  actual.daif, expected.daif);
> +	}
> +}
> +
> +static __always_inline
> +void arm64_debug_exc_context(arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t expected = arm64_exc_hwstate_of_context(context);
> +
> +	arm64_debug_exc_hwstate(expected);
> +}
> +
> +static __always_inline
> +void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
> +{
> +	if (system_uses_irq_prio_masking() &&
> +	    update_pmr &&
> +	    hwstate.pmr == GIC_PRIO_IRQOFF) {
> +		/*
> +		 * There has been concern that the write to daif
> +		 * might be reordered before this write to PMR.
> +		 * From the ARM ARM DDI 0487D.a, section D1.7.1
> +		 * "Accessing PSTATE fields":
> +		 *   Writes to the PSTATE fields have side-effects on
> +		 *   various aspects of the PE operation. All of these
> +		 *   side-effects are guaranteed:
> +		 *     - Not to be visible to earlier instructions in
> +		 *       the execution stream.
> +		 *     - To be visible to later instructions in the
> +		 *       execution stream
> +		 *
> +		 * Also, writes to PMR are self-synchronizing, so no
> +		 * interrupts with a lower priority than PMR is signaled
> +		 * to the PE after the write.
> +		 *
> +		 * So we don't need additional synchronization here.
> +		 */
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +		pmr_sync();

According to the comment above, pmr_sync() seems unnecessary here, can
we safely remove it?

> +	}
> +
> +	write_sysreg(hwstate.daif, daif);
> +
> +	if (system_uses_irq_prio_masking() &&
> +	    update_pmr &&
> +	    hwstate.pmr == GIC_PRIO_IRQON) {
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +		pmr_sync();
> +	}

I notice the sequence of manipulating DAIF and PMR for IRQ disabling
has been reordered from the old one, is there any subtle side effect
we should be aware of?

> +}
> +
> +static __always_inline
> +void arm64_update_exc_context(arm64_exc_context_t context, bool update_pmr)
> +{
> +	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(context);
> +
> +	arm64_update_exc_hwstate(hwstate, update_pmr);
> +}
> +#endif /* __ASM_INTERRUPTS_COMMON_FLAGS_H */
> diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
> new file mode 100644
> index 000000000000..3034c490ed66
> --- /dev/null
> +++ b/arch/arm64/include/asm/interrupts/entry.h
> @@ -0,0 +1,110 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2025 Arm Ltd.
> + */
> +#ifndef __ASM_INTERRUPTS_ENTRY_H
> +#define __ASM_INTERRUPTS_ENTRY_H
> +
> +#include <asm/arch_gicv3.h>
> +#include <asm/bug.h>
> +#include <asm/cpufeature.h>
> +#include <asm/interrupts/common_flags.h>
> +
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
> +						  arm64_exc_hwstate_t next)
> +{
> +	bool update_pmr = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
> +
> +	arm64_debug_exc_hwstate(prev);
> +
> +	if (prev.flags == next.flags)
> +		return next;
> +
> +	if (!arch_irqs_disabled_flags(next.flags))
> +		trace_hardirqs_on();
> +
> +	arm64_update_exc_hwstate(next, update_pmr);
> +
> +	if (arch_irqs_disabled_flags(next.flags))

Could we save the disabled flags in a local variable to avoid the duplicate calls?

> +		trace_hardirqs_off();
> +
> +	return next;
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_inherit_exc_context(struct pt_regs *regs)
> +{
> +	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
> +	arm64_exc_hwstate_t next = arm64_inherit_exc_hwstate(regs);
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
> +
> +	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
> +		bool pnmi = system_uses_irq_prio_masking();
> +
> +		WARN_ON_ONCE(context > ERROR_CONTEXT &&
> +			     prev.daif == DAIF_ERRCTX);
> +
> +		WARN_ON_ONCE(context > NONMI_CONTEXT &&
> +			     prev.daif == DAIF_PROCCTX_NOIRQ);
> +
> +		WARN_ON_ONCE(context > NOIRQ_CONTEXT &&
> +			     pnmi && prev.pmr == GIC_PRIO_IRQOFF);
> +
> +		WARN_ON_ONCE(context > PROCESS_CONTEXT &&
> +			     ((pnmi && prev.daif == DAIF_PROCCTX && prev.pmr == GIC_PRIO_IRQON) ||
> +			      (!pnmi && prev.daif == DAIF_PROCCTX)));
> +	}
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
> +
> +	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
> +		bool pnmi = system_uses_irq_prio_masking();
> +
> +		WARN_ON_ONCE(context < CRITICAL_CONTEXT &&
> +			     prev.daif == DAIF_MASK);
> +
> +		WARN_ON_ONCE(context < ERROR_CONTEXT &&
> +			     prev.daif == DAIF_ERRCTX);
> +
> +		WARN_ON_ONCE(context < NONMI_CONTEXT &&
> +			     pnmi && prev.daif == DAIF_PROCCTX_NOIRQ);
> +
> +		WARN_ON_ONCE(context < NOIRQ_CONTEXT &&
> +			     ((pnmi && prev.pmr == GIC_PRIO_IRQOFF) ||
> +			      (!pnmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
> +	}
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_unmask_exc_context(arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
> +
> +	return arm64_drop_exc_context(prev, context);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_mask_exc_context(arm64_exc_hwstate_t prev)
> +{
> +	return arm64_lift_exc_context(prev, CRITICAL_CONTEXT);
> +}
> +
> +#endif /* __ASM_INTERRUPTS_ENTRY_H */


-- 
BR
Liao, Chang

Re: [RFC PATCH 11/36] arm64: interrupts: introduce interrupt masking helpers for entry code

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> The entry code handles interrupt masking differently from the rest of
> the kernel. Exception handlers enter and exit with all exceptions
> masked, but they must temporarily unmask the appropriate set of
> exceptions so that the rest of the handler executes with the expected
> exception state.
> 
> For EL0 handlers, this means dropping to masking context appropriate
> for the work to be performed. For EL1 handlers, this means restoring
> the masking context of the interrupted task. In both cases, all
> exceptions must be masked again before returning from the exception
> handler.
> 
> The rest of the kernel typically follows the opposite pattern: it
> raises the masking context to protect a critical section and later
> restores the previous context.
> 
> Given these different usage patterns, introduce a dedicated set of
> exception masking helpers for the entry code. Keeping these helpers
> separate from the generic interrupt masking APIs makes the intended
> usage explicit and helps avoid mixing the two masking models.
> 
> To make the masking logic easier to reason about, introduce exception
> contexts that map directly to the corresponding hardware exception
> state. Along with these contexts, provide helpers to:
> 
> - translate an exception context into the corresponding hardware
>   state,
> 
> - verify that the current hardware exception state matches the
>   expected exception context,
> 
> - raise or lower the current exception context, and
> 
> - perform the common mask/unmask operations when the starting or
>   target exception context is already known.
> 
> Tracking the current exception context also provides two additional
> benefits:
> 
> - improved debugging by verifying that the hardware exception state
>   matches the expected exception context, and
> 
> - avoiding unnecessary writes to the hardware exception state.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  .../include/asm/interrupts/common_flags.h     | 197 ++++++++++++++++++
>  arch/arm64/include/asm/interrupts/entry.h     | 110 ++++++++++
>  2 files changed, 307 insertions(+)
>  create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
>  create mode 100644 arch/arm64/include/asm/interrupts/entry.h
> 
> diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
> new file mode 100644
> index 000000000000..6ce60d1519e8
> --- /dev/null
> +++ b/arch/arm64/include/asm/interrupts/common_flags.h
> @@ -0,0 +1,197 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2025 Arm Ltd.
> + */
> +#ifndef __ASM_INTERRUPTS_COMMON_FLAGS_H
> +#define __ASM_INTERRUPTS_COMMON_FLAGS_H
> +
> +#include <asm/arch_gicv3.h>
> +#include <asm/bug.h>
> +#include <asm/cpufeature.h>
> +#include <asm/ptrace.h>
> +#include <asm/sysreg.h>
> +#include <asm/irqflags.h>
> +
> +#define DAIF_PROCCTX		0
> +#define DAIF_PROCCTX_NOIRQ	(PSR_I_BIT | PSR_F_BIT)
> +#define DAIF_ERRCTX		(PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
> +#define DAIF_MASK		(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
> +
> +/*
> + * Exception context mapping
> + *
> + * pseudo-NMI
> + *
> + * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
> + * ERROR    ->  AIF + IRQON
> + * NONMI    ->   IF + IRQON
> + * NOIRQ    ->    0 + IRQOFF
> + * PROCESS  ->    0 + IRQON
> + *
> + * Otherwise
> + *
> + * CRITICAL -> DAIF           (corresponds to the state on exception entry)
> + * ERROR    ->  AIF
> + * NONMI    ->   IF
> + * NOIRQ    ->   IF
> + * PROCESS  ->    0
> + */
> +typedef enum arm64_exc_context {
> +	PROCESS_CONTEXT,
> +	NOIRQ_CONTEXT,
> +	NONMI_CONTEXT,
> +	ERROR_CONTEXT,
> +	CRITICAL_CONTEXT,
> +} arm64_exc_context_t;
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_process_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_exc_hwstate_of_context(arm64_exc_context_t context) {
> +	switch (context) {
> +	case PROCESS_CONTEXT:
> +		return __arm64_exc_hwstate_of_process_context();
> +	case NOIRQ_CONTEXT:
> +		return __arm64_exc_hwstate_of_noirq_context();
> +	case NONMI_CONTEXT:
> +		return __arm64_exc_hwstate_of_nonmi_context();
> +	case ERROR_CONTEXT:
> +		return __arm64_exc_hwstate_of_error_context();
> +	case CRITICAL_CONTEXT:
> +		return __arm64_exc_hwstate_of_critical_context();
> +	default:
> +		BUG();
> +	}
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
> +{
> +	arm64_exc_hwstate_t state = {.daif=regs->pstate & DAIF_MASK};
> +
> +	if (system_uses_irq_prio_masking())
> +		state.pmr = regs->pmr;
> +
> +	return state;
> +}
> +
> +static __always_inline
> +void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
> +{
> +	arm64_exc_hwstate_t actual;
> +
> +	if (!IS_ENABLED(CONFIG_DEBUG_IRQFLAGS))
> +		return;
> +
> +	actual.flags = arch_local_save_flags();
> +
> +	if (expected.flags == actual.flags)
> +		return;
> +
> +	if (system_uses_irq_prio_masking()) {
> +		WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
> +			  actual.daif, actual.pmr, expected.daif, expected.pmr);
> +	} else {
> +		WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
> +			  actual.daif, expected.daif);
> +	}
> +}
> +
> +static __always_inline
> +void arm64_debug_exc_context(arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t expected = arm64_exc_hwstate_of_context(context);
> +
> +	arm64_debug_exc_hwstate(expected);
> +}
> +
> +static __always_inline
> +void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
> +{
> +	if (system_uses_irq_prio_masking() &&
> +	    update_pmr &&
> +	    hwstate.pmr == GIC_PRIO_IRQOFF) {
> +		/*
> +		 * There has been concern that the write to daif
> +		 * might be reordered before this write to PMR.
> +		 * From the ARM ARM DDI 0487D.a, section D1.7.1
> +		 * "Accessing PSTATE fields":
> +		 *   Writes to the PSTATE fields have side-effects on
> +		 *   various aspects of the PE operation. All of these
> +		 *   side-effects are guaranteed:
> +		 *     - Not to be visible to earlier instructions in
> +		 *       the execution stream.
> +		 *     - To be visible to later instructions in the
> +		 *       execution stream
> +		 *
> +		 * Also, writes to PMR are self-synchronizing, so no
> +		 * interrupts with a lower priority than PMR is signaled
> +		 * to the PE after the write.
> +		 *
> +		 * So we don't need additional synchronization here.
> +		 */
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +		pmr_sync();
> +	}
> +
> +	write_sysreg(hwstate.daif, daif);
> +
> +	if (system_uses_irq_prio_masking() &&
> +	    update_pmr &&
> +	    hwstate.pmr == GIC_PRIO_IRQON) {
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +		pmr_sync();
> +	}
The default behavior of both local_daif_restore() and
local_daif_inherit() is to first restore the PMR register and then
restore DAIF. It appears that there has been a functional change here.

> +}
> +
> +static __always_inline
> +void arm64_update_exc_context(arm64_exc_context_t context, bool update_pmr)
> +{
> +	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(context);
> +
> +	arm64_update_exc_hwstate(hwstate, update_pmr);
> +}
> +#endif /* __ASM_INTERRUPTS_COMMON_FLAGS_H */
> diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
> new file mode 100644
> index 000000000000..3034c490ed66
> --- /dev/null
> +++ b/arch/arm64/include/asm/interrupts/entry.h
> @@ -0,0 +1,110 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2025 Arm Ltd.
> + */
> +#ifndef __ASM_INTERRUPTS_ENTRY_H
> +#define __ASM_INTERRUPTS_ENTRY_H
> +
> +#include <asm/arch_gicv3.h>
> +#include <asm/bug.h>
> +#include <asm/cpufeature.h>
> +#include <asm/interrupts/common_flags.h>
> +
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
> +						  arm64_exc_hwstate_t next)
> +{
> +	bool update_pmr = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
> +
> +	arm64_debug_exc_hwstate(prev);
> +
> +	if (prev.flags == next.flags)
> +		return next;
> +
> +	if (!arch_irqs_disabled_flags(next.flags))
> +		trace_hardirqs_on();
> +
> +	arm64_update_exc_hwstate(next, update_pmr);
> +
> +	if (arch_irqs_disabled_flags(next.flags))
> +		trace_hardirqs_off();
> +
> +	return next;
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_inherit_exc_context(struct pt_regs *regs)
> +{
> +	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
> +	arm64_exc_hwstate_t next = arm64_inherit_exc_hwstate(regs);
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
> +
> +	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
> +		bool pnmi = system_uses_irq_prio_masking();
> +
> +		WARN_ON_ONCE(context > ERROR_CONTEXT &&
> +			     prev.daif == DAIF_ERRCTX);
> +
> +		WARN_ON_ONCE(context > NONMI_CONTEXT &&
> +			     prev.daif == DAIF_PROCCTX_NOIRQ);
> +
> +		WARN_ON_ONCE(context > NOIRQ_CONTEXT &&
> +			     pnmi && prev.pmr == GIC_PRIO_IRQOFF);
> +
> +		WARN_ON_ONCE(context > PROCESS_CONTEXT &&
> +			     ((pnmi && prev.daif == DAIF_PROCCTX && prev.pmr == GIC_PRIO_IRQON) ||
> +			      (!pnmi && prev.daif == DAIF_PROCCTX)));
> +	}
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
> +
> +	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
> +		bool pnmi = system_uses_irq_prio_masking();
> +
> +		WARN_ON_ONCE(context < CRITICAL_CONTEXT &&
> +			     prev.daif == DAIF_MASK);
> +
> +		WARN_ON_ONCE(context < ERROR_CONTEXT &&
> +			     prev.daif == DAIF_ERRCTX);
> +
> +		WARN_ON_ONCE(context < NONMI_CONTEXT &&
> +			     pnmi && prev.daif == DAIF_PROCCTX_NOIRQ);
> +
> +		WARN_ON_ONCE(context < NOIRQ_CONTEXT &&
> +			     ((pnmi && prev.pmr == GIC_PRIO_IRQOFF) ||
> +			      (!pnmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
> +	}
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_unmask_exc_context(arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
> +
> +	return arm64_drop_exc_context(prev, context);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_mask_exc_context(arm64_exc_hwstate_t prev)
> +{
> +	return arm64_lift_exc_context(prev, CRITICAL_CONTEXT);
> +}
> +
> +#endif /* __ASM_INTERRUPTS_ENTRY_H */

Re: [RFC PATCH 11/36] arm64: interrupts: introduce interrupt masking helpers for entry code

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> The entry code handles interrupt masking differently from the rest of
> the kernel. Exception handlers enter and exit with all exceptions
> masked, but they must temporarily unmask the appropriate set of
> exceptions so that the rest of the handler executes with the expected
> exception state.
> 
> For EL0 handlers, this means dropping to masking context appropriate
> for the work to be performed. For EL1 handlers, this means restoring
> the masking context of the interrupted task. In both cases, all
> exceptions must be masked again before returning from the exception
> handler.
> 
> The rest of the kernel typically follows the opposite pattern: it
> raises the masking context to protect a critical section and later
> restores the previous context.
> 
> Given these different usage patterns, introduce a dedicated set of
> exception masking helpers for the entry code. Keeping these helpers
> separate from the generic interrupt masking APIs makes the intended
> usage explicit and helps avoid mixing the two masking models.
> 
> To make the masking logic easier to reason about, introduce exception
> contexts that map directly to the corresponding hardware exception
> state. Along with these contexts, provide helpers to:
> 
> - translate an exception context into the corresponding hardware
>   state,
> 
> - verify that the current hardware exception state matches the
>   expected exception context,
> 
> - raise or lower the current exception context, and
> 
> - perform the common mask/unmask operations when the starting or
>   target exception context is already known.
> 
> Tracking the current exception context also provides two additional
> benefits:
> 
> - improved debugging by verifying that the hardware exception state
>   matches the expected exception context, and
> 
> - avoiding unnecessary writes to the hardware exception state.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  .../include/asm/interrupts/common_flags.h     | 197 ++++++++++++++++++
>  arch/arm64/include/asm/interrupts/entry.h     | 110 ++++++++++
>  2 files changed, 307 insertions(+)
>  create mode 100644 arch/arm64/include/asm/interrupts/common_flags.h
>  create mode 100644 arch/arm64/include/asm/interrupts/entry.h
> 
> diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
> new file mode 100644
> index 000000000000..6ce60d1519e8
> --- /dev/null
> +++ b/arch/arm64/include/asm/interrupts/common_flags.h
> @@ -0,0 +1,197 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2025 Arm Ltd.
> + */
> +#ifndef __ASM_INTERRUPTS_COMMON_FLAGS_H
> +#define __ASM_INTERRUPTS_COMMON_FLAGS_H
> +
> +#include <asm/arch_gicv3.h>
> +#include <asm/bug.h>
> +#include <asm/cpufeature.h>
> +#include <asm/ptrace.h>
> +#include <asm/sysreg.h>
> +#include <asm/irqflags.h>
> +
> +#define DAIF_PROCCTX		0
> +#define DAIF_PROCCTX_NOIRQ	(PSR_I_BIT | PSR_F_BIT)
> +#define DAIF_ERRCTX		(PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
> +#define DAIF_MASK		(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
> +
> +/*
> + * Exception context mapping
> + *
> + * pseudo-NMI
> + *
> + * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
> + * ERROR    ->  AIF + IRQON
> + * NONMI    ->   IF + IRQON
> + * NOIRQ    ->    0 + IRQOFF
> + * PROCESS  ->    0 + IRQON
> + *
> + * Otherwise
> + *
> + * CRITICAL -> DAIF           (corresponds to the state on exception entry)
> + * ERROR    ->  AIF
> + * NONMI    ->   IF
> + * NOIRQ    ->   IF
> + * PROCESS  ->    0
> + */
> +typedef enum arm64_exc_context {
> +	PROCESS_CONTEXT,
> +	NOIRQ_CONTEXT,
> +	NONMI_CONTEXT,
> +	ERROR_CONTEXT,
> +	CRITICAL_CONTEXT,
> +} arm64_exc_context_t;
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_process_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
> +{
> +	if (system_uses_irq_prio_masking())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
> +
> +	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_exc_hwstate_of_context(arm64_exc_context_t context) {
> +	switch (context) {
> +	case PROCESS_CONTEXT:
> +		return __arm64_exc_hwstate_of_process_context();
> +	case NOIRQ_CONTEXT:
> +		return __arm64_exc_hwstate_of_noirq_context();
> +	case NONMI_CONTEXT:
> +		return __arm64_exc_hwstate_of_nonmi_context();
> +	case ERROR_CONTEXT:
> +		return __arm64_exc_hwstate_of_error_context();
> +	case CRITICAL_CONTEXT:
> +		return __arm64_exc_hwstate_of_critical_context();
> +	default:
> +		BUG();
> +	}
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
> +{
> +	arm64_exc_hwstate_t state = {.daif=regs->pstate & DAIF_MASK};
> +
> +	if (system_uses_irq_prio_masking())
> +		state.pmr = regs->pmr;
> +
> +	return state;
> +}
> +
> +static __always_inline
> +void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
> +{
> +	arm64_exc_hwstate_t actual;
> +
> +	if (!IS_ENABLED(CONFIG_DEBUG_IRQFLAGS))
> +		return;
> +
> +	actual.flags = arch_local_save_flags();
> +
> +	if (expected.flags == actual.flags)
> +		return;
> +
> +	if (system_uses_irq_prio_masking()) {
> +		WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
> +			  actual.daif, actual.pmr, expected.daif, expected.pmr);
> +	} else {
> +		WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
> +			  actual.daif, expected.daif);
> +	}
> +}
> +
> +static __always_inline
> +void arm64_debug_exc_context(arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t expected = arm64_exc_hwstate_of_context(context);
> +
> +	arm64_debug_exc_hwstate(expected);
> +}
> +
> +static __always_inline
> +void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
> +{
> +	if (system_uses_irq_prio_masking() &&
> +	    update_pmr &&
> +	    hwstate.pmr == GIC_PRIO_IRQOFF) {
> +		/*
> +		 * There has been concern that the write to daif
> +		 * might be reordered before this write to PMR.
> +		 * From the ARM ARM DDI 0487D.a, section D1.7.1
> +		 * "Accessing PSTATE fields":
> +		 *   Writes to the PSTATE fields have side-effects on
> +		 *   various aspects of the PE operation. All of these
> +		 *   side-effects are guaranteed:
> +		 *     - Not to be visible to earlier instructions in
> +		 *       the execution stream.
> +		 *     - To be visible to later instructions in the
> +		 *       execution stream
> +		 *
> +		 * Also, writes to PMR are self-synchronizing, so no
> +		 * interrupts with a lower priority than PMR is signaled
> +		 * to the PE after the write.
> +		 *
> +		 * So we don't need additional synchronization here.
> +		 */
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +		pmr_sync();
> +	}
> +
> +	write_sysreg(hwstate.daif, daif);
> +
> +	if (system_uses_irq_prio_masking() &&
> +	    update_pmr &&
> +	    hwstate.pmr == GIC_PRIO_IRQON) {
> +		write_sysreg_s(hwstate.pmr, SYS_ICC_PMR_EL1);
> +		pmr_sync();

In the original local_daif_restore, pmr_sync is only required when
interrupts are enabled. Is there any performance issue?

If all are using pmr_sync, why not merge it with the previous pmr
operations?

> +	}
> +}
> +
> +static __always_inline
> +void arm64_update_exc_context(arm64_exc_context_t context, bool update_pmr)
> +{
> +	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(context);
> +
> +	arm64_update_exc_hwstate(hwstate, update_pmr);
> +}
> +#endif /* __ASM_INTERRUPTS_COMMON_FLAGS_H */
> diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
> new file mode 100644
> index 000000000000..3034c490ed66
> --- /dev/null
> +++ b/arch/arm64/include/asm/interrupts/entry.h
> @@ -0,0 +1,110 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2025 Arm Ltd.
> + */
> +#ifndef __ASM_INTERRUPTS_ENTRY_H
> +#define __ASM_INTERRUPTS_ENTRY_H
> +
> +#include <asm/arch_gicv3.h>
> +#include <asm/bug.h>
> +#include <asm/cpufeature.h>
> +#include <asm/interrupts/common_flags.h>
> +
> +
> +static __always_inline
> +arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
> +						  arm64_exc_hwstate_t next)
> +{
> +	bool update_pmr = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
> +
> +	arm64_debug_exc_hwstate(prev);
> +
> +	if (prev.flags == next.flags)
> +		return next;
> +
> +	if (!arch_irqs_disabled_flags(next.flags))
> +		trace_hardirqs_on();
> +
> +	arm64_update_exc_hwstate(next, update_pmr);
> +
> +	if (arch_irqs_disabled_flags(next.flags))
> +		trace_hardirqs_off();
> +
> +	return next;
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_inherit_exc_context(struct pt_regs *regs)
> +{
> +	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
> +	arm64_exc_hwstate_t next = arm64_inherit_exc_hwstate(regs);
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
> +
> +	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
> +		bool pnmi = system_uses_irq_prio_masking();
> +
> +		WARN_ON_ONCE(context > ERROR_CONTEXT &&
> +			     prev.daif == DAIF_ERRCTX);
> +
> +		WARN_ON_ONCE(context > NONMI_CONTEXT &&
> +			     prev.daif == DAIF_PROCCTX_NOIRQ);
> +
> +		WARN_ON_ONCE(context > NOIRQ_CONTEXT &&
> +			     pnmi && prev.pmr == GIC_PRIO_IRQOFF);
> +
> +		WARN_ON_ONCE(context > PROCESS_CONTEXT &&
> +			     ((pnmi && prev.daif == DAIF_PROCCTX && prev.pmr == GIC_PRIO_IRQON) ||
> +			      (!pnmi && prev.daif == DAIF_PROCCTX)));
> +	}
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t next = arm64_exc_hwstate_of_context(context);
> +
> +	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
> +		bool pnmi = system_uses_irq_prio_masking();
> +
> +		WARN_ON_ONCE(context < CRITICAL_CONTEXT &&
> +			     prev.daif == DAIF_MASK);
> +
> +		WARN_ON_ONCE(context < ERROR_CONTEXT &&
> +			     prev.daif == DAIF_ERRCTX);
> +
> +		WARN_ON_ONCE(context < NONMI_CONTEXT &&
> +			     pnmi && prev.daif == DAIF_PROCCTX_NOIRQ);
> +
> +		WARN_ON_ONCE(context < NOIRQ_CONTEXT &&
> +			     ((pnmi && prev.pmr == GIC_PRIO_IRQOFF) ||
> +			      (!pnmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
> +	}
> +
> +	return __arm64_switch_exc_hwstate_to(prev, next);
> +}
> +
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_unmask_exc_context(arm64_exc_context_t context)
> +{
> +	arm64_exc_hwstate_t prev = arm64_exc_hwstate_of_context(CRITICAL_CONTEXT);
> +
> +	return arm64_drop_exc_context(prev, context);
> +}
> +
> +static __always_inline
> +arm64_exc_hwstate_t arm64_mask_exc_context(arm64_exc_hwstate_t prev)
> +{
> +	return arm64_lift_exc_context(prev, CRITICAL_CONTEXT);
> +}
> +
> +#endif /* __ASM_INTERRUPTS_ENTRY_H */

Re: [RFC PATCH 13/36] arm64: process: Use helper to check exception state

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> debug_switch_state() explicitly checks the hardware exception state
> for both the pseudo-NMI and non-NMI paths. The expected state matches
> the NOIRQ_CONTEXT exception context introduced alongside the exception
> masking helpers and state checkers.
> 
> Use the helper instead of open-coding these checks.
> 
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/kernel/process.c | 21 ++-------------------
>  1 file changed, 2 insertions(+), 19 deletions(-)
> 
> diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
> index 033643cd4e5e..ddda8d7aee24 100644
> --- a/arch/arm64/kernel/process.c
> +++ b/arch/arm64/kernel/process.c
> @@ -51,6 +51,7 @@
>  #include <asm/exec.h>
>  #include <asm/fpsimd.h>
>  #include <asm/gcs.h>
> +#include <asm/interrupts/common_flags.h>
>  #include <asm/mmu_context.h>
>  #include <asm/mpam.h>
>  #include <asm/mte.h>
> @@ -738,25 +739,7 @@ void update_sctlr_el1(u64 sctlr)
>  
>  static inline void debug_switch_state(void)
>  {
> -	if (system_uses_irq_prio_masking()) {
> -		unsigned long daif_expected = 0;
> -		unsigned long daif_actual = read_sysreg(daif);
> -		unsigned long pmr_expected = GIC_PRIO_IRQOFF;
> -		unsigned long pmr_actual = read_sysreg_s(SYS_ICC_PMR_EL1);
> -
> -		WARN_ONCE(daif_actual != daif_expected ||
> -			  pmr_actual != pmr_expected,
> -			  "Unexpected DAIF + PMR: 0x%lx + 0x%lx (expected 0x%lx + 0x%lx)\n",
> -			  daif_actual, pmr_actual,
> -			  daif_expected, pmr_expected);
> -	} else {
> -		unsigned long daif_expected = DAIF_PROCCTX_NOIRQ;
> -		unsigned long daif_actual = read_sysreg(daif);
> -
> -		WARN_ONCE(daif_actual != daif_expected,
> -			  "Unexpected DAIF value: 0x%lx (expected 0x%lx)\n",
> -			  daif_actual, daif_expected);
> -	}
> +	arm64_debug_exc_context(NOIRQ_CONTEXT);


With CONFIG_DEBUG_IRQFLAGS，we can avoid the overhead of this check.

LGTM
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

>  }
>  
>  /*

Re: [RFC PATCH 12/36] arm64: entry: replace DAIF helpers with entry helpers

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Replace all uses of the `local_daif_...` helpers in `entry-common.c`
> with the new entry-specific exception masking helpers.
> 
> Also replace `local_irq_disable()` with a switch to NOIRQ context
> using the new helpers. This provides:
> 
> - consistency checks for hardware state
> 
> - a unified style for managing exception context
> 
> Now that exception context is tracked explicitly, involuntary kernel
> preemption can be optimized. We either preempt:
> 
> - from PROCESS exception context, or
> 
> - when returning from an interrupt handler, where the GIC code
>   switches from NONMI to NOIRQ exception context
> 
> To support this split, divide `arm64_exit_to_kernel_mode()` into
> preempt, non-preempt, and dispatch parts.
> 
> Remove `local_daif_inherit()`, since it is no longer used.
> 
> Now that both the irqflags API and entry code handle DAIF and PMR
> correctly, remove `GIC_PRIO_PSR_I_SET` from the PMR value set by
> `kernel_entry` in `entry.S` and by `init_gic_priority_masking()` in
> `kernel/smp.c`.
> 
> The `local_daif_...` helpers and other specialized code still use
> `GIC_PRIO_PSR_I_SET` for now; this commit does not change their
> behavior.
> 
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>


I think this patch and the previous one have too many changes, making it
difficult to review.

Can it be broken down into smaller patches with modifications that
include the original context? Otherwise, it will be hard to identify
what changes have been made.

> ---
>  arch/arm64/include/asm/daifflags.h |  22 ---
>  arch/arm64/kernel/entry-common.c   | 272 ++++++++++++++++++++---------
>  arch/arm64/kernel/entry.S          |   2 +-
>  arch/arm64/kernel/smp.c            |   2 +-
>  4 files changed, 190 insertions(+), 108 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
> index 56341578e7e3..6b14ec4d4dbe 100644
> --- a/arch/arm64/include/asm/daifflags.h
> +++ b/arch/arm64/include/asm/daifflags.h
> @@ -120,28 +120,6 @@ static __always_inline void local_daif_restore(unsigned long flags)
>  		trace_hardirqs_off();
>  }
>  
> -/*
> - * Called by synchronous exception handlers to restore the DAIF bits that were
> - * modified by taking an exception.
> - */
> -static __always_inline void local_daif_inherit(struct pt_regs *regs)
> -{
> -	unsigned long flags = regs->pstate & DAIF_MASK;
> -
> -	if (!regs_irqs_disabled(regs))
> -		trace_hardirqs_on();
> -
> -	if (system_uses_irq_prio_masking())
> -		gic_write_pmr(regs->pmr);
> -
> -	/*
> -	 * We can't use local_daif_restore(regs->pstate) here as
> -	 * system_has_prio_mask_debugging() won't restore the I bit if it can
> -	 * use the pmr instead.
> -	 */
> -	write_sysreg(flags, daif);
> -}
> -
>  /*
>   * During early boot, we unmask PSR.DA before the GIC has been set up.
>   * If we use IRQ priority masking, the PMR and PSR will be out of sync
> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> index 2be42d7f4eaa..a13653b228b7 100644
> --- a/arch/arm64/kernel/entry-common.c
> +++ b/arch/arm64/kernel/entry-common.c
> @@ -18,11 +18,11 @@
>  #include <linux/thread_info.h>
>  
>  #include <asm/cpufeature.h>
> -#include <asm/daifflags.h>
>  #include <asm/esr.h>
>  #include <asm/exception.h>
>  #include <asm/fpsimd.h>
>  #include <asm/irq_regs.h>
> +#include <asm/interrupts/entry.h>
>  #include <asm/kprobes.h>
>  #include <asm/mmu.h>
>  #include <asm/processor.h>
> @@ -52,12 +52,11 @@ static noinstr irqentry_state_t arm64_enter_from_kernel_mode(struct pt_regs *reg
>   * After this function returns it is not safe to call regular kernel code,
>   * instrumentable code, or any code which may trigger an exception.
>   */
> -static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
> -					      irqentry_state_t state)
> +static void noinstr __arm64_exit_to_kernel_mode(struct pt_regs *regs,
> +						irqentry_state_t state,
> +						arm64_exc_hwstate_t hwstate)
>  {
> -	local_irq_disable();
> -	irqentry_exit_to_kernel_mode_preempt(regs, state);
> -	local_daif_mask();
> +	arm64_mask_exc_context(hwstate);
>  	mte_check_tfsr_exit();
>  	irqentry_exit_to_kernel_mode_after_preempt(regs, state);
>  }
> @@ -69,6 +68,30 @@ static __always_inline void arm64_syscall_enter_from_user_mode(struct pt_regs *r
>  	sme_enter_from_user_mode();
>  }
>  
> +/*
> + * We are returning from the context which allows involuntary kernel preemption
> + */
> +static void noinstr arm64_exit_to_kernel_mode_preempt(struct pt_regs *regs,
> +						      irqentry_state_t state,
> +						      arm64_exc_hwstate_t hwstate)
> +{
> +	irqentry_exit_to_kernel_mode_preempt(regs, state);
> +	__arm64_exit_to_kernel_mode(regs, state, hwstate);
> +}
> +
> +static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
> +					      irqentry_state_t state,
> +					      arm64_exc_hwstate_t hwstate)
> +{
> +	if (!arch_irqs_disabled_flags(hwstate.flags)) {
> +		hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
> +		arm64_exit_to_kernel_mode_preempt(regs, state, hwstate);
> +		return;
> +	}
> +
> +	__arm64_exit_to_kernel_mode(regs, state, hwstate);
> +}
> +
>  /*
>   * Handle IRQ/context state management when entering from user mode.
>   * Before this function is called it is not safe to call regular kernel code,
> @@ -82,11 +105,12 @@ static __always_inline void arm64_enter_from_user_mode(struct pt_regs *regs)
>  	sme_enter_from_user_mode();
>  }
>  
> -static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs)
> +static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs,
> +							    arm64_exc_hwstate_t hwstate)
>  {
> -	local_irq_disable();
> +	hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
>  	syscall_exit_to_user_mode_prepare(regs);
> -	local_daif_mask();
> +	arm64_mask_exc_context(hwstate);
>  	sme_exit_to_user_mode();
>  	mte_check_tfsr_exit();
>  	exit_to_user_mode();
> @@ -97,11 +121,12 @@ static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs
>   * After this function returns it is not safe to call regular kernel code,
>   * instrumentable code, or any code which may trigger an exception.
>   */
> -static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
> +static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs,
> +						    arm64_exc_hwstate_t hwstate)
>  {
> -	local_irq_disable();
> +	hwstate = arm64_lift_exc_context(hwstate, NOIRQ_CONTEXT);
>  	irqentry_exit_to_user_mode_prepare(regs);
> -	local_daif_mask();
> +	arm64_mask_exc_context(hwstate);
>  	sme_exit_to_user_mode();
>  	mte_check_tfsr_exit();
>  	exit_to_user_mode();
> @@ -109,7 +134,10 @@ static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
>  
>  asmlinkage void noinstr asm_exit_to_user_mode(struct pt_regs *regs)
>  {
> -	arm64_syscall_exit_to_user_mode(regs);
> +	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(PROCESS_CONTEXT);
> +
> +	arm64_syscall_exit_to_user_mode(regs, hwstate);
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  /*
> @@ -315,63 +343,69 @@ UNHANDLED(el1t, 64, error)
>  static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
>  {
>  	unsigned long far = read_sysreg(far_el1);
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);
>  	do_mem_abort(far, esr, regs);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  static void noinstr el1_pc(struct pt_regs *regs, unsigned long esr)
>  {
>  	unsigned long far = read_sysreg(far_el1);
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);
>  	do_sp_pc_abort(far, esr, regs);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  static void noinstr el1_undef(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);
>  	do_el1_undef(regs, esr);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  static void noinstr el1_bti(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);
>  	do_el1_bti(regs, esr);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  static void noinstr el1_gcs(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);
>  	do_el1_gcs(regs, esr);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  static void noinstr el1_mops(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);
>  	do_el1_mops(regs, esr);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  static void noinstr el1_breakpt(struct pt_regs *regs, unsigned long esr)
> @@ -431,12 +465,13 @@ static void noinstr el1_brk64(struct pt_regs *regs, unsigned long esr)
>  
>  static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	hwstate = arm64_inherit_exc_context(regs);
>  	do_el1_fpac(regs, esr);
> -	arm64_exit_to_kernel_mode(regs, state);
> +	arm64_exit_to_kernel_mode(regs, state, hwstate);
>  }
>  
>  asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
> @@ -486,16 +521,20 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
>  	default:
>  		__panic_unhandled(regs, "64-bit el1h sync", esr);
>  	}
> +
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  static __always_inline void __el1_pnmi(struct pt_regs *regs,
>  				       void (*handler)(struct pt_regs *))
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	state = irqentry_nmi_enter(regs);
> +	hwstate = arm64_unmask_exc_context(NONMI_CONTEXT);
>  	do_interrupt_handler(regs, handler);
> -	local_daif_mask();
> +	arm64_mask_exc_context(hwstate);
>  	irqentry_nmi_exit(regs, state);
>  }
>  
> @@ -506,21 +545,32 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
>  
>  	state = arm64_enter_from_kernel_mode(regs);
>  
> +	arm64_unmask_exc_context(NONMI_CONTEXT);
> +
>  	irq_enter_rcu();
>  	do_interrupt_handler(regs, handler);
>  	irq_exit_rcu();
>  
> -	arm64_exit_to_kernel_mode(regs, state);
> +	/*
> +	 * If pseudo-NMIs are enabled and the interrupted context had
> +	 * IRQs unmasked, the interrupt handler will have cleared DAIF
> +	 * and switched to PMR masking in order to handle
> +	 * NMIs. Otherwise it would keep IF.  In both cases on return
> +	 * we effectivly have NOIRQ_CONTEXT - keep track of it
> +	 */
> +	arm64_debug_exc_context(NOIRQ_CONTEXT);
> +	arm64_exit_to_kernel_mode_preempt(regs, state, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
>  }
> +
>  static void noinstr el1_interrupt(struct pt_regs *regs,
>  				  void (*handler)(struct pt_regs *))
>  {
> -	write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
> -
>  	if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && regs_irqs_disabled(regs))
>  		__el1_pnmi(regs, handler);
>  	else
>  		__el1_irq(regs, handler);
> +
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
> @@ -536,28 +586,31 @@ asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
>  asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
>  {
>  	unsigned long esr = read_sysreg(esr_el1);
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
> -	local_daif_restore(DAIF_ERRCTX);
> +	hwstate = arm64_unmask_exc_context(ERROR_CONTEXT);
>  	state = irqentry_nmi_enter(regs);
>  	do_serror(regs, esr);
> -	local_daif_mask();
> +	arm64_mask_exc_context(hwstate);
>  	irqentry_nmi_exit(regs, state);
>  }
>  
>  static void noinstr el0_da(struct pt_regs *regs, unsigned long esr)
>  {
>  	unsigned long far = read_sysreg(far_el1);
> +	arm64_exc_hwstate_t hwstate;
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_mem_abort(far, esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
>  {
>  	unsigned long far = read_sysreg(far_el1);
> +	arm64_exc_hwstate_t hwstate;
>  
>  	/*
>  	 * We've taken an instruction abort from userspace and not yet
> @@ -568,114 +621,139 @@ static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
>  		arm64_apply_bp_hardening();
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_mem_abort(far, esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_fpsimd_acc(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_fpsimd_acc(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_sve_acc(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_sve_acc(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_sme_acc(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_sme_acc(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_fpsimd_exc(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_fpsimd_exc(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_sys(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_sys(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_pc(struct pt_regs *regs, unsigned long esr)
>  {
>  	unsigned long far = read_sysreg(far_el1);
> +	arm64_exc_hwstate_t hwstate;
>  
>  	if (!is_ttbr0_addr(instruction_pointer(regs)))
>  		arm64_apply_bp_hardening();
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_sp_pc_abort(far, esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_sp(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_sp_pc_abort(regs->sp, esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_undef(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_undef(regs, esr);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_bti(struct pt_regs *regs)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_bti(regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_mops(regs, esr);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_gcs(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_gcs(regs, esr);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_inv(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	bad_el0_sync(regs, 0, esr);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	if (!is_ttbr0_addr(regs->pc))
>  		arm64_apply_bp_hardening();
>  
> @@ -683,12 +761,13 @@ static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
>  	debug_exception_enter(regs);
>  	do_breakpoint(esr, regs);
>  	debug_exception_exit(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> -	arm64_exit_to_user_mode(regs);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	bool step_done;
>  
>  	if (!is_ttbr0_addr(regs->pc))
> @@ -702,50 +781,56 @@ static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
>  	 * the single-step is complete.
>  	 */
>  	step_done = try_step_suspended_breakpoints(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	if (!step_done)
>  		do_el0_softstep(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_watchpt(struct pt_regs *regs, unsigned long esr)
>  {
>  	/* Watchpoints are the only debug exception to write FAR_EL1 */
>  	unsigned long far = read_sysreg(far_el1);
> +	arm64_exc_hwstate_t hwstate;
>  
>  	arm64_enter_from_user_mode(regs);
>  	debug_exception_enter(regs);
>  	do_watchpoint(far, esr, regs);
>  	debug_exception_exit(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> -	arm64_exit_to_user_mode(regs);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_brk64(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_brk64(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_svc(struct pt_regs *regs)
>  {
> +	arm64_exc_hwstate_t hwstate;
>  	arm64_syscall_enter_from_user_mode(regs);
>  	cortex_a76_erratum_1463225_svc_handler();
>  	fpsimd_syscall_enter();
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_svc(regs);
> -	arm64_syscall_exit_to_user_mode(regs);
> +	arm64_syscall_exit_to_user_mode(regs, hwstate);
>  	fpsimd_syscall_exit();
>  }
>  
>  static void noinstr el0_fpac(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_fpac(regs, esr);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
> @@ -814,6 +899,8 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
>  	default:
>  		el0_inv(regs, esr);
>  	}
> +
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  static void noinstr el0_interrupt(struct pt_regs *regs,
> @@ -821,7 +908,7 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
>  {
>  	arm64_enter_from_user_mode(regs);
>  
> -	write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
> +	arm64_unmask_exc_context(NONMI_CONTEXT);
>  
>  	if (regs->pc & BIT(55))
>  		arm64_apply_bp_hardening();
> @@ -830,7 +917,14 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
>  	do_interrupt_handler(regs, handler);
>  	irq_exit_rcu();
>  
> -	arm64_exit_to_user_mode(regs);
> +	/*
> +	 * For the same reason as in el1_irq() we effectivly
> +	 * have NOIRQ_CONTEXT on return from handler - keep
> +	 * track of it
> +	 */
> +	arm64_debug_exc_context(NOIRQ_CONTEXT);
> +	arm64_exit_to_user_mode(regs, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  static void noinstr __el0_irq_handler_common(struct pt_regs *regs)
> @@ -856,15 +950,17 @@ asmlinkage void noinstr el0t_64_fiq_handler(struct pt_regs *regs)
>  static void noinstr __el0_error_handler_common(struct pt_regs *regs)
>  {
>  	unsigned long esr = read_sysreg(esr_el1);
> +	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_ERRCTX);
> +	hwstate = arm64_unmask_exc_context(ERROR_CONTEXT);
>  	state = irqentry_nmi_enter(regs);
>  	do_serror(regs, esr);
>  	irqentry_nmi_exit(regs, state);
> -	local_daif_restore(DAIF_PROCCTX);
> -	arm64_exit_to_user_mode(regs);
> +	hwstate = arm64_drop_exc_context(hwstate, PROCESS_CONTEXT);
> +	arm64_exit_to_user_mode(regs, hwstate);
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
> @@ -875,27 +971,33 @@ asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
>  #ifdef CONFIG_COMPAT
>  static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_cp15(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_svc_compat(struct pt_regs *regs)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_syscall_enter_from_user_mode(regs);
>  	cortex_a76_erratum_1463225_svc_handler();
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_el0_svc_compat(regs);
> -	arm64_syscall_exit_to_user_mode(regs);
> +	arm64_syscall_exit_to_user_mode(regs, hwstate);
>  }
>  
>  static void noinstr el0_bkpt32(struct pt_regs *regs, unsigned long esr)
>  {
> +	arm64_exc_hwstate_t hwstate;
> +
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	hwstate = arm64_unmask_exc_context(PROCESS_CONTEXT);
>  	do_bkpt32(esr, regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_exit_to_user_mode(regs, hwstate);
>  }
>  
>  asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
> @@ -946,6 +1048,8 @@ asmlinkage void noinstr el0t_32_sync_handler(struct pt_regs *regs)
>  	default:
>  		el0_inv(regs, esr);
>  	}
> +
> +	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
>  asmlinkage void noinstr el0t_32_irq_handler(struct pt_regs *regs)
> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> index f63049ac32dc..cb3be770f2d0 100644
> --- a/arch/arm64/kernel/entry.S
> +++ b/arch/arm64/kernel/entry.S
> @@ -316,7 +316,7 @@ alternative_else_nop_endif
>  
>  	mrs_s	x20, SYS_ICC_PMR_EL1
>  	str	w20, [sp, #S_PMR]
> -	mov	x20, #GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET
> +	mov	x20, #GIC_PRIO_IRQON
>  	msr_s	SYS_ICC_PMR_EL1, x20
>  
>  .Lskip_pmr_save\@:
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index d46022f72075..b91cf163aac7 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -185,7 +185,7 @@ static void init_gic_priority_masking(void)
>  	WARN_ON(!(cpuflags & PSR_I_BIT));
>  	WARN_ON(!(cpuflags & PSR_F_BIT));
>  
> -	gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
> +	gic_write_pmr(GIC_PRIO_IRQON);
>  }
>  
>  /*

Re: [RFC PATCH 29/36] arm64: nmi: Manage masking for superpriority interrupts

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Extend logic to handle and debug exception context/state with knowlage
> of FEAT_NMI.
> 
> Take care to order writes to ALLINT relative to DAIF, as clearing
> ALLINT before DAIF could result in taking an NMI while DAIF is fully
> masked, as could setting it after DAIF.
> 
> Since superpriority interrupts are not masked through DAIF like pseduo
> NMIs are, we also need to modify the assembler macros for managing
> DAIF to ensure that the masking is done in the assembly code.
> 
> Note that save_and_disable_irq/restore_irq and
> save_and_disable_daif/restore_irq pairs are used in distinct
> contextes:
> 
> - former is used in context of SW PAN to quickly disable/enable
>   preemption
> 
> - latter is used to completely mask all exceptions.
> 
> For that reason split save_and_disable_daif/restore_irq into more
> generic exception save restore pair and plumb with FEAT_NMI logic.
> 
> Co-developed-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> ---
>  arch/arm64/include/asm/assembler.h            | 20 ++++++++-
>  .../include/asm/interrupts/common_flags.h     | 41 +++++++++++++++++--
>  arch/arm64/include/asm/interrupts/entry.h     | 18 +++++---
>  arch/arm64/include/asm/interrupts/masking.h   | 25 ++++++-----
>  arch/arm64/include/asm/irqflags.h             |  1 +
>  arch/arm64/kernel/entry.S                     | 12 +++---
>  6 files changed, 89 insertions(+), 28 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 0b58b550e8dc..bcdbc308afba 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -37,11 +37,27 @@
>  /*
>   * Save/restore interrupts.
>   */
> -	.macro save_and_disable_daif, flags
> -	mrs	\flags, daif
> +	.macro save_and_disable_exceptions, flags, tmp
> +	mrs     \flags, daif		// updates flags[9:6] with DAIF
> +#ifdef CONFIG_ARM64_NMI
> +alternative_if ARM64_NMI
> +	mrs_s	\tmp, SYS_ALLINT	// updates tmp[13] with AllInt
> +	msr_s	SYS_ALLINT_SET, xzr
> +	orr	\flags, \flags, \tmp	// now flags[13,9:6] carry pair of AllInt,DAIF
> +alternative_else_nop_endif
> +#endif
>  	msr	daifset, #0xf
>  	.endm
>  
> +	.macro  restore_exceptions, flags
> +	msr     daif, \flags		// bits other than flags[9:6] are ignored
> +#ifdef CONFIG_ARM64_NMI
> +alternative_if ARM64_NMI
> +	msr_s	SYS_ALLINT, \flags	// bits other than flags[13] are ignored
> +alternative_else_nop_endif
> +#endif
> +	.endm
> +
>  	.macro	save_and_disable_irq, flags
>  	mrs	\flags, daif
>  	msr	daifset, #3
> diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
> index 72ed6e75d146..315aaec3ea34 100644
> --- a/arch/arm64/include/asm/interrupts/common_flags.h
> +++ b/arch/arm64/include/asm/interrupts/common_flags.h
> @@ -20,6 +20,14 @@
>  /*
>   * Exception context mapping
>   *
> + * FEAT_NMI
> + *
> + * CRITICAL -> DAIF + AllInt (corresponds to the state on exception entry)
> + * ERROR    ->  AIF + AllInt
> + * NONMI    ->   IF + AllInt
> + * NOIRQ    ->   IF
> + * PROCESS  ->    0

This is consistent with what I understand.

> + *
>   * pseudo-NMI
>   *
>   * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
> @@ -76,6 +84,7 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
>  		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
>  
>  	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +
>  }
>  
>  static __always_inline
> @@ -84,6 +93,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
>  	if (system_uses_irq_prio_masking())
>  		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
>  
> +	if (system_uses_nmi())
> +               return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .allint=ALLINT_ALLINT};
> +
>  	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
>  }
>  
> @@ -93,6 +105,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
>  	if (system_uses_irq_prio_masking())
>  		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
>  
> +	if (system_uses_nmi())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .allint=ALLINT_ALLINT};
> +
>  	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
>  }
>  
> @@ -102,6 +117,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
>  	if (system_uses_irq_prio_masking())
>  		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
>  
> +	if (system_uses_nmi())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .allint=ALLINT_ALLINT};
> +
>  	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
>  }
>  
> @@ -131,6 +149,9 @@ arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
>  	if (system_uses_irq_prio_masking())
>  		state.pmr = regs->pmr;
>  
> +	if (system_uses_nmi())
> +		state.allint = regs->pstate & PSR_ALLINT_BIT;
> +
>  	return state;
>  }
>  
> @@ -150,6 +171,9 @@ void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
>  	if (system_uses_irq_prio_masking()) {
>  		WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
>  			  actual.daif, actual.pmr, expected.daif, expected.pmr);
> +	} else if (system_uses_nmi()) {
> +		WARN_ONCE(1, "Unexpected DAIF+ALLINT: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
> +			  actual.daif, actual.allint, expected.daif, expected.allint);
>  	} else {
>  		WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
>  			  actual.daif, expected.daif);
> @@ -165,7 +189,7 @@ void arm64_debug_exc_context(arm64_exc_context_t context)
>  }
>  
>  static __always_inline
> -void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
> +void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr, bool update_allint)
>  {
>  	if (system_uses_irq_prio_masking() &&
>  	    update_pmr &&
> @@ -193,8 +217,19 @@ void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
>  		pmr_sync();
>  	}
>  
> +	/*
> +	 * Try to order ALLINT writes to be consistent with the DAIF state :
> +	 * we don't want to take an NMI with DAIF masked or when it should
> +	 * be masked but isn't yet.
> +	 */
> +	if (system_uses_nmi() && update_allint && hwstate.allint)
> +		_allint_set();
> +
>  	write_sysreg(hwstate.daif, daif);
>  
> +	if (system_uses_nmi() && update_allint && !hwstate.allint)
> +		_allint_clear();
> +
>  	if (system_uses_irq_prio_masking() &&
>  	    update_pmr &&
>  	    hwstate.pmr == GIC_PRIO_IRQON) {
> @@ -204,10 +239,10 @@ void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
>  }
>  
>  static __always_inline
> -void arm64_update_exc_context(arm64_exc_context_t context, bool update_pmr)
> +void arm64_update_exc_context(arm64_exc_context_t context, bool update_pmr,  bool update_allint)
>  {
>  	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(context);
>  
> -	arm64_update_exc_hwstate(hwstate, update_pmr);
> +	arm64_update_exc_hwstate(hwstate, update_pmr, update_allint);
>  }
>  #endif /* __ASM_INTERRUPTS_COMMON_FLAGS_H */
> diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
> index 3034c490ed66..986536ecd058 100644
> --- a/arch/arm64/include/asm/interrupts/entry.h
> +++ b/arch/arm64/include/asm/interrupts/entry.h
> @@ -10,12 +10,12 @@
>  #include <asm/cpufeature.h>
>  #include <asm/interrupts/common_flags.h>
>  
> -
>  static __always_inline
>  arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
>  						  arm64_exc_hwstate_t next)
>  {
>  	bool update_pmr = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
> +	bool update_allint = system_uses_nmi() && prev.allint != next.allint;
>  
>  	arm64_debug_exc_hwstate(prev);
>  
> @@ -25,7 +25,7 @@ arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
>  	if (!arch_irqs_disabled_flags(next.flags))
>  		trace_hardirqs_on();
>  
> -	arm64_update_exc_hwstate(next, update_pmr);
> +	arm64_update_exc_hwstate(next, update_pmr, update_allint);
>  
>  	if (arch_irqs_disabled_flags(next.flags))
>  		trace_hardirqs_off();
> @@ -49,15 +49,18 @@ arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
>  
>  	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
>  		bool pnmi = system_uses_irq_prio_masking();
> +		bool nmi = system_uses_nmi();
>  
>  		WARN_ON_ONCE(context > ERROR_CONTEXT &&
>  			     prev.daif == DAIF_ERRCTX);
>  
>  		WARN_ON_ONCE(context > NONMI_CONTEXT &&
> -			     prev.daif == DAIF_PROCCTX_NOIRQ);
> +			     ((nmi && prev.daif == DAIF_PROCCTX_NOIRQ && prev.allint == ALLINT_ALLINT) ||
> +			      (!nmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
>  
>  		WARN_ON_ONCE(context > NOIRQ_CONTEXT &&
> -			     pnmi && prev.pmr == GIC_PRIO_IRQOFF);
> +			     ((pnmi && prev.pmr == GIC_PRIO_IRQOFF) ||
> +			      (nmi && prev.daif == DAIF_PROCCTX_NOIRQ && prev.allint != ALLINT_ALLINT)));
>  
>  		WARN_ON_ONCE(context > PROCESS_CONTEXT &&
>  			     ((pnmi && prev.daif == DAIF_PROCCTX && prev.pmr == GIC_PRIO_IRQON) ||
> @@ -74,6 +77,7 @@ arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
>  
>  	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
>  		bool pnmi = system_uses_irq_prio_masking();
> +		bool nmi = system_uses_nmi();
>  
>  		WARN_ON_ONCE(context < CRITICAL_CONTEXT &&
>  			     prev.daif == DAIF_MASK);
> @@ -82,11 +86,13 @@ arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
>  			     prev.daif == DAIF_ERRCTX);
>  
>  		WARN_ON_ONCE(context < NONMI_CONTEXT &&
> -			     pnmi && prev.daif == DAIF_PROCCTX_NOIRQ);
> +			     ((pnmi && prev.daif == DAIF_PROCCTX_NOIRQ) ||
> +			      (nmi &&  prev.daif == DAIF_PROCCTX_NOIRQ && prev.allint == ALLINT_ALLINT)));
>  
>  		WARN_ON_ONCE(context < NOIRQ_CONTEXT &&
>  			     ((pnmi && prev.pmr == GIC_PRIO_IRQOFF) ||
> -			      (!pnmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
> +			      (nmi && prev.daif == DAIF_PROCCTX_NOIRQ && prev.allint != ALLINT_ALLINT) ||
> +			      (!pnmi && !nmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
>  	}
>  
>  	return __arm64_switch_exc_hwstate_to(prev, next);
> diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
> index 9dc37a571094..178e915f43b9 100644
> --- a/arch/arm64/include/asm/interrupts/masking.h
> +++ b/arch/arm64/include/asm/interrupts/masking.h
> @@ -30,6 +30,7 @@ arm64_exc_hwstates_t local_all_irqs_save_mask(arm64_exc_context_t new)
>  
>  	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
>  		bool pnmi = system_uses_irq_prio_masking();
> +		bool nmi = system_uses_nmi();
>  
>  		WARN_ON_ONCE(new < CRITICAL_CONTEXT &&
>  			     actual.daif == DAIF_MASK);
> @@ -38,14 +39,16 @@ arm64_exc_hwstates_t local_all_irqs_save_mask(arm64_exc_context_t new)
>  			     actual.daif == DAIF_ERRCTX);
>  
>  		WARN_ON_ONCE(new < NONMI_CONTEXT &&
> -			     pnmi && actual.daif == DAIF_PROCCTX_NOIRQ);
> +			     ((pnmi && actual.daif == DAIF_PROCCTX_NOIRQ) ||
> +			      (nmi &&  actual.daif == DAIF_PROCCTX_NOIRQ && actual.allint == ALLINT_ALLINT)));
>  
>  		WARN_ON_ONCE(new < NOIRQ_CONTEXT &&
>  			     ((pnmi && actual.pmr == GIC_PRIO_IRQOFF) ||
> -			      (!pnmi && actual.daif == DAIF_PROCCTX_NOIRQ)));
> +			      (nmi && actual.daif == DAIF_PROCCTX_NOIRQ && actual.allint != ALLINT_ALLINT) ||
> +			      (!pnmi && !nmi && actual.daif == DAIF_PROCCTX_NOIRQ)));
>  	}
>  
> -	arm64_update_exc_hwstate(state, actual.pmr != state.pmr);
> +	arm64_update_exc_hwstate(state, actual.pmr != state.pmr, actual.allint != state.allint);
>  
>  	if (!arch_irqs_disabled_flags(actual.flags))
>  		trace_hardirqs_off();
> @@ -60,7 +63,7 @@ static inline void local_all_irqs_restore(arm64_exc_hwstates_t states)
>  	if (!arch_irqs_disabled_flags(states.saved.flags))
>  		trace_hardirqs_on();
>  
> -	arm64_update_exc_hwstate(states.saved, true);
> +	arm64_update_exc_hwstate(states.saved, true, true);
>  }
>  
>  #ifdef CONFIG_DEBUG_IRQFLAGS
> @@ -70,7 +73,7 @@ void local_all_irqs_cpu_init_mask(arm64_exc_context_t context)
>  	WARN_ON(__this_cpu_read(irqs_masks_cpu_init_done));
>  	if (context == PROCESS_CONTEXT)
>  		trace_hardirqs_on();
> -	arm64_update_exc_context(context, true);
> +	arm64_update_exc_context(context, true, true);
>  	__this_cpu_write(irqs_masks_cpu_init_done, true);
>  	__this_cpu_write(irqs_masks_cpu_final_done, false);
>  }
> @@ -78,7 +81,7 @@ void local_all_irqs_cpu_init_mask(arm64_exc_context_t context)
>  static inline void local_all_irqs_final_mask(void)
>  {
>  	WARN_ON(__this_cpu_read(irqs_masks_cpu_final_done));
> -	arm64_update_exc_context(CRITICAL_CONTEXT, true);
> +	arm64_update_exc_context(CRITICAL_CONTEXT, true, true);
>  	trace_hardirqs_off();
>  	__this_cpu_write(irqs_masks_cpu_final_done, true);
>  	__this_cpu_write(irqs_masks_cpu_init_done, false);
> @@ -89,12 +92,12 @@ void local_all_irqs_cpu_init_mask(arm64_exc_context_t context)
>  {
>  	if (context == PROCESS_CONTEXT)
>  		trace_hardirqs_on();
> -	arm64_update_exc_context(context, true);
> +	arm64_update_exc_context(context, true, true);
>  }
>  
>  static inline void local_all_irqs_final_mask(void)
>  {
> -	arm64_update_exc_context(CRITICAL_CONTEXT, true);
> +	arm64_update_exc_context(CRITICAL_CONTEXT, true, true);
>  	trace_hardirqs_off();
>  }
>  #endif /* CONFIG_DEBUG_IRQFLAGS */
> @@ -130,7 +133,7 @@ static inline arm64_exc_hwstates_t local_all_irqs_force_daif_save(void)
>  		states.expected.daif = states.saved.daif | DAIF_PROCCTX_NOIRQ;
>  		states.expected.pmr = GIC_PRIO_IRQON;
>  
> -		arm64_update_exc_hwstate(states.expected, true);
> +		arm64_update_exc_hwstate(states.expected, true, false);
>  	}
>  
>  	return states;
> @@ -153,7 +156,7 @@ void local_all_irqs_force_daif_restore(arm64_exc_hwstates_t states)
>  
>  	if (system_uses_irq_prio_masking()) {
>  		arm64_debug_exc_hwstate(states.expected);
> -		arm64_update_exc_hwstate(states.saved, true);
> +		arm64_update_exc_hwstate(states.saved, true, false);
>  	}
>  }
>  
> @@ -167,7 +170,7 @@ static inline void local_interrupt_priority_init(void)
>  	WARN_ON(read_sysreg(daif) & PSR_A_BIT);
>  	lockdep_assert_irqs_disabled();
>  
> -	arm64_update_exc_context(NOIRQ_CONTEXT, true);
> +	arm64_update_exc_context(NOIRQ_CONTEXT, true, true);
>  }
>  
>  #endif /* __ASM_INTERRUPTS_MASKING_H */
> diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
> index 585d3ce15d80..b571d34bf11d 100644
> --- a/arch/arm64/include/asm/irqflags.h
> +++ b/arch/arm64/include/asm/irqflags.h
> @@ -31,6 +31,7 @@
>  typedef union arm64_exc_hwstate {
>  	struct {
>  		u16 daif;
> +		u16 allint;
>  		u8 pmr;
>  	};
>  	unsigned long flags;
> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> index cb3be770f2d0..39ea3fdeb03a 100644
> --- a/arch/arm64/kernel/entry.S
> +++ b/arch/arm64/kernel/entry.S
> @@ -815,7 +815,7 @@ SYM_CODE_END(__bp_harden_el1_vectors)
>   *
>   */
>  SYM_FUNC_START(cpu_switch_to)
> -	save_and_disable_daif x11
> +	save_and_disable_exceptions x11, x12
>  	mov	x10, #THREAD_CPU_CONTEXT
>  	add	x8, x0, x10
>  	mov	x9, sp
> @@ -839,7 +839,7 @@ SYM_FUNC_START(cpu_switch_to)
>  	ptrauth_keys_install_kernel x1, x8, x9, x10
>  	scs_save x0
>  	scs_load_current
> -	restore_irq x11
> +	restore_exceptions x11
>  	ret
>  SYM_FUNC_END(cpu_switch_to)
>  NOKPROBE(cpu_switch_to)
> @@ -866,7 +866,7 @@ NOKPROBE(ret_from_fork)
>   * Calls func(regs) using this CPU's irq stack and shadow irq stack.
>   */
>  SYM_FUNC_START(call_on_irq_stack)
> -	save_and_disable_daif x9
> +	save_and_disable_exceptions x9, x10
>  #ifdef CONFIG_SHADOW_CALL_STACK
>  	get_current_task x16
>  	scs_save x16
> @@ -881,10 +881,10 @@ SYM_FUNC_START(call_on_irq_stack)
>  
>  	/* Move to the new stack and call the function there */
>  	add	sp, x16, #IRQ_STACK_SIZE
> -	restore_irq x9
> +	restore_exceptions x9
>  	blr	x1
>  
> -	save_and_disable_daif x9
> +	save_and_disable_exceptions x9, x10
>  	/*
>  	 * Restore the SP from the FP, and restore the FP and LR from the frame
>  	 * record.
> @@ -892,7 +892,7 @@ SYM_FUNC_START(call_on_irq_stack)
>  	mov	sp, x29
>  	ldp	x29, x30, [sp], #16
>  	scs_load_current
> -	restore_irq x9
> +	restore_exceptions x9
>  	ret
>  SYM_FUNC_END(call_on_irq_stack)
>  NOKPROBE(call_on_irq_stack)

Re: [RFC PATCH 29/36] arm64: nmi: Manage masking for superpriority interrupts

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> 
> Extend logic to handle and debug exception context/state with knowlage
> of FEAT_NMI.
> 
> Take care to order writes to ALLINT relative to DAIF, as clearing
> ALLINT before DAIF could result in taking an NMI while DAIF is fully
> masked, as could setting it after DAIF.
> 
> Since superpriority interrupts are not masked through DAIF like pseduo
> NMIs are, we also need to modify the assembler macros for managing
> DAIF to ensure that the masking is done in the assembly code.
> 
> Note that save_and_disable_irq/restore_irq and
> save_and_disable_daif/restore_irq pairs are used in distinct
> contextes:
> 
> - former is used in context of SW PAN to quickly disable/enable
>   preemption
> 
> - latter is used to completely mask all exceptions.
> 
> For that reason split save_and_disable_daif/restore_irq into more
> generic exception save restore pair and plumb with FEAT_NMI logic.
> 
> Co-developed-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> ---
>  arch/arm64/include/asm/assembler.h            | 20 ++++++++-
>  .../include/asm/interrupts/common_flags.h     | 41 +++++++++++++++++--
>  arch/arm64/include/asm/interrupts/entry.h     | 18 +++++---
>  arch/arm64/include/asm/interrupts/masking.h   | 25 ++++++-----
>  arch/arm64/include/asm/irqflags.h             |  1 +
>  arch/arm64/kernel/entry.S                     | 12 +++---
>  6 files changed, 89 insertions(+), 28 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 0b58b550e8dc..bcdbc308afba 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -37,11 +37,27 @@
>  /*
>   * Save/restore interrupts.
>   */
> -	.macro save_and_disable_daif, flags
> -	mrs	\flags, daif
> +	.macro save_and_disable_exceptions, flags, tmp
> +	mrs     \flags, daif		// updates flags[9:6] with DAIF
> +#ifdef CONFIG_ARM64_NMI
> +alternative_if ARM64_NMI
> +	mrs_s	\tmp, SYS_ALLINT	// updates tmp[13] with AllInt
> +	msr_s	SYS_ALLINT_SET, xzr
> +	orr	\flags, \flags, \tmp	// now flags[13,9:6] carry pair of AllInt,DAIF
> +alternative_else_nop_endif
> +#endif
>  	msr	daifset, #0xf
>  	.endm
>  
> +	.macro  restore_exceptions, flags
> +	msr     daif, \flags		// bits other than flags[9:6] are ignored
> +#ifdef CONFIG_ARM64_NMI
> +alternative_if ARM64_NMI
> +	msr_s	SYS_ALLINT, \flags	// bits other than flags[13] are ignored
> +alternative_else_nop_endif
> +#endif
> +	.endm
> +
>  	.macro	save_and_disable_irq, flags
>  	mrs	\flags, daif
>  	msr	daifset, #3
> diff --git a/arch/arm64/include/asm/interrupts/common_flags.h b/arch/arm64/include/asm/interrupts/common_flags.h
> index 72ed6e75d146..315aaec3ea34 100644
> --- a/arch/arm64/include/asm/interrupts/common_flags.h
> +++ b/arch/arm64/include/asm/interrupts/common_flags.h
> @@ -20,6 +20,14 @@
>  /*
>   * Exception context mapping
>   *
> + * FEAT_NMI
> + *
> + * CRITICAL -> DAIF + AllInt (corresponds to the state on exception entry)
> + * ERROR    ->  AIF + AllInt
> + * NONMI    ->   IF + AllInt
> + * NOIRQ    ->   IF
> + * PROCESS  ->    0
> + *
>   * pseudo-NMI
>   *
>   * CRITICAL -> DAIF + IRQON  (corresponds to the state on exception entry)
> @@ -76,6 +84,7 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_noirq_context(void)
>  		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX, .pmr=GIC_PRIO_IRQOFF};
>  
>  	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};
> +
>  }
>  
>  static __always_inline
> @@ -84,6 +93,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
>  	if (system_uses_irq_prio_masking())
>  		return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .pmr=GIC_PRIO_IRQON};
>  
> +	if (system_uses_nmi())
> +               return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ, .allint=ALLINT_ALLINT};
> +
>  	return (arm64_exc_hwstate_t){.daif=DAIF_PROCCTX_NOIRQ};


DAIF_PROCCTX_NOIRQ is universal; can it be initialized with a loal
varible and set at the beginning?

90 static __always_inline
 91 arm64_exc_hwstate_t __arm64_exc_hwstate_of_nonmi_context(void)
 92 {
 93 >-------arm64_exc_hwstate_t state = { .daif=DAIF_PROCCTX_NOIRQ };
 94
 95 >-------if (system_uses_irq_prio_masking())
 96 >------->-------state.pmr = GIC_PRIO_IRQON;
 97
 98 >-------if (system_uses_nmi())
 99 >------->-------state.allint = ALLINT_ALLINT;
100
101 >-------return state;
102 }

>  }
>  
> @@ -93,6 +105,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_error_context(void)
>  	if (system_uses_irq_prio_masking())
>  		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .pmr=GIC_PRIO_IRQON};
>  
> +	if (system_uses_nmi())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX, .allint=ALLINT_ALLINT};
> +
>  	return (arm64_exc_hwstate_t){.daif=DAIF_ERRCTX};
>  }
>  
> @@ -102,6 +117,9 @@ arm64_exc_hwstate_t __arm64_exc_hwstate_of_critical_context(void)
>  	if (system_uses_irq_prio_masking())
>  		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .pmr=GIC_PRIO_IRQON};
>  
> +	if (system_uses_nmi())
> +		return (arm64_exc_hwstate_t){.daif=DAIF_MASK, .allint=ALLINT_ALLINT};
> +
>  	return (arm64_exc_hwstate_t){.daif=DAIF_MASK};
>  }
>  
> @@ -131,6 +149,9 @@ arm64_exc_hwstate_t arm64_inherit_exc_hwstate(struct pt_regs *regs)
>  	if (system_uses_irq_prio_masking())
>  		state.pmr = regs->pmr;
>  
> +	if (system_uses_nmi())
> +		state.allint = regs->pstate & PSR_ALLINT_BIT;
> +
>  	return state;
>  }
>  
> @@ -150,6 +171,9 @@ void arm64_debug_exc_hwstate(arm64_exc_hwstate_t expected)
>  	if (system_uses_irq_prio_masking()) {
>  		WARN_ONCE(1, "Unexpected DAIF+PMR: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
>  			  actual.daif, actual.pmr, expected.daif, expected.pmr);
> +	} else if (system_uses_nmi()) {
> +		WARN_ONCE(1, "Unexpected DAIF+ALLINT: 0x%x + 0x%x (expected 0x%x + 0x%x)\n",
> +			  actual.daif, actual.allint, expected.daif, expected.allint);
>  	} else {
>  		WARN_ONCE(1, "Unexpected DAIF: 0x%x (expected 0x%x)\n",
>  			  actual.daif, expected.daif);
> @@ -165,7 +189,7 @@ void arm64_debug_exc_context(arm64_exc_context_t context)
>  }
>  
>  static __always_inline
> -void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
> +void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr, bool update_allint)
>  {
>  	if (system_uses_irq_prio_masking() &&
>  	    update_pmr &&
> @@ -193,8 +217,19 @@ void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
>  		pmr_sync();
>  	}
>  
> +	/*
> +	 * Try to order ALLINT writes to be consistent with the DAIF state :
> +	 * we don't want to take an NMI with DAIF masked or when it should
> +	 * be masked but isn't yet.
> +	 */
> +	if (system_uses_nmi() && update_allint && hwstate.allint)
> +		_allint_set();
> +
>  	write_sysreg(hwstate.daif, daif);
>  
> +	if (system_uses_nmi() && update_allint && !hwstate.allint)
> +		_allint_clear();
> +
>  	if (system_uses_irq_prio_masking() &&
>  	    update_pmr &&
>  	    hwstate.pmr == GIC_PRIO_IRQON) {
> @@ -204,10 +239,10 @@ void arm64_update_exc_hwstate(arm64_exc_hwstate_t hwstate, bool update_pmr)
>  }
>  
>  static __always_inline
> -void arm64_update_exc_context(arm64_exc_context_t context, bool update_pmr)
> +void arm64_update_exc_context(arm64_exc_context_t context, bool update_pmr,  bool update_allint)
>  {
>  	arm64_exc_hwstate_t hwstate = arm64_exc_hwstate_of_context(context);
>  
> -	arm64_update_exc_hwstate(hwstate, update_pmr);
> +	arm64_update_exc_hwstate(hwstate, update_pmr, update_allint);
>  }
>  #endif /* __ASM_INTERRUPTS_COMMON_FLAGS_H */
> diff --git a/arch/arm64/include/asm/interrupts/entry.h b/arch/arm64/include/asm/interrupts/entry.h
> index 3034c490ed66..986536ecd058 100644
> --- a/arch/arm64/include/asm/interrupts/entry.h
> +++ b/arch/arm64/include/asm/interrupts/entry.h
> @@ -10,12 +10,12 @@
>  #include <asm/cpufeature.h>
>  #include <asm/interrupts/common_flags.h>
>  
> -
>  static __always_inline
>  arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
>  						  arm64_exc_hwstate_t next)
>  {
>  	bool update_pmr = system_uses_irq_prio_masking() && prev.pmr != next.pmr;
> +	bool update_allint = system_uses_nmi() && prev.allint != next.allint;
>  
>  	arm64_debug_exc_hwstate(prev);
>  
> @@ -25,7 +25,7 @@ arm64_exc_hwstate_t __arm64_switch_exc_hwstate_to(arm64_exc_hwstate_t prev,
>  	if (!arch_irqs_disabled_flags(next.flags))
>  		trace_hardirqs_on();
>  
> -	arm64_update_exc_hwstate(next, update_pmr);
> +	arm64_update_exc_hwstate(next, update_pmr, update_allint);
>  
>  	if (arch_irqs_disabled_flags(next.flags))
>  		trace_hardirqs_off();
> @@ -49,15 +49,18 @@ arm64_exc_hwstate_t arm64_drop_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
>  
>  	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
>  		bool pnmi = system_uses_irq_prio_masking();
> +		bool nmi = system_uses_nmi();
>  
>  		WARN_ON_ONCE(context > ERROR_CONTEXT &&
>  			     prev.daif == DAIF_ERRCTX);
>  
>  		WARN_ON_ONCE(context > NONMI_CONTEXT &&
> -			     prev.daif == DAIF_PROCCTX_NOIRQ);
> +			     ((nmi && prev.daif == DAIF_PROCCTX_NOIRQ && prev.allint == ALLINT_ALLINT) ||
> +			      (!nmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
>  
>  		WARN_ON_ONCE(context > NOIRQ_CONTEXT &&
> -			     pnmi && prev.pmr == GIC_PRIO_IRQOFF);
> +			     ((pnmi && prev.pmr == GIC_PRIO_IRQOFF) ||
> +			      (nmi && prev.daif == DAIF_PROCCTX_NOIRQ && prev.allint != ALLINT_ALLINT)));
>  
>  		WARN_ON_ONCE(context > PROCESS_CONTEXT &&
>  			     ((pnmi && prev.daif == DAIF_PROCCTX && prev.pmr == GIC_PRIO_IRQON) ||
> @@ -74,6 +77,7 @@ arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
>  
>  	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
>  		bool pnmi = system_uses_irq_prio_masking();
> +		bool nmi = system_uses_nmi();
>  
>  		WARN_ON_ONCE(context < CRITICAL_CONTEXT &&
>  			     prev.daif == DAIF_MASK);
> @@ -82,11 +86,13 @@ arm64_exc_hwstate_t arm64_lift_exc_context(arm64_exc_hwstate_t prev, arm64_exc_c
>  			     prev.daif == DAIF_ERRCTX);
>  
>  		WARN_ON_ONCE(context < NONMI_CONTEXT &&
> -			     pnmi && prev.daif == DAIF_PROCCTX_NOIRQ);
> +			     ((pnmi && prev.daif == DAIF_PROCCTX_NOIRQ) ||
> +			      (nmi &&  prev.daif == DAIF_PROCCTX_NOIRQ && prev.allint == ALLINT_ALLINT)));
>  
>  		WARN_ON_ONCE(context < NOIRQ_CONTEXT &&
>  			     ((pnmi && prev.pmr == GIC_PRIO_IRQOFF) ||
> -			      (!pnmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
> +			      (nmi && prev.daif == DAIF_PROCCTX_NOIRQ && prev.allint != ALLINT_ALLINT) ||
> +			      (!pnmi && !nmi && prev.daif == DAIF_PROCCTX_NOIRQ)));
>  	}
>  
>  	return __arm64_switch_exc_hwstate_to(prev, next);
> diff --git a/arch/arm64/include/asm/interrupts/masking.h b/arch/arm64/include/asm/interrupts/masking.h
> index 9dc37a571094..178e915f43b9 100644
> --- a/arch/arm64/include/asm/interrupts/masking.h
> +++ b/arch/arm64/include/asm/interrupts/masking.h
> @@ -30,6 +30,7 @@ arm64_exc_hwstates_t local_all_irqs_save_mask(arm64_exc_context_t new)
>  
>  	if (IS_ENABLED(CONFIG_DEBUG_IRQFLAGS)) {
>  		bool pnmi = system_uses_irq_prio_masking();
> +		bool nmi = system_uses_nmi();
>  
>  		WARN_ON_ONCE(new < CRITICAL_CONTEXT &&
>  			     actual.daif == DAIF_MASK);
> @@ -38,14 +39,16 @@ arm64_exc_hwstates_t local_all_irqs_save_mask(arm64_exc_context_t new)
>  			     actual.daif == DAIF_ERRCTX);
>  
>  		WARN_ON_ONCE(new < NONMI_CONTEXT &&
> -			     pnmi && actual.daif == DAIF_PROCCTX_NOIRQ);
> +			     ((pnmi && actual.daif == DAIF_PROCCTX_NOIRQ) ||
> +			      (nmi &&  actual.daif == DAIF_PROCCTX_NOIRQ && actual.allint == ALLINT_ALLINT)));
>  
>  		WARN_ON_ONCE(new < NOIRQ_CONTEXT &&
>  			     ((pnmi && actual.pmr == GIC_PRIO_IRQOFF) ||
> -			      (!pnmi && actual.daif == DAIF_PROCCTX_NOIRQ)));
> +			      (nmi && actual.daif == DAIF_PROCCTX_NOIRQ && actual.allint != ALLINT_ALLINT) ||
> +			      (!pnmi && !nmi && actual.daif == DAIF_PROCCTX_NOIRQ)));
>  	}
>  
> -	arm64_update_exc_hwstate(state, actual.pmr != state.pmr);
> +	arm64_update_exc_hwstate(state, actual.pmr != state.pmr, actual.allint != state.allint);
>  
>  	if (!arch_irqs_disabled_flags(actual.flags))
>  		trace_hardirqs_off();
> @@ -60,7 +63,7 @@ static inline void local_all_irqs_restore(arm64_exc_hwstates_t states)
>  	if (!arch_irqs_disabled_flags(states.saved.flags))
>  		trace_hardirqs_on();
>  
> -	arm64_update_exc_hwstate(states.saved, true);
> +	arm64_update_exc_hwstate(states.saved, true, true);
>  }
>  
>  #ifdef CONFIG_DEBUG_IRQFLAGS
> @@ -70,7 +73,7 @@ void local_all_irqs_cpu_init_mask(arm64_exc_context_t context)
>  	WARN_ON(__this_cpu_read(irqs_masks_cpu_init_done));
>  	if (context == PROCESS_CONTEXT)
>  		trace_hardirqs_on();
> -	arm64_update_exc_context(context, true);
> +	arm64_update_exc_context(context, true, true);
>  	__this_cpu_write(irqs_masks_cpu_init_done, true);
>  	__this_cpu_write(irqs_masks_cpu_final_done, false);
>  }
> @@ -78,7 +81,7 @@ void local_all_irqs_cpu_init_mask(arm64_exc_context_t context)
>  static inline void local_all_irqs_final_mask(void)
>  {
>  	WARN_ON(__this_cpu_read(irqs_masks_cpu_final_done));
> -	arm64_update_exc_context(CRITICAL_CONTEXT, true);
> +	arm64_update_exc_context(CRITICAL_CONTEXT, true, true);
>  	trace_hardirqs_off();
>  	__this_cpu_write(irqs_masks_cpu_final_done, true);
>  	__this_cpu_write(irqs_masks_cpu_init_done, false);
> @@ -89,12 +92,12 @@ void local_all_irqs_cpu_init_mask(arm64_exc_context_t context)
>  {
>  	if (context == PROCESS_CONTEXT)
>  		trace_hardirqs_on();
> -	arm64_update_exc_context(context, true);
> +	arm64_update_exc_context(context, true, true);
>  }
>  
>  static inline void local_all_irqs_final_mask(void)
>  {
> -	arm64_update_exc_context(CRITICAL_CONTEXT, true);
> +	arm64_update_exc_context(CRITICAL_CONTEXT, true, true);
>  	trace_hardirqs_off();
>  }
>  #endif /* CONFIG_DEBUG_IRQFLAGS */
> @@ -130,7 +133,7 @@ static inline arm64_exc_hwstates_t local_all_irqs_force_daif_save(void)
>  		states.expected.daif = states.saved.daif | DAIF_PROCCTX_NOIRQ;
>  		states.expected.pmr = GIC_PRIO_IRQON;
>  
> -		arm64_update_exc_hwstate(states.expected, true);
> +		arm64_update_exc_hwstate(states.expected, true, false);
>  	}
>  
>  	return states;
> @@ -153,7 +156,7 @@ void local_all_irqs_force_daif_restore(arm64_exc_hwstates_t states)
>  
>  	if (system_uses_irq_prio_masking()) {
>  		arm64_debug_exc_hwstate(states.expected);
> -		arm64_update_exc_hwstate(states.saved, true);
> +		arm64_update_exc_hwstate(states.saved, true, false);
>  	}
>  }
>  
> @@ -167,7 +170,7 @@ static inline void local_interrupt_priority_init(void)
>  	WARN_ON(read_sysreg(daif) & PSR_A_BIT);
>  	lockdep_assert_irqs_disabled();
>  
> -	arm64_update_exc_context(NOIRQ_CONTEXT, true);
> +	arm64_update_exc_context(NOIRQ_CONTEXT, true, true);
>  }
>  
>  #endif /* __ASM_INTERRUPTS_MASKING_H */
> diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
> index 585d3ce15d80..b571d34bf11d 100644
> --- a/arch/arm64/include/asm/irqflags.h
> +++ b/arch/arm64/include/asm/irqflags.h
> @@ -31,6 +31,7 @@
>  typedef union arm64_exc_hwstate {
>  	struct {
>  		u16 daif;
> +		u16 allint;
>  		u8 pmr;
>  	};
>  	unsigned long flags;
> diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
> index cb3be770f2d0..39ea3fdeb03a 100644
> --- a/arch/arm64/kernel/entry.S
> +++ b/arch/arm64/kernel/entry.S
> @@ -815,7 +815,7 @@ SYM_CODE_END(__bp_harden_el1_vectors)
>   *
>   */
>  SYM_FUNC_START(cpu_switch_to)
> -	save_and_disable_daif x11
> +	save_and_disable_exceptions x11, x12
>  	mov	x10, #THREAD_CPU_CONTEXT
>  	add	x8, x0, x10
>  	mov	x9, sp
> @@ -839,7 +839,7 @@ SYM_FUNC_START(cpu_switch_to)
>  	ptrauth_keys_install_kernel x1, x8, x9, x10
>  	scs_save x0
>  	scs_load_current
> -	restore_irq x11
> +	restore_exceptions x11
>  	ret
>  SYM_FUNC_END(cpu_switch_to)
>  NOKPROBE(cpu_switch_to)
> @@ -866,7 +866,7 @@ NOKPROBE(ret_from_fork)
>   * Calls func(regs) using this CPU's irq stack and shadow irq stack.
>   */
>  SYM_FUNC_START(call_on_irq_stack)
> -	save_and_disable_daif x9
> +	save_and_disable_exceptions x9, x10
>  #ifdef CONFIG_SHADOW_CALL_STACK
>  	get_current_task x16
>  	scs_save x16
> @@ -881,10 +881,10 @@ SYM_FUNC_START(call_on_irq_stack)
>  
>  	/* Move to the new stack and call the function there */
>  	add	sp, x16, #IRQ_STACK_SIZE
> -	restore_irq x9
> +	restore_exceptions x9
>  	blr	x1
>  
> -	save_and_disable_daif x9
> +	save_and_disable_exceptions x9, x10
>  	/*
>  	 * Restore the SP from the FP, and restore the FP and LR from the frame
>  	 * record.
> @@ -892,7 +892,7 @@ SYM_FUNC_START(call_on_irq_stack)
>  	mov	sp, x29
>  	ldp	x29, x30, [sp], #16
>  	scs_load_current
> -	restore_irq x9
> +	restore_exceptions x9
>  	ret
>  SYM_FUNC_END(call_on_irq_stack)
>  NOKPROBE(call_on_irq_stack)

Re: [RFC PATCH 31/36] arm64: nmi: Add handling of superpriority interrupts as NMIs

[Jinjie Ruan]

On 7/9/2026 8:13 PM, Vladimir Murzin wrote:
> From: Mark Brown <broonie@kernel.org>
> 
> Our goal with superpriority interrupts is to use them as NMIs, taking
> advantage of the much smaller regions where they are masked to allow
> prompt handling of the most time-critical interrupts.
> 
> When an interrupt is configured with superpriority, we enter EL1 as we
> do for any other interrupt. The presence of a superpriority interrupt
> is indicated by a status bit in ISR_EL1. We check this bit before
> unmasking interrupts in elX_interrupt(), and if a superpriority
> interrupt is pending, we handle it as an NMI. Otherwise, normal
> interrupts are handled as usual.
> 
> Since superpriority interrupts are always handled as NMIs, the
> interrupt controller can rely on in_nmi() to distinguish them from
> ordinary interrupts.
> 
> Enable IPIs to use superpriority interrupts as NMIs, matching the
> existing pseudo-NMI behaviour.
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
> ---
>  arch/arm64/include/asm/entry-common.h |  7 ++++
>  arch/arm64/kernel/entry-common.c      | 59 ++++++++++++++++++---------
>  arch/arm64/kernel/smp.c               |  2 +-
>  3 files changed, 48 insertions(+), 20 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
> index 73d82a8d8e95..0681ba91ac3b 100644
> --- a/arch/arm64/include/asm/entry-common.h
> +++ b/arch/arm64/include/asm/entry-common.h
> @@ -37,6 +37,13 @@ static inline bool arch_irqentry_exit_need_resched(void)
>  	if (system_uses_irq_prio_masking() && read_sysreg(daif))
>  		return false;
>  
> +	/*
> +	 * If AllInt is set then we must have handled an NMI, so skip
> +	 * preemption
> +	 */
> +	if (system_uses_nmi() && read_sysreg_s(SYS_ALLINT))
> +		return false;
> +
>  	/*
>  	 * Preempting a task from an IRQ means we leave copies of PSTATE
>  	 * on the stack. cpufeature's enable calls may modify PSTATE, but
> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> index a13653b228b7..de71d5a3a6a1 100644
> --- a/arch/arm64/kernel/entry-common.c
> +++ b/arch/arm64/kernel/entry-common.c
> @@ -525,8 +525,8 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
>  	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
> -static __always_inline void __el1_pnmi(struct pt_regs *regs,
> -				       void (*handler)(struct pt_regs *))
> +static __always_inline void __el1_nmi(struct pt_regs *regs,
> +				      void (*handler)(struct pt_regs *))
>  {
>  	arm64_exc_hwstate_t hwstate;
>  	irqentry_state_t state;
> @@ -545,7 +545,10 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
>  
>  	state = arm64_enter_from_kernel_mode(regs);
>  
> -	arm64_unmask_exc_context(NONMI_CONTEXT);
> +	if (system_uses_nmi())
> +		arm64_unmask_exc_context(NOIRQ_CONTEXT);
> +	else
> +		arm64_unmask_exc_context(NONMI_CONTEXT);

This looks strange; whether it's a pseudo NMI or FEAT_NMI, the behavior
should be consistent.

>  
>  	irq_enter_rcu();
>  	do_interrupt_handler(regs, handler);
> @@ -565,8 +568,11 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
>  static void noinstr el1_interrupt(struct pt_regs *regs,
>  				  void (*handler)(struct pt_regs *))
>  {
> -	if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && regs_irqs_disabled(regs))
> -		__el1_pnmi(regs, handler);
> +	/* Is there a NMI to handle? */
> +	if (regs_irqs_disabled(regs))
> +		__el1_nmi(regs, handler);
> +	else if (system_uses_nmi() && (read_sysreg(isr_el1) & (ISR_EL1_IS | ISR_EL1_FS)))
> +		__el1_nmi(regs, handler);
>  	else
>  		__el1_irq(regs, handler);
>  
> @@ -906,24 +912,39 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
>  static void noinstr el0_interrupt(struct pt_regs *regs,
>  				  void (*handler)(struct pt_regs *))
>  {
> -	arm64_enter_from_user_mode(regs);
> -
> -	arm64_unmask_exc_context(NONMI_CONTEXT);
> -
>  	if (regs->pc & BIT(55))
>  		arm64_apply_bp_hardening();
>  
> -	irq_enter_rcu();
> -	do_interrupt_handler(regs, handler);
> -	irq_exit_rcu();
> +	/* Is there a NMI to handle? */
> +	if (system_uses_nmi() && (read_sysreg(isr_el1) & (ISR_EL1_IS | ISR_EL1_FS))) {
> +		irqentry_state_t state;
> +		arm64_exc_hwstate_t hwstate;
> +
> +		state = irqentry_nmi_enter(regs);
> +		hwstate = arm64_unmask_exc_context(NONMI_CONTEXT);
> +		do_interrupt_handler(regs, handler);
> +		arm64_mask_exc_context(hwstate);
> +		irqentry_nmi_exit(regs, state);
> +	} else {
> +		arm64_enter_from_user_mode(regs);
> +
> +		if (system_uses_nmi())
> +			arm64_unmask_exc_context(NOIRQ_CONTEXT);
> +		else
> +			arm64_unmask_exc_context(NONMI_CONTEXT);
> +
> +		irq_enter_rcu();
> +		do_interrupt_handler(regs, handler);
> +		irq_exit_rcu();
> +		/*
> +		 * For the same reason as in el1_irq() we effectivly
> +		 * have NOIRQ_CONTEXT on return from handler - keep
> +		 * track of it
> +		 */
> +		arm64_debug_exc_context(NOIRQ_CONTEXT);
> +		arm64_exit_to_user_mode(regs, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
> +	}
>  
> -	/*
> -	 * For the same reason as in el1_irq() we effectivly
> -	 * have NOIRQ_CONTEXT on return from handler - keep
> -	 * track of it
> -	 */
> -	arm64_debug_exc_context(NOIRQ_CONTEXT);
> -	arm64_exit_to_user_mode(regs, arm64_exc_hwstate_of_context(NOIRQ_CONTEXT));
>  	arm64_debug_exc_context(CRITICAL_CONTEXT);
>  }
>  
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index 257d50529d14..80e35a8e5c8f 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -1035,7 +1035,7 @@ static void smp_cross_call(const struct cpumask *target, unsigned int ipinr)
>  
>  static bool ipi_should_be_nmi(enum ipi_msg_type ipi)
>  {
> -	if (!system_uses_irq_prio_masking())
> +	if (!system_uses_nmi() && !system_uses_irq_prio_masking())
>  		return false;
>  
>  	switch (ipi) {