[PATCH 00/17] arm64: Support FEAT_NMI and Rework Exception Masking

[PATCH 00/17] arm64: Support FEAT_NMI and Rework Exception Masking

[Jinjie Ruan]

Hi all,

This patch series implements support for the ARMv8.8-A/v9.3-A
hardware NMI extension (FEAT_NMI), introducing the use of PSTATE.ALLINT
to manage superpriority interrupts on arm64.

Background and Current Status
-----------------------------

The hardware ecosystem is already mature and fully ready for FEAT_NMI:

- Production Hardware: Huawei's Kunpeng servers (such as the HIP12 based
  platforms) already feature native hardware support for FEAT_NMI.

- Emulation: QEMU has integrated robust FEAT_NMI emulation[1] support
  since 2024.

Despite the hardware readiness, upstream support for FEAT_NMI has been
delayed. Previous patch[2] attempts tried to bundle ALLINT management
directly into the existing DAIF abstraction layers. As Mark Rutland
pointed out[3], pretending ALLINT or pseudo-NMI (GIC PMR) is part of
DAIF creates convoluted, unmaintainable hacks that frequently fail to
handle complex edge cases correctly (such as state escape during
context switching).

Reworking Exception Masking (Mark's Feedback)
---------------------------------------------

Following Mark Rutland's strong recommendation ("We must clean up
the existing approach before we add the real NMI support"), this series
does not simply stack FEAT_NMI on top of the old framework. Instead, it
completely reworks how the arm64 kernel manages abstract and logical
exception masks.

Per Mark's guidelines, this series achieves the following architectural
improvements:

1. Entry/Exit Specific Helpers (a):

   Introduces abstract exception mask helpers specifically for exception
   boundaries. They handle unified unmask-at-entry and mask-at-exit
   behaviors. This decouples the entry/exit paths from raw DAIF
   manipulation. In this series, these helpers are first refactored to
   manage DAIF + PMR cleanly, preparing the ground before any FEAT_NMI
   code is introduced.

2. Logical Exception Mask Separation (b):

   Introduces a decoupled logical mask tracking mechanism that treats DAIF,
   PMR, and ALLINT as separate, distinct elements. This enables accurate
   irqflag tracking and debug assertions to save, restore, and validate all
   elements without forcing them to fake or pollute a traditional DAIF
   layout.

Production Bug Fixes & Integration
-----------------------------------

On top of this solid architectural foundation, this series adds the actual
support for FEAT_NMI (ALLINT management). Crucially, during baisc testing
and validation on production Kunpeng (HIP12) servers, we identified and
resolved several critical bugs.

The series is structured as follows:

- Patches 1-5: Clean up and rework the existing DAIF/PMR masking into
  separate logical exception helpers (Pre-requisite refactoring).

- Patches 6-17: Add FEAT_NMI support for ARM64, including specific
  stability fixes found on Kunpeng hardware and QEmu.

Any feedback, testing, or review, especially regarding the exception
masking refactoring, is highly appreciated.

[1]: https://lore.kernel.org/all/20240407081733.3231820-1-ruanjinjie@huawei.com/
[2]: https://lore.kernel.org/linux-arm-kernel/20221112151708.175147-1-broonie@kernel.org/
[3]: https://lore.kernel.org/linux-arm-kernel/Y5c9SLeJacLYHmP7@FVFF77S0Q05N/

Jinjie Ruan (5):
  arm64: Move DAIF macros to ptrace.h and use them centrally
  arm64: Rework exception masking into abstract logical mask
  arm64: entry: arm64: entry: Move DAIF masking for EL1 exit to C code
  arm64: entry: Add entry-specific helpers
  arm64: Introduce helpers for restoring standard exception masks

Lorenzo Pieralisi (1):
  irqchip/gic-v3: Implement FEAT_GICv3_NMI support

Mark Brown (11):
  arm64/booting: Document boot requirements for FEAT_NMI
  arm64/sysreg: Add definitions for immediate versions of MSR ALLINT
  arm64/hyp-stub: Enable access to ALLINT
  arm64/idreg: Add an override for FEAT_NMI
  arm64/cpufeature: Detect PE support for FEAT_NMI
  KVM: arm64: Hide FEAT_NMI from guests
  arm64/nmi: Manage masking for superpriority interrupts along with DAIF
  arm64/entry: Don't call preempt_schedule_irq() with NMIs masked
  arm64/irq: Document handling of FEAT_NMI in irqflags.h
  arm64/nmi: Add handling of superpriority interrupts as NMIs
  arm64/nmi: Add Kconfig for NMI

 Documentation/arch/arm64/booting.rst     |   6 +
 arch/arm64/Kconfig                       |  17 ++
 arch/arm64/include/asm/arch_gicv3.h      |   7 +-
 arch/arm64/include/asm/assembler.h       |  17 +-
 arch/arm64/include/asm/cpufeature.h      |   5 +
 arch/arm64/include/asm/cpuidle.h         |  30 ++-
 arch/arm64/include/asm/daifflags.h       | 144 --------------
 arch/arm64/include/asm/efi.h             |  22 ++-
 arch/arm64/include/asm/el2_setup.h       |  13 ++
 arch/arm64/include/asm/entry-common.h    |  11 +-
 arch/arm64/include/asm/exception_masks.h | 232 +++++++++++++++++++++++
 arch/arm64/include/asm/irq.h             |   2 +
 arch/arm64/include/asm/irqflags.h        |  10 +
 arch/arm64/include/asm/kvm_host.h        |   2 +-
 arch/arm64/include/asm/mmu_context.h     |   2 +-
 arch/arm64/include/asm/nmi.h             |  23 +++
 arch/arm64/include/asm/ptrace.h          |  11 +-
 arch/arm64/include/asm/sysreg.h          |   2 +
 arch/arm64/include/uapi/asm/ptrace.h     |   1 +
 arch/arm64/kernel/acpi.c                 |  14 +-
 arch/arm64/kernel/cpufeature.c           |  58 +++++-
 arch/arm64/kernel/debug-monitors.c       |   9 +-
 arch/arm64/kernel/entry-common.c         | 167 +++++++++++-----
 arch/arm64/kernel/entry.S                |   4 -
 arch/arm64/kernel/hibernate.c            |  10 +-
 arch/arm64/kernel/idle.c                 |   7 +-
 arch/arm64/kernel/irq.c                  |  36 +++-
 arch/arm64/kernel/machine_kexec.c        |   4 +-
 arch/arm64/kernel/pi/idreg-override.c    |   1 +
 arch/arm64/kernel/probes/kprobes.c       |   9 +-
 arch/arm64/kernel/process.c              |   7 +-
 arch/arm64/kernel/setup.c                |   4 +-
 arch/arm64/kernel/signal.c               |   2 +-
 arch/arm64/kernel/smp.c                  |  22 +--
 arch/arm64/kernel/suspend.c              |  15 +-
 arch/arm64/kernel/traps.c                |   2 +-
 arch/arm64/kvm/emulate-nested.c          |   6 +-
 arch/arm64/kvm/hyp/include/hyp/switch.h  |   6 +
 arch/arm64/kvm/hyp/nvhe/host.S           |   4 +-
 arch/arm64/kvm/hyp/nvhe/hyp-init.S       |   3 +-
 arch/arm64/kvm/hyp/nvhe/hyp-main.c       |   4 +-
 arch/arm64/kvm/hyp/vgic-v3-sr.c          |   6 +-
 arch/arm64/kvm/hyp/vhe/switch.c          |   6 +-
 arch/arm64/kvm/reset.c                   |   6 +-
 arch/arm64/mm/fault.c                    |   2 +-
 arch/arm64/mm/mmu.c                      |   6 +-
 arch/arm64/tools/cpucaps                 |   2 +
 drivers/firmware/psci/psci.c             |   7 +-
 drivers/irqchip/irq-gic-v3.c             | 152 +++++++++++++--
 include/linux/irqchip/arm-gic-v3.h       |   4 +
 50 files changed, 822 insertions(+), 320 deletions(-)
 delete mode 100644 arch/arm64/include/asm/daifflags.h
 create mode 100644 arch/arm64/include/asm/exception_masks.h
 create mode 100644 arch/arm64/include/asm/nmi.h

-- 
2.34.1

[PATCH 01/17] arm64: Move DAIF macros to ptrace.h and use them centrally

[Jinjie Ruan]

Currently, various subsystems and low-level headers explicitly combine
PSR_I_BIT, PSR_F_BIT, and other DAIF bits (such as D and A) to mask
or unmask interrupts and exceptions. This hardcoded approach reduces
code readability and increases the maintenance burden when managing
processor contexts.

So move the definitions of DAIF_PROCCTX, DAIF_PROCCTX_NOIRQ, DAIF_ERRCTX,
and DAIF_MASK from <asm/daifflags.h> to <asm/ptrace.h>. And clean this up
by replacing explicit bitmask combinations with the unified
macros defined in <asm/daifflags.h>. Specifically:

- Replace `PSR_I_BIT | PSR_F_BIT` with `DAIF_PROCCTX_NOIRQ` where
  normal interrupts are disabled but NMIs/asynchronous errors can
  still be taken.

- Replace the full explicit DAIF bit mask with `DAIF_MASK`.

This unifies DAIF flag manipulation across kvm, cpuidle, daifflags, and
efi subsystems, making the code more readable.

Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/cpuidle.h   |  3 ++-
 arch/arm64/include/asm/daifflags.h | 12 +++---------
 arch/arm64/include/asm/efi.h       |  2 +-
 arch/arm64/include/asm/ptrace.h    | 11 +++++++----
 arch/arm64/kvm/emulate-nested.c    |  6 +++---
 arch/arm64/kvm/hyp/nvhe/host.S     |  4 ++--
 arch/arm64/kvm/hyp/nvhe/hyp-init.S |  3 ++-
 arch/arm64/kvm/hyp/nvhe/hyp-main.c |  4 ++--
 arch/arm64/kvm/reset.c             |  6 ++----
 9 files changed, 24 insertions(+), 27 deletions(-)

diff --git a/arch/arm64/include/asm/cpuidle.h b/arch/arm64/include/asm/cpuidle.h
index 2047713e097d..64ddb533d3ec 100644
--- a/arch/arm64/include/asm/cpuidle.h
+++ b/arch/arm64/include/asm/cpuidle.h
@@ -6,6 +6,7 @@
 
 #ifdef CONFIG_ARM64_PSEUDO_NMI
 #include <asm/arch_gicv3.h>
+#include <asm/ptrace.h>
 
 struct arm_cpuidle_irq_context {
 	unsigned long pmr;
@@ -17,7 +18,7 @@ struct arm_cpuidle_irq_context {
 		struct arm_cpuidle_irq_context *c = __c;		\
 		if (system_uses_irq_prio_masking()) {			\
 			c->daif_bits = read_sysreg(daif);		\
-			write_sysreg(c->daif_bits | PSR_I_BIT | PSR_F_BIT, \
+			write_sysreg(c->daif_bits | DAIF_PROCCTX_NOIRQ, \
 				     daif);				\
 			c->pmr = gic_read_pmr();			\
 			gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); \
diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
index 795b35128467..3a4b63a948ac 100644
--- a/arch/arm64/include/asm/daifflags.h
+++ b/arch/arm64/include/asm/daifflags.h
@@ -12,12 +12,6 @@
 #include <asm/cpufeature.h>
 #include <asm/ptrace.h>
 
-#define DAIF_PROCCTX		0
-#define DAIF_PROCCTX_NOIRQ	(PSR_I_BIT | PSR_F_BIT)
-#define DAIF_ERRCTX		(PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
-#define DAIF_MASK		(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
-
-
 /* mask/save/unmask/restore all exceptions, including interrupts. */
 static __always_inline void local_daif_mask(void)
 {
@@ -47,7 +41,7 @@ static __always_inline unsigned long local_daif_save_flags(void)
 	if (system_uses_irq_prio_masking()) {
 		/* If IRQs are masked with PMR, reflect it in the flags */
 		if (read_sysreg_s(SYS_ICC_PMR_EL1) != GIC_PRIO_IRQON)
-			flags |= PSR_I_BIT | PSR_F_BIT;
+			flags |= DAIF_PROCCTX_NOIRQ;
 	}
 
 	return flags;
@@ -69,7 +63,7 @@ static __always_inline void local_daif_restore(unsigned long flags)
 	bool irq_disabled = flags & PSR_I_BIT;
 
 	WARN_ON(system_has_prio_mask_debugging() &&
-		(read_sysreg(daif) & (PSR_I_BIT | PSR_F_BIT)) != (PSR_I_BIT | PSR_F_BIT));
+		(read_sysreg(daif) & DAIF_PROCCTX_NOIRQ) != DAIF_PROCCTX_NOIRQ);
 
 	if (!irq_disabled) {
 		trace_hardirqs_on();
@@ -86,7 +80,7 @@ static __always_inline void local_daif_restore(unsigned long flags)
 			 * If interrupts are disabled but we can take
 			 * asynchronous errors, we can take NMIs
 			 */
-			flags &= ~(PSR_I_BIT | PSR_F_BIT);
+			flags &= ~DAIF_PROCCTX_NOIRQ;
 			pmr = GIC_PRIO_IRQOFF;
 		} else {
 			pmr = GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET;
diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h
index e8a9783235cb..ae2a24868ee7 100644
--- a/arch/arm64/include/asm/efi.h
+++ b/arch/arm64/include/asm/efi.h
@@ -48,7 +48,7 @@ void arch_efi_call_virt_teardown(void);
 	(efi_rt_stack_top != NULL &&					\
 	 on_task_stack(current, READ_ONCE(efi_rt_stack_top[-1]), 1))
 
-#define ARCH_EFI_IRQ_FLAGS_MASK (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
+#define ARCH_EFI_IRQ_FLAGS_MASK (DAIF_MASK)
 
 /*
  * Even when Linux uses IRQ priorities for IRQ disabling, EFI does not.
diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index 39582511ad72..f33f75881c9a 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -12,14 +12,17 @@
 
 #include <uapi/asm/ptrace.h>
 
+#define DAIF_PROCCTX		0
+#define DAIF_PROCCTX_NOIRQ	(PSR_I_BIT | PSR_F_BIT)
+#define DAIF_ERRCTX		(PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
+#define DAIF_MASK		(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT)
+
 /* Current Exception Level values, as contained in CurrentEL */
 #define CurrentEL_EL1		(1 << 2)
 #define CurrentEL_EL2		(2 << 2)
 
-#define INIT_PSTATE_EL1 \
-	(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL1h)
-#define INIT_PSTATE_EL2 \
-	(PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT | PSR_MODE_EL2h)
+#define INIT_PSTATE_EL1	(DAIF_MASK | PSR_MODE_EL1h)
+#define INIT_PSTATE_EL2	(DAIF_MASK | PSR_MODE_EL2h)
 
 #include <linux/irqchip/arm-gic-v3-prio.h>
 
diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
index e688bc5139c1..356b67aa8dc0 100644
--- a/arch/arm64/kvm/emulate-nested.c
+++ b/arch/arm64/kvm/emulate-nested.c
@@ -9,6 +9,7 @@
 
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_nested.h>
+#include <asm/ptrace.h>
 
 #include "hyp/include/hyp/adjust_pc.h"
 
@@ -2754,9 +2755,8 @@ static u64 kvm_check_illegal_exception_return(struct kvm_vcpu *vcpu, u64 spsr)
 		 */
 		spsr = *vcpu_cpsr(vcpu);
 
-		spsr &= (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT |
-			 PSR_N_BIT | PSR_Z_BIT | PSR_C_BIT | PSR_V_BIT |
-			 PSR_MODE_MASK | PSR_MODE32_BIT);
+		spsr &= (DAIF_MASK | PSR_N_BIT | PSR_Z_BIT | PSR_C_BIT |
+			 PSR_V_BIT | PSR_MODE_MASK | PSR_MODE32_BIT);
 		spsr |= PSR_IL_BIT;
 	}
 
diff --git a/arch/arm64/kvm/hyp/nvhe/host.S b/arch/arm64/kvm/hyp/nvhe/host.S
index 9393fe3ea6a1..cbe2a616c726 100644
--- a/arch/arm64/kvm/hyp/nvhe/host.S
+++ b/arch/arm64/kvm/hyp/nvhe/host.S
@@ -11,6 +11,7 @@
 #include <asm/kvm_asm.h>
 #include <asm/kvm_mmu.h>
 #include <asm/kvm_ptrauth.h>
+#include <asm/ptrace.h>
 
 	.text
 
@@ -111,8 +112,7 @@ SYM_FUNC_END(__host_enter)
  */
 SYM_FUNC_START(__hyp_do_panic)
 	/* Prepare and exit to the host's panic function. */
-	mov	lr, #(PSR_F_BIT | PSR_I_BIT | PSR_A_BIT | PSR_D_BIT |\
-		      PSR_MODE_EL1h)
+	mov	lr, #(DAIF_MASK | PSR_MODE_EL1h)
 	msr	spsr_el2, lr
 	adr_l	lr, nvhe_hyp_panic_handler
 	hyp_kimg_va lr, x6
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S b/arch/arm64/kvm/hyp/nvhe/hyp-init.S
index 89cb553be1e5..26ea02e7f5fd 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S
@@ -15,6 +15,7 @@
 #include <asm/kvm_asm.h>
 #include <asm/kvm_mmu.h>
 #include <asm/pgtable-hwdef.h>
+#include <asm/ptrace.h>
 #include <asm/sysreg.h>
 #include <asm/virt.h>
 
@@ -224,7 +225,7 @@ SYM_CODE_START(__kvm_handle_stub_hvc)
 
 	/* This is where we're about to jump, staying at EL2 */
 	msr	elr_el2, x1
-	mov	x0, #(PSR_F_BIT | PSR_I_BIT | PSR_A_BIT | PSR_D_BIT | PSR_MODE_EL2h)
+	mov	x0, #(DAIF_MASK | PSR_MODE_EL2h)
 	msr	spsr_el2, x0
 
 	/* Shuffle the arguments, and don't come back */
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
index d3c69de698f4..69dfbbe109ad 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c
+++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c
@@ -8,6 +8,7 @@
 #include <hyp/switch.h>
 
 #include <asm/pgtable-types.h>
+#include <asm/ptrace.h>
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_host.h>
@@ -859,8 +860,7 @@ void inject_host_exception(u64 esr)
 	}
 
 	spsr_el2 &= spsr_mask;
-	spsr_el2 |= PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT |
-		    PSR_MODE_EL1h;
+	spsr_el2 |= DAIF_MASK | PSR_MODE_EL1h;
 
 	sctlr = read_sysreg_el1(SYS_SCTLR);
 	if (!(sctlr & SCTLR_EL1_SPAN))
diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
index b963fd975aac..0e1167eee5c0 100644
--- a/arch/arm64/kvm/reset.c
+++ b/arch/arm64/kvm/reset.c
@@ -37,11 +37,9 @@ unsigned int __ro_after_init kvm_host_sve_max_vl;
 /*
  * ARMv8 Reset Values
  */
-#define VCPU_RESET_PSTATE_EL1	(PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
-				 PSR_F_BIT | PSR_D_BIT)
+#define VCPU_RESET_PSTATE_EL1	(PSR_MODE_EL1h | DAIF_MASK)
 
-#define VCPU_RESET_PSTATE_EL2	(PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
-				 PSR_F_BIT | PSR_D_BIT)
+#define VCPU_RESET_PSTATE_EL2	(PSR_MODE_EL2h | DAIF_MASK)
 
 #define VCPU_RESET_PSTATE_SVC	(PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
 				 PSR_AA32_I_BIT | PSR_AA32_F_BIT)
-- 
2.34.1

[PATCH 02/17] arm64: Rework exception masking into abstract logical mask

[Jinjie Ruan]

The current pseudo-NMI implementation in arm64 bodges the GICv3 PMR
state into the DAIF management code. This approach forces local_daif_save()
and local_daif_restore() to pretend that PMR is a part of DAIF. This has
proven to be convoluted, difficult to maintain, and prone to architectural
corner cases where it cannot do the right thing (e.g., faking up DAIF
elements during irqflags/lockdep tracking).

As we prepare for the hardware-based NMI (FEAT_NMI, which use ALLINT
to mask NMI) support, continuing down this path will significantly
increase technical debt. We must clean up the exception masking
abstraction ahead of ALLINT integration.

Introduce a new abstract `struct exception_mask` that explicitly treats
DAIF, PMR, and ALLINT as separate, independent elements of the logical
exception state. This allows the kernel to track and manipulate the exact
hardware masking state cleanly without faking up DAIF flags.

Introduce the following new APIs to replace the legacy DAIF-centric
helpers:

 - arm64_make_{procctx,errctx,noirq}_mask() to generate logical masks.

 - local_exception_save_and_mask() / local_exception_restore() to replace
   the old local_daif_save() / local_daif_restore() pattern.

 - local_exception_inherit() to replace local_daif_inherit().

Mechanically convert all existing callers across the arm64 architecture
to use the new logical exception mask helpers. The conversion spans:

- Core exception entry/exit handling and traps.

- KVM nVHE/VHE hypervisor switch and GICv3 save/restore code

- CPU idle, suspend, hibernate, and PSCI firmware invocations.

- Subsystems including SMP, ACPI, kprobes, and debug monitors

Link: https://lore.kernel.org/linux-arm-kernel/Y5c9SLeJacLYHmP7@FVFF77S0Q05N/
Suggested-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/cpuidle.h         |  29 ++---
 arch/arm64/include/asm/daifflags.h       | 138 ----------------------
 arch/arm64/include/asm/entry-common.h    |   2 +-
 arch/arm64/include/asm/exception_masks.h | 141 +++++++++++++++++++++++
 arch/arm64/include/asm/kvm_host.h        |   2 +-
 arch/arm64/include/asm/mmu_context.h     |   2 +-
 arch/arm64/kernel/acpi.c                 |  14 +--
 arch/arm64/kernel/debug-monitors.c       |   9 +-
 arch/arm64/kernel/entry-common.c         |  74 ++++++------
 arch/arm64/kernel/hibernate.c            |  10 +-
 arch/arm64/kernel/idle.c                 |   7 +-
 arch/arm64/kernel/irq.c                  |   4 +-
 arch/arm64/kernel/machine_kexec.c        |   4 +-
 arch/arm64/kernel/probes/kprobes.c       |   2 +-
 arch/arm64/kernel/setup.c                |   4 +-
 arch/arm64/kernel/signal.c               |   2 +-
 arch/arm64/kernel/smp.c                  |  22 ++--
 arch/arm64/kernel/suspend.c              |  15 ++-
 arch/arm64/kernel/traps.c                |   2 +-
 arch/arm64/kvm/hyp/vgic-v3-sr.c          |   6 +-
 arch/arm64/kvm/hyp/vhe/switch.c          |   6 +-
 arch/arm64/mm/fault.c                    |   2 +-
 arch/arm64/mm/mmu.c                      |   6 +-
 drivers/firmware/psci/psci.c             |   7 +-
 24 files changed, 254 insertions(+), 256 deletions(-)
 delete mode 100644 arch/arm64/include/asm/daifflags.h
 create mode 100644 arch/arm64/include/asm/exception_masks.h

diff --git a/arch/arm64/include/asm/cpuidle.h b/arch/arm64/include/asm/cpuidle.h
index 64ddb533d3ec..4c19d5b79678 100644
--- a/arch/arm64/include/asm/cpuidle.h
+++ b/arch/arm64/include/asm/cpuidle.h
@@ -6,37 +6,30 @@
 
 #ifdef CONFIG_ARM64_PSEUDO_NMI
 #include <asm/arch_gicv3.h>
+#include <asm/exception_masks.h>
 #include <asm/ptrace.h>
 
-struct arm_cpuidle_irq_context {
-	unsigned long pmr;
-	unsigned long daif_bits;
-};
-
-#define arm_cpuidle_save_irq_context(__c)				\
+#define arm_cpuidle_save_irq_context(__m)				\
 	do {								\
-		struct arm_cpuidle_irq_context *c = __c;		\
+		struct exception_mask *m = __m;				\
 		if (system_uses_irq_prio_masking()) {			\
-			c->daif_bits = read_sysreg(daif);		\
-			write_sysreg(c->daif_bits | DAIF_PROCCTX_NOIRQ, \
+			local_exception_save_mask(m);			\
+			write_sysreg(m->daif | DAIF_PROCCTX_NOIRQ,	\
 				     daif);				\
-			c->pmr = gic_read_pmr();			\
 			gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); \
 		}							\
 	} while (0)
 
-#define arm_cpuidle_restore_irq_context(__c)				\
+#define arm_cpuidle_restore_irq_context(__m)				\
 	do {								\
-		struct arm_cpuidle_irq_context *c = __c;		\
+		struct exception_mask *m = __m;				\
 		if (system_uses_irq_prio_masking()) {			\
-			gic_write_pmr(c->pmr);				\
-			write_sysreg(c->daif_bits, daif);		\
+			gic_write_pmr(m->pmr);				\
+			write_sysreg(m->daif, daif);			\
 		}							\
 	} while (0)
 #else
-struct arm_cpuidle_irq_context { };
-
-#define arm_cpuidle_save_irq_context(c)		(void)c
-#define arm_cpuidle_restore_irq_context(c)	(void)c
+#define arm_cpuidle_save_irq_context(m)		((void)m)
+#define arm_cpuidle_restore_irq_context(m)	((void)m)
 #endif
 #endif
diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
deleted file mode 100644
index 3a4b63a948ac..000000000000
--- a/arch/arm64/include/asm/daifflags.h
+++ /dev/null
@@ -1,138 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (C) 2017 ARM Ltd.
- */
-#ifndef __ASM_DAIFFLAGS_H
-#define __ASM_DAIFFLAGS_H
-
-#include <linux/irqflags.h>
-
-#include <asm/arch_gicv3.h>
-#include <asm/barrier.h>
-#include <asm/cpufeature.h>
-#include <asm/ptrace.h>
-
-/* mask/save/unmask/restore all exceptions, including interrupts. */
-static __always_inline void local_daif_mask(void)
-{
-	WARN_ON(system_has_prio_mask_debugging() &&
-		(read_sysreg_s(SYS_ICC_PMR_EL1) == (GIC_PRIO_IRQOFF |
-						    GIC_PRIO_PSR_I_SET)));
-
-	asm volatile(
-		"msr	daifset, #0xf		// local_daif_mask\n"
-		:
-		:
-		: "memory");
-
-	/* Don't really care for a dsb here, we don't intend to enable IRQs */
-	if (system_uses_irq_prio_masking())
-		gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
-
-	trace_hardirqs_off();
-}
-
-static __always_inline unsigned long local_daif_save_flags(void)
-{
-	unsigned long flags;
-
-	flags = read_sysreg(daif);
-
-	if (system_uses_irq_prio_masking()) {
-		/* If IRQs are masked with PMR, reflect it in the flags */
-		if (read_sysreg_s(SYS_ICC_PMR_EL1) != GIC_PRIO_IRQON)
-			flags |= DAIF_PROCCTX_NOIRQ;
-	}
-
-	return flags;
-}
-
-static __always_inline unsigned long local_daif_save(void)
-{
-	unsigned long flags;
-
-	flags = local_daif_save_flags();
-
-	local_daif_mask();
-
-	return flags;
-}
-
-static __always_inline void local_daif_restore(unsigned long flags)
-{
-	bool irq_disabled = flags & PSR_I_BIT;
-
-	WARN_ON(system_has_prio_mask_debugging() &&
-		(read_sysreg(daif) & DAIF_PROCCTX_NOIRQ) != DAIF_PROCCTX_NOIRQ);
-
-	if (!irq_disabled) {
-		trace_hardirqs_on();
-
-		if (system_uses_irq_prio_masking()) {
-			gic_write_pmr(GIC_PRIO_IRQON);
-			pmr_sync();
-		}
-	} else if (system_uses_irq_prio_masking()) {
-		u64 pmr;
-
-		if (!(flags & PSR_A_BIT)) {
-			/*
-			 * If interrupts are disabled but we can take
-			 * asynchronous errors, we can take NMIs
-			 */
-			flags &= ~DAIF_PROCCTX_NOIRQ;
-			pmr = GIC_PRIO_IRQOFF;
-		} else {
-			pmr = GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET;
-		}
-
-		/*
-		 * There has been concern that the write to daif
-		 * might be reordered before this write to PMR.
-		 * From the ARM ARM DDI 0487D.a, section D1.7.1
-		 * "Accessing PSTATE fields":
-		 *   Writes to the PSTATE fields have side-effects on
-		 *   various aspects of the PE operation. All of these
-		 *   side-effects are guaranteed:
-		 *     - Not to be visible to earlier instructions in
-		 *       the execution stream.
-		 *     - To be visible to later instructions in the
-		 *       execution stream
-		 *
-		 * Also, writes to PMR are self-synchronizing, so no
-		 * interrupts with a lower priority than PMR is signaled
-		 * to the PE after the write.
-		 *
-		 * So we don't need additional synchronization here.
-		 */
-		gic_write_pmr(pmr);
-	}
-
-	write_sysreg(flags, daif);
-
-	if (irq_disabled)
-		trace_hardirqs_off();
-}
-
-/*
- * Called by synchronous exception handlers to restore the DAIF bits that were
- * modified by taking an exception.
- */
-static __always_inline void local_daif_inherit(struct pt_regs *regs)
-{
-	unsigned long flags = regs->pstate & DAIF_MASK;
-
-	if (!regs_irqs_disabled(regs))
-		trace_hardirqs_on();
-
-	if (system_uses_irq_prio_masking())
-		gic_write_pmr(regs->pmr);
-
-	/*
-	 * We can't use local_daif_restore(regs->pstate) here as
-	 * system_has_prio_mask_debugging() won't restore the I bit if it can
-	 * use the pmr instead.
-	 */
-	write_sysreg(flags, daif);
-}
-#endif
diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
index cab8cd78f693..3d8b38ce7afb 100644
--- a/arch/arm64/include/asm/entry-common.h
+++ b/arch/arm64/include/asm/entry-common.h
@@ -6,7 +6,7 @@
 #include <linux/thread_info.h>
 
 #include <asm/cpufeature.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/fpsimd.h>
 #include <asm/mte.h>
 #include <asm/stacktrace.h>
diff --git a/arch/arm64/include/asm/exception_masks.h b/arch/arm64/include/asm/exception_masks.h
new file mode 100644
index 000000000000..0a21fb97c3ca
--- /dev/null
+++ b/arch/arm64/include/asm/exception_masks.h
@@ -0,0 +1,141 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2017 ARM Ltd.
+ */
+#ifndef __ASM_EXCEPTION_MASKS_H
+#define __ASM_EXCEPTION_MASKS_H
+
+#include <linux/irqflags.h>
+
+#include <asm/arch_gicv3.h>
+#include <asm/barrier.h>
+#include <asm/cpufeature.h>
+#include <asm/ptrace.h>
+
+/*
+ * Logical exception mask: tracks the three independent exception
+ * masking controls on arm64:
+ *  - DAIF (PSTATE.{D,A,I,F} bits)
+ *  - PMR  (ICC_PMR_EL1)
+ *  - ALLINT (PSTATE.ALLINT)
+ */
+struct exception_mask {
+	unsigned long daif;
+	unsigned long pmr;
+	unsigned long allint;		// for future FEAT_NMI use
+};
+
+static inline struct exception_mask arm64_make_procctx_mask(void)
+{
+	struct exception_mask mask;
+
+	mask.daif = DAIF_PROCCTX;
+	if (system_uses_irq_prio_masking())
+		mask.pmr = GIC_PRIO_IRQON;
+
+	mask.allint = 0;
+
+	return mask;
+}
+
+static inline struct exception_mask arm64_make_errctx_mask(void)
+{
+	struct exception_mask mask;
+
+	mask.daif = DAIF_ERRCTX;
+	if (system_uses_irq_prio_masking())
+		mask.pmr = GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET;
+
+	mask.allint = 0;
+
+	return mask;
+}
+
+static inline struct exception_mask arm64_make_noirq_mask(void)
+{
+	struct exception_mask mask;
+
+	if (system_uses_irq_prio_masking()) {
+		mask.daif = 0;
+		mask.pmr = GIC_PRIO_IRQOFF;
+	} else
+		mask.daif = DAIF_PROCCTX_NOIRQ;
+
+	mask.allint = 0;
+
+	return mask;
+}
+
+/* Mask all exceptions immediately */
+static inline void local_exception_mask(void)
+{
+	WARN_ON(system_has_prio_mask_debugging() &&
+		(read_sysreg_s(SYS_ICC_PMR_EL1) == (GIC_PRIO_IRQOFF |
+						    GIC_PRIO_PSR_I_SET)));
+
+	asm volatile("msr daifset, #0xf" ::: "memory");
+
+	if (system_uses_irq_prio_masking())
+		gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
+
+	trace_hardirqs_off();
+}
+
+static inline void local_exception_save_mask(struct exception_mask *mask)
+{
+	mask->daif = read_sysreg(daif);
+	if (system_uses_irq_prio_masking())
+		mask->pmr = gic_read_pmr();
+
+	mask->allint = 0;
+}
+
+static inline struct exception_mask local_exception_save_and_mask(void)
+{
+	struct exception_mask mask;
+
+	local_exception_save_mask(&mask);
+	local_exception_mask();
+
+	return mask;
+}
+
+static inline void local_exception_restore(const struct exception_mask mask)
+{
+	bool irq_disabled = mask.daif & PSR_I_BIT;
+
+	if (system_uses_irq_prio_masking())
+		irq_disabled |= (mask.pmr == GIC_PRIO_IRQOFF);
+
+	WARN_ON(system_has_prio_mask_debugging() &&
+		(read_sysreg(daif) & DAIF_PROCCTX_NOIRQ) != DAIF_PROCCTX_NOIRQ);
+
+	if (!irq_disabled)
+		trace_hardirqs_on();
+
+	if (system_uses_irq_prio_masking()) {
+		gic_write_pmr(mask.pmr);
+		pmr_sync();
+	}
+
+	write_sysreg(mask.daif, daif);
+
+	if (irq_disabled)
+		trace_hardirqs_off();
+}
+
+/*
+ * Called by synchronous exception handlers to restore the DAIF bits that were
+ * modified by taking an exception.
+ */
+static inline void local_exception_inherit(struct pt_regs *regs)
+{
+	if (!regs_irqs_disabled(regs))
+		trace_hardirqs_on();
+
+	if (system_uses_irq_prio_masking())
+		gic_write_pmr(regs->pmr);
+
+	write_sysreg(regs->pstate & DAIF_MASK, daif);
+}
+#endif /* __ASM_EXCEPTION_MASKS_H */
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index bae2c4f92ef5..8a443a379689 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -23,7 +23,7 @@
 #include <asm/barrier.h>
 #include <asm/cpufeature.h>
 #include <asm/cputype.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/fpsimd.h>
 #include <asm/kvm.h>
 #include <asm/kvm_asm.h>
diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index 803b68758152..6103925f85b7 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -19,7 +19,7 @@
 
 #include <asm/cacheflush.h>
 #include <asm/cpufeature.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/gcs.h>
 #include <asm/proc-fns.h>
 #include <asm/cputype.h>
diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
index 5891f92c2035..4d413419309d 100644
--- a/arch/arm64/kernel/acpi.c
+++ b/arch/arm64/kernel/acpi.c
@@ -33,7 +33,7 @@
 #include <acpi/processor.h>
 #include <asm/cputype.h>
 #include <asm/cpu_ops.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/smp_plat.h>
 
 int acpi_noirq = 1;		/* skip ACPI IRQ initialization */
@@ -388,14 +388,14 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
  */
 int apei_claim_sea(struct pt_regs *regs)
 {
-	int err = -ENOENT;
+	struct exception_mask current_mask;
 	bool return_to_irqs_enabled;
-	unsigned long current_flags;
+	int err = -ENOENT;
 
 	if (!IS_ENABLED(CONFIG_ACPI_APEI_GHES))
 		return err;
 
-	current_flags = local_daif_save_flags();
+	local_exception_save_mask(&current_mask);
 
 	/* current_flags isn't useful here as daif doesn't tell us about pNMI */
 	return_to_irqs_enabled = !irqs_disabled_flags(arch_local_save_flags());
@@ -407,7 +407,7 @@ int apei_claim_sea(struct pt_regs *regs)
 	 * SEA can interrupt SError, mask it and describe this as an NMI so
 	 * that APEI defers the handling.
 	 */
-	local_daif_restore(DAIF_ERRCTX);
+	local_exception_restore(arm64_make_errctx_mask());
 	nmi_enter();
 	err = ghes_notify_sea();
 	nmi_exit();
@@ -418,7 +418,7 @@ int apei_claim_sea(struct pt_regs *regs)
 	 */
 	if (!err) {
 		if (return_to_irqs_enabled) {
-			local_daif_restore(DAIF_PROCCTX_NOIRQ);
+			local_exception_restore(arm64_make_noirq_mask());
 			__irq_enter();
 			irq_work_run();
 			__irq_exit();
@@ -428,7 +428,7 @@ int apei_claim_sea(struct pt_regs *regs)
 		}
 	}
 
-	local_daif_restore(current_flags);
+	local_exception_restore(current_mask);
 
 	return err;
 }
diff --git a/arch/arm64/kernel/debug-monitors.c b/arch/arm64/kernel/debug-monitors.c
index 29307642f4c9..5b53034428e4 100644
--- a/arch/arm64/kernel/debug-monitors.c
+++ b/arch/arm64/kernel/debug-monitors.c
@@ -19,7 +19,7 @@
 
 #include <asm/cpufeature.h>
 #include <asm/cputype.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/debug-monitors.h>
 #include <asm/exception.h>
 #include <asm/kgdb.h>
@@ -40,10 +40,11 @@ u8 debug_monitors_arch(void)
  */
 static void mdscr_write(u64 mdscr)
 {
-	unsigned long flags;
-	flags = local_daif_save();
+	struct exception_mask mask;
+
+	mask = local_exception_save_and_mask();
 	write_sysreg(mdscr, mdscr_el1);
-	local_daif_restore(flags);
+	local_exception_restore(mask);
 }
 NOKPROBE_SYMBOL(mdscr_write);
 
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index ceb4eb11232a..195af3f8001e 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -18,7 +18,7 @@
 #include <linux/thread_info.h>
 
 #include <asm/cpufeature.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/esr.h>
 #include <asm/exception.h>
 #include <asm/fpsimd.h>
@@ -57,7 +57,7 @@ static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
 {
 	local_irq_disable();
 	irqentry_exit_to_kernel_mode_preempt(regs, state);
-	local_daif_mask();
+	local_exception_mask();
 	mte_check_tfsr_exit();
 	irqentry_exit_to_kernel_mode_after_preempt(regs, state);
 }
@@ -86,7 +86,7 @@ static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs
 {
 	local_irq_disable();
 	syscall_exit_to_user_mode_prepare(regs);
-	local_daif_mask();
+	local_exception_mask();
 	sme_exit_to_user_mode();
 	mte_check_tfsr_exit();
 	exit_to_user_mode();
@@ -101,7 +101,7 @@ static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
 {
 	local_irq_disable();
 	irqentry_exit_to_user_mode_prepare(regs);
-	local_daif_mask();
+	local_exception_mask();
 	sme_exit_to_user_mode();
 	mte_check_tfsr_exit();
 	exit_to_user_mode();
@@ -318,7 +318,7 @@ static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	local_exception_inherit(regs);
 	do_mem_abort(far, esr, regs);
 	arm64_exit_to_kernel_mode(regs, state);
 }
@@ -329,7 +329,7 @@ static void noinstr el1_pc(struct pt_regs *regs, unsigned long esr)
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	local_exception_inherit(regs);
 	do_sp_pc_abort(far, esr, regs);
 	arm64_exit_to_kernel_mode(regs, state);
 }
@@ -339,7 +339,7 @@ static void noinstr el1_undef(struct pt_regs *regs, unsigned long esr)
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	local_exception_inherit(regs);
 	do_el1_undef(regs, esr);
 	arm64_exit_to_kernel_mode(regs, state);
 }
@@ -349,7 +349,7 @@ static void noinstr el1_bti(struct pt_regs *regs, unsigned long esr)
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	local_exception_inherit(regs);
 	do_el1_bti(regs, esr);
 	arm64_exit_to_kernel_mode(regs, state);
 }
@@ -359,7 +359,7 @@ static void noinstr el1_gcs(struct pt_regs *regs, unsigned long esr)
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	local_exception_inherit(regs);
 	do_el1_gcs(regs, esr);
 	arm64_exit_to_kernel_mode(regs, state);
 }
@@ -369,7 +369,7 @@ static void noinstr el1_mops(struct pt_regs *regs, unsigned long esr)
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	local_exception_inherit(regs);
 	do_el1_mops(regs, esr);
 	arm64_exit_to_kernel_mode(regs, state);
 }
@@ -434,7 +434,7 @@ static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr)
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_daif_inherit(regs);
+	local_exception_inherit(regs);
 	do_el1_fpac(regs, esr);
 	arm64_exit_to_kernel_mode(regs, state);
 }
@@ -537,7 +537,7 @@ asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
 	unsigned long esr = read_sysreg(esr_el1);
 	irqentry_state_t state;
 
-	local_daif_restore(DAIF_ERRCTX);
+	local_exception_restore(arm64_make_errctx_mask());
 	state = irqentry_nmi_enter(regs);
 	do_serror(regs, esr);
 	irqentry_nmi_exit(regs, state);
@@ -548,7 +548,7 @@ static void noinstr el0_da(struct pt_regs *regs, unsigned long esr)
 	unsigned long far = read_sysreg(far_el1);
 
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_mem_abort(far, esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -566,7 +566,7 @@ static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
 		arm64_apply_bp_hardening();
 
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_mem_abort(far, esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -574,7 +574,7 @@ static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_fpsimd_acc(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_fpsimd_acc(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -582,7 +582,7 @@ static void noinstr el0_fpsimd_acc(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_sve_acc(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_sve_acc(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -590,7 +590,7 @@ static void noinstr el0_sve_acc(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_sme_acc(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_sme_acc(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -598,7 +598,7 @@ static void noinstr el0_sme_acc(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_fpsimd_exc(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_fpsimd_exc(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -606,7 +606,7 @@ static void noinstr el0_fpsimd_exc(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_sys(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_el0_sys(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -619,7 +619,7 @@ static void noinstr el0_pc(struct pt_regs *regs, unsigned long esr)
 		arm64_apply_bp_hardening();
 
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_sp_pc_abort(far, esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -627,7 +627,7 @@ static void noinstr el0_pc(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_sp(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_sp_pc_abort(regs->sp, esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -635,7 +635,7 @@ static void noinstr el0_sp(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_undef(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_el0_undef(regs, esr);
 	arm64_exit_to_user_mode(regs);
 }
@@ -643,7 +643,7 @@ static void noinstr el0_undef(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_bti(struct pt_regs *regs)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_el0_bti(regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -651,7 +651,7 @@ static void noinstr el0_bti(struct pt_regs *regs)
 static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_el0_mops(regs, esr);
 	arm64_exit_to_user_mode(regs);
 }
@@ -659,7 +659,7 @@ static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_gcs(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_el0_gcs(regs, esr);
 	arm64_exit_to_user_mode(regs);
 }
@@ -667,7 +667,7 @@ static void noinstr el0_gcs(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_inv(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	bad_el0_sync(regs, 0, esr);
 	arm64_exit_to_user_mode(regs);
 }
@@ -681,7 +681,7 @@ static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
 	debug_exception_enter(regs);
 	do_breakpoint(esr, regs);
 	debug_exception_exit(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	arm64_exit_to_user_mode(regs);
 }
 
@@ -700,7 +700,7 @@ static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
 	 * the single-step is complete.
 	 */
 	step_done = try_step_suspended_breakpoints(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	if (!step_done)
 		do_el0_softstep(esr, regs);
 	arm64_exit_to_user_mode(regs);
@@ -715,14 +715,14 @@ static void noinstr el0_watchpt(struct pt_regs *regs, unsigned long esr)
 	debug_exception_enter(regs);
 	do_watchpoint(far, esr, regs);
 	debug_exception_exit(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	arm64_exit_to_user_mode(regs);
 }
 
 static void noinstr el0_brk64(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_el0_brk64(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -732,7 +732,7 @@ static void noinstr el0_svc(struct pt_regs *regs)
 	arm64_syscall_enter_from_user_mode(regs);
 	cortex_a76_erratum_1463225_svc_handler();
 	fpsimd_syscall_enter();
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_el0_svc(regs);
 	arm64_syscall_exit_to_user_mode(regs);
 	fpsimd_syscall_exit();
@@ -741,7 +741,7 @@ static void noinstr el0_svc(struct pt_regs *regs)
 static void noinstr el0_fpac(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_el0_fpac(regs, esr);
 	arm64_exit_to_user_mode(regs);
 }
@@ -857,11 +857,11 @@ static void noinstr __el0_error_handler_common(struct pt_regs *regs)
 	irqentry_state_t state;
 
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_ERRCTX);
+	local_exception_restore(arm64_make_errctx_mask());
 	state = irqentry_nmi_enter(regs);
 	do_serror(regs, esr);
 	irqentry_nmi_exit(regs, state);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	arm64_exit_to_user_mode(regs);
 }
 
@@ -874,7 +874,7 @@ asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
 static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_el0_cp15(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -883,7 +883,7 @@ static void noinstr el0_svc_compat(struct pt_regs *regs)
 {
 	arm64_syscall_enter_from_user_mode(regs);
 	cortex_a76_erratum_1463225_svc_handler();
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_el0_svc_compat(regs);
 	arm64_syscall_exit_to_user_mode(regs);
 }
@@ -891,7 +891,7 @@ static void noinstr el0_svc_compat(struct pt_regs *regs)
 static void noinstr el0_bkpt32(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 	do_bkpt32(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 9717568518ba..f6996b22fa29 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -20,7 +20,7 @@
 #include <asm/barrier.h>
 #include <asm/cacheflush.h>
 #include <asm/cputype.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/irqflags.h>
 #include <asm/kexec.h>
 #include <asm/memory.h>
@@ -332,16 +332,16 @@ static void swsusp_mte_restore_tags(void)
 
 int swsusp_arch_suspend(void)
 {
-	int ret = 0;
-	unsigned long flags;
 	struct sleep_stack_data state;
+	struct exception_mask mask;
+	int ret = 0;
 
 	if (cpus_are_stuck_in_kernel()) {
 		pr_err("Can't hibernate: no mechanism to offline secondary CPUs.\n");
 		return -EBUSY;
 	}
 
-	flags = local_daif_save();
+	mask = local_exception_save_and_mask();
 
 	if (__cpu_suspend_enter(&state)) {
 		/* make the crash dump kernel image visible/saveable */
@@ -391,7 +391,7 @@ int swsusp_arch_suspend(void)
 		spectre_v4_enable_mitigation(NULL);
 	}
 
-	local_daif_restore(flags);
+	local_exception_restore(mask);
 
 	return ret;
 }
diff --git a/arch/arm64/kernel/idle.c b/arch/arm64/kernel/idle.c
index 05cfb347ec26..3997e297691e 100644
--- a/arch/arm64/kernel/idle.c
+++ b/arch/arm64/kernel/idle.c
@@ -6,6 +6,7 @@
 #include <linux/cpu.h>
 #include <linux/irqflags.h>
 
+#include <asm/exception_masks.h>
 #include <asm/barrier.h>
 #include <asm/cpuidle.h>
 #include <asm/cpufeature.h>
@@ -22,14 +23,14 @@
  */
 void __cpuidle cpu_do_idle(void)
 {
-	struct arm_cpuidle_irq_context context;
+	struct exception_mask mask;
 
-	arm_cpuidle_save_irq_context(&context);
+	arm_cpuidle_save_irq_context(&mask);
 
 	dsb(sy);
 	wfi();
 
-	arm_cpuidle_restore_irq_context(&context);
+	arm_cpuidle_restore_irq_context(&mask);
 }
 
 /*
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index 9fafd826002b..9e7fb0d22586 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -21,7 +21,7 @@
 #include <linux/seq_file.h>
 #include <linux/smp.h>
 #include <linux/vmalloc.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/exception.h>
 #include <asm/numa.h>
 #include <asm/softirq_stack.h>
@@ -130,6 +130,6 @@ void __init init_IRQ(void)
 		 * the PMR/PSR pair to a consistent state.
 		 */
 		WARN_ON(read_sysreg(daif) & PSR_A_BIT);
-		local_daif_restore(DAIF_PROCCTX_NOIRQ);
+		local_exception_restore(arm64_make_noirq_mask());
 	}
 }
diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
index c5693a32e49b..a8676ee10ea2 100644
--- a/arch/arm64/kernel/machine_kexec.c
+++ b/arch/arm64/kernel/machine_kexec.c
@@ -17,7 +17,7 @@
 
 #include <asm/cacheflush.h>
 #include <asm/cpu_ops.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/memory.h>
 #include <asm/mmu.h>
 #include <asm/mmu_context.h>
@@ -173,7 +173,7 @@ void machine_kexec(struct kimage *kimage)
 
 	pr_info("Bye!\n");
 
-	local_daif_mask();
+	local_exception_mask();
 
 	/*
 	 * Both restart and kernel_reloc will shutdown the MMU, disable data
diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c
index 43a0361a8bf0..70d311e415ac 100644
--- a/arch/arm64/kernel/probes/kprobes.c
+++ b/arch/arm64/kernel/probes/kprobes.c
@@ -24,7 +24,7 @@
 #include <linux/vmalloc.h>
 
 #include <asm/cacheflush.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/debug-monitors.h>
 #include <asm/insn.h>
 #include <asm/irq.h>
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 23c05dc7a8f2..10507e55e2ce 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -37,7 +37,7 @@
 #include <asm/fixmap.h>
 #include <asm/cpu.h>
 #include <asm/cputype.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/elf.h>
 #include <asm/cpufeature.h>
 #include <asm/cpu_ops.h>
@@ -311,7 +311,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 	 * IRQ and FIQ will be unmasked after the root irqchip has been
 	 * detected and initialized.
 	 */
-	local_daif_restore(DAIF_PROCCTX_NOIRQ);
+	local_exception_restore(arm64_make_noirq_mask());
 
 	/*
 	 * TTBR0 is only used for the identity mapping at this stage. Make it
diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
index 38e6fa204c17..ea39b47ae0eb 100644
--- a/arch/arm64/kernel/signal.c
+++ b/arch/arm64/kernel/signal.c
@@ -22,7 +22,7 @@
 #include <linux/syscalls.h>
 #include <linux/pkeys.h>
 
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/debug-monitors.h>
 #include <asm/elf.h>
 #include <asm/exception.h>
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index d46022f72075..d153ff77d25c 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -42,7 +42,7 @@
 #include <asm/cpu.h>
 #include <asm/cputype.h>
 #include <asm/cpu_ops.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/kvm_mmu.h>
 #include <asm/mmu_context.h>
 #include <asm/numa.h>
@@ -263,7 +263,7 @@ asmlinkage notrace void secondary_start_kernel(void)
 	 * as the root irqchip has already been detected and initialized we can
 	 * unmask IRQ and FIQ at the same time.
 	 */
-	local_daif_restore(DAIF_PROCCTX);
+	local_exception_restore(arm64_make_procctx_mask());
 
 	/*
 	 * OK, it's off to the idle thread for us
@@ -370,7 +370,7 @@ void __noreturn cpu_die(void)
 
 	idle_task_exit();
 
-	local_daif_mask();
+	local_exception_mask();
 
 	/* Tell cpuhp_bp_sync_dead() that this CPU is now safe to dispose of */
 	cpuhp_ap_report_dead();
@@ -870,7 +870,7 @@ static void __noreturn local_cpu_stop(unsigned int cpu)
 {
 	set_cpu_online(cpu, false);
 
-	local_daif_mask();
+	local_exception_mask();
 	sdei_mask_local_cpu();
 	cpu_park_loop();
 }
@@ -889,14 +889,14 @@ static void __noreturn ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs
 {
 #ifdef CONFIG_KEXEC_CORE
 	/*
-	 * Use local_daif_mask() instead of local_irq_disable() to make sure
-	 * that pseudo-NMIs are disabled. The "crash stop" code starts with
-	 * an IRQ and falls back to NMI (which might be pseudo). If the IRQ
-	 * finally goes through right as we're timing out then the NMI could
-	 * interrupt us. It's better to prevent the NMI and let the IRQ
-	 * finish since the pt_regs will be better.
+	 * Use local_exception_mask() instead of local_irq_disable()
+	 * to make sure that pseudo-NMIs are disabled. The "crash stop" code
+	 * starts with an IRQ and falls back to NMI (which might be pseudo).
+	 * If the IRQ finally goes through right as we're timing out then
+	 * the NMI could interrupt us. It's better to prevent the NMI and let
+	 * the IRQ finish since the pt_regs will be better.
 	 */
-	local_daif_mask();
+	local_exception_mask();
 
 	crash_save_cpu(regs, cpu);
 
diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c
index eaaff94329cd..af16120818ac 100644
--- a/arch/arm64/kernel/suspend.c
+++ b/arch/arm64/kernel/suspend.c
@@ -9,7 +9,7 @@
 #include <asm/cacheflush.h>
 #include <asm/cpufeature.h>
 #include <asm/cpuidle.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/debug-monitors.h>
 #include <asm/exec.h>
 #include <asm/fpsimd.h>
@@ -96,10 +96,9 @@ void notrace __cpu_suspend_exit(void)
  */
 int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 {
-	int ret = 0;
-	unsigned long flags;
+	struct exception_mask mask, cpuidle_mask;
 	struct sleep_stack_data state;
-	struct arm_cpuidle_irq_context context;
+	int ret = 0;
 
 	/*
 	 * Some portions of CPU state (e.g. PSTATE.{PAN,DIT}) are initialized
@@ -122,7 +121,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 	 * hardirqs should be firmly off by now. This really ought to use
 	 * something like raw_local_daif_save().
 	 */
-	flags = local_daif_save();
+	mask = local_exception_save_and_mask();
 
 	/*
 	 * Function graph tracer state gets inconsistent when the kernel
@@ -135,7 +134,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 	 * Switch to using DAIF.IF instead of PMR in order to reliably
 	 * resume if we're using pseudo-NMIs.
 	 */
-	arm_cpuidle_save_irq_context(&context);
+	arm_cpuidle_save_irq_context(&cpuidle_mask);
 
 	ct_cpuidle_enter();
 
@@ -159,7 +158,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 		__cpu_suspend_exit();
 	}
 
-	arm_cpuidle_restore_irq_context(&context);
+	arm_cpuidle_restore_irq_context(&cpuidle_mask);
 
 	unpause_graph_tracing();
 
@@ -168,7 +167,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
 	 * restored, so from this point onwards, debugging is fully
 	 * reenabled if it was enabled when core started shutdown.
 	 */
-	local_daif_restore(flags);
+	local_exception_restore(mask);
 
 	return ret;
 }
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index 914282016069..fcc18b182eb2 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -33,7 +33,7 @@
 #include <asm/atomic.h>
 #include <asm/bug.h>
 #include <asm/cpufeature.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/debug-monitors.h>
 #include <asm/efi.h>
 #include <asm/esr.h>
diff --git a/arch/arm64/kvm/hyp/vgic-v3-sr.c b/arch/arm64/kvm/hyp/vgic-v3-sr.c
index c4d2f1feea8b..96187ef5a025 100644
--- a/arch/arm64/kvm/hyp/vgic-v3-sr.c
+++ b/arch/arm64/kvm/hyp/vgic-v3-sr.c
@@ -445,8 +445,8 @@ void __vgic_v3_init_lrs(void)
  */
 u64 __vgic_v3_get_gic_config(void)
 {
+	struct exception_mask mask;
 	u64 val, sre;
-	unsigned long flags = 0;
 
 	/*
 	 * In compat mode, we cannot access ICC_SRE_EL1 at any EL
@@ -476,7 +476,7 @@ u64 __vgic_v3_get_gic_config(void)
 	 * of the exception entry to EL2.
 	 */
 	if (has_vhe()) {
-		flags = local_daif_save();
+		mask = local_exception_save_and_mask();
 	} else {
 		sysreg_clear_set_hcr(0, HCR_AMO | HCR_FMO | HCR_IMO);
 		isb();
@@ -491,7 +491,7 @@ u64 __vgic_v3_get_gic_config(void)
 	isb();
 
 	if (has_vhe()) {
-		local_daif_restore(flags);
+		local_exception_restore(mask);
 	} else {
 		sysreg_clear_set_hcr(HCR_AMO | HCR_FMO | HCR_IMO, 0);
 		isb();
diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
index bbe9cebd3d9d..024876efe0c3 100644
--- a/arch/arm64/kvm/hyp/vhe/switch.c
+++ b/arch/arm64/kvm/hyp/vhe/switch.c
@@ -631,7 +631,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
 {
 	int ret;
 
-	local_daif_mask();
+	local_exception_mask();
 
 	/*
 	 * Having IRQs masked via PMR when entering the guest means the GIC
@@ -647,10 +647,10 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
 	ret = __kvm_vcpu_run_vhe(vcpu);
 
 	/*
-	 * local_daif_restore() takes care to properly restore PSTATE.DAIF
+	 * local_exception_restore() takes care to properly restore PSTATE.DAIF
 	 * and the GIC PMR if the host is using IRQ priorities.
 	 */
-	local_daif_restore(DAIF_PROCCTX_NOIRQ);
+	local_exception_restore(arm64_make_noirq_mask());
 
 	return ret;
 }
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 85e23388f9bb..f7b8e87a099e 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -34,7 +34,7 @@
 #include <asm/cpufeature.h>
 #include <asm/efi.h>
 #include <asm/exception.h>
-#include <asm/daifflags.h>
+#include <asm/exception_masks.h>
 #include <asm/debug-monitors.h>
 #include <asm/esr.h>
 #include <asm/kprobes.h>
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index f2be501468ce..9afcd2bc2fd1 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -2307,7 +2307,7 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
 	typedef void (ttbr_replace_func)(phys_addr_t);
 	extern ttbr_replace_func idmap_cpu_replace_ttbr1;
 	ttbr_replace_func *replace_phys;
-	unsigned long daif;
+	struct exception_mask mask;
 
 	/* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */
 	phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp));
@@ -2323,9 +2323,9 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
 	 * We really don't want to take *any* exceptions while TTBR1 is
 	 * in the process of being replaced so mask everything.
 	 */
-	daif = local_daif_save();
+	mask = local_exception_save_and_mask();
 	replace_phys(ttbr1);
-	local_daif_restore(daif);
+	local_exception_restore(mask);
 
 	cpu_uninstall_idmap();
 }
diff --git a/drivers/firmware/psci/psci.c b/drivers/firmware/psci/psci.c
index e73bae6cb23a..4cc834db7c15 100644
--- a/drivers/firmware/psci/psci.c
+++ b/drivers/firmware/psci/psci.c
@@ -24,6 +24,7 @@
 
 #include <asm/cpuidle.h>
 #include <asm/cputype.h>
+#include <asm/exception_masks.h>
 #include <asm/hypervisor.h>
 #include <asm/system_misc.h>
 #include <asm/smp_plat.h>
@@ -503,12 +504,12 @@ int psci_cpu_suspend_enter(u32 state)
 	int ret;
 
 	if (!psci_power_state_loses_context(state)) {
-		struct arm_cpuidle_irq_context context;
+		struct exception_mask mask;
 
 		ct_cpuidle_enter();
-		arm_cpuidle_save_irq_context(&context);
+		arm_cpuidle_save_irq_context(&mask);
 		ret = psci_ops.cpu_suspend(state, 0);
-		arm_cpuidle_restore_irq_context(&context);
+		arm_cpuidle_restore_irq_context(&mask);
 		ct_cpuidle_exit();
 	} else {
 		/*
-- 
2.34.1

[PATCH 03/17] arm64: entry: arm64: entry: Move DAIF masking for EL1 exit to C code

[Jinjie Ruan]

Move the DAIF masking logic on exception exit from the assembly
macro kernel_exit into the respective EL1 C handler functions.

Originally, when returning to a privileged mode (el != 0), kernel_exit
invoked the disable_daif macro ("msr daifset, #0xf") to ensure that the
subsequent registers and PMR state restoration occurred with all local
interrupts fully masked.

Move the DAIF masking out of the assembly macro and into the individual
C exception handlers, immediately after the core handler logic.
Specifically, add `write_sysreg(DAIF_MASK, daif)` at the end of:

- el1h_64_sync_handler().
- el1_interrupt().
- el1h_64_error_handler).

We achieve identical masking behavior before entering the low-level
assembly exit path.

This change cleans up the assembly macro and consolidates the exception
lifecycle management in C code.

No functional change intended.

Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/assembler.h | 4 ----
 arch/arm64/kernel/entry-common.c   | 5 +++++
 arch/arm64/kernel/entry.S          | 4 ----
 3 files changed, 5 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index effae53e9739..0b58b550e8dc 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -34,10 +34,6 @@
 	wx\n	.req	w\n
 	.endr
 
-	.macro disable_daif
-	msr	daifset, #0xf
-	.endm
-
 /*
  * Save/restore interrupts.
  */
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 195af3f8001e..6fdf74ff4835 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -486,6 +486,8 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
 	default:
 		__panic_unhandled(regs, "64-bit el1h sync", esr);
 	}
+
+	write_sysreg(DAIF_MASK, daif);
 }
 
 static __always_inline void __el1_pnmi(struct pt_regs *regs,
@@ -520,6 +522,8 @@ static void noinstr el1_interrupt(struct pt_regs *regs,
 		__el1_pnmi(regs, handler);
 	else
 		__el1_irq(regs, handler);
+
+	write_sysreg(DAIF_MASK, daif);
 }
 
 asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
@@ -541,6 +545,7 @@ asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
 	state = irqentry_nmi_enter(regs);
 	do_serror(regs, esr);
 	irqentry_nmi_exit(regs, state);
+	write_sysreg(DAIF_MASK, daif);
 }
 
 static void noinstr el0_da(struct pt_regs *regs, unsigned long esr)
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index e0db14e9c843..f63049ac32dc 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -333,10 +333,6 @@ alternative_else_nop_endif
 	.endm
 
 	.macro	kernel_exit, el
-	.if	\el != 0
-	disable_daif
-	.endif
-
 #ifdef CONFIG_ARM64_PSEUDO_NMI
 alternative_if_not ARM64_HAS_GIC_PRIO_MASKING
 	b	.Lskip_pmr_restore\@
-- 
2.34.1

[PATCH 04/17] arm64: entry: Add entry-specific helpers

[Jinjie Ruan]

The existing hacks to bodge pseudo-NMI into the DAIF management code
are convoluted and difficult to maintain. To pave the way for real
hardware NMI support (FEAT_NMI) without compounding the existing
complexity, we need to abstract the low-level exception masking
and unmasking operations into entry-specific helpers.

Introduce a set of high-level abstract exception mask helpers to
encapsulate the "unmask-at-entry" and "mask-at-exit" behaviors across
different exception paths (IRQ, SError, EL1 Sync, and EL0 Sync):

- irq_entry_unmask_debug_serror(): Used at IRQ entry to save the original
  mask state and switch to a safe NOIRQ context.

- error_entry_unmask_debug(): Used at SError entry to mask all exceptions
  except Debug.

- el1_sync_entry_unmask_inherit(): Used at EL1 synchronous exception entry
  to inherit the interrupted kernel context's masks.

- el0_sync_entry_unmask_all() / el0_sync_exit_unmask_all(): Used
  at EL0 synchronous exception entry/exit boundaries to cleanly allow
  all exceptions under a standard process context.

- exception_exit_restore_mask(): A unified exit-side helper to safely
  restore the hardware exception mask states (DAIF and PMR) before
  returning to the interrupted context.

By replacing the raw register bit manipulations with these abstract
helpers in entry-common.c, we achieve better code encapsulation, cleaner
lifecycle symmetry, and a clear path forward for ALLINT integration.

No functional change intended.

Link: https://lore.kernel.org/linux-arm-kernel/Y5c9SLeJacLYHmP7@FVFF77S0Q05N/
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/exception_masks.h |  58 +++++++++++++
 arch/arm64/kernel/entry-common.c         | 103 ++++++++++++++---------
 2 files changed, 122 insertions(+), 39 deletions(-)

diff --git a/arch/arm64/include/asm/exception_masks.h b/arch/arm64/include/asm/exception_masks.h
index 0a21fb97c3ca..2c87f7c90f62 100644
--- a/arch/arm64/include/asm/exception_masks.h
+++ b/arch/arm64/include/asm/exception_masks.h
@@ -138,4 +138,62 @@ static inline void local_exception_inherit(struct pt_regs *regs)
 
 	write_sysreg(regs->pstate & DAIF_MASK, daif);
 }
+
+/*
+ * Allow Debug exceptions and SError, mask IRQ/FIQ
+ */
+static __always_inline struct exception_mask irq_entry_unmask_debug_serror(struct pt_regs *regs)
+{
+	struct exception_mask orig;
+
+	local_exception_save_mask(&orig);
+	write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
+
+	return orig;
+}
+
+static __always_inline struct exception_mask error_entry_unmask_debug(struct pt_regs *regs)
+{
+	struct exception_mask orig;
+
+	local_exception_save_mask(&orig);
+	local_exception_restore(arm64_make_errctx_mask());
+
+	return orig;
+}
+
+static __always_inline struct exception_mask el1_sync_entry_unmask_inherit(struct pt_regs *regs)
+{
+	struct exception_mask orig;
+
+	local_exception_save_mask(&orig);
+	local_exception_inherit(regs);
+
+	return orig;
+}
+
+/*
+ * Unmask all exceptions to establish a standard process context.
+ * Suitable for EL0 sync entry and secondary CPU boot streaming.
+ */
+static __always_inline void el0_sync_entry_unmask_all(struct pt_regs *regs)
+{
+	local_exception_restore(arm64_make_procctx_mask());
+}
+
+/*
+ * Retained for symmetric naming, used before returning to EL0
+ */
+static __always_inline void el0_sync_exit_unmask_all(struct pt_regs *regs)
+{
+	local_exception_restore(arm64_make_procctx_mask());
+}
+
+/*
+ * Mask all exceptions, ready to return to interrupted context
+ */
+static __always_inline void exception_exit_restore_mask(struct exception_mask mask)
+{
+	write_sysreg(mask.daif, daif);
+}
 #endif /* __ASM_EXCEPTION_MASKS_H */
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 6fdf74ff4835..dc223085e773 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -315,80 +315,97 @@ UNHANDLED(el1t, 64, error)
 static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
 {
 	unsigned long far = read_sysreg(far_el1);
+	struct exception_mask mask;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_exception_inherit(regs);
+	mask = el1_sync_entry_unmask_inherit(regs);
 	do_mem_abort(far, esr, regs);
 	arm64_exit_to_kernel_mode(regs, state);
+	exception_exit_restore_mask(mask);
 }
 
 static void noinstr el1_pc(struct pt_regs *regs, unsigned long esr)
 {
 	unsigned long far = read_sysreg(far_el1);
+	struct exception_mask mask;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_exception_inherit(regs);
+	mask = el1_sync_entry_unmask_inherit(regs);
 	do_sp_pc_abort(far, esr, regs);
 	arm64_exit_to_kernel_mode(regs, state);
+	exception_exit_restore_mask(mask);
 }
 
 static void noinstr el1_undef(struct pt_regs *regs, unsigned long esr)
 {
+	struct exception_mask mask;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_exception_inherit(regs);
+	mask = el1_sync_entry_unmask_inherit(regs);
 	do_el1_undef(regs, esr);
 	arm64_exit_to_kernel_mode(regs, state);
+	exception_exit_restore_mask(mask);
 }
 
 static void noinstr el1_bti(struct pt_regs *regs, unsigned long esr)
 {
+	struct exception_mask mask;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_exception_inherit(regs);
+	mask = el1_sync_entry_unmask_inherit(regs);
 	do_el1_bti(regs, esr);
 	arm64_exit_to_kernel_mode(regs, state);
+	exception_exit_restore_mask(mask);
 }
 
 static void noinstr el1_gcs(struct pt_regs *regs, unsigned long esr)
 {
+	struct exception_mask mask;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_exception_inherit(regs);
+	mask = el1_sync_entry_unmask_inherit(regs);
 	do_el1_gcs(regs, esr);
 	arm64_exit_to_kernel_mode(regs, state);
+	exception_exit_restore_mask(mask);
 }
 
 static void noinstr el1_mops(struct pt_regs *regs, unsigned long esr)
 {
+	struct exception_mask mask;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_exception_inherit(regs);
+	mask = el1_sync_entry_unmask_inherit(regs);
 	do_el1_mops(regs, esr);
 	arm64_exit_to_kernel_mode(regs, state);
+	exception_exit_restore_mask(mask);
 }
 
 static void noinstr el1_breakpt(struct pt_regs *regs, unsigned long esr)
 {
+	struct exception_mask mask;
 	irqentry_state_t state;
 
+	local_exception_save_mask(&mask);
 	state = arm64_enter_el1_dbg(regs);
 	debug_exception_enter(regs);
 	do_breakpoint(esr, regs);
 	debug_exception_exit(regs);
 	arm64_exit_el1_dbg(regs, state);
+	exception_exit_restore_mask(mask);
 }
 
 static void noinstr el1_softstp(struct pt_regs *regs, unsigned long esr)
 {
+	struct exception_mask mask;
 	irqentry_state_t state;
 
+	local_exception_save_mask(&mask);
 	state = arm64_enter_el1_dbg(regs);
 	if (!cortex_a76_erratum_1463225_debug_handler(regs)) {
 		debug_exception_enter(regs);
@@ -403,40 +420,49 @@ static void noinstr el1_softstp(struct pt_regs *regs, unsigned long esr)
 		debug_exception_exit(regs);
 	}
 	arm64_exit_el1_dbg(regs, state);
+	exception_exit_restore_mask(mask);
 }
 
 static void noinstr el1_watchpt(struct pt_regs *regs, unsigned long esr)
 {
 	/* Watchpoints are the only debug exception to write FAR_EL1 */
 	unsigned long far = read_sysreg(far_el1);
+	struct exception_mask mask;
 	irqentry_state_t state;
 
+	local_exception_save_mask(&mask);
 	state = arm64_enter_el1_dbg(regs);
 	debug_exception_enter(regs);
 	do_watchpoint(far, esr, regs);
 	debug_exception_exit(regs);
 	arm64_exit_el1_dbg(regs, state);
+	exception_exit_restore_mask(mask);
 }
 
 static void noinstr el1_brk64(struct pt_regs *regs, unsigned long esr)
 {
+	struct exception_mask mask;
 	irqentry_state_t state;
 
+	local_exception_save_mask(&mask);
 	state = arm64_enter_el1_dbg(regs);
 	debug_exception_enter(regs);
 	do_el1_brk64(esr, regs);
 	debug_exception_exit(regs);
 	arm64_exit_el1_dbg(regs, state);
+	exception_exit_restore_mask(mask);
 }
 
 static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr)
 {
+	struct exception_mask mask;
 	irqentry_state_t state;
 
 	state = arm64_enter_from_kernel_mode(regs);
-	local_exception_inherit(regs);
+	mask = el1_sync_entry_unmask_inherit(regs);
 	do_el1_fpac(regs, esr);
 	arm64_exit_to_kernel_mode(regs, state);
+	exception_exit_restore_mask(mask);
 }
 
 asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
@@ -486,8 +512,6 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
 	default:
 		__panic_unhandled(regs, "64-bit el1h sync", esr);
 	}
-
-	write_sysreg(DAIF_MASK, daif);
 }
 
 static __always_inline void __el1_pnmi(struct pt_regs *regs,
@@ -516,14 +540,14 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
 static void noinstr el1_interrupt(struct pt_regs *regs,
 				  void (*handler)(struct pt_regs *))
 {
-	write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
+	struct exception_mask mask = irq_entry_unmask_debug_serror(regs);
 
 	if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && regs_irqs_disabled(regs))
 		__el1_pnmi(regs, handler);
 	else
 		__el1_irq(regs, handler);
 
-	write_sysreg(DAIF_MASK, daif);
+	exception_exit_restore_mask(mask);
 }
 
 asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
@@ -539,13 +563,14 @@ asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
 asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
 {
 	unsigned long esr = read_sysreg(esr_el1);
+	struct exception_mask mask;
 	irqentry_state_t state;
 
-	local_exception_restore(arm64_make_errctx_mask());
+	mask = error_entry_unmask_debug(regs);
 	state = irqentry_nmi_enter(regs);
 	do_serror(regs, esr);
 	irqentry_nmi_exit(regs, state);
-	write_sysreg(DAIF_MASK, daif);
+	exception_exit_restore_mask(mask);
 }
 
 static void noinstr el0_da(struct pt_regs *regs, unsigned long esr)
@@ -553,7 +578,7 @@ static void noinstr el0_da(struct pt_regs *regs, unsigned long esr)
 	unsigned long far = read_sysreg(far_el1);
 
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_mem_abort(far, esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -571,7 +596,7 @@ static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
 		arm64_apply_bp_hardening();
 
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_mem_abort(far, esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -579,7 +604,7 @@ static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_fpsimd_acc(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_fpsimd_acc(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -587,7 +612,7 @@ static void noinstr el0_fpsimd_acc(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_sve_acc(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_sve_acc(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -595,7 +620,7 @@ static void noinstr el0_sve_acc(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_sme_acc(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_sme_acc(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -603,7 +628,7 @@ static void noinstr el0_sme_acc(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_fpsimd_exc(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_fpsimd_exc(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -611,7 +636,7 @@ static void noinstr el0_fpsimd_exc(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_sys(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_el0_sys(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -624,7 +649,7 @@ static void noinstr el0_pc(struct pt_regs *regs, unsigned long esr)
 		arm64_apply_bp_hardening();
 
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_sp_pc_abort(far, esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -632,7 +657,7 @@ static void noinstr el0_pc(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_sp(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_sp_pc_abort(regs->sp, esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -640,7 +665,7 @@ static void noinstr el0_sp(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_undef(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_el0_undef(regs, esr);
 	arm64_exit_to_user_mode(regs);
 }
@@ -648,7 +673,7 @@ static void noinstr el0_undef(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_bti(struct pt_regs *regs)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_el0_bti(regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -656,7 +681,7 @@ static void noinstr el0_bti(struct pt_regs *regs)
 static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_el0_mops(regs, esr);
 	arm64_exit_to_user_mode(regs);
 }
@@ -664,7 +689,7 @@ static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_gcs(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_el0_gcs(regs, esr);
 	arm64_exit_to_user_mode(regs);
 }
@@ -672,7 +697,7 @@ static void noinstr el0_gcs(struct pt_regs *regs, unsigned long esr)
 static void noinstr el0_inv(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	bad_el0_sync(regs, 0, esr);
 	arm64_exit_to_user_mode(regs);
 }
@@ -686,7 +711,7 @@ static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
 	debug_exception_enter(regs);
 	do_breakpoint(esr, regs);
 	debug_exception_exit(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_exit_unmask_all(regs);
 	arm64_exit_to_user_mode(regs);
 }
 
@@ -705,7 +730,7 @@ static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
 	 * the single-step is complete.
 	 */
 	step_done = try_step_suspended_breakpoints(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	if (!step_done)
 		do_el0_softstep(esr, regs);
 	arm64_exit_to_user_mode(regs);
@@ -720,14 +745,14 @@ static void noinstr el0_watchpt(struct pt_regs *regs, unsigned long esr)
 	debug_exception_enter(regs);
 	do_watchpoint(far, esr, regs);
 	debug_exception_exit(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_exit_unmask_all(regs);
 	arm64_exit_to_user_mode(regs);
 }
 
 static void noinstr el0_brk64(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_el0_brk64(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -737,7 +762,7 @@ static void noinstr el0_svc(struct pt_regs *regs)
 	arm64_syscall_enter_from_user_mode(regs);
 	cortex_a76_erratum_1463225_svc_handler();
 	fpsimd_syscall_enter();
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_el0_svc(regs);
 	arm64_syscall_exit_to_user_mode(regs);
 	fpsimd_syscall_exit();
@@ -746,7 +771,7 @@ static void noinstr el0_svc(struct pt_regs *regs)
 static void noinstr el0_fpac(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_el0_fpac(regs, esr);
 	arm64_exit_to_user_mode(regs);
 }
@@ -824,7 +849,7 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
 {
 	arm64_enter_from_user_mode(regs);
 
-	write_sysreg(DAIF_PROCCTX_NOIRQ, daif);
+	irq_entry_unmask_debug_serror(regs);
 
 	if (regs->pc & BIT(55))
 		arm64_apply_bp_hardening();
@@ -862,11 +887,11 @@ static void noinstr __el0_error_handler_common(struct pt_regs *regs)
 	irqentry_state_t state;
 
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_errctx_mask());
+	error_entry_unmask_debug(regs);
 	state = irqentry_nmi_enter(regs);
 	do_serror(regs, esr);
 	irqentry_nmi_exit(regs, state);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_exit_unmask_all(regs);
 	arm64_exit_to_user_mode(regs);
 }
 
@@ -879,7 +904,7 @@ asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
 static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_el0_cp15(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
@@ -888,7 +913,7 @@ static void noinstr el0_svc_compat(struct pt_regs *regs)
 {
 	arm64_syscall_enter_from_user_mode(regs);
 	cortex_a76_erratum_1463225_svc_handler();
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_el0_svc_compat(regs);
 	arm64_syscall_exit_to_user_mode(regs);
 }
@@ -896,7 +921,7 @@ static void noinstr el0_svc_compat(struct pt_regs *regs)
 static void noinstr el0_bkpt32(struct pt_regs *regs, unsigned long esr)
 {
 	arm64_enter_from_user_mode(regs);
-	local_exception_restore(arm64_make_procctx_mask());
+	el0_sync_entry_unmask_all(regs);
 	do_bkpt32(esr, regs);
 	arm64_exit_to_user_mode(regs);
 }
-- 
2.34.1

[PATCH 05/17] arm64: Introduce helpers for restoring standard exception masks

[Jinjie Ruan]

Several non-entry call sites construct and restore standard exception
masks by open-coding patterns like

	local_exception_restore(arm64_make_noirq_mask());
	local_exception_restore(arm64_make_procctx_mask());
	local_exception_restore(arm64_make_errctx_mask());

These scattered calls directly manipulate DAIF and are blind to the
ongoing rework of exception mask management (DAIF + PMR + future
ALLINT).  Encapsulate each pattern into a dedicated helper:

	local_exception_restore_noirq()
	local_exception_restore_procctx()
	local_exception_restore_errctx()

This centralises the mask construction, avoids open-coding of
arch-specific mask details, and prepares for the entry-specific
exception mask framework that will manage DAIF, PMR and eventually
ALLINT in a consistent way.

Callers in ACPI, IRQ setup, SMP, KVM VHE switch and elsewhere are
converted to use the new helpers.  No functional change.

No functional change.

Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/exception_masks.h | 15 +++++++++++++++
 arch/arm64/kernel/acpi.c                 |  4 ++--
 arch/arm64/kernel/irq.c                  |  2 +-
 arch/arm64/kernel/setup.c                |  2 +-
 arch/arm64/kernel/smp.c                  |  2 +-
 arch/arm64/kvm/hyp/vhe/switch.c          |  2 +-
 6 files changed, 21 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/exception_masks.h b/arch/arm64/include/asm/exception_masks.h
index 2c87f7c90f62..fbbba769ca03 100644
--- a/arch/arm64/include/asm/exception_masks.h
+++ b/arch/arm64/include/asm/exception_masks.h
@@ -196,4 +196,19 @@ static __always_inline void exception_exit_restore_mask(struct exception_mask ma
 {
 	write_sysreg(mask.daif, daif);
 }
+
+static inline void local_exception_restore_noirq(void)
+{
+	local_exception_restore(arm64_make_noirq_mask());
+}
+
+static inline void local_exception_restore_errctx(void)
+{
+	local_exception_restore(arm64_make_errctx_mask());
+}
+
+static inline void local_exception_restore_procctx(void)
+{
+	local_exception_restore(arm64_make_procctx_mask());
+}
 #endif /* __ASM_EXCEPTION_MASKS_H */
diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
index 4d413419309d..88883d074ecd 100644
--- a/arch/arm64/kernel/acpi.c
+++ b/arch/arm64/kernel/acpi.c
@@ -407,7 +407,7 @@ int apei_claim_sea(struct pt_regs *regs)
 	 * SEA can interrupt SError, mask it and describe this as an NMI so
 	 * that APEI defers the handling.
 	 */
-	local_exception_restore(arm64_make_errctx_mask());
+	local_exception_restore_errctx();
 	nmi_enter();
 	err = ghes_notify_sea();
 	nmi_exit();
@@ -418,7 +418,7 @@ int apei_claim_sea(struct pt_regs *regs)
 	 */
 	if (!err) {
 		if (return_to_irqs_enabled) {
-			local_exception_restore(arm64_make_noirq_mask());
+			local_exception_restore_noirq();
 			__irq_enter();
 			irq_work_run();
 			__irq_exit();
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index 9e7fb0d22586..fff0efd5a195 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -130,6 +130,6 @@ void __init init_IRQ(void)
 		 * the PMR/PSR pair to a consistent state.
 		 */
 		WARN_ON(read_sysreg(daif) & PSR_A_BIT);
-		local_exception_restore(arm64_make_noirq_mask());
+		local_exception_restore_noirq();
 	}
 }
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 10507e55e2ce..bd7aa8263ebe 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -311,7 +311,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 	 * IRQ and FIQ will be unmasked after the root irqchip has been
 	 * detected and initialized.
 	 */
-	local_exception_restore(arm64_make_noirq_mask());
+	local_exception_restore_noirq();
 
 	/*
 	 * TTBR0 is only used for the identity mapping at this stage. Make it
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index d153ff77d25c..a94ebb1bb6d7 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -263,7 +263,7 @@ asmlinkage notrace void secondary_start_kernel(void)
 	 * as the root irqchip has already been detected and initialized we can
 	 * unmask IRQ and FIQ at the same time.
 	 */
-	local_exception_restore(arm64_make_procctx_mask());
+	local_exception_restore_procctx();
 
 	/*
 	 * OK, it's off to the idle thread for us
diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
index 024876efe0c3..d11fb05e558b 100644
--- a/arch/arm64/kvm/hyp/vhe/switch.c
+++ b/arch/arm64/kvm/hyp/vhe/switch.c
@@ -650,7 +650,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
 	 * local_exception_restore() takes care to properly restore PSTATE.DAIF
 	 * and the GIC PMR if the host is using IRQ priorities.
 	 */
-	local_exception_restore(arm64_make_noirq_mask());
+	local_exception_restore_noirq();
 
 	return ret;
 }
-- 
2.34.1

[PATCH 06/17] arm64/booting: Document boot requirements for FEAT_NMI

[Jinjie Ruan]

From: Mark Brown <broonie@kernel.org>

In order to use FEAT_NMI we must be able to use ALLINT, require that it
behave as though not trapped when it is present.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20221112151708.175147-2-broonie@kernel.org
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 Documentation/arch/arm64/booting.rst | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Documentation/arch/arm64/booting.rst b/Documentation/arch/arm64/booting.rst
index 13ef311dace8..aaabe8fa4cdd 100644
--- a/Documentation/arch/arm64/booting.rst
+++ b/Documentation/arch/arm64/booting.rst
@@ -419,6 +419,12 @@ Before jumping into the kernel, the following conditions must be met:
 
     - HCR_EL2.ATA (bit 56) must be initialised to 0b1.
 
+  For CPUs with Non-maskable Interrupts (FEAT_NMI):
+
+  - If the kernel is entered at EL1 and EL2 is present:
+
+    - HCRX_EL2.TALLINT must be initialised to 0b0.
+
   For CPUs with the Scalable Matrix Extension version 2 (FEAT_SME2):
 
   - If EL3 is present:
-- 
2.34.1

[PATCH 07/17] arm64/sysreg: Add definitions for immediate versions of MSR ALLINT

[Jinjie Ruan]

From: Mark Brown <broonie@kernel.org>

Encodings are provided for ALLINT which allow setting of ALLINT.ALLINT
using an immediate rather than requiring that a register be loaded with
the value to write. Since these don't currently fit within the scheme we
have for sysreg generation add manual encodings like we currently do for
other similar registers such as SVCR.

Since it is required that these immediate versions be encoded with xzr
as the source register provide asm wrapper which ensure this is the
case.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20221112151708.175147-5-broonie@kernel.org
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/nmi.h    | 23 +++++++++++++++++++++++
 arch/arm64/include/asm/sysreg.h |  2 ++
 2 files changed, 25 insertions(+)
 create mode 100644 arch/arm64/include/asm/nmi.h

diff --git a/arch/arm64/include/asm/nmi.h b/arch/arm64/include/asm/nmi.h
new file mode 100644
index 000000000000..eb2c9a8350cc
--- /dev/null
+++ b/arch/arm64/include/asm/nmi.h
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2022 ARM Ltd.
+ */
+#ifndef __ASM_NMI_H
+#define __ASM_NMI_H
+
+#ifdef CONFIG_ARM64_NMI
+static __always_inline void _allint_clear(void)
+{
+	asm volatile(__msr_s(SYS_ALLINT_CLR, "xzr"));
+}
+
+static __always_inline void _allint_set(void)
+{
+	asm volatile(__msr_s(SYS_ALLINT_SET, "xzr"));
+}
+#else
+static __always_inline void _allint_clear(void) { }
+static __always_inline void _allint_set(void) { }
+#endif /* CONFIG_ARM64_NMI */
+
+#endif
diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index 7aa08d59d494..6d2c6bc1b985 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -179,6 +179,8 @@
  * System registers, organised loosely by encoding but grouped together
  * where the architected name contains an index. e.g. ID_MMFR<n>_EL1.
  */
+#define SYS_ALLINT_CLR			sys_reg(0, 1, 4, 0, 0)
+#define SYS_ALLINT_SET			sys_reg(0, 1, 4, 1, 0)
 #define SYS_SVCR_SMSTOP_SM_EL0		sys_reg(0, 3, 4, 2, 3)
 #define SYS_SVCR_SMSTART_SM_EL0		sys_reg(0, 3, 4, 3, 3)
 #define SYS_SVCR_SMSTOP_SMZA_EL0	sys_reg(0, 3, 4, 6, 3)
-- 
2.34.1

[PATCH 08/17] arm64/hyp-stub: Enable access to ALLINT

[Jinjie Ruan]

From: Mark Brown <broonie@kernel.org>

In order to use NMIs we need to ensure that traps are disabled for it so
update HCRX_EL2 to ensure that TALLINT is not set when we detect support
for NMIs.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20221112151708.175147-7-broonie@kernel.org
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/el2_setup.h | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h
index aa8ec9df8024..73b8997f8d43 100644
--- a/arch/arm64/include/asm/el2_setup.h
+++ b/arch/arm64/include/asm/el2_setup.h
@@ -600,6 +600,19 @@
 
 	msr_s	SYS_SMPRIMAP_EL2, xzr		// Make all priorities equal
 .Lskip_sme_\@:
+
+	// NMIs
+	__check_override id_aa64pfr1, ID_AA64PFR1_EL1_NMI_SHIFT, 4, .Linit_nmi_\@, .Lskip_nmi_\@, x1, x2
+
+.Linit_nmi_\@:
+	mrs	x1, id_aa64mmfr1_el1		// HCRX_EL2 present?
+	ubfx	x1, x1, #ID_AA64MMFR1_EL1_HCX_SHIFT, #4
+	cbz	x1, .Lskip_nmi_\@
+
+	mrs_s	x1, SYS_HCRX_EL2
+	and	x1, x1, # ~HCRX_EL2_TALLINT_MASK	// Don't trap ALLINT
+	msr_s	SYS_HCRX_EL2, x1
+.Lskip_nmi_\@:
 .endm
 
 #endif /* __ARM_KVM_INIT_H__ */
-- 
2.34.1

[PATCH 09/17] arm64/idreg: Add an override for FEAT_NMI

[Jinjie Ruan]

From: Mark Brown <broonie@kernel.org>

Add a named override for FEAT_NMI, allowing it to be explicitly disabled
in case of problems.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20221112151708.175147-8-broonie@kernel.org
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/kernel/pi/idreg-override.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c
index bc57b290e5e7..e5ea280452c3 100644
--- a/arch/arm64/kernel/pi/idreg-override.c
+++ b/arch/arm64/kernel/pi/idreg-override.c
@@ -154,6 +154,7 @@ static const struct ftr_set_desc pfr1 __prel64_initconst = {
 		FIELD("bt", ID_AA64PFR1_EL1_BT_SHIFT, NULL ),
 		FIELD("gcs", ID_AA64PFR1_EL1_GCS_SHIFT, NULL),
 		FIELD("mte", ID_AA64PFR1_EL1_MTE_SHIFT, NULL),
+		FIELD("nmi", ID_AA64PFR1_EL1_NMI_SHIFT, NULL),
 		FIELD("sme", ID_AA64PFR1_EL1_SME_SHIFT, pfr1_sme_filter),
 		FIELD("mpam_frac", ID_AA64PFR1_EL1_MPAM_frac_SHIFT, NULL),
 		{}
-- 
2.34.1

[PATCH 10/17] arm64/cpufeature: Detect PE support for FEAT_NMI

[Jinjie Ruan]

From: Mark Brown <broonie@kernel.org>

Use of FEAT_NMI requires that all the PEs in the system and the GIC
have NMI support. This patch implements the PE part of that detection.

In order to avoid problematic interactions between real and pseudo NMIs
we disable the architected feature if the user has enabled pseudo NMIs
on the command line. If this is done on a system where support for the
architected feature is detected then a warning is printed during boot in
order to help users spot what is likely to be a misconfiguration.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20221112151708.175147-9-broonie@kernel.org
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/cpufeature.h |  5 +++
 arch/arm64/kernel/cpufeature.c      | 58 ++++++++++++++++++++++++++++-
 arch/arm64/tools/cpucaps            |  2 +
 3 files changed, 64 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index a57870fa96db..8ce12c21e223 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -816,6 +816,11 @@ static __always_inline bool system_uses_irq_prio_masking(void)
 	return alternative_has_cap_unlikely(ARM64_HAS_GIC_PRIO_MASKING);
 }
 
+static __always_inline bool system_uses_nmi(void)
+{
+	return alternative_has_cap_unlikely(ARM64_USES_NMI);
+}
+
 static inline bool system_supports_mte(void)
 {
 	return alternative_has_cap_unlikely(ARM64_MTE);
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 9a22df0c5120..33d7cb43269f 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -89,6 +89,7 @@
 #include <asm/mmu_context.h>
 #include <asm/mpam.h>
 #include <asm/mte.h>
+#include <asm/nmi.h>
 #include <asm/hypervisor.h>
 #include <asm/processor.h>
 #include <asm/smp.h>
@@ -315,6 +316,7 @@ static const struct arm64_ftr_bits ftr_id_aa64pfr1[] = {
 	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_GCS),
 		       FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_GCS_SHIFT, 4, 0),
 	S_ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MTE_frac_SHIFT, 4, 0),
+	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_NMI_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_SME),
 		       FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_SME_SHIFT, 4, 0),
 	ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR1_EL1_MPAM_frac_SHIFT, 4, 0),
@@ -1137,9 +1139,11 @@ static void init_32bit_cpu_features(struct cpuinfo_32bit *info)
 	init_cpu_ftr_reg(SYS_MVFR2_EL1, info->reg_mvfr2);
 }
 
-#ifdef CONFIG_ARM64_PSEUDO_NMI
+#if IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) || IS_ENABLED(CONFIG_ARM64_NMI)
 static bool enable_pseudo_nmi;
+#endif
 
+#ifdef CONFIG_ARM64_PSEUDO_NMI
 static int __init early_enable_pseudo_nmi(char *p)
 {
 	return kstrtobool(p, &enable_pseudo_nmi);
@@ -2320,6 +2324,41 @@ static bool has_gic_prio_relaxed_sync(const struct arm64_cpu_capabilities *entry
 }
 #endif
 
+#ifdef CONFIG_ARM64_NMI
+static bool use_nmi(const struct arm64_cpu_capabilities *entry, int scope)
+{
+	if (!has_cpuid_feature(entry, scope))
+		return false;
+
+	/*
+	 * Having both real and pseudo NMIs enabled simultaneously is
+	 * likely to cause confusion.  Since pseudo NMIs must be
+	 * enabled with an explicit command line option, if the user
+	 * has set that option on a system with real NMIs for some
+	 * reason assume they know what they're doing.
+	 */
+	if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && enable_pseudo_nmi) {
+		pr_info_once("Pseudo NMI enabled, not using architected NMI\n");
+		return false;
+	}
+
+	return true;
+}
+
+static void nmi_enable(const struct arm64_cpu_capabilities *__unused)
+{
+	/*
+	 * Enable use of NMIs controlled by ALLINT, SPINTMASK should
+	 * be clear by default but make it explicit that we are using
+	 * this mode.  Ensure that ALLINT is clear first in order to
+	 * avoid leaving things masked.
+	 */
+	_allint_clear();
+	sysreg_clear_set(sctlr_el1, SCTLR_EL1_SPINTMASK, SCTLR_EL1_NMI);
+	isb();
+}
+#endif
+
 static bool can_trap_icv_dir_el1(const struct arm64_cpu_capabilities *entry,
 				 int scope)
 {
@@ -3198,6 +3237,23 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.matches = has_cpuid_feature,
 		ARM64_CPUID_FIELDS(ID_AA64ISAR3_EL1, LSUI, IMP)
 	},
+#endif
+#ifdef CONFIG_ARM64_NMI
+	{
+		.desc = "Non-maskable Interrupts present",
+		.capability = ARM64_HAS_NMI,
+		.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
+		.matches = has_cpuid_feature,
+		ARM64_CPUID_FIELDS(ID_AA64PFR1_EL1, NMI, IMP)
+	},
+	{
+		.desc = "Non-maskable Interrupts enabled",
+		.capability = ARM64_USES_NMI,
+		.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
+		.matches = use_nmi,
+		.cpu_enable = nmi_enable,
+		ARM64_CPUID_FIELDS(ID_AA64PFR1_EL1, NMI, IMP)
+	},
 #endif
 	{},
 };
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index 9b85a84f6fd4..676d00b5fd14 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -51,6 +51,7 @@ HAS_LS64_V
 HAS_LSUI
 HAS_MOPS
 HAS_NESTED_VIRT
+HAS_NMI
 HAS_BBML2_NOABORT
 HAS_PAN
 HAS_PMUV3
@@ -90,6 +91,7 @@ SPECTRE_BHB
 SSBS
 SVE
 UNMAP_KERNEL_AT_EL0
+USES_NMI
 WORKAROUND_834220
 WORKAROUND_843419
 WORKAROUND_845719
-- 
2.34.1

[PATCH 11/17] KVM: arm64: Hide FEAT_NMI from guests

[Jinjie Ruan]

From: Mark Brown <broonie@kernel.org>

FEAT_NMI is not yet useful to guests pending implementation of vGIC
support. Mask out the feature from the ID register and prevent guests
creating state in ALLINT.ALLINT by activating the trap on write provided
in HCRX_EL2.TALLINT when they are running. There is no trap available
for reads from ALLINT.

We do not need to check for FEAT_HCRX since it is mandatory since v8.7
and FEAT_NMI is a v8.8 feature.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20221112151708.175147-10-broonie@kernel.org
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/kvm/hyp/include/hyp/switch.h | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
index 18131e395e24..21b19f331a12 100644
--- a/arch/arm64/kvm/hyp/include/hyp/switch.h
+++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
@@ -346,6 +346,9 @@ static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
 		vcpu_set_flag(vcpu, PMUSERENR_ON_CPU);
 	}
 
+	if (cpus_have_final_cap(ARM64_HAS_NMI))
+		sysreg_clear_set_s(SYS_HCRX_EL2, 0, HCRX_EL2_TALLINT);
+
 	if (cpus_have_final_cap(ARM64_HAS_HCX)) {
 		u64 hcrx = vcpu->arch.hcrx_el2;
 		if (is_nested_ctxt(vcpu)) {
@@ -367,6 +370,9 @@ static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu)
 {
 	struct kvm_cpu_context *hctxt = host_data_ptr(host_ctxt);
 
+	if (cpus_have_final_cap(ARM64_HAS_NMI))
+		sysreg_clear_set_s(SYS_HCRX_EL2, HCRX_EL2_TALLINT, 0);
+
 	write_sysreg(0, hstr_el2);
 	if (system_supports_pmuv3()) {
 		write_sysreg(ctxt_sys_reg(hctxt, PMUSERENR_EL0), pmuserenr_el0);
-- 
2.34.1

[PATCH 12/17] arm64/nmi: Manage masking for superpriority interrupts along with DAIF

[Jinjie Ruan]

From: Mark Brown <broonie@kernel.org>

As we do for pseudo NMIs add code to our DAIF management which keeps
superpriority interrupts unmasked when we have asynchronous exceptions
enabled. Since superpriority interrupts are not masked through DAIF like
pseduo NMIs are we also need to modify the assembler macros for managing
DAIF to ensure that the masking is done in the assembly code. At present
users of the assembly macros always mask pseudo NMIs.

There is a difference to the actual handling between pseudo NMIs
and superpriority interrupts in the assembly save_and_disable_irq and
restore_irq macros, these cover both interrupts and FIQs using DAIF
without regard for the use of pseudo NMIs so also mask those but are not
updated here to mask superpriority interrupts. Given the names it is not
clear that the behaviour with pseudo NMIs is particularly intentional,
and in any case these macros are only used in the implementation of
alternatives for software PAN while hardware PAN has been mandatory
since v8.1 so it is not anticipated that practical systems with support
for FEAT_NMI will ever execute the affected code.

This should be a conservative set of masked regions, we may be able to
relax this in future, but this should represent a good starting point.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20221112151708.175147-11-broonie@kernel.org
Co-developed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/arch_gicv3.h      |  7 ++++++-
 arch/arm64/include/asm/assembler.h       | 13 +++++++++++++
 arch/arm64/include/asm/efi.h             | 22 ++++++++++++++++------
 arch/arm64/include/asm/exception_masks.h | 22 ++++++++++++++++++++--
 arch/arm64/include/uapi/asm/ptrace.h     |  1 +
 arch/arm64/kernel/probes/kprobes.c       |  7 +++++++
 arch/arm64/kernel/process.c              |  3 ++-
 drivers/irqchip/irq-gic-v3.c             |  6 +++---
 8 files changed, 68 insertions(+), 13 deletions(-)

diff --git a/arch/arm64/include/asm/arch_gicv3.h b/arch/arm64/include/asm/arch_gicv3.h
index d20b03931a8d..78405aa0d233 100644
--- a/arch/arm64/include/asm/arch_gicv3.h
+++ b/arch/arm64/include/asm/arch_gicv3.h
@@ -15,6 +15,7 @@
 #include <linux/stringify.h>
 #include <asm/barrier.h>
 #include <asm/cacheflush.h>
+#include <asm/nmi.h>
 
 #define read_gicreg(r)			read_sysreg_s(SYS_ ## r)
 #define write_gicreg(v, r)		write_sysreg_s(v, SYS_ ## r)
@@ -180,7 +181,11 @@ static inline void gic_pmr_mask_irqs(void)
 
 static inline void gic_arch_enable_irqs(void)
 {
-	asm volatile ("msr daifclr, #3" : : : "memory");
+	if (gic_prio_masking_enabled())
+		asm volatile ("msr daifclr, #3" : : : "memory");
+
+	if (system_uses_nmi())
+		_allint_clear();
 }
 
 static inline bool gic_has_relaxed_pmr_sync(void)
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 0b58b550e8dc..b7293844bbc5 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -39,6 +39,11 @@
  */
 	.macro save_and_disable_daif, flags
 	mrs	\flags, daif
+#ifdef CONFIG_ARM64_NMI
+alternative_if ARM64_HAS_NMI
+	msr_s	SYS_ALLINT_SET, xzr
+alternative_else_nop_endif
+#endif
 	msr	daifset, #0xf
 	.endm
 
@@ -49,6 +54,14 @@
 
 	.macro	restore_irq, flags
 	msr	daif, \flags
+#ifdef CONFIG_ARM64_NMI
+alternative_if ARM64_HAS_NMI
+	/* If async exceptions are unmasked we can take NMIs */
+	tbnz	\flags, #8, 2004f
+	msr_s	SYS_ALLINT_CLR, xzr
+2004:
+alternative_else_nop_endif
+#endif
 	.endm
 
 	.macro	disable_step_tsk, flgs, tmp
diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h
index ae2a24868ee7..9435b64124e7 100644
--- a/arch/arm64/include/asm/efi.h
+++ b/arch/arm64/include/asm/efi.h
@@ -48,18 +48,28 @@ void arch_efi_call_virt_teardown(void);
 	(efi_rt_stack_top != NULL &&					\
 	 on_task_stack(current, READ_ONCE(efi_rt_stack_top[-1]), 1))
 
-#define ARCH_EFI_IRQ_FLAGS_MASK (DAIF_MASK)
+#define ARCH_EFI_IRQ_FLAGS_MASK		\
+	(system_uses_nmi() ? (DAIF_MASK | ALLINT_ALLINT) : (DAIF_MASK))
 
 /*
  * Even when Linux uses IRQ priorities for IRQ disabling, EFI does not.
  * And EFI shouldn't really play around with priority masking as it is not aware
  * which priorities the OS has assigned to its interrupts.
  */
-#define arch_efi_save_flags(state_flags)		\
-	((void)((state_flags) = read_sysreg(daif)))
-
-#define arch_efi_restore_flags(state_flags)	write_sysreg(state_flags, daif)
-
+#define arch_efi_save_flags(state_flags)			\
+	do {							\
+		((void)((state_flags) = read_sysreg(daif)));	\
+		if (system_uses_nmi())				\
+			(state_flags) |= (read_sysreg_s(SYS_ALLINT) & ALLINT_ALLINT);	\
+	} while (0)
+
+#define arch_efi_restore_flags(state_flags)			\
+	do {							\
+		u64 __flags = (state_flags);			\
+		write_sysreg(__flags & DAIF_MASK, daif);	\
+		if (system_uses_nmi())				\
+			write_sysreg_s(__flags & ALLINT_ALLINT, SYS_ALLINT);	\
+	} while (0)
 
 /* arch specific definitions used by the stub code */
 
diff --git a/arch/arm64/include/asm/exception_masks.h b/arch/arm64/include/asm/exception_masks.h
index fbbba769ca03..3dd41998f705 100644
--- a/arch/arm64/include/asm/exception_masks.h
+++ b/arch/arm64/include/asm/exception_masks.h
@@ -10,6 +10,7 @@
 #include <asm/arch_gicv3.h>
 #include <asm/barrier.h>
 #include <asm/cpufeature.h>
+#include <asm/nmi.h>
 #include <asm/ptrace.h>
 
 /*
@@ -46,7 +47,10 @@ static inline struct exception_mask arm64_make_errctx_mask(void)
 	if (system_uses_irq_prio_masking())
 		mask.pmr = GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET;
 
-	mask.allint = 0;
+	if (system_uses_nmi())
+		mask.allint = ALLINT_ALLINT;
+	else
+		mask.allint = 0;
 
 	return mask;
 }
@@ -78,6 +82,9 @@ static inline void local_exception_mask(void)
 	if (system_uses_irq_prio_masking())
 		gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
 
+	if (system_uses_nmi())
+		_allint_set();
+
 	trace_hardirqs_off();
 }
 
@@ -87,7 +94,8 @@ static inline void local_exception_save_mask(struct exception_mask *mask)
 	if (system_uses_irq_prio_masking())
 		mask->pmr = gic_read_pmr();
 
-	mask->allint = 0;
+	if (system_uses_nmi())
+		mask->allint = read_sysreg_s(SYS_ALLINT);
 }
 
 static inline struct exception_mask local_exception_save_and_mask(void)
@@ -118,6 +126,9 @@ static inline void local_exception_restore(const struct exception_mask mask)
 		pmr_sync();
 	}
 
+	if (system_uses_nmi())
+		write_sysreg_s(mask.allint, SYS_ALLINT);
+
 	write_sysreg(mask.daif, daif);
 
 	if (irq_disabled)
@@ -137,6 +148,13 @@ static inline void local_exception_inherit(struct pt_regs *regs)
 		gic_write_pmr(regs->pmr);
 
 	write_sysreg(regs->pstate & DAIF_MASK, daif);
+
+	if (system_uses_nmi()) {
+		if (regs->pstate & ALLINT_ALLINT)
+			_allint_set();
+		else
+			_allint_clear();
+	}
 }
 
 /*
diff --git a/arch/arm64/include/uapi/asm/ptrace.h b/arch/arm64/include/uapi/asm/ptrace.h
index 6fed93fb2536..99352dd823cc 100644
--- a/arch/arm64/include/uapi/asm/ptrace.h
+++ b/arch/arm64/include/uapi/asm/ptrace.h
@@ -48,6 +48,7 @@
 #define PSR_D_BIT	0x00000200
 #define PSR_BTYPE_MASK	0x00000c00
 #define PSR_SSBS_BIT	0x00001000
+#define PSR_ALLINT_BIT	0x00002000
 #define PSR_PAN_BIT	0x00400000
 #define PSR_UAO_BIT	0x00800000
 #define PSR_DIT_BIT	0x01000000
diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c
index 70d311e415ac..8045245ebbdd 100644
--- a/arch/arm64/kernel/probes/kprobes.c
+++ b/arch/arm64/kernel/probes/kprobes.c
@@ -197,12 +197,19 @@ static void __kprobes kprobes_save_local_irqflag(struct kprobe_ctlblk *kcb,
 {
 	kcb->saved_irqflag = regs->pstate & DAIF_MASK;
 	regs->pstate |= DAIF_MASK;
+
+	if (system_uses_nmi())
+		regs->pstate |= ALLINT_ALLINT;
 }
 
 static void __kprobes kprobes_restore_local_irqflag(struct kprobe_ctlblk *kcb,
 						struct pt_regs *regs)
 {
 	regs->pstate &= ~DAIF_MASK;
+
+	if (system_uses_nmi())
+		regs->pstate &= ~ALLINT_ALLINT;
+
 	regs->pstate |= kcb->saved_irqflag;
 }
 
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 033643cd4e5e..1aa8f0f66484 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -182,7 +182,7 @@ static void print_pstate(struct pt_regs *regs)
 		const char *btype_str = btypes[(pstate & PSR_BTYPE_MASK) >>
 					       PSR_BTYPE_SHIFT];
 
-		printk("pstate: %08llx (%c%c%c%c %c%c%c%c %cPAN %cUAO %cTCO %cDIT %cSSBS BTYPE=%s)\n",
+		printk("pstate: %08llx (%c%c%c%c %c%c%c%c %cPAN %cUAO %cTCO %cDIT %cSSBS %cALLINT BTYPE=%s)\n",
 			pstate,
 			pstate & PSR_N_BIT ? 'N' : 'n',
 			pstate & PSR_Z_BIT ? 'Z' : 'z',
@@ -197,6 +197,7 @@ static void print_pstate(struct pt_regs *regs)
 			pstate & PSR_TCO_BIT ? '+' : '-',
 			pstate & PSR_DIT_BIT ? '+' : '-',
 			pstate & PSR_SSBS_BIT ? '+' : '-',
+			pstate & PSR_ALLINT_BIT ? '+' : '-',
 			btype_str);
 	}
 }
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 99444a1b2ffa..111cd882c7a9 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -867,10 +867,10 @@ static void __gic_handle_irq_from_irqson(struct pt_regs *regs)
 		nmi_exit();
 	}
 
-	if (gic_prio_masking_enabled()) {
+	if (gic_prio_masking_enabled())
 		gic_pmr_mask_irqs();
-		gic_arch_enable_irqs();
-	}
+
+	gic_arch_enable_irqs();
 
 	if (!is_nmi)
 		__gic_handle_irq(irqnr, regs);
-- 
2.34.1

[PATCH 13/17] arm64/entry: Don't call preempt_schedule_irq() with NMIs masked

[Jinjie Ruan]

From: Mark Brown <broonie@kernel.org>

As we do for pseudo NMIs don't call preempt_schedule_irq()
when architechted NMIs are masked. If they are masked then we are
calling from a preempting context so skip preemption.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/entry-common.h | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
index 3d8b38ce7afb..4efcc7cfdd9b 100644
--- a/arch/arm64/include/asm/entry-common.h
+++ b/arch/arm64/include/asm/entry-common.h
@@ -29,6 +29,15 @@ static __always_inline void arch_exit_to_user_mode_work(struct pt_regs *regs,
 
 static inline bool arch_irqentry_exit_need_resched(void)
 {
+	/*
+	 * Architected NMIs are unmasked prior to handling regular
+	 * IRQs and masked while handling FIQs. If ALLINT is set then
+	 * we are in a NMI or other preempting context so skip
+	 * preemption.
+	 */
+	if (system_uses_nmi() && (read_sysreg_s(SYS_ALLINT) & ALLINT_ALLINT))
+		return false;
+
 	/*
 	 * DAIF.DA are cleared at the start of IRQ/FIQ handling, and when GIC
 	 * priority masking is used the GIC irqchip driver will clear DAIF.IF
-- 
2.34.1

[PATCH 14/17] arm64/irq: Document handling of FEAT_NMI in irqflags.h

[Jinjie Ruan]

From: Mark Brown <broonie@kernel.org>

We have documentation at the top of irqflags.h which explains the DAIF
masking. Since the additional masking with NMIs is related and also
covers the IF in DAIF extend the comment to note what's going on with NMIs
though none of the code in irqflags.h is updated to handle NMIs.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20221112151708.175147-12-broonie@kernel.org
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/irqflags.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/arm64/include/asm/irqflags.h b/arch/arm64/include/asm/irqflags.h
index a8cb5a5c93b7..f2af9f423d88 100644
--- a/arch/arm64/include/asm/irqflags.h
+++ b/arch/arm64/include/asm/irqflags.h
@@ -18,6 +18,16 @@
  * always masked and unmasked together, and have no side effects for other
  * flags. Keeping to this order makes it easier for entry.S to know which
  * exceptions should be unmasked.
+ *
+ * With the addition of the FEAT_NMI extension we gain an additional
+ * class of superpriority IRQ/FIQ which is separately masked with a
+ * choice of modes controlled by SCTLR_ELn.{SPINTMASK,NMI}.  Linux
+ * sets SPINTMASK to 0 and NMI to 1 which results in ALLINT.ALLINT
+ * masking both superpriority interrupts and IRQ/FIQ regardless of the
+ * I and F settings. Since these superpriority interrupts are being
+ * used as NMIs we do not include them in the interrupt masking here,
+ * anything that requires that NMIs be masked needs to explicitly do
+ * so.
  */
 
 static __always_inline void __daif_local_irq_enable(void)
-- 
2.34.1

[PATCH 15/17] arm64/nmi: Add handling of superpriority interrupts as NMIs

[Jinjie Ruan]

From: Mark Brown <broonie@kernel.org>

Our goal with superpriority interrupts is to use them as NMIs, taking
advantage of the much smaller regions where they are masked to allow
prompt handling of the most time critical interrupts.

When an interrupt configured with superpriority we will enter EL1 as
normal for any interrupt, the presence of a superpriority interrupt is
indicated with a status bit in ISR_EL1. We use this to check for the
presence of a superpriority interrupt before we unmask anything in
elX_interrupt(), reporting without unmasking any interrupts. If no
superpriority interrupt is present then we handle normal interrupts as
normal, superpriority interrupts will be unmasked while doing so as a
result of setting DAIF_PROCCTX.

Both IRQs and FIQs may be configured with superpriority so we handle
both, passing an additional root handler into the elX_interrupt()
function along with the mask for the bit in ISR_EL1 which indicates the
presence of the relevant kind of superpriority interrupt. These root
handlers can be configured by the interrupt controller similarly to the
root handlers for normal interrupts using the newly added
set_handle_nmi_irq() and set_handle_nmi_fiq() functions.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20221112151708.175147-13-broonie@kernel.org
Co-developed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/include/asm/irq.h     |  2 ++
 arch/arm64/kernel/entry-common.c | 59 +++++++++++++++++++++++++++-----
 arch/arm64/kernel/irq.c          | 32 +++++++++++++++++
 arch/arm64/kernel/process.c      |  4 +++
 4 files changed, 89 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h
index e93548914c36..bf77769c5dfb 100644
--- a/arch/arm64/include/asm/irq.h
+++ b/arch/arm64/include/asm/irq.h
@@ -13,6 +13,8 @@ void arch_trigger_cpumask_backtrace(const cpumask_t *mask, int exclude_cpu);
 
 struct pt_regs;
 
+int set_handle_nmi_irq(void (*handle_irq)(struct pt_regs *));
+int set_handle_nmi_fiq(void (*handle_fiq)(struct pt_regs *));
 int set_handle_irq(void (*handle_irq)(struct pt_regs *));
 #define set_handle_irq	set_handle_irq
 int set_handle_fiq(void (*handle_fiq)(struct pt_regs *));
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index dc223085e773..4115a5e7c392 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -162,6 +162,8 @@ static void do_interrupt_handler(struct pt_regs *regs,
 	set_irq_regs(old_regs);
 }
 
+extern void (*handle_arch_nmi_irq)(struct pt_regs *);
+extern void (*handle_arch_nmi_fiq)(struct pt_regs *);
 extern void (*handle_arch_irq)(struct pt_regs *);
 extern void (*handle_arch_fiq)(struct pt_regs *);
 
@@ -514,6 +516,16 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
 	}
 }
 
+static __always_inline void __el1_nmi(struct pt_regs *regs,
+				      void (*handler)(struct pt_regs *))
+{
+	irqentry_state_t state;
+
+	state = irqentry_nmi_enter(regs);
+	do_interrupt_handler(regs, handler);
+	irqentry_nmi_exit(regs, state);
+}
+
 static __always_inline void __el1_pnmi(struct pt_regs *regs,
 				       void (*handler)(struct pt_regs *))
 {
@@ -537,11 +549,20 @@ static __always_inline void __el1_irq(struct pt_regs *regs,
 
 	arm64_exit_to_kernel_mode(regs, state);
 }
-static void noinstr el1_interrupt(struct pt_regs *regs,
-				  void (*handler)(struct pt_regs *))
+static void noinstr el1_interrupt(struct pt_regs *regs, u64 nmi_flag,
+				  void (*handler)(struct pt_regs *),
+				  void (*nmi_handler)(struct pt_regs *))
 {
 	struct exception_mask mask = irq_entry_unmask_debug_serror(regs);
 
+	if (system_uses_nmi()) {
+		/* Is there a NMI to handle? */
+		if (read_sysreg(isr_el1) & nmi_flag) {
+			__el1_nmi(regs, nmi_handler);
+			return;
+		}
+	}
+
 	if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && regs_irqs_disabled(regs))
 		__el1_pnmi(regs, handler);
 	else
@@ -552,12 +573,12 @@ static void noinstr el1_interrupt(struct pt_regs *regs,
 
 asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs)
 {
-	el1_interrupt(regs, handle_arch_irq);
+	el1_interrupt(regs, ISR_EL1_IS, handle_arch_irq, handle_arch_nmi_irq);
 }
 
 asmlinkage void noinstr el1h_64_fiq_handler(struct pt_regs *regs)
 {
-	el1_interrupt(regs, handle_arch_fiq);
+	el1_interrupt(regs, ISR_EL1_FS, handle_arch_fiq, handle_arch_nmi_fiq);
 }
 
 asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
@@ -844,13 +865,35 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs)
 	}
 }
 
-static void noinstr el0_interrupt(struct pt_regs *regs,
-				  void (*handler)(struct pt_regs *))
+static void noinstr el0_interrupt(struct pt_regs *regs, u64 nmi_flag,
+				  void (*handler)(struct pt_regs *),
+				  void (*nmi_handler)(struct pt_regs *))
 {
 	arm64_enter_from_user_mode(regs);
 
 	irq_entry_unmask_debug_serror(regs);
 
+	if (system_uses_nmi()) {
+		irqentry_state_t state;
+
+		/* Is there a NMI to handle? */
+		if (read_sysreg(isr_el1) & nmi_flag) {
+			/*
+			 * Any system with FEAT_NMI should not be
+			 * affected by Spectre v2 so we don't mitigate
+			 * here.
+			 */
+			state = irqentry_nmi_enter(regs);
+			do_interrupt_handler(regs, nmi_handler);
+			irqentry_nmi_exit(regs, state);
+
+			_allint_clear();
+
+			arm64_exit_to_user_mode(regs);
+			return;
+		}
+	}
+
 	if (regs->pc & BIT(55))
 		arm64_apply_bp_hardening();
 
@@ -863,7 +906,7 @@ static void noinstr el0_interrupt(struct pt_regs *regs,
 
 static void noinstr __el0_irq_handler_common(struct pt_regs *regs)
 {
-	el0_interrupt(regs, handle_arch_irq);
+	el0_interrupt(regs, ISR_EL1_IS, handle_arch_irq, handle_arch_nmi_irq);
 }
 
 asmlinkage void noinstr el0t_64_irq_handler(struct pt_regs *regs)
@@ -873,7 +916,7 @@ asmlinkage void noinstr el0t_64_irq_handler(struct pt_regs *regs)
 
 static void noinstr __el0_fiq_handler_common(struct pt_regs *regs)
 {
-	el0_interrupt(regs, handle_arch_fiq);
+	el0_interrupt(regs, ISR_EL1_FS, handle_arch_fiq, handle_arch_nmi_fiq);
 }
 
 asmlinkage void noinstr el0t_64_fiq_handler(struct pt_regs *regs)
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index fff0efd5a195..54e25b4c0698 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -84,6 +84,16 @@ void do_softirq_own_stack(void)
 }
 #endif
 
+static void default_handle_nmi_irq(struct pt_regs *regs)
+{
+	panic("Superpriority IRQ taken without a root NMI IRQ handler\n");
+}
+
+static void default_handle_nmi_fiq(struct pt_regs *regs)
+{
+	panic("Superpriority FIQ taken without a root NMI FIQ handler\n");
+}
+
 static void default_handle_irq(struct pt_regs *regs)
 {
 	panic("IRQ taken without a root IRQ handler\n");
@@ -94,9 +104,31 @@ static void default_handle_fiq(struct pt_regs *regs)
 	panic("FIQ taken without a root FIQ handler\n");
 }
 
+void (*handle_arch_nmi_irq)(struct pt_regs *) __ro_after_init = default_handle_nmi_irq;
+void (*handle_arch_nmi_fiq)(struct pt_regs *) __ro_after_init = default_handle_nmi_fiq;
 void (*handle_arch_irq)(struct pt_regs *) __ro_after_init = default_handle_irq;
 void (*handle_arch_fiq)(struct pt_regs *) __ro_after_init = default_handle_fiq;
 
+int __init set_handle_nmi_irq(void (*handle_nmi_irq)(struct pt_regs *))
+{
+	if (handle_arch_nmi_irq != default_handle_nmi_irq)
+		return -EBUSY;
+
+	handle_arch_nmi_irq = handle_nmi_irq;
+	pr_info("Root superpriority IRQ handler: %ps\n", handle_nmi_irq);
+	return 0;
+}
+
+int __init set_handle_nmi_fiq(void (*handle_nmi_fiq)(struct pt_regs *))
+{
+	if (handle_arch_nmi_fiq != default_handle_nmi_fiq)
+		return -EBUSY;
+
+	handle_arch_nmi_fiq = handle_nmi_fiq;
+	pr_info("Root superpriority FIQ handler: %ps\n", handle_nmi_fiq);
+	return 0;
+}
+
 int __init set_handle_irq(void (*handle_irq)(struct pt_regs *))
 {
 	if (handle_arch_irq != default_handle_irq)
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 1aa8f0f66484..b0244005e20f 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -757,6 +757,10 @@ static inline void debug_switch_state(void)
 		WARN_ONCE(daif_actual != daif_expected,
 			  "Unexpected DAIF value: 0x%lx (expected 0x%lx)\n",
 			  daif_actual, daif_expected);
+
+		if (system_uses_nmi())
+			WARN_ONCE(read_sysreg_s(SYS_ALLINT) & ALLINT_ALLINT,
+				  "Unexpected ALLINT set\n");
 	}
 }
 
-- 
2.34.1

[PATCH 16/17] arm64/nmi: Add Kconfig for NMI

[Jinjie Ruan]

From: Mark Brown <broonie@kernel.org>

Since NMI handling is in some fairly hot paths we provide a Kconfig option
which allows support to be compiled out when not needed.

Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20221112151708.175147-14-broonie@kernel.org
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 arch/arm64/Kconfig | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b3afe0688919..218996752079 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -2195,6 +2195,23 @@ endmenu # "ARMv8.7 architectural features"
 config AS_HAS_MOPS
 	def_bool $(as-instr,.arch_extension mops)
 
+menu "ARMv8.8 architectural features"
+
+config ARM64_NMI
+	bool "Enable support for Non-maskable Interrupts (NMI)"
+	default y
+	help
+	  Non-maskable interrupts are an architecture and GIC feature
+	  which allow the system to configure some interrupts to be
+	  configured to have superpriority, allowing them to be handled
+	  before other interrupts and masked for shorter periods of time.
+
+	  The feature is detected at runtime, and will remain disabled
+	  if the cpu does not implement the feature. It will also be
+	  disabled if pseudo NMIs are enabled at runtime.
+
+endmenu # "ARMv8.8 architectural features"
+
 menu "ARMv8.9 architectural features"
 
 config ARM64_POE
-- 
2.34.1

[PATCH 17/17] irqchip/gic-v3: Implement FEAT_GICv3_NMI support

[Jinjie Ruan]

From: Lorenzo Pieralisi <lpieralisi@kernel.org>

The FEAT_GICv3_NMI GIC feature coupled with the CPU FEAT_NMI enables
handling NMI interrupts in HW on aarch64, by adding a superpriority
interrupt to the existing GIC priority scheme.

Implement GIC driver support for the FEAT_GICv3_NMI feature.

Rename gic_supports_nmi() helper function to gic_supports_pseudo_nmis()
to make the pseudo NMIs code path clearer and more explicit.

Check, through the ARM64 capabilitity infrastructure, if support
for FEAT_NMI was detected on the core and the system has not overridden
the detection and forced pseudo-NMIs enablement.

If FEAT_NMI is detected, it was not overridden (check embedded in the
system_uses_nmi() call) and the GIC supports the FEAT_GICv3_NMI feature,
install an NMI handler and initialize NMIs related HW GIC registers.

Signed-off-by: Lorenzo Pieralisi <lpieralisi@kernel.org>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20221112151708.175147-15-broonie@kernel.org
Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
---
 drivers/irqchip/irq-gic-v3.c       | 146 ++++++++++++++++++++++++++---
 include/linux/irqchip/arm-gic-v3.h |   4 +
 2 files changed, 138 insertions(+), 12 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 111cd882c7a9..6e48dd999021 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -64,6 +64,7 @@ struct gic_chip_data {
 	u32			nr_redist_regions;
 	u64			flags;
 	bool			has_rss;
+	bool			has_nmi;
 	unsigned int		ppi_nr;
 	struct partition_affinity *parts;
 	unsigned int		nr_parts;
@@ -252,6 +253,20 @@ enum gic_intid_range {
 	__INVALID_RANGE__
 };
 
+#ifdef CONFIG_ARM64
+#include <linux/cpufeature.h>
+
+static inline bool has_v3_3_nmi(void)
+{
+	return gic_data.has_nmi && system_uses_nmi();
+}
+#else
+static inline bool has_v3_3_nmi(void)
+{
+	return false;
+}
+#endif
+
 static enum gic_intid_range __get_intid_range(irq_hw_number_t hwirq)
 {
 	switch (hwirq) {
@@ -467,6 +482,42 @@ static int gic_peek_irq(struct irq_data *d, u32 offset)
 	return !!(readl_relaxed(base + offset + (index / 32) * 4) & mask);
 }
 
+static DEFINE_RAW_SPINLOCK(irq_controller_lock);
+
+static void gic_irq_configure_nmi(struct irq_data *d, bool enable)
+{
+	void __iomem *base, *addr;
+	u32 offset, index, mask, val;
+
+	offset = convert_offset_index(d, GICD_INMIR, &index);
+	mask = 1 << (index % 32);
+
+	if (gic_irq_in_rdist(d))
+		base = gic_data_rdist_sgi_base();
+	else
+		base = gic_data.dist_base;
+
+	addr = base + offset + (index / 32) * 4;
+
+	raw_spin_lock(&irq_controller_lock);
+
+	val = readl_relaxed(addr);
+	val = enable ? (val | mask) : (val & ~mask);
+	writel_relaxed(val, addr);
+
+	raw_spin_unlock(&irq_controller_lock);
+}
+
+static void gic_irq_enable_nmi(struct irq_data *d)
+{
+	gic_irq_configure_nmi(d, true);
+}
+
+static void gic_irq_disable_nmi(struct irq_data *d)
+{
+	gic_irq_configure_nmi(d, false);
+}
+
 static void gic_poke_irq(struct irq_data *d, u32 offset)
 {
 	void __iomem *base;
@@ -512,7 +563,7 @@ static void gic_unmask_irq(struct irq_data *d)
 	gic_poke_irq(d, GICD_ISENABLER);
 }
 
-static inline bool gic_supports_nmi(void)
+static inline bool gic_supports_pseudo_nmis(void)
 {
 	return IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) &&
 	       static_branch_likely(&supports_pseudo_nmis);
@@ -598,7 +649,7 @@ static int gic_irq_nmi_setup(struct irq_data *d)
 {
 	struct irq_desc *desc = irq_to_desc(d->irq);
 
-	if (!gic_supports_nmi())
+	if (!gic_supports_pseudo_nmis() && !has_v3_3_nmi())
 		return -EINVAL;
 
 	if (gic_peek_irq(d, GICD_ISENABLER)) {
@@ -617,7 +668,10 @@ static int gic_irq_nmi_setup(struct irq_data *d)
 	if (!gic_irq_in_rdist(d))
 		desc->handle_irq = handle_fasteoi_nmi;
 
-	gic_irq_set_prio(d, dist_prio_nmi);
+	if (has_v3_3_nmi())
+		gic_irq_enable_nmi(d);
+	else
+		gic_irq_set_prio(d, dist_prio_nmi);
 
 	return 0;
 }
@@ -626,7 +680,7 @@ static void gic_irq_nmi_teardown(struct irq_data *d)
 {
 	struct irq_desc *desc = irq_to_desc(d->irq);
 
-	if (WARN_ON(!gic_supports_nmi()))
+	if (WARN_ON(!gic_supports_pseudo_nmis() && !has_v3_3_nmi()))
 		return;
 
 	if (gic_peek_irq(d, GICD_ISENABLER)) {
@@ -645,7 +699,10 @@ static void gic_irq_nmi_teardown(struct irq_data *d)
 	if (!gic_irq_in_rdist(d))
 		desc->handle_irq = handle_fasteoi_irq;
 
-	gic_irq_set_prio(d, dist_prio_irq);
+	if (has_v3_3_nmi())
+		gic_irq_disable_nmi(d);
+	else
+		gic_irq_set_prio(d, dist_prio_irq);
 }
 
 static bool gic_arm64_erratum_2941627_needed(struct irq_data *d)
@@ -804,7 +861,7 @@ static inline void gic_complete_ack(u32 irqnr)
 
 static bool gic_rpr_is_nmi_prio(void)
 {
-	if (!gic_supports_nmi())
+	if (!gic_supports_pseudo_nmis())
 		return false;
 
 	return unlikely(gic_read_rpr() == GICV3_PRIO_NMI);
@@ -836,7 +893,8 @@ static void __gic_handle_nmi(u32 irqnr, struct pt_regs *regs)
 	gic_complete_ack(irqnr);
 
 	if (generic_handle_domain_nmi(gic_data.domain, irqnr)) {
-		WARN_ONCE(true, "Unexpected pseudo-NMI (irqnr %u)\n", irqnr);
+		WARN_ONCE(true, "Unexpected %sNMI (irqnr %u)\n",
+			  gic_supports_pseudo_nmis() ? "pseudo-" : "", irqnr);
 		gic_deactivate_unhandled(irqnr);
 	}
 }
@@ -857,6 +915,28 @@ static void __gic_handle_irq_from_irqson(struct pt_regs *regs)
 	bool is_nmi;
 	u32 irqnr;
 
+	/*
+	 * We should enter here with interrupts disabled, otherwise we may met
+	 * a race here with FEAT_NMI/FEAT_GICv3_NMI:
+	 *
+	 * [interrupt disabled]
+	 *                   <- normal interrupt pending, for example timer interrupt
+	 *                   <- NMI occurs, ISR_EL1.nmi = 1
+	 * do_el1_interrupt()
+	 *                   <- NMI withdraw, ISR_EL1.nmi = 0
+	 *   ISR_EL1.nmi = 0, not an NMI interrupt
+	 *   gic_handle_irq()
+	 *     __gic_handle_irq_from_irqson()
+	 *       irqnr = gic_read_iar() <- Oops, ack and handle an normal interrupt
+	 *                                 in interrupt disabled context!
+	 *
+	 * So if we met this case here, just return from the interrupt context.
+	 * Since the interrupt is still pending, we can handle it once the
+	 * interrupt re-enabled and it'll not be missing.
+	 */
+	if (!interrupts_enabled(regs))
+		return;
+
 	irqnr = gic_read_iar();
 
 	is_nmi = gic_rpr_is_nmi_prio();
@@ -912,9 +992,37 @@ static void __gic_handle_irq_from_irqsoff(struct pt_regs *regs)
 	__gic_handle_nmi(irqnr, regs);
 }
 
+#ifdef CONFIG_ARM64
+static inline u64 gic_read_nmiar(void)
+{
+	u64 irqstat;
+
+	irqstat = read_sysreg_s(SYS_ICC_NMIAR1_EL1);
+
+	dsb(sy);
+
+	return irqstat;
+}
+
+static asmlinkage void __exception_irq_entry gic_handle_nmi_irq(struct pt_regs *regs)
+{
+	u32 irqnr = gic_read_nmiar();
+
+	__gic_handle_nmi(irqnr, regs);
+}
+
+static inline void gic_setup_nmi_handler(void)
+{
+	if (has_v3_3_nmi())
+		set_handle_nmi_irq(gic_handle_nmi_irq);
+}
+#else
+static inline void gic_setup_nmi_handler(void) { }
+#endif
+
 static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
 {
-	if (unlikely(gic_supports_nmi() && !interrupts_enabled(regs)))
+	if (unlikely(gic_supports_pseudo_nmis() && !interrupts_enabled(regs)))
 		__gic_handle_irq_from_irqsoff(regs);
 	else
 		__gic_handle_irq_from_irqson(regs);
@@ -1165,7 +1273,7 @@ static void gic_cpu_sys_reg_init(void)
 	/* Set priority mask register */
 	if (!gic_prio_masking_enabled()) {
 		write_gicreg(DEFAULT_PMR_VALUE, ICC_PMR_EL1);
-	} else if (gic_supports_nmi()) {
+	} else if (gic_supports_pseudo_nmis()) {
 		/*
 		 * Check that all CPUs use the same priority space.
 		 *
@@ -1958,15 +2066,27 @@ static const struct gic_quirk gic_quirks[] = {
 	}
 };
 
-static void gic_enable_nmi_support(void)
+static void gic_enable_pseudo_nmis(void)
 {
-	if (!gic_prio_masking_enabled() || nmi_support_forbidden)
-		return;
 
 	pr_info("Pseudo-NMIs enabled using %s ICC_PMR_EL1 synchronisation\n",
 		gic_has_relaxed_pmr_sync() ? "relaxed" : "forced");
 
 	static_branch_enable(&supports_pseudo_nmis);
+}
+
+static void gic_enable_nmi_support(void)
+{
+	if ((!gic_prio_masking_enabled() || nmi_support_forbidden) &&
+	     !has_v3_3_nmi())
+		return;
+
+	/*
+	 * Initialize pseudo-NMIs only if GIC driver cannot take advantage
+	 * of core (FEAT_NMI) and GIC (FEAT_GICv3_NMI) in HW
+	 */
+	if (!has_v3_3_nmi())
+		gic_enable_pseudo_nmis();
 
 	if (static_branch_likely(&supports_deactivate_key))
 		gic_eoimode1_chip.flags |= IRQCHIP_SUPPORTS_NMI;
@@ -2035,6 +2155,7 @@ static int __init gic_init_bases(phys_addr_t dist_phys_base,
 	irq_domain_update_bus_token(gic_data.domain, DOMAIN_BUS_WIRED);
 
 	gic_data.has_rss = !!(typer & GICD_TYPER_RSS);
+	gic_data.has_nmi = !!(typer & GICD_TYPER_NMI);
 
 	if (typer & GICD_TYPER_MBIS) {
 		err = mbi_init(handle, gic_data.domain);
@@ -2043,6 +2164,7 @@ static int __init gic_init_bases(phys_addr_t dist_phys_base,
 	}
 
 	set_handle_irq(gic_handle_irq);
+	gic_setup_nmi_handler();
 
 	gic_update_rdist_properties();
 
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index ea5fd2374ebe..ebccd76cadda 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -30,6 +30,7 @@
 #define GICD_ICFGR			0x0C00
 #define GICD_IGRPMODR			0x0D00
 #define GICD_NSACR			0x0E00
+#define GICD_INMIR			0x0F80
 #define GICD_IGROUPRnE			0x1000
 #define GICD_ISENABLERnE		0x1200
 #define GICD_ICENABLERnE		0x1400
@@ -39,6 +40,7 @@
 #define GICD_ICACTIVERnE		0x1C00
 #define GICD_IPRIORITYRnE		0x2000
 #define GICD_ICFGRnE			0x3000
+#define GICD_INMIRnE			0x3B00
 #define GICD_IROUTER			0x6000
 #define GICD_IROUTERnE			0x8000
 #define GICD_IDREGS			0xFFD0
@@ -83,6 +85,7 @@
 #define GICD_TYPER_LPIS			(1U << 17)
 #define GICD_TYPER_MBIS			(1U << 16)
 #define GICD_TYPER_ESPI			(1U << 8)
+#define GICD_TYPER_NMI			(1U << 9)
 
 #define GICD_TYPER_ID_BITS(typer)	((((typer) >> 19) & 0x1f) + 1)
 #define GICD_TYPER_NUM_LPIS(typer)	((((typer) >> 11) & 0x1f) + 1)
@@ -238,6 +241,7 @@
 #define GICR_ICFGR0			GICD_ICFGR
 #define GICR_IGRPMODR0			GICD_IGRPMODR
 #define GICR_NSACR			GICD_NSACR
+#define GICR_INMIR0			GICD_INMIR
 
 #define GICR_TYPER_PLPIS		(1U << 0)
 #define GICR_TYPER_VLPIS		(1U << 1)
-- 
2.34.1

Re: [PATCH 02/17] arm64: Rework exception masking into abstract logical mask

[Leonardo Bras]

On Fri, Jul 03, 2026 at 06:01:20PM +0800, Jinjie Ruan wrote:
> The current pseudo-NMI implementation in arm64 bodges the GICv3 PMR
> state into the DAIF management code. This approach forces local_daif_save()
> and local_daif_restore() to pretend that PMR is a part of DAIF. This has
> proven to be convoluted, difficult to maintain, and prone to architectural
> corner cases where it cannot do the right thing (e.g., faking up DAIF
> elements during irqflags/lockdep tracking).
> 
> As we prepare for the hardware-based NMI (FEAT_NMI, which use ALLINT
> to mask NMI) support, continuing down this path will significantly
> increase technical debt. We must clean up the exception masking
> abstraction ahead of ALLINT integration.
> 
> Introduce a new abstract `struct exception_mask` that explicitly treats
> DAIF, PMR, and ALLINT as separate, independent elements of the logical
> exception state. This allows the kernel to track and manipulate the exact
> hardware masking state cleanly without faking up DAIF flags.
> 
> Introduce the following new APIs to replace the legacy DAIF-centric
> helpers:
> 
>  - arm64_make_{procctx,errctx,noirq}_mask() to generate logical masks.
> 
>  - local_exception_save_and_mask() / local_exception_restore() to replace
>    the old local_daif_save() / local_daif_restore() pattern.
> 
>  - local_exception_inherit() to replace local_daif_inherit().
> 
> Mechanically convert all existing callers across the arm64 architecture
> to use the new logical exception mask helpers. The conversion spans:
> 
> - Core exception entry/exit handling and traps.
> 
> - KVM nVHE/VHE hypervisor switch and GICv3 save/restore code
> 
> - CPU idle, suspend, hibernate, and PSCI firmware invocations.
> 
> - Subsystems including SMP, ACPI, kprobes, and debug monitors
> 
> Link: https://lore.kernel.org/linux-arm-kernel/Y5c9SLeJacLYHmP7@FVFF77S0Q05N/
> Suggested-by: Mark Rutland <mark.rutland@arm.com>
> Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
> ---
>  arch/arm64/include/asm/cpuidle.h         |  29 ++---
>  arch/arm64/include/asm/daifflags.h       | 138 ----------------------
>  arch/arm64/include/asm/entry-common.h    |   2 +-
>  arch/arm64/include/asm/exception_masks.h | 141 +++++++++++++++++++++++
>  arch/arm64/include/asm/kvm_host.h        |   2 +-
>  arch/arm64/include/asm/mmu_context.h     |   2 +-
>  arch/arm64/kernel/acpi.c                 |  14 +--
>  arch/arm64/kernel/debug-monitors.c       |   9 +-
>  arch/arm64/kernel/entry-common.c         |  74 ++++++------
>  arch/arm64/kernel/hibernate.c            |  10 +-
>  arch/arm64/kernel/idle.c                 |   7 +-
>  arch/arm64/kernel/irq.c                  |   4 +-
>  arch/arm64/kernel/machine_kexec.c        |   4 +-
>  arch/arm64/kernel/probes/kprobes.c       |   2 +-
>  arch/arm64/kernel/setup.c                |   4 +-
>  arch/arm64/kernel/signal.c               |   2 +-
>  arch/arm64/kernel/smp.c                  |  22 ++--
>  arch/arm64/kernel/suspend.c              |  15 ++-
>  arch/arm64/kernel/traps.c                |   2 +-
>  arch/arm64/kvm/hyp/vgic-v3-sr.c          |   6 +-
>  arch/arm64/kvm/hyp/vhe/switch.c          |   6 +-
>  arch/arm64/mm/fault.c                    |   2 +-
>  arch/arm64/mm/mmu.c                      |   6 +-
>  drivers/firmware/psci/psci.c             |   7 +-
>  24 files changed, 254 insertions(+), 256 deletions(-)
>  delete mode 100644 arch/arm64/include/asm/daifflags.h
>  create mode 100644 arch/arm64/include/asm/exception_masks.h
> 
> diff --git a/arch/arm64/include/asm/cpuidle.h b/arch/arm64/include/asm/cpuidle.h
> index 64ddb533d3ec..4c19d5b79678 100644
> --- a/arch/arm64/include/asm/cpuidle.h
> +++ b/arch/arm64/include/asm/cpuidle.h
> @@ -6,37 +6,30 @@
>  
>  #ifdef CONFIG_ARM64_PSEUDO_NMI
>  #include <asm/arch_gicv3.h>
> +#include <asm/exception_masks.h>
>  #include <asm/ptrace.h>
>  
> -struct arm_cpuidle_irq_context {
> -	unsigned long pmr;
> -	unsigned long daif_bits;
> -};
> -
> -#define arm_cpuidle_save_irq_context(__c)				\
> +#define arm_cpuidle_save_irq_context(__m)				\
>  	do {								\
> -		struct arm_cpuidle_irq_context *c = __c;		\
> +		struct exception_mask *m = __m;				\
>  		if (system_uses_irq_prio_masking()) {			\
> -			c->daif_bits = read_sysreg(daif);		\
> -			write_sysreg(c->daif_bits | DAIF_PROCCTX_NOIRQ, \
> +			local_exception_save_mask(m);			\
> +			write_sysreg(m->daif | DAIF_PROCCTX_NOIRQ,	\
>  				     daif);				\
> -			c->pmr = gic_read_pmr();			\
>  			gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); \
>  		}							\
>  	} while (0)
>  
> -#define arm_cpuidle_restore_irq_context(__c)				\
> +#define arm_cpuidle_restore_irq_context(__m)				\
>  	do {								\
> -		struct arm_cpuidle_irq_context *c = __c;		\
> +		struct exception_mask *m = __m;				\
>  		if (system_uses_irq_prio_masking()) {			\
> -			gic_write_pmr(c->pmr);				\
> -			write_sysreg(c->daif_bits, daif);		\
> +			gic_write_pmr(m->pmr);				\
> +			write_sysreg(m->daif, daif);			\
>  		}							\
>  	} while (0)
>  #else
> -struct arm_cpuidle_irq_context { };
> -
> -#define arm_cpuidle_save_irq_context(c)		(void)c
> -#define arm_cpuidle_restore_irq_context(c)	(void)c
> +#define arm_cpuidle_save_irq_context(m)		((void)m)
> +#define arm_cpuidle_restore_irq_context(m)	((void)m)
>  #endif
>  #endif
> diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
> deleted file mode 100644
> index 3a4b63a948ac..000000000000
> --- a/arch/arm64/include/asm/daifflags.h
> +++ /dev/null
> @@ -1,138 +0,0 @@
> -/* SPDX-License-Identifier: GPL-2.0-only */
> -/*
> - * Copyright (C) 2017 ARM Ltd.
> - */
> -#ifndef __ASM_DAIFFLAGS_H
> -#define __ASM_DAIFFLAGS_H
> -
> -#include <linux/irqflags.h>
> -
> -#include <asm/arch_gicv3.h>
> -#include <asm/barrier.h>
> -#include <asm/cpufeature.h>
> -#include <asm/ptrace.h>
> -
> -/* mask/save/unmask/restore all exceptions, including interrupts. */
> -static __always_inline void local_daif_mask(void)
> -{
> -	WARN_ON(system_has_prio_mask_debugging() &&
> -		(read_sysreg_s(SYS_ICC_PMR_EL1) == (GIC_PRIO_IRQOFF |
> -						    GIC_PRIO_PSR_I_SET)));
> -
> -	asm volatile(
> -		"msr	daifset, #0xf		// local_daif_mask\n"
> -		:
> -		:
> -		: "memory");
> -
> -	/* Don't really care for a dsb here, we don't intend to enable IRQs */
> -	if (system_uses_irq_prio_masking())
> -		gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
> -
> -	trace_hardirqs_off();
> -}
> -
> -static __always_inline unsigned long local_daif_save_flags(void)
> -{
> -	unsigned long flags;
> -
> -	flags = read_sysreg(daif);
> -
> -	if (system_uses_irq_prio_masking()) {
> -		/* If IRQs are masked with PMR, reflect it in the flags */
> -		if (read_sysreg_s(SYS_ICC_PMR_EL1) != GIC_PRIO_IRQON)
> -			flags |= DAIF_PROCCTX_NOIRQ;
> -	}
> -
> -	return flags;
> -}
> -
> -static __always_inline unsigned long local_daif_save(void)
> -{
> -	unsigned long flags;
> -
> -	flags = local_daif_save_flags();
> -
> -	local_daif_mask();
> -
> -	return flags;
> -}
> -
> -static __always_inline void local_daif_restore(unsigned long flags)
> -{
> -	bool irq_disabled = flags & PSR_I_BIT;
> -
> -	WARN_ON(system_has_prio_mask_debugging() &&
> -		(read_sysreg(daif) & DAIF_PROCCTX_NOIRQ) != DAIF_PROCCTX_NOIRQ);
> -
> -	if (!irq_disabled) {
> -		trace_hardirqs_on();
> -
> -		if (system_uses_irq_prio_masking()) {
> -			gic_write_pmr(GIC_PRIO_IRQON);
> -			pmr_sync();
> -		}
> -	} else if (system_uses_irq_prio_masking()) {
> -		u64 pmr;
> -
> -		if (!(flags & PSR_A_BIT)) {
> -			/*
> -			 * If interrupts are disabled but we can take
> -			 * asynchronous errors, we can take NMIs
> -			 */
> -			flags &= ~DAIF_PROCCTX_NOIRQ;
> -			pmr = GIC_PRIO_IRQOFF;
> -		} else {
> -			pmr = GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET;
> -		}
> -
> -		/*
> -		 * There has been concern that the write to daif
> -		 * might be reordered before this write to PMR.
> -		 * From the ARM ARM DDI 0487D.a, section D1.7.1
> -		 * "Accessing PSTATE fields":
> -		 *   Writes to the PSTATE fields have side-effects on
> -		 *   various aspects of the PE operation. All of these
> -		 *   side-effects are guaranteed:
> -		 *     - Not to be visible to earlier instructions in
> -		 *       the execution stream.
> -		 *     - To be visible to later instructions in the
> -		 *       execution stream
> -		 *
> -		 * Also, writes to PMR are self-synchronizing, so no
> -		 * interrupts with a lower priority than PMR is signaled
> -		 * to the PE after the write.
> -		 *
> -		 * So we don't need additional synchronization here.
> -		 */
> -		gic_write_pmr(pmr);
> -	}
> -
> -	write_sysreg(flags, daif);
> -
> -	if (irq_disabled)
> -		trace_hardirqs_off();
> -}
> -
> -/*
> - * Called by synchronous exception handlers to restore the DAIF bits that were
> - * modified by taking an exception.
> - */
> -static __always_inline void local_daif_inherit(struct pt_regs *regs)
> -{
> -	unsigned long flags = regs->pstate & DAIF_MASK;
> -
> -	if (!regs_irqs_disabled(regs))
> -		trace_hardirqs_on();
> -
> -	if (system_uses_irq_prio_masking())
> -		gic_write_pmr(regs->pmr);
> -
> -	/*
> -	 * We can't use local_daif_restore(regs->pstate) here as
> -	 * system_has_prio_mask_debugging() won't restore the I bit if it can
> -	 * use the pmr instead.
> -	 */
> -	write_sysreg(flags, daif);
> -}
> -#endif
> diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
> index cab8cd78f693..3d8b38ce7afb 100644
> --- a/arch/arm64/include/asm/entry-common.h
> +++ b/arch/arm64/include/asm/entry-common.h
> @@ -6,7 +6,7 @@
>  #include <linux/thread_info.h>
>  
>  #include <asm/cpufeature.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/fpsimd.h>
>  #include <asm/mte.h>
>  #include <asm/stacktrace.h>
> diff --git a/arch/arm64/include/asm/exception_masks.h b/arch/arm64/include/asm/exception_masks.h
> new file mode 100644
> index 000000000000..0a21fb97c3ca
> --- /dev/null
> +++ b/arch/arm64/include/asm/exception_masks.h
> @@ -0,0 +1,141 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2017 ARM Ltd.
> + */
> +#ifndef __ASM_EXCEPTION_MASKS_H
> +#define __ASM_EXCEPTION_MASKS_H
> +
> +#include <linux/irqflags.h>
> +
> +#include <asm/arch_gicv3.h>
> +#include <asm/barrier.h>
> +#include <asm/cpufeature.h>
> +#include <asm/ptrace.h>
> +
> +/*
> + * Logical exception mask: tracks the three independent exception
> + * masking controls on arm64:
> + *  - DAIF (PSTATE.{D,A,I,F} bits)
> + *  - PMR  (ICC_PMR_EL1)
> + *  - ALLINT (PSTATE.ALLINT)
> + */
> +struct exception_mask {
> +	unsigned long daif;
> +	unsigned long pmr;
> +	unsigned long allint;		// for future FEAT_NMI use
> +};
> +
> +static inline struct exception_mask arm64_make_procctx_mask(void)

By the code you removed, I understand that those new functions should not 
be marked as 'inline' but as '__always_inline'.

Is there any reason why you decided to use 'inline' instead?

Thanks!
Leo


> +{
> +	struct exception_mask mask;
> +
> +	mask.daif = DAIF_PROCCTX;
> +	if (system_uses_irq_prio_masking())
> +		mask.pmr = GIC_PRIO_IRQON;
> +
> +	mask.allint = 0;
> +
> +	return mask;
> +}
> +
> +static inline struct exception_mask arm64_make_errctx_mask(void)
> +{
> +	struct exception_mask mask;
> +
> +	mask.daif = DAIF_ERRCTX;
> +	if (system_uses_irq_prio_masking())
> +		mask.pmr = GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET;
> +
> +	mask.allint = 0;
> +
> +	return mask;
> +}
> +
> +static inline struct exception_mask arm64_make_noirq_mask(void)
> +{
> +	struct exception_mask mask;
> +
> +	if (system_uses_irq_prio_masking()) {
> +		mask.daif = 0;
> +		mask.pmr = GIC_PRIO_IRQOFF;
> +	} else
> +		mask.daif = DAIF_PROCCTX_NOIRQ;
> +
> +	mask.allint = 0;
> +
> +	return mask;
> +}
> +
> +/* Mask all exceptions immediately */
> +static inline void local_exception_mask(void)
> +{
> +	WARN_ON(system_has_prio_mask_debugging() &&
> +		(read_sysreg_s(SYS_ICC_PMR_EL1) == (GIC_PRIO_IRQOFF |
> +						    GIC_PRIO_PSR_I_SET)));
> +
> +	asm volatile("msr daifset, #0xf" ::: "memory");
> +
> +	if (system_uses_irq_prio_masking())
> +		gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
> +
> +	trace_hardirqs_off();
> +}
> +
> +static inline void local_exception_save_mask(struct exception_mask *mask)
> +{
> +	mask->daif = read_sysreg(daif);
> +	if (system_uses_irq_prio_masking())
> +		mask->pmr = gic_read_pmr();
> +
> +	mask->allint = 0;
> +}
> +
> +static inline struct exception_mask local_exception_save_and_mask(void)
> +{
> +	struct exception_mask mask;
> +
> +	local_exception_save_mask(&mask);
> +	local_exception_mask();
> +
> +	return mask;
> +}
> +
> +static inline void local_exception_restore(const struct exception_mask mask)
> +{
> +	bool irq_disabled = mask.daif & PSR_I_BIT;
> +
> +	if (system_uses_irq_prio_masking())
> +		irq_disabled |= (mask.pmr == GIC_PRIO_IRQOFF);
> +
> +	WARN_ON(system_has_prio_mask_debugging() &&
> +		(read_sysreg(daif) & DAIF_PROCCTX_NOIRQ) != DAIF_PROCCTX_NOIRQ);
> +
> +	if (!irq_disabled)
> +		trace_hardirqs_on();
> +
> +	if (system_uses_irq_prio_masking()) {
> +		gic_write_pmr(mask.pmr);
> +		pmr_sync();
> +	}
> +
> +	write_sysreg(mask.daif, daif);
> +
> +	if (irq_disabled)
> +		trace_hardirqs_off();
> +}
> +
> +/*
> + * Called by synchronous exception handlers to restore the DAIF bits that were
> + * modified by taking an exception.
> + */
> +static inline void local_exception_inherit(struct pt_regs *regs)
> +{
> +	if (!regs_irqs_disabled(regs))
> +		trace_hardirqs_on();
> +
> +	if (system_uses_irq_prio_masking())
> +		gic_write_pmr(regs->pmr);
> +
> +	write_sysreg(regs->pstate & DAIF_MASK, daif);
> +}
> +#endif /* __ASM_EXCEPTION_MASKS_H */
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index bae2c4f92ef5..8a443a379689 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -23,7 +23,7 @@
>  #include <asm/barrier.h>
>  #include <asm/cpufeature.h>
>  #include <asm/cputype.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/fpsimd.h>
>  #include <asm/kvm.h>
>  #include <asm/kvm_asm.h>
> diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
> index 803b68758152..6103925f85b7 100644
> --- a/arch/arm64/include/asm/mmu_context.h
> +++ b/arch/arm64/include/asm/mmu_context.h
> @@ -19,7 +19,7 @@
>  
>  #include <asm/cacheflush.h>
>  #include <asm/cpufeature.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/gcs.h>
>  #include <asm/proc-fns.h>
>  #include <asm/cputype.h>
> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> index 5891f92c2035..4d413419309d 100644
> --- a/arch/arm64/kernel/acpi.c
> +++ b/arch/arm64/kernel/acpi.c
> @@ -33,7 +33,7 @@
>  #include <acpi/processor.h>
>  #include <asm/cputype.h>
>  #include <asm/cpu_ops.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/smp_plat.h>
>  
>  int acpi_noirq = 1;		/* skip ACPI IRQ initialization */
> @@ -388,14 +388,14 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
>   */
>  int apei_claim_sea(struct pt_regs *regs)
>  {
> -	int err = -ENOENT;
> +	struct exception_mask current_mask;
>  	bool return_to_irqs_enabled;
> -	unsigned long current_flags;
> +	int err = -ENOENT;
>  
>  	if (!IS_ENABLED(CONFIG_ACPI_APEI_GHES))
>  		return err;
>  
> -	current_flags = local_daif_save_flags();
> +	local_exception_save_mask(&current_mask);
>  
>  	/* current_flags isn't useful here as daif doesn't tell us about pNMI */
>  	return_to_irqs_enabled = !irqs_disabled_flags(arch_local_save_flags());
> @@ -407,7 +407,7 @@ int apei_claim_sea(struct pt_regs *regs)
>  	 * SEA can interrupt SError, mask it and describe this as an NMI so
>  	 * that APEI defers the handling.
>  	 */
> -	local_daif_restore(DAIF_ERRCTX);
> +	local_exception_restore(arm64_make_errctx_mask());
>  	nmi_enter();
>  	err = ghes_notify_sea();
>  	nmi_exit();
> @@ -418,7 +418,7 @@ int apei_claim_sea(struct pt_regs *regs)
>  	 */
>  	if (!err) {
>  		if (return_to_irqs_enabled) {
> -			local_daif_restore(DAIF_PROCCTX_NOIRQ);
> +			local_exception_restore(arm64_make_noirq_mask());
>  			__irq_enter();
>  			irq_work_run();
>  			__irq_exit();
> @@ -428,7 +428,7 @@ int apei_claim_sea(struct pt_regs *regs)
>  		}
>  	}
>  
> -	local_daif_restore(current_flags);
> +	local_exception_restore(current_mask);
>  
>  	return err;
>  }
> diff --git a/arch/arm64/kernel/debug-monitors.c b/arch/arm64/kernel/debug-monitors.c
> index 29307642f4c9..5b53034428e4 100644
> --- a/arch/arm64/kernel/debug-monitors.c
> +++ b/arch/arm64/kernel/debug-monitors.c
> @@ -19,7 +19,7 @@
>  
>  #include <asm/cpufeature.h>
>  #include <asm/cputype.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/debug-monitors.h>
>  #include <asm/exception.h>
>  #include <asm/kgdb.h>
> @@ -40,10 +40,11 @@ u8 debug_monitors_arch(void)
>   */
>  static void mdscr_write(u64 mdscr)
>  {
> -	unsigned long flags;
> -	flags = local_daif_save();
> +	struct exception_mask mask;
> +
> +	mask = local_exception_save_and_mask();
>  	write_sysreg(mdscr, mdscr_el1);
> -	local_daif_restore(flags);
> +	local_exception_restore(mask);
>  }
>  NOKPROBE_SYMBOL(mdscr_write);
>  
> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> index ceb4eb11232a..195af3f8001e 100644
> --- a/arch/arm64/kernel/entry-common.c
> +++ b/arch/arm64/kernel/entry-common.c
> @@ -18,7 +18,7 @@
>  #include <linux/thread_info.h>
>  
>  #include <asm/cpufeature.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/esr.h>
>  #include <asm/exception.h>
>  #include <asm/fpsimd.h>
> @@ -57,7 +57,7 @@ static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
>  {
>  	local_irq_disable();
>  	irqentry_exit_to_kernel_mode_preempt(regs, state);
> -	local_daif_mask();
> +	local_exception_mask();
>  	mte_check_tfsr_exit();
>  	irqentry_exit_to_kernel_mode_after_preempt(regs, state);
>  }
> @@ -86,7 +86,7 @@ static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs
>  {
>  	local_irq_disable();
>  	syscall_exit_to_user_mode_prepare(regs);
> -	local_daif_mask();
> +	local_exception_mask();
>  	sme_exit_to_user_mode();
>  	mte_check_tfsr_exit();
>  	exit_to_user_mode();
> @@ -101,7 +101,7 @@ static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
>  {
>  	local_irq_disable();
>  	irqentry_exit_to_user_mode_prepare(regs);
> -	local_daif_mask();
> +	local_exception_mask();
>  	sme_exit_to_user_mode();
>  	mte_check_tfsr_exit();
>  	exit_to_user_mode();
> @@ -318,7 +318,7 @@ static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_mem_abort(far, esr, regs);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -329,7 +329,7 @@ static void noinstr el1_pc(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_sp_pc_abort(far, esr, regs);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -339,7 +339,7 @@ static void noinstr el1_undef(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_el1_undef(regs, esr);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -349,7 +349,7 @@ static void noinstr el1_bti(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_el1_bti(regs, esr);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -359,7 +359,7 @@ static void noinstr el1_gcs(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_el1_gcs(regs, esr);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -369,7 +369,7 @@ static void noinstr el1_mops(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_el1_mops(regs, esr);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -434,7 +434,7 @@ static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_el1_fpac(regs, esr);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -537,7 +537,7 @@ asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
>  	unsigned long esr = read_sysreg(esr_el1);
>  	irqentry_state_t state;
>  
> -	local_daif_restore(DAIF_ERRCTX);
> +	local_exception_restore(arm64_make_errctx_mask());
>  	state = irqentry_nmi_enter(regs);
>  	do_serror(regs, esr);
>  	irqentry_nmi_exit(regs, state);
> @@ -548,7 +548,7 @@ static void noinstr el0_da(struct pt_regs *regs, unsigned long esr)
>  	unsigned long far = read_sysreg(far_el1);
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_mem_abort(far, esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -566,7 +566,7 @@ static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
>  		arm64_apply_bp_hardening();
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_mem_abort(far, esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -574,7 +574,7 @@ static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_fpsimd_acc(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_fpsimd_acc(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -582,7 +582,7 @@ static void noinstr el0_fpsimd_acc(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_sve_acc(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_sve_acc(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -590,7 +590,7 @@ static void noinstr el0_sve_acc(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_sme_acc(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_sme_acc(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -598,7 +598,7 @@ static void noinstr el0_sme_acc(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_fpsimd_exc(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_fpsimd_exc(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -606,7 +606,7 @@ static void noinstr el0_fpsimd_exc(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_sys(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_sys(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -619,7 +619,7 @@ static void noinstr el0_pc(struct pt_regs *regs, unsigned long esr)
>  		arm64_apply_bp_hardening();
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_sp_pc_abort(far, esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -627,7 +627,7 @@ static void noinstr el0_pc(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_sp(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_sp_pc_abort(regs->sp, esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -635,7 +635,7 @@ static void noinstr el0_sp(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_undef(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_undef(regs, esr);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -643,7 +643,7 @@ static void noinstr el0_undef(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_bti(struct pt_regs *regs)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_bti(regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -651,7 +651,7 @@ static void noinstr el0_bti(struct pt_regs *regs)
>  static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_mops(regs, esr);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -659,7 +659,7 @@ static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_gcs(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_gcs(regs, esr);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -667,7 +667,7 @@ static void noinstr el0_gcs(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_inv(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	bad_el0_sync(regs, 0, esr);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -681,7 +681,7 @@ static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
>  	debug_exception_enter(regs);
>  	do_breakpoint(esr, regs);
>  	debug_exception_exit(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	arm64_exit_to_user_mode(regs);
>  }
>  
> @@ -700,7 +700,7 @@ static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
>  	 * the single-step is complete.
>  	 */
>  	step_done = try_step_suspended_breakpoints(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	if (!step_done)
>  		do_el0_softstep(esr, regs);
>  	arm64_exit_to_user_mode(regs);
> @@ -715,14 +715,14 @@ static void noinstr el0_watchpt(struct pt_regs *regs, unsigned long esr)
>  	debug_exception_enter(regs);
>  	do_watchpoint(far, esr, regs);
>  	debug_exception_exit(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	arm64_exit_to_user_mode(regs);
>  }
>  
>  static void noinstr el0_brk64(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_brk64(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -732,7 +732,7 @@ static void noinstr el0_svc(struct pt_regs *regs)
>  	arm64_syscall_enter_from_user_mode(regs);
>  	cortex_a76_erratum_1463225_svc_handler();
>  	fpsimd_syscall_enter();
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_svc(regs);
>  	arm64_syscall_exit_to_user_mode(regs);
>  	fpsimd_syscall_exit();
> @@ -741,7 +741,7 @@ static void noinstr el0_svc(struct pt_regs *regs)
>  static void noinstr el0_fpac(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_fpac(regs, esr);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -857,11 +857,11 @@ static void noinstr __el0_error_handler_common(struct pt_regs *regs)
>  	irqentry_state_t state;
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_ERRCTX);
> +	local_exception_restore(arm64_make_errctx_mask());
>  	state = irqentry_nmi_enter(regs);
>  	do_serror(regs, esr);
>  	irqentry_nmi_exit(regs, state);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	arm64_exit_to_user_mode(regs);
>  }
>  
> @@ -874,7 +874,7 @@ asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
>  static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_cp15(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -883,7 +883,7 @@ static void noinstr el0_svc_compat(struct pt_regs *regs)
>  {
>  	arm64_syscall_enter_from_user_mode(regs);
>  	cortex_a76_erratum_1463225_svc_handler();
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_svc_compat(regs);
>  	arm64_syscall_exit_to_user_mode(regs);
>  }
> @@ -891,7 +891,7 @@ static void noinstr el0_svc_compat(struct pt_regs *regs)
>  static void noinstr el0_bkpt32(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_bkpt32(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
> index 9717568518ba..f6996b22fa29 100644
> --- a/arch/arm64/kernel/hibernate.c
> +++ b/arch/arm64/kernel/hibernate.c
> @@ -20,7 +20,7 @@
>  #include <asm/barrier.h>
>  #include <asm/cacheflush.h>
>  #include <asm/cputype.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/irqflags.h>
>  #include <asm/kexec.h>
>  #include <asm/memory.h>
> @@ -332,16 +332,16 @@ static void swsusp_mte_restore_tags(void)
>  
>  int swsusp_arch_suspend(void)
>  {
> -	int ret = 0;
> -	unsigned long flags;
>  	struct sleep_stack_data state;
> +	struct exception_mask mask;
> +	int ret = 0;
>  
>  	if (cpus_are_stuck_in_kernel()) {
>  		pr_err("Can't hibernate: no mechanism to offline secondary CPUs.\n");
>  		return -EBUSY;
>  	}
>  
> -	flags = local_daif_save();
> +	mask = local_exception_save_and_mask();
>  
>  	if (__cpu_suspend_enter(&state)) {
>  		/* make the crash dump kernel image visible/saveable */
> @@ -391,7 +391,7 @@ int swsusp_arch_suspend(void)
>  		spectre_v4_enable_mitigation(NULL);
>  	}
>  
> -	local_daif_restore(flags);
> +	local_exception_restore(mask);
>  
>  	return ret;
>  }
> diff --git a/arch/arm64/kernel/idle.c b/arch/arm64/kernel/idle.c
> index 05cfb347ec26..3997e297691e 100644
> --- a/arch/arm64/kernel/idle.c
> +++ b/arch/arm64/kernel/idle.c
> @@ -6,6 +6,7 @@
>  #include <linux/cpu.h>
>  #include <linux/irqflags.h>
>  
> +#include <asm/exception_masks.h>
>  #include <asm/barrier.h>
>  #include <asm/cpuidle.h>
>  #include <asm/cpufeature.h>
> @@ -22,14 +23,14 @@
>   */
>  void __cpuidle cpu_do_idle(void)
>  {
> -	struct arm_cpuidle_irq_context context;
> +	struct exception_mask mask;
>  
> -	arm_cpuidle_save_irq_context(&context);
> +	arm_cpuidle_save_irq_context(&mask);
>  
>  	dsb(sy);
>  	wfi();
>  
> -	arm_cpuidle_restore_irq_context(&context);
> +	arm_cpuidle_restore_irq_context(&mask);
>  }
>  
>  /*
> diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
> index 9fafd826002b..9e7fb0d22586 100644
> --- a/arch/arm64/kernel/irq.c
> +++ b/arch/arm64/kernel/irq.c
> @@ -21,7 +21,7 @@
>  #include <linux/seq_file.h>
>  #include <linux/smp.h>
>  #include <linux/vmalloc.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/exception.h>
>  #include <asm/numa.h>
>  #include <asm/softirq_stack.h>
> @@ -130,6 +130,6 @@ void __init init_IRQ(void)
>  		 * the PMR/PSR pair to a consistent state.
>  		 */
>  		WARN_ON(read_sysreg(daif) & PSR_A_BIT);
> -		local_daif_restore(DAIF_PROCCTX_NOIRQ);
> +		local_exception_restore(arm64_make_noirq_mask());
>  	}
>  }
> diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
> index c5693a32e49b..a8676ee10ea2 100644
> --- a/arch/arm64/kernel/machine_kexec.c
> +++ b/arch/arm64/kernel/machine_kexec.c
> @@ -17,7 +17,7 @@
>  
>  #include <asm/cacheflush.h>
>  #include <asm/cpu_ops.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/memory.h>
>  #include <asm/mmu.h>
>  #include <asm/mmu_context.h>
> @@ -173,7 +173,7 @@ void machine_kexec(struct kimage *kimage)
>  
>  	pr_info("Bye!\n");
>  
> -	local_daif_mask();
> +	local_exception_mask();
>  
>  	/*
>  	 * Both restart and kernel_reloc will shutdown the MMU, disable data
> diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c
> index 43a0361a8bf0..70d311e415ac 100644
> --- a/arch/arm64/kernel/probes/kprobes.c
> +++ b/arch/arm64/kernel/probes/kprobes.c
> @@ -24,7 +24,7 @@
>  #include <linux/vmalloc.h>
>  
>  #include <asm/cacheflush.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/debug-monitors.h>
>  #include <asm/insn.h>
>  #include <asm/irq.h>
> diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
> index 23c05dc7a8f2..10507e55e2ce 100644
> --- a/arch/arm64/kernel/setup.c
> +++ b/arch/arm64/kernel/setup.c
> @@ -37,7 +37,7 @@
>  #include <asm/fixmap.h>
>  #include <asm/cpu.h>
>  #include <asm/cputype.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/elf.h>
>  #include <asm/cpufeature.h>
>  #include <asm/cpu_ops.h>
> @@ -311,7 +311,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
>  	 * IRQ and FIQ will be unmasked after the root irqchip has been
>  	 * detected and initialized.
>  	 */
> -	local_daif_restore(DAIF_PROCCTX_NOIRQ);
> +	local_exception_restore(arm64_make_noirq_mask());
>  
>  	/*
>  	 * TTBR0 is only used for the identity mapping at this stage. Make it
> diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
> index 38e6fa204c17..ea39b47ae0eb 100644
> --- a/arch/arm64/kernel/signal.c
> +++ b/arch/arm64/kernel/signal.c
> @@ -22,7 +22,7 @@
>  #include <linux/syscalls.h>
>  #include <linux/pkeys.h>
>  
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/debug-monitors.h>
>  #include <asm/elf.h>
>  #include <asm/exception.h>
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index d46022f72075..d153ff77d25c 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -42,7 +42,7 @@
>  #include <asm/cpu.h>
>  #include <asm/cputype.h>
>  #include <asm/cpu_ops.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/kvm_mmu.h>
>  #include <asm/mmu_context.h>
>  #include <asm/numa.h>
> @@ -263,7 +263,7 @@ asmlinkage notrace void secondary_start_kernel(void)
>  	 * as the root irqchip has already been detected and initialized we can
>  	 * unmask IRQ and FIQ at the same time.
>  	 */
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  
>  	/*
>  	 * OK, it's off to the idle thread for us
> @@ -370,7 +370,7 @@ void __noreturn cpu_die(void)
>  
>  	idle_task_exit();
>  
> -	local_daif_mask();
> +	local_exception_mask();
>  
>  	/* Tell cpuhp_bp_sync_dead() that this CPU is now safe to dispose of */
>  	cpuhp_ap_report_dead();
> @@ -870,7 +870,7 @@ static void __noreturn local_cpu_stop(unsigned int cpu)
>  {
>  	set_cpu_online(cpu, false);
>  
> -	local_daif_mask();
> +	local_exception_mask();
>  	sdei_mask_local_cpu();
>  	cpu_park_loop();
>  }
> @@ -889,14 +889,14 @@ static void __noreturn ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs
>  {
>  #ifdef CONFIG_KEXEC_CORE
>  	/*
> -	 * Use local_daif_mask() instead of local_irq_disable() to make sure
> -	 * that pseudo-NMIs are disabled. The "crash stop" code starts with
> -	 * an IRQ and falls back to NMI (which might be pseudo). If the IRQ
> -	 * finally goes through right as we're timing out then the NMI could
> -	 * interrupt us. It's better to prevent the NMI and let the IRQ
> -	 * finish since the pt_regs will be better.
> +	 * Use local_exception_mask() instead of local_irq_disable()
> +	 * to make sure that pseudo-NMIs are disabled. The "crash stop" code
> +	 * starts with an IRQ and falls back to NMI (which might be pseudo).
> +	 * If the IRQ finally goes through right as we're timing out then
> +	 * the NMI could interrupt us. It's better to prevent the NMI and let
> +	 * the IRQ finish since the pt_regs will be better.
>  	 */
> -	local_daif_mask();
> +	local_exception_mask();
>  
>  	crash_save_cpu(regs, cpu);
>  
> diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c
> index eaaff94329cd..af16120818ac 100644
> --- a/arch/arm64/kernel/suspend.c
> +++ b/arch/arm64/kernel/suspend.c
> @@ -9,7 +9,7 @@
>  #include <asm/cacheflush.h>
>  #include <asm/cpufeature.h>
>  #include <asm/cpuidle.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/debug-monitors.h>
>  #include <asm/exec.h>
>  #include <asm/fpsimd.h>
> @@ -96,10 +96,9 @@ void notrace __cpu_suspend_exit(void)
>   */
>  int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  {
> -	int ret = 0;
> -	unsigned long flags;
> +	struct exception_mask mask, cpuidle_mask;
>  	struct sleep_stack_data state;
> -	struct arm_cpuidle_irq_context context;
> +	int ret = 0;
>  
>  	/*
>  	 * Some portions of CPU state (e.g. PSTATE.{PAN,DIT}) are initialized
> @@ -122,7 +121,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	 * hardirqs should be firmly off by now. This really ought to use
>  	 * something like raw_local_daif_save().
>  	 */
> -	flags = local_daif_save();
> +	mask = local_exception_save_and_mask();
>  
>  	/*
>  	 * Function graph tracer state gets inconsistent when the kernel
> @@ -135,7 +134,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	 * Switch to using DAIF.IF instead of PMR in order to reliably
>  	 * resume if we're using pseudo-NMIs.
>  	 */
> -	arm_cpuidle_save_irq_context(&context);
> +	arm_cpuidle_save_irq_context(&cpuidle_mask);
>  
>  	ct_cpuidle_enter();
>  
> @@ -159,7 +158,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  		__cpu_suspend_exit();
>  	}
>  
> -	arm_cpuidle_restore_irq_context(&context);
> +	arm_cpuidle_restore_irq_context(&cpuidle_mask);
>  
>  	unpause_graph_tracing();
>  
> @@ -168,7 +167,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	 * restored, so from this point onwards, debugging is fully
>  	 * reenabled if it was enabled when core started shutdown.
>  	 */
> -	local_daif_restore(flags);
> +	local_exception_restore(mask);
>  
>  	return ret;
>  }
> diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> index 914282016069..fcc18b182eb2 100644
> --- a/arch/arm64/kernel/traps.c
> +++ b/arch/arm64/kernel/traps.c
> @@ -33,7 +33,7 @@
>  #include <asm/atomic.h>
>  #include <asm/bug.h>
>  #include <asm/cpufeature.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/debug-monitors.h>
>  #include <asm/efi.h>
>  #include <asm/esr.h>
> diff --git a/arch/arm64/kvm/hyp/vgic-v3-sr.c b/arch/arm64/kvm/hyp/vgic-v3-sr.c
> index c4d2f1feea8b..96187ef5a025 100644
> --- a/arch/arm64/kvm/hyp/vgic-v3-sr.c
> +++ b/arch/arm64/kvm/hyp/vgic-v3-sr.c
> @@ -445,8 +445,8 @@ void __vgic_v3_init_lrs(void)
>   */
>  u64 __vgic_v3_get_gic_config(void)
>  {
> +	struct exception_mask mask;
>  	u64 val, sre;
> -	unsigned long flags = 0;
>  
>  	/*
>  	 * In compat mode, we cannot access ICC_SRE_EL1 at any EL
> @@ -476,7 +476,7 @@ u64 __vgic_v3_get_gic_config(void)
>  	 * of the exception entry to EL2.
>  	 */
>  	if (has_vhe()) {
> -		flags = local_daif_save();
> +		mask = local_exception_save_and_mask();
>  	} else {
>  		sysreg_clear_set_hcr(0, HCR_AMO | HCR_FMO | HCR_IMO);
>  		isb();
> @@ -491,7 +491,7 @@ u64 __vgic_v3_get_gic_config(void)
>  	isb();
>  
>  	if (has_vhe()) {
> -		local_daif_restore(flags);
> +		local_exception_restore(mask);
>  	} else {
>  		sysreg_clear_set_hcr(HCR_AMO | HCR_FMO | HCR_IMO, 0);
>  		isb();
> diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
> index bbe9cebd3d9d..024876efe0c3 100644
> --- a/arch/arm64/kvm/hyp/vhe/switch.c
> +++ b/arch/arm64/kvm/hyp/vhe/switch.c
> @@ -631,7 +631,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
>  {
>  	int ret;
>  
> -	local_daif_mask();
> +	local_exception_mask();
>  
>  	/*
>  	 * Having IRQs masked via PMR when entering the guest means the GIC
> @@ -647,10 +647,10 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
>  	ret = __kvm_vcpu_run_vhe(vcpu);
>  
>  	/*
> -	 * local_daif_restore() takes care to properly restore PSTATE.DAIF
> +	 * local_exception_restore() takes care to properly restore PSTATE.DAIF
>  	 * and the GIC PMR if the host is using IRQ priorities.
>  	 */
> -	local_daif_restore(DAIF_PROCCTX_NOIRQ);
> +	local_exception_restore(arm64_make_noirq_mask());
>  
>  	return ret;
>  }
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 85e23388f9bb..f7b8e87a099e 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -34,7 +34,7 @@
>  #include <asm/cpufeature.h>
>  #include <asm/efi.h>
>  #include <asm/exception.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/debug-monitors.h>
>  #include <asm/esr.h>
>  #include <asm/kprobes.h>
> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> index f2be501468ce..9afcd2bc2fd1 100644
> --- a/arch/arm64/mm/mmu.c
> +++ b/arch/arm64/mm/mmu.c
> @@ -2307,7 +2307,7 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
>  	typedef void (ttbr_replace_func)(phys_addr_t);
>  	extern ttbr_replace_func idmap_cpu_replace_ttbr1;
>  	ttbr_replace_func *replace_phys;
> -	unsigned long daif;
> +	struct exception_mask mask;
>  
>  	/* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */
>  	phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp));
> @@ -2323,9 +2323,9 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
>  	 * We really don't want to take *any* exceptions while TTBR1 is
>  	 * in the process of being replaced so mask everything.
>  	 */
> -	daif = local_daif_save();
> +	mask = local_exception_save_and_mask();
>  	replace_phys(ttbr1);
> -	local_daif_restore(daif);
> +	local_exception_restore(mask);
>  
>  	cpu_uninstall_idmap();
>  }
> diff --git a/drivers/firmware/psci/psci.c b/drivers/firmware/psci/psci.c
> index e73bae6cb23a..4cc834db7c15 100644
> --- a/drivers/firmware/psci/psci.c
> +++ b/drivers/firmware/psci/psci.c
> @@ -24,6 +24,7 @@
>  
>  #include <asm/cpuidle.h>
>  #include <asm/cputype.h>
> +#include <asm/exception_masks.h>
>  #include <asm/hypervisor.h>
>  #include <asm/system_misc.h>
>  #include <asm/smp_plat.h>
> @@ -503,12 +504,12 @@ int psci_cpu_suspend_enter(u32 state)
>  	int ret;
>  
>  	if (!psci_power_state_loses_context(state)) {
> -		struct arm_cpuidle_irq_context context;
> +		struct exception_mask mask;
>  
>  		ct_cpuidle_enter();
> -		arm_cpuidle_save_irq_context(&context);
> +		arm_cpuidle_save_irq_context(&mask);
>  		ret = psci_ops.cpu_suspend(state, 0);
> -		arm_cpuidle_restore_irq_context(&context);
> +		arm_cpuidle_restore_irq_context(&mask);
>  		ct_cpuidle_exit();
>  	} else {
>  		/*
> -- 
> 2.34.1
> 

Re: [PATCH 02/17] arm64: Rework exception masking into abstract logical mask

[Leonardo Bras]

On Fri, Jul 03, 2026 at 06:01:20PM +0800, Jinjie Ruan wrote:
> The current pseudo-NMI implementation in arm64 bodges the GICv3 PMR
> state into the DAIF management code. This approach forces local_daif_save()
> and local_daif_restore() to pretend that PMR is a part of DAIF. This has
> proven to be convoluted, difficult to maintain, and prone to architectural
> corner cases where it cannot do the right thing (e.g., faking up DAIF
> elements during irqflags/lockdep tracking).
> 
> As we prepare for the hardware-based NMI (FEAT_NMI, which use ALLINT
> to mask NMI) support, continuing down this path will significantly
> increase technical debt. We must clean up the exception masking
> abstraction ahead of ALLINT integration.
> 
> Introduce a new abstract `struct exception_mask` that explicitly treats
> DAIF, PMR, and ALLINT as separate, independent elements of the logical
> exception state. This allows the kernel to track and manipulate the exact
> hardware masking state cleanly without faking up DAIF flags.
> 
> Introduce the following new APIs to replace the legacy DAIF-centric
> helpers:
> 
>  - arm64_make_{procctx,errctx,noirq}_mask() to generate logical masks.
> 
>  - local_exception_save_and_mask() / local_exception_restore() to replace
>    the old local_daif_save() / local_daif_restore() pattern.
> 
>  - local_exception_inherit() to replace local_daif_inherit().
> 
> Mechanically convert all existing callers across the arm64 architecture
> to use the new logical exception mask helpers. The conversion spans:
> 
> - Core exception entry/exit handling and traps.
> 
> - KVM nVHE/VHE hypervisor switch and GICv3 save/restore code
> 
> - CPU idle, suspend, hibernate, and PSCI firmware invocations.
> 
> - Subsystems including SMP, ACPI, kprobes, and debug monitors
> 
> Link: https://lore.kernel.org/linux-arm-kernel/Y5c9SLeJacLYHmP7@FVFF77S0Q05N/
> Suggested-by: Mark Rutland <mark.rutland@arm.com>
> Signed-off-by: Jinjie Ruan <ruanjinjie@huawei.com>
> ---
>  arch/arm64/include/asm/cpuidle.h         |  29 ++---
>  arch/arm64/include/asm/daifflags.h       | 138 ----------------------
>  arch/arm64/include/asm/entry-common.h    |   2 +-
>  arch/arm64/include/asm/exception_masks.h | 141 +++++++++++++++++++++++
>  arch/arm64/include/asm/kvm_host.h        |   2 +-
>  arch/arm64/include/asm/mmu_context.h     |   2 +-
>  arch/arm64/kernel/acpi.c                 |  14 +--
>  arch/arm64/kernel/debug-monitors.c       |   9 +-
>  arch/arm64/kernel/entry-common.c         |  74 ++++++------
>  arch/arm64/kernel/hibernate.c            |  10 +-
>  arch/arm64/kernel/idle.c                 |   7 +-
>  arch/arm64/kernel/irq.c                  |   4 +-
>  arch/arm64/kernel/machine_kexec.c        |   4 +-
>  arch/arm64/kernel/probes/kprobes.c       |   2 +-
>  arch/arm64/kernel/setup.c                |   4 +-
>  arch/arm64/kernel/signal.c               |   2 +-
>  arch/arm64/kernel/smp.c                  |  22 ++--
>  arch/arm64/kernel/suspend.c              |  15 ++-
>  arch/arm64/kernel/traps.c                |   2 +-
>  arch/arm64/kvm/hyp/vgic-v3-sr.c          |   6 +-
>  arch/arm64/kvm/hyp/vhe/switch.c          |   6 +-
>  arch/arm64/mm/fault.c                    |   2 +-
>  arch/arm64/mm/mmu.c                      |   6 +-
>  drivers/firmware/psci/psci.c             |   7 +-
>  24 files changed, 254 insertions(+), 256 deletions(-)
>  delete mode 100644 arch/arm64/include/asm/daifflags.h
>  create mode 100644 arch/arm64/include/asm/exception_masks.h
> 
> diff --git a/arch/arm64/include/asm/cpuidle.h b/arch/arm64/include/asm/cpuidle.h
> index 64ddb533d3ec..4c19d5b79678 100644
> --- a/arch/arm64/include/asm/cpuidle.h
> +++ b/arch/arm64/include/asm/cpuidle.h
> @@ -6,37 +6,30 @@
>  
>  #ifdef CONFIG_ARM64_PSEUDO_NMI
>  #include <asm/arch_gicv3.h>
> +#include <asm/exception_masks.h>
>  #include <asm/ptrace.h>
>  
> -struct arm_cpuidle_irq_context {
> -	unsigned long pmr;
> -	unsigned long daif_bits;
> -};
> -
> -#define arm_cpuidle_save_irq_context(__c)				\
> +#define arm_cpuidle_save_irq_context(__m)				\
>  	do {								\
> -		struct arm_cpuidle_irq_context *c = __c;		\
> +		struct exception_mask *m = __m;				\
>  		if (system_uses_irq_prio_masking()) {			\
> -			c->daif_bits = read_sysreg(daif);		\
> -			write_sysreg(c->daif_bits | DAIF_PROCCTX_NOIRQ, \
> +			local_exception_save_mask(m);			\
> +			write_sysreg(m->daif | DAIF_PROCCTX_NOIRQ,	\
>  				     daif);				\
> -			c->pmr = gic_read_pmr();			\
>  			gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET); \
>  		}							\
>  	} while (0)
>  
> -#define arm_cpuidle_restore_irq_context(__c)				\
> +#define arm_cpuidle_restore_irq_context(__m)				\
>  	do {								\
> -		struct arm_cpuidle_irq_context *c = __c;		\
> +		struct exception_mask *m = __m;				\
>  		if (system_uses_irq_prio_masking()) {			\
> -			gic_write_pmr(c->pmr);				\
> -			write_sysreg(c->daif_bits, daif);		\
> +			gic_write_pmr(m->pmr);				\
> +			write_sysreg(m->daif, daif);			\
>  		}							\
>  	} while (0)
>  #else
> -struct arm_cpuidle_irq_context { };
> -
> -#define arm_cpuidle_save_irq_context(c)		(void)c
> -#define arm_cpuidle_restore_irq_context(c)	(void)c
> +#define arm_cpuidle_save_irq_context(m)		((void)m)
> +#define arm_cpuidle_restore_irq_context(m)	((void)m)
>  #endif
>  #endif
> diff --git a/arch/arm64/include/asm/daifflags.h b/arch/arm64/include/asm/daifflags.h
> deleted file mode 100644
> index 3a4b63a948ac..000000000000
> --- a/arch/arm64/include/asm/daifflags.h
> +++ /dev/null
> @@ -1,138 +0,0 @@
> -/* SPDX-License-Identifier: GPL-2.0-only */
> -/*
> - * Copyright (C) 2017 ARM Ltd.
> - */
> -#ifndef __ASM_DAIFFLAGS_H
> -#define __ASM_DAIFFLAGS_H
> -
> -#include <linux/irqflags.h>
> -
> -#include <asm/arch_gicv3.h>
> -#include <asm/barrier.h>
> -#include <asm/cpufeature.h>
> -#include <asm/ptrace.h>
> -
> -/* mask/save/unmask/restore all exceptions, including interrupts. */
> -static __always_inline void local_daif_mask(void)
> -{
> -	WARN_ON(system_has_prio_mask_debugging() &&
> -		(read_sysreg_s(SYS_ICC_PMR_EL1) == (GIC_PRIO_IRQOFF |
> -						    GIC_PRIO_PSR_I_SET)));
> -
> -	asm volatile(
> -		"msr	daifset, #0xf		// local_daif_mask\n"
> -		:
> -		:
> -		: "memory");
> -
> -	/* Don't really care for a dsb here, we don't intend to enable IRQs */
> -	if (system_uses_irq_prio_masking())
> -		gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
> -
> -	trace_hardirqs_off();
> -}
> -
> -static __always_inline unsigned long local_daif_save_flags(void)
> -{
> -	unsigned long flags;
> -
> -	flags = read_sysreg(daif);
> -
> -	if (system_uses_irq_prio_masking()) {
> -		/* If IRQs are masked with PMR, reflect it in the flags */
> -		if (read_sysreg_s(SYS_ICC_PMR_EL1) != GIC_PRIO_IRQON)
> -			flags |= DAIF_PROCCTX_NOIRQ;
> -	}
> -
> -	return flags;
> -}
> -
> -static __always_inline unsigned long local_daif_save(void)
> -{
> -	unsigned long flags;
> -
> -	flags = local_daif_save_flags();
> -
> -	local_daif_mask();
> -
> -	return flags;
> -}
> -
> -static __always_inline void local_daif_restore(unsigned long flags)
> -{
> -	bool irq_disabled = flags & PSR_I_BIT;
> -
> -	WARN_ON(system_has_prio_mask_debugging() &&
> -		(read_sysreg(daif) & DAIF_PROCCTX_NOIRQ) != DAIF_PROCCTX_NOIRQ);
> -
> -	if (!irq_disabled) {
> -		trace_hardirqs_on();
> -
> -		if (system_uses_irq_prio_masking()) {
> -			gic_write_pmr(GIC_PRIO_IRQON);
> -			pmr_sync();
> -		}
> -	} else if (system_uses_irq_prio_masking()) {
> -		u64 pmr;
> -
> -		if (!(flags & PSR_A_BIT)) {
> -			/*
> -			 * If interrupts are disabled but we can take
> -			 * asynchronous errors, we can take NMIs
> -			 */
> -			flags &= ~DAIF_PROCCTX_NOIRQ;
> -			pmr = GIC_PRIO_IRQOFF;
> -		} else {
> -			pmr = GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET;
> -		}
> -
> -		/*
> -		 * There has been concern that the write to daif
> -		 * might be reordered before this write to PMR.
> -		 * From the ARM ARM DDI 0487D.a, section D1.7.1
> -		 * "Accessing PSTATE fields":
> -		 *   Writes to the PSTATE fields have side-effects on
> -		 *   various aspects of the PE operation. All of these
> -		 *   side-effects are guaranteed:
> -		 *     - Not to be visible to earlier instructions in
> -		 *       the execution stream.
> -		 *     - To be visible to later instructions in the
> -		 *       execution stream
> -		 *
> -		 * Also, writes to PMR are self-synchronizing, so no
> -		 * interrupts with a lower priority than PMR is signaled
> -		 * to the PE after the write.
> -		 *
> -		 * So we don't need additional synchronization here.
> -		 */
> -		gic_write_pmr(pmr);
> -	}
> -
> -	write_sysreg(flags, daif);
> -
> -	if (irq_disabled)
> -		trace_hardirqs_off();
> -}
> -
> -/*
> - * Called by synchronous exception handlers to restore the DAIF bits that were
> - * modified by taking an exception.
> - */
> -static __always_inline void local_daif_inherit(struct pt_regs *regs)
> -{
> -	unsigned long flags = regs->pstate & DAIF_MASK;
> -
> -	if (!regs_irqs_disabled(regs))
> -		trace_hardirqs_on();
> -
> -	if (system_uses_irq_prio_masking())
> -		gic_write_pmr(regs->pmr);
> -
> -	/*
> -	 * We can't use local_daif_restore(regs->pstate) here as
> -	 * system_has_prio_mask_debugging() won't restore the I bit if it can
> -	 * use the pmr instead.
> -	 */
> -	write_sysreg(flags, daif);
> -}
> -#endif
> diff --git a/arch/arm64/include/asm/entry-common.h b/arch/arm64/include/asm/entry-common.h
> index cab8cd78f693..3d8b38ce7afb 100644
> --- a/arch/arm64/include/asm/entry-common.h
> +++ b/arch/arm64/include/asm/entry-common.h
> @@ -6,7 +6,7 @@
>  #include <linux/thread_info.h>
>  
>  #include <asm/cpufeature.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/fpsimd.h>
>  #include <asm/mte.h>
>  #include <asm/stacktrace.h>
> diff --git a/arch/arm64/include/asm/exception_masks.h b/arch/arm64/include/asm/exception_masks.h
> new file mode 100644
> index 000000000000..0a21fb97c3ca
> --- /dev/null
> +++ b/arch/arm64/include/asm/exception_masks.h
> @@ -0,0 +1,141 @@
> +/* SPDX-License-Identifier: GPL-2.0-only */
> +/*
> + * Copyright (C) 2017 ARM Ltd.
> + */
> +#ifndef __ASM_EXCEPTION_MASKS_H
> +#define __ASM_EXCEPTION_MASKS_H
> +
> +#include <linux/irqflags.h>
> +
> +#include <asm/arch_gicv3.h>
> +#include <asm/barrier.h>
> +#include <asm/cpufeature.h>
> +#include <asm/ptrace.h>
> +
> +/*
> + * Logical exception mask: tracks the three independent exception
> + * masking controls on arm64:
> + *  - DAIF (PSTATE.{D,A,I,F} bits)
> + *  - PMR  (ICC_PMR_EL1)
> + *  - ALLINT (PSTATE.ALLINT)
> + */
> +struct exception_mask {
> +	unsigned long daif;
> +	unsigned long pmr;
> +	unsigned long allint;		// for future FEAT_NMI use
> +};

Also, out of curiosity: it looks like all those bits would fit a single 
u32. Have you considered using a bit-field, instead of 3 ulongs?

Thanks!
Leo

> +
> +static inline struct exception_mask arm64_make_procctx_mask(void)
> +{
> +	struct exception_mask mask;
> +
> +	mask.daif = DAIF_PROCCTX;
> +	if (system_uses_irq_prio_masking())
> +		mask.pmr = GIC_PRIO_IRQON;
> +
> +	mask.allint = 0;
> +
> +	return mask;
> +}
> +
> +static inline struct exception_mask arm64_make_errctx_mask(void)
> +{
> +	struct exception_mask mask;
> +
> +	mask.daif = DAIF_ERRCTX;
> +	if (system_uses_irq_prio_masking())
> +		mask.pmr = GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET;
> +
> +	mask.allint = 0;
> +
> +	return mask;
> +}
> +
> +static inline struct exception_mask arm64_make_noirq_mask(void)
> +{
> +	struct exception_mask mask;
> +
> +	if (system_uses_irq_prio_masking()) {
> +		mask.daif = 0;
> +		mask.pmr = GIC_PRIO_IRQOFF;
> +	} else
> +		mask.daif = DAIF_PROCCTX_NOIRQ;
> +
> +	mask.allint = 0;
> +
> +	return mask;
> +}
> +
> +/* Mask all exceptions immediately */
> +static inline void local_exception_mask(void)
> +{
> +	WARN_ON(system_has_prio_mask_debugging() &&
> +		(read_sysreg_s(SYS_ICC_PMR_EL1) == (GIC_PRIO_IRQOFF |
> +						    GIC_PRIO_PSR_I_SET)));
> +
> +	asm volatile("msr daifset, #0xf" ::: "memory");
> +
> +	if (system_uses_irq_prio_masking())
> +		gic_write_pmr(GIC_PRIO_IRQON | GIC_PRIO_PSR_I_SET);
> +
> +	trace_hardirqs_off();
> +}
> +
> +static inline void local_exception_save_mask(struct exception_mask *mask)
> +{
> +	mask->daif = read_sysreg(daif);
> +	if (system_uses_irq_prio_masking())
> +		mask->pmr = gic_read_pmr();
> +
> +	mask->allint = 0;
> +}
> +
> +static inline struct exception_mask local_exception_save_and_mask(void)
> +{
> +	struct exception_mask mask;
> +
> +	local_exception_save_mask(&mask);
> +	local_exception_mask();
> +
> +	return mask;
> +}
> +
> +static inline void local_exception_restore(const struct exception_mask mask)
> +{
> +	bool irq_disabled = mask.daif & PSR_I_BIT;
> +
> +	if (system_uses_irq_prio_masking())
> +		irq_disabled |= (mask.pmr == GIC_PRIO_IRQOFF);
> +
> +	WARN_ON(system_has_prio_mask_debugging() &&
> +		(read_sysreg(daif) & DAIF_PROCCTX_NOIRQ) != DAIF_PROCCTX_NOIRQ);
> +
> +	if (!irq_disabled)
> +		trace_hardirqs_on();
> +
> +	if (system_uses_irq_prio_masking()) {
> +		gic_write_pmr(mask.pmr);
> +		pmr_sync();
> +	}
> +
> +	write_sysreg(mask.daif, daif);
> +
> +	if (irq_disabled)
> +		trace_hardirqs_off();
> +}
> +
> +/*
> + * Called by synchronous exception handlers to restore the DAIF bits that were
> + * modified by taking an exception.
> + */
> +static inline void local_exception_inherit(struct pt_regs *regs)
> +{
> +	if (!regs_irqs_disabled(regs))
> +		trace_hardirqs_on();
> +
> +	if (system_uses_irq_prio_masking())
> +		gic_write_pmr(regs->pmr);
> +
> +	write_sysreg(regs->pstate & DAIF_MASK, daif);
> +}
> +#endif /* __ASM_EXCEPTION_MASKS_H */
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index bae2c4f92ef5..8a443a379689 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -23,7 +23,7 @@
>  #include <asm/barrier.h>
>  #include <asm/cpufeature.h>
>  #include <asm/cputype.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/fpsimd.h>
>  #include <asm/kvm.h>
>  #include <asm/kvm_asm.h>
> diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
> index 803b68758152..6103925f85b7 100644
> --- a/arch/arm64/include/asm/mmu_context.h
> +++ b/arch/arm64/include/asm/mmu_context.h
> @@ -19,7 +19,7 @@
>  
>  #include <asm/cacheflush.h>
>  #include <asm/cpufeature.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/gcs.h>
>  #include <asm/proc-fns.h>
>  #include <asm/cputype.h>
> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c
> index 5891f92c2035..4d413419309d 100644
> --- a/arch/arm64/kernel/acpi.c
> +++ b/arch/arm64/kernel/acpi.c
> @@ -33,7 +33,7 @@
>  #include <acpi/processor.h>
>  #include <asm/cputype.h>
>  #include <asm/cpu_ops.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/smp_plat.h>
>  
>  int acpi_noirq = 1;		/* skip ACPI IRQ initialization */
> @@ -388,14 +388,14 @@ void __iomem *acpi_os_ioremap(acpi_physical_address phys, acpi_size size)
>   */
>  int apei_claim_sea(struct pt_regs *regs)
>  {
> -	int err = -ENOENT;
> +	struct exception_mask current_mask;
>  	bool return_to_irqs_enabled;
> -	unsigned long current_flags;
> +	int err = -ENOENT;
>  
>  	if (!IS_ENABLED(CONFIG_ACPI_APEI_GHES))
>  		return err;
>  
> -	current_flags = local_daif_save_flags();
> +	local_exception_save_mask(&current_mask);
>  
>  	/* current_flags isn't useful here as daif doesn't tell us about pNMI */
>  	return_to_irqs_enabled = !irqs_disabled_flags(arch_local_save_flags());
> @@ -407,7 +407,7 @@ int apei_claim_sea(struct pt_regs *regs)
>  	 * SEA can interrupt SError, mask it and describe this as an NMI so
>  	 * that APEI defers the handling.
>  	 */
> -	local_daif_restore(DAIF_ERRCTX);
> +	local_exception_restore(arm64_make_errctx_mask());
>  	nmi_enter();
>  	err = ghes_notify_sea();
>  	nmi_exit();
> @@ -418,7 +418,7 @@ int apei_claim_sea(struct pt_regs *regs)
>  	 */
>  	if (!err) {
>  		if (return_to_irqs_enabled) {
> -			local_daif_restore(DAIF_PROCCTX_NOIRQ);
> +			local_exception_restore(arm64_make_noirq_mask());
>  			__irq_enter();
>  			irq_work_run();
>  			__irq_exit();
> @@ -428,7 +428,7 @@ int apei_claim_sea(struct pt_regs *regs)
>  		}
>  	}
>  
> -	local_daif_restore(current_flags);
> +	local_exception_restore(current_mask);
>  
>  	return err;
>  }
> diff --git a/arch/arm64/kernel/debug-monitors.c b/arch/arm64/kernel/debug-monitors.c
> index 29307642f4c9..5b53034428e4 100644
> --- a/arch/arm64/kernel/debug-monitors.c
> +++ b/arch/arm64/kernel/debug-monitors.c
> @@ -19,7 +19,7 @@
>  
>  #include <asm/cpufeature.h>
>  #include <asm/cputype.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/debug-monitors.h>
>  #include <asm/exception.h>
>  #include <asm/kgdb.h>
> @@ -40,10 +40,11 @@ u8 debug_monitors_arch(void)
>   */
>  static void mdscr_write(u64 mdscr)
>  {
> -	unsigned long flags;
> -	flags = local_daif_save();
> +	struct exception_mask mask;
> +
> +	mask = local_exception_save_and_mask();
>  	write_sysreg(mdscr, mdscr_el1);
> -	local_daif_restore(flags);
> +	local_exception_restore(mask);
>  }
>  NOKPROBE_SYMBOL(mdscr_write);
>  
> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> index ceb4eb11232a..195af3f8001e 100644
> --- a/arch/arm64/kernel/entry-common.c
> +++ b/arch/arm64/kernel/entry-common.c
> @@ -18,7 +18,7 @@
>  #include <linux/thread_info.h>
>  
>  #include <asm/cpufeature.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/esr.h>
>  #include <asm/exception.h>
>  #include <asm/fpsimd.h>
> @@ -57,7 +57,7 @@ static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
>  {
>  	local_irq_disable();
>  	irqentry_exit_to_kernel_mode_preempt(regs, state);
> -	local_daif_mask();
> +	local_exception_mask();
>  	mte_check_tfsr_exit();
>  	irqentry_exit_to_kernel_mode_after_preempt(regs, state);
>  }
> @@ -86,7 +86,7 @@ static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs
>  {
>  	local_irq_disable();
>  	syscall_exit_to_user_mode_prepare(regs);
> -	local_daif_mask();
> +	local_exception_mask();
>  	sme_exit_to_user_mode();
>  	mte_check_tfsr_exit();
>  	exit_to_user_mode();
> @@ -101,7 +101,7 @@ static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
>  {
>  	local_irq_disable();
>  	irqentry_exit_to_user_mode_prepare(regs);
> -	local_daif_mask();
> +	local_exception_mask();
>  	sme_exit_to_user_mode();
>  	mte_check_tfsr_exit();
>  	exit_to_user_mode();
> @@ -318,7 +318,7 @@ static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_mem_abort(far, esr, regs);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -329,7 +329,7 @@ static void noinstr el1_pc(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_sp_pc_abort(far, esr, regs);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -339,7 +339,7 @@ static void noinstr el1_undef(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_el1_undef(regs, esr);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -349,7 +349,7 @@ static void noinstr el1_bti(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_el1_bti(regs, esr);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -359,7 +359,7 @@ static void noinstr el1_gcs(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_el1_gcs(regs, esr);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -369,7 +369,7 @@ static void noinstr el1_mops(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_el1_mops(regs, esr);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -434,7 +434,7 @@ static void noinstr el1_fpac(struct pt_regs *regs, unsigned long esr)
>  	irqentry_state_t state;
>  
>  	state = arm64_enter_from_kernel_mode(regs);
> -	local_daif_inherit(regs);
> +	local_exception_inherit(regs);
>  	do_el1_fpac(regs, esr);
>  	arm64_exit_to_kernel_mode(regs, state);
>  }
> @@ -537,7 +537,7 @@ asmlinkage void noinstr el1h_64_error_handler(struct pt_regs *regs)
>  	unsigned long esr = read_sysreg(esr_el1);
>  	irqentry_state_t state;
>  
> -	local_daif_restore(DAIF_ERRCTX);
> +	local_exception_restore(arm64_make_errctx_mask());
>  	state = irqentry_nmi_enter(regs);
>  	do_serror(regs, esr);
>  	irqentry_nmi_exit(regs, state);
> @@ -548,7 +548,7 @@ static void noinstr el0_da(struct pt_regs *regs, unsigned long esr)
>  	unsigned long far = read_sysreg(far_el1);
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_mem_abort(far, esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -566,7 +566,7 @@ static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
>  		arm64_apply_bp_hardening();
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_mem_abort(far, esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -574,7 +574,7 @@ static void noinstr el0_ia(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_fpsimd_acc(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_fpsimd_acc(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -582,7 +582,7 @@ static void noinstr el0_fpsimd_acc(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_sve_acc(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_sve_acc(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -590,7 +590,7 @@ static void noinstr el0_sve_acc(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_sme_acc(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_sme_acc(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -598,7 +598,7 @@ static void noinstr el0_sme_acc(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_fpsimd_exc(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_fpsimd_exc(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -606,7 +606,7 @@ static void noinstr el0_fpsimd_exc(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_sys(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_sys(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -619,7 +619,7 @@ static void noinstr el0_pc(struct pt_regs *regs, unsigned long esr)
>  		arm64_apply_bp_hardening();
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_sp_pc_abort(far, esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -627,7 +627,7 @@ static void noinstr el0_pc(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_sp(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_sp_pc_abort(regs->sp, esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -635,7 +635,7 @@ static void noinstr el0_sp(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_undef(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_undef(regs, esr);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -643,7 +643,7 @@ static void noinstr el0_undef(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_bti(struct pt_regs *regs)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_bti(regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -651,7 +651,7 @@ static void noinstr el0_bti(struct pt_regs *regs)
>  static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_mops(regs, esr);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -659,7 +659,7 @@ static void noinstr el0_mops(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_gcs(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_gcs(regs, esr);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -667,7 +667,7 @@ static void noinstr el0_gcs(struct pt_regs *regs, unsigned long esr)
>  static void noinstr el0_inv(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	bad_el0_sync(regs, 0, esr);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -681,7 +681,7 @@ static void noinstr el0_breakpt(struct pt_regs *regs, unsigned long esr)
>  	debug_exception_enter(regs);
>  	do_breakpoint(esr, regs);
>  	debug_exception_exit(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	arm64_exit_to_user_mode(regs);
>  }
>  
> @@ -700,7 +700,7 @@ static void noinstr el0_softstp(struct pt_regs *regs, unsigned long esr)
>  	 * the single-step is complete.
>  	 */
>  	step_done = try_step_suspended_breakpoints(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	if (!step_done)
>  		do_el0_softstep(esr, regs);
>  	arm64_exit_to_user_mode(regs);
> @@ -715,14 +715,14 @@ static void noinstr el0_watchpt(struct pt_regs *regs, unsigned long esr)
>  	debug_exception_enter(regs);
>  	do_watchpoint(far, esr, regs);
>  	debug_exception_exit(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	arm64_exit_to_user_mode(regs);
>  }
>  
>  static void noinstr el0_brk64(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_brk64(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -732,7 +732,7 @@ static void noinstr el0_svc(struct pt_regs *regs)
>  	arm64_syscall_enter_from_user_mode(regs);
>  	cortex_a76_erratum_1463225_svc_handler();
>  	fpsimd_syscall_enter();
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_svc(regs);
>  	arm64_syscall_exit_to_user_mode(regs);
>  	fpsimd_syscall_exit();
> @@ -741,7 +741,7 @@ static void noinstr el0_svc(struct pt_regs *regs)
>  static void noinstr el0_fpac(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_fpac(regs, esr);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -857,11 +857,11 @@ static void noinstr __el0_error_handler_common(struct pt_regs *regs)
>  	irqentry_state_t state;
>  
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_ERRCTX);
> +	local_exception_restore(arm64_make_errctx_mask());
>  	state = irqentry_nmi_enter(regs);
>  	do_serror(regs, esr);
>  	irqentry_nmi_exit(regs, state);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	arm64_exit_to_user_mode(regs);
>  }
>  
> @@ -874,7 +874,7 @@ asmlinkage void noinstr el0t_64_error_handler(struct pt_regs *regs)
>  static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_cp15(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> @@ -883,7 +883,7 @@ static void noinstr el0_svc_compat(struct pt_regs *regs)
>  {
>  	arm64_syscall_enter_from_user_mode(regs);
>  	cortex_a76_erratum_1463225_svc_handler();
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_el0_svc_compat(regs);
>  	arm64_syscall_exit_to_user_mode(regs);
>  }
> @@ -891,7 +891,7 @@ static void noinstr el0_svc_compat(struct pt_regs *regs)
>  static void noinstr el0_bkpt32(struct pt_regs *regs, unsigned long esr)
>  {
>  	arm64_enter_from_user_mode(regs);
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  	do_bkpt32(esr, regs);
>  	arm64_exit_to_user_mode(regs);
>  }
> diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
> index 9717568518ba..f6996b22fa29 100644
> --- a/arch/arm64/kernel/hibernate.c
> +++ b/arch/arm64/kernel/hibernate.c
> @@ -20,7 +20,7 @@
>  #include <asm/barrier.h>
>  #include <asm/cacheflush.h>
>  #include <asm/cputype.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/irqflags.h>
>  #include <asm/kexec.h>
>  #include <asm/memory.h>
> @@ -332,16 +332,16 @@ static void swsusp_mte_restore_tags(void)
>  
>  int swsusp_arch_suspend(void)
>  {
> -	int ret = 0;
> -	unsigned long flags;
>  	struct sleep_stack_data state;
> +	struct exception_mask mask;
> +	int ret = 0;
>  
>  	if (cpus_are_stuck_in_kernel()) {
>  		pr_err("Can't hibernate: no mechanism to offline secondary CPUs.\n");
>  		return -EBUSY;
>  	}
>  
> -	flags = local_daif_save();
> +	mask = local_exception_save_and_mask();
>  
>  	if (__cpu_suspend_enter(&state)) {
>  		/* make the crash dump kernel image visible/saveable */
> @@ -391,7 +391,7 @@ int swsusp_arch_suspend(void)
>  		spectre_v4_enable_mitigation(NULL);
>  	}
>  
> -	local_daif_restore(flags);
> +	local_exception_restore(mask);
>  
>  	return ret;
>  }
> diff --git a/arch/arm64/kernel/idle.c b/arch/arm64/kernel/idle.c
> index 05cfb347ec26..3997e297691e 100644
> --- a/arch/arm64/kernel/idle.c
> +++ b/arch/arm64/kernel/idle.c
> @@ -6,6 +6,7 @@
>  #include <linux/cpu.h>
>  #include <linux/irqflags.h>
>  
> +#include <asm/exception_masks.h>
>  #include <asm/barrier.h>
>  #include <asm/cpuidle.h>
>  #include <asm/cpufeature.h>
> @@ -22,14 +23,14 @@
>   */
>  void __cpuidle cpu_do_idle(void)
>  {
> -	struct arm_cpuidle_irq_context context;
> +	struct exception_mask mask;
>  
> -	arm_cpuidle_save_irq_context(&context);
> +	arm_cpuidle_save_irq_context(&mask);
>  
>  	dsb(sy);
>  	wfi();
>  
> -	arm_cpuidle_restore_irq_context(&context);
> +	arm_cpuidle_restore_irq_context(&mask);
>  }
>  
>  /*
> diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
> index 9fafd826002b..9e7fb0d22586 100644
> --- a/arch/arm64/kernel/irq.c
> +++ b/arch/arm64/kernel/irq.c
> @@ -21,7 +21,7 @@
>  #include <linux/seq_file.h>
>  #include <linux/smp.h>
>  #include <linux/vmalloc.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/exception.h>
>  #include <asm/numa.h>
>  #include <asm/softirq_stack.h>
> @@ -130,6 +130,6 @@ void __init init_IRQ(void)
>  		 * the PMR/PSR pair to a consistent state.
>  		 */
>  		WARN_ON(read_sysreg(daif) & PSR_A_BIT);
> -		local_daif_restore(DAIF_PROCCTX_NOIRQ);
> +		local_exception_restore(arm64_make_noirq_mask());
>  	}
>  }
> diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c
> index c5693a32e49b..a8676ee10ea2 100644
> --- a/arch/arm64/kernel/machine_kexec.c
> +++ b/arch/arm64/kernel/machine_kexec.c
> @@ -17,7 +17,7 @@
>  
>  #include <asm/cacheflush.h>
>  #include <asm/cpu_ops.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/memory.h>
>  #include <asm/mmu.h>
>  #include <asm/mmu_context.h>
> @@ -173,7 +173,7 @@ void machine_kexec(struct kimage *kimage)
>  
>  	pr_info("Bye!\n");
>  
> -	local_daif_mask();
> +	local_exception_mask();
>  
>  	/*
>  	 * Both restart and kernel_reloc will shutdown the MMU, disable data
> diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c
> index 43a0361a8bf0..70d311e415ac 100644
> --- a/arch/arm64/kernel/probes/kprobes.c
> +++ b/arch/arm64/kernel/probes/kprobes.c
> @@ -24,7 +24,7 @@
>  #include <linux/vmalloc.h>
>  
>  #include <asm/cacheflush.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/debug-monitors.h>
>  #include <asm/insn.h>
>  #include <asm/irq.h>
> diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
> index 23c05dc7a8f2..10507e55e2ce 100644
> --- a/arch/arm64/kernel/setup.c
> +++ b/arch/arm64/kernel/setup.c
> @@ -37,7 +37,7 @@
>  #include <asm/fixmap.h>
>  #include <asm/cpu.h>
>  #include <asm/cputype.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/elf.h>
>  #include <asm/cpufeature.h>
>  #include <asm/cpu_ops.h>
> @@ -311,7 +311,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
>  	 * IRQ and FIQ will be unmasked after the root irqchip has been
>  	 * detected and initialized.
>  	 */
> -	local_daif_restore(DAIF_PROCCTX_NOIRQ);
> +	local_exception_restore(arm64_make_noirq_mask());
>  
>  	/*
>  	 * TTBR0 is only used for the identity mapping at this stage. Make it
> diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
> index 38e6fa204c17..ea39b47ae0eb 100644
> --- a/arch/arm64/kernel/signal.c
> +++ b/arch/arm64/kernel/signal.c
> @@ -22,7 +22,7 @@
>  #include <linux/syscalls.h>
>  #include <linux/pkeys.h>
>  
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/debug-monitors.h>
>  #include <asm/elf.h>
>  #include <asm/exception.h>
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index d46022f72075..d153ff77d25c 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -42,7 +42,7 @@
>  #include <asm/cpu.h>
>  #include <asm/cputype.h>
>  #include <asm/cpu_ops.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/kvm_mmu.h>
>  #include <asm/mmu_context.h>
>  #include <asm/numa.h>
> @@ -263,7 +263,7 @@ asmlinkage notrace void secondary_start_kernel(void)
>  	 * as the root irqchip has already been detected and initialized we can
>  	 * unmask IRQ and FIQ at the same time.
>  	 */
> -	local_daif_restore(DAIF_PROCCTX);
> +	local_exception_restore(arm64_make_procctx_mask());
>  
>  	/*
>  	 * OK, it's off to the idle thread for us
> @@ -370,7 +370,7 @@ void __noreturn cpu_die(void)
>  
>  	idle_task_exit();
>  
> -	local_daif_mask();
> +	local_exception_mask();
>  
>  	/* Tell cpuhp_bp_sync_dead() that this CPU is now safe to dispose of */
>  	cpuhp_ap_report_dead();
> @@ -870,7 +870,7 @@ static void __noreturn local_cpu_stop(unsigned int cpu)
>  {
>  	set_cpu_online(cpu, false);
>  
> -	local_daif_mask();
> +	local_exception_mask();
>  	sdei_mask_local_cpu();
>  	cpu_park_loop();
>  }
> @@ -889,14 +889,14 @@ static void __noreturn ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs
>  {
>  #ifdef CONFIG_KEXEC_CORE
>  	/*
> -	 * Use local_daif_mask() instead of local_irq_disable() to make sure
> -	 * that pseudo-NMIs are disabled. The "crash stop" code starts with
> -	 * an IRQ and falls back to NMI (which might be pseudo). If the IRQ
> -	 * finally goes through right as we're timing out then the NMI could
> -	 * interrupt us. It's better to prevent the NMI and let the IRQ
> -	 * finish since the pt_regs will be better.
> +	 * Use local_exception_mask() instead of local_irq_disable()
> +	 * to make sure that pseudo-NMIs are disabled. The "crash stop" code
> +	 * starts with an IRQ and falls back to NMI (which might be pseudo).
> +	 * If the IRQ finally goes through right as we're timing out then
> +	 * the NMI could interrupt us. It's better to prevent the NMI and let
> +	 * the IRQ finish since the pt_regs will be better.
>  	 */
> -	local_daif_mask();
> +	local_exception_mask();
>  
>  	crash_save_cpu(regs, cpu);
>  
> diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c
> index eaaff94329cd..af16120818ac 100644
> --- a/arch/arm64/kernel/suspend.c
> +++ b/arch/arm64/kernel/suspend.c
> @@ -9,7 +9,7 @@
>  #include <asm/cacheflush.h>
>  #include <asm/cpufeature.h>
>  #include <asm/cpuidle.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/debug-monitors.h>
>  #include <asm/exec.h>
>  #include <asm/fpsimd.h>
> @@ -96,10 +96,9 @@ void notrace __cpu_suspend_exit(void)
>   */
>  int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  {
> -	int ret = 0;
> -	unsigned long flags;
> +	struct exception_mask mask, cpuidle_mask;
>  	struct sleep_stack_data state;
> -	struct arm_cpuidle_irq_context context;
> +	int ret = 0;
>  
>  	/*
>  	 * Some portions of CPU state (e.g. PSTATE.{PAN,DIT}) are initialized
> @@ -122,7 +121,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	 * hardirqs should be firmly off by now. This really ought to use
>  	 * something like raw_local_daif_save().
>  	 */
> -	flags = local_daif_save();
> +	mask = local_exception_save_and_mask();
>  
>  	/*
>  	 * Function graph tracer state gets inconsistent when the kernel
> @@ -135,7 +134,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	 * Switch to using DAIF.IF instead of PMR in order to reliably
>  	 * resume if we're using pseudo-NMIs.
>  	 */
> -	arm_cpuidle_save_irq_context(&context);
> +	arm_cpuidle_save_irq_context(&cpuidle_mask);
>  
>  	ct_cpuidle_enter();
>  
> @@ -159,7 +158,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  		__cpu_suspend_exit();
>  	}
>  
> -	arm_cpuidle_restore_irq_context(&context);
> +	arm_cpuidle_restore_irq_context(&cpuidle_mask);
>  
>  	unpause_graph_tracing();
>  
> @@ -168,7 +167,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
>  	 * restored, so from this point onwards, debugging is fully
>  	 * reenabled if it was enabled when core started shutdown.
>  	 */
> -	local_daif_restore(flags);
> +	local_exception_restore(mask);
>  
>  	return ret;
>  }
> diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> index 914282016069..fcc18b182eb2 100644
> --- a/arch/arm64/kernel/traps.c
> +++ b/arch/arm64/kernel/traps.c
> @@ -33,7 +33,7 @@
>  #include <asm/atomic.h>
>  #include <asm/bug.h>
>  #include <asm/cpufeature.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/debug-monitors.h>
>  #include <asm/efi.h>
>  #include <asm/esr.h>
> diff --git a/arch/arm64/kvm/hyp/vgic-v3-sr.c b/arch/arm64/kvm/hyp/vgic-v3-sr.c
> index c4d2f1feea8b..96187ef5a025 100644
> --- a/arch/arm64/kvm/hyp/vgic-v3-sr.c
> +++ b/arch/arm64/kvm/hyp/vgic-v3-sr.c
> @@ -445,8 +445,8 @@ void __vgic_v3_init_lrs(void)
>   */
>  u64 __vgic_v3_get_gic_config(void)
>  {
> +	struct exception_mask mask;
>  	u64 val, sre;
> -	unsigned long flags = 0;
>  
>  	/*
>  	 * In compat mode, we cannot access ICC_SRE_EL1 at any EL
> @@ -476,7 +476,7 @@ u64 __vgic_v3_get_gic_config(void)
>  	 * of the exception entry to EL2.
>  	 */
>  	if (has_vhe()) {
> -		flags = local_daif_save();
> +		mask = local_exception_save_and_mask();
>  	} else {
>  		sysreg_clear_set_hcr(0, HCR_AMO | HCR_FMO | HCR_IMO);
>  		isb();
> @@ -491,7 +491,7 @@ u64 __vgic_v3_get_gic_config(void)
>  	isb();
>  
>  	if (has_vhe()) {
> -		local_daif_restore(flags);
> +		local_exception_restore(mask);
>  	} else {
>  		sysreg_clear_set_hcr(HCR_AMO | HCR_FMO | HCR_IMO, 0);
>  		isb();
> diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
> index bbe9cebd3d9d..024876efe0c3 100644
> --- a/arch/arm64/kvm/hyp/vhe/switch.c
> +++ b/arch/arm64/kvm/hyp/vhe/switch.c
> @@ -631,7 +631,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
>  {
>  	int ret;
>  
> -	local_daif_mask();
> +	local_exception_mask();
>  
>  	/*
>  	 * Having IRQs masked via PMR when entering the guest means the GIC
> @@ -647,10 +647,10 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
>  	ret = __kvm_vcpu_run_vhe(vcpu);
>  
>  	/*
> -	 * local_daif_restore() takes care to properly restore PSTATE.DAIF
> +	 * local_exception_restore() takes care to properly restore PSTATE.DAIF
>  	 * and the GIC PMR if the host is using IRQ priorities.
>  	 */
> -	local_daif_restore(DAIF_PROCCTX_NOIRQ);
> +	local_exception_restore(arm64_make_noirq_mask());
>  
>  	return ret;
>  }
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index 85e23388f9bb..f7b8e87a099e 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -34,7 +34,7 @@
>  #include <asm/cpufeature.h>
>  #include <asm/efi.h>
>  #include <asm/exception.h>
> -#include <asm/daifflags.h>
> +#include <asm/exception_masks.h>
>  #include <asm/debug-monitors.h>
>  #include <asm/esr.h>
>  #include <asm/kprobes.h>
> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> index f2be501468ce..9afcd2bc2fd1 100644
> --- a/arch/arm64/mm/mmu.c
> +++ b/arch/arm64/mm/mmu.c
> @@ -2307,7 +2307,7 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
>  	typedef void (ttbr_replace_func)(phys_addr_t);
>  	extern ttbr_replace_func idmap_cpu_replace_ttbr1;
>  	ttbr_replace_func *replace_phys;
> -	unsigned long daif;
> +	struct exception_mask mask;
>  
>  	/* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */
>  	phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp));
> @@ -2323,9 +2323,9 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, bool cnp)
>  	 * We really don't want to take *any* exceptions while TTBR1 is
>  	 * in the process of being replaced so mask everything.
>  	 */
> -	daif = local_daif_save();
> +	mask = local_exception_save_and_mask();
>  	replace_phys(ttbr1);
> -	local_daif_restore(daif);
> +	local_exception_restore(mask);
>  
>  	cpu_uninstall_idmap();
>  }
> diff --git a/drivers/firmware/psci/psci.c b/drivers/firmware/psci/psci.c
> index e73bae6cb23a..4cc834db7c15 100644
> --- a/drivers/firmware/psci/psci.c
> +++ b/drivers/firmware/psci/psci.c
> @@ -24,6 +24,7 @@
>  
>  #include <asm/cpuidle.h>
>  #include <asm/cputype.h>
> +#include <asm/exception_masks.h>
>  #include <asm/hypervisor.h>
>  #include <asm/system_misc.h>
>  #include <asm/smp_plat.h>
> @@ -503,12 +504,12 @@ int psci_cpu_suspend_enter(u32 state)
>  	int ret;
>  
>  	if (!psci_power_state_loses_context(state)) {
> -		struct arm_cpuidle_irq_context context;
> +		struct exception_mask mask;
>  
>  		ct_cpuidle_enter();
> -		arm_cpuidle_save_irq_context(&context);
> +		arm_cpuidle_save_irq_context(&mask);
>  		ret = psci_ops.cpu_suspend(state, 0);
> -		arm_cpuidle_restore_irq_context(&context);
> +		arm_cpuidle_restore_irq_context(&mask);
>  		ct_cpuidle_exit();
>  	} else {
>  		/*
> -- 
> 2.34.1
> 

Re: [PATCH 00/17] arm64: Support FEAT_NMI and Rework Exception Masking

[Mark Rutland]

Hi Jinjie
On Fri, Jul 03, 2026 at 06:01:18PM +0800, Jinjie Ruan wrote:
> Hi all,

Vladimir (cc'd) has also been working on NMI support, but hadn't posted
to the list yet (due to some delays with testing).

I haven't had the chance to look at this yet, but at a high-level we'll
need to figure out what to do going forwards.

Mark.

> This patch series implements support for the ARMv8.8-A/v9.3-A
> hardware NMI extension (FEAT_NMI), introducing the use of PSTATE.ALLINT
> to manage superpriority interrupts on arm64.
> 
> Background and Current Status
> -----------------------------
> 
> The hardware ecosystem is already mature and fully ready for FEAT_NMI:
> 
> - Production Hardware: Huawei's Kunpeng servers (such as the HIP12 based
>   platforms) already feature native hardware support for FEAT_NMI.
> 
> - Emulation: QEMU has integrated robust FEAT_NMI emulation[1] support
>   since 2024.
> 
> Despite the hardware readiness, upstream support for FEAT_NMI has been
> delayed. Previous patch[2] attempts tried to bundle ALLINT management
> directly into the existing DAIF abstraction layers. As Mark Rutland
> pointed out[3], pretending ALLINT or pseudo-NMI (GIC PMR) is part of
> DAIF creates convoluted, unmaintainable hacks that frequently fail to
> handle complex edge cases correctly (such as state escape during
> context switching).
> 
> Reworking Exception Masking (Mark's Feedback)
> ---------------------------------------------
> 
> Following Mark Rutland's strong recommendation ("We must clean up
> the existing approach before we add the real NMI support"), this series
> does not simply stack FEAT_NMI on top of the old framework. Instead, it
> completely reworks how the arm64 kernel manages abstract and logical
> exception masks.
> 
> Per Mark's guidelines, this series achieves the following architectural
> improvements:
> 
> 1. Entry/Exit Specific Helpers (a):
> 
>    Introduces abstract exception mask helpers specifically for exception
>    boundaries. They handle unified unmask-at-entry and mask-at-exit
>    behaviors. This decouples the entry/exit paths from raw DAIF
>    manipulation. In this series, these helpers are first refactored to
>    manage DAIF + PMR cleanly, preparing the ground before any FEAT_NMI
>    code is introduced.
> 
> 2. Logical Exception Mask Separation (b):
> 
>    Introduces a decoupled logical mask tracking mechanism that treats DAIF,
>    PMR, and ALLINT as separate, distinct elements. This enables accurate
>    irqflag tracking and debug assertions to save, restore, and validate all
>    elements without forcing them to fake or pollute a traditional DAIF
>    layout.
> 
> Production Bug Fixes & Integration
> -----------------------------------
> 
> On top of this solid architectural foundation, this series adds the actual
> support for FEAT_NMI (ALLINT management). Crucially, during baisc testing
> and validation on production Kunpeng (HIP12) servers, we identified and
> resolved several critical bugs.
> 
> The series is structured as follows:
> 
> - Patches 1-5: Clean up and rework the existing DAIF/PMR masking into
>   separate logical exception helpers (Pre-requisite refactoring).
> 
> - Patches 6-17: Add FEAT_NMI support for ARM64, including specific
>   stability fixes found on Kunpeng hardware and QEmu.
> 
> Any feedback, testing, or review, especially regarding the exception
> masking refactoring, is highly appreciated.
> 
> [1]: https://lore.kernel.org/all/20240407081733.3231820-1-ruanjinjie@huawei.com/
> [2]: https://lore.kernel.org/linux-arm-kernel/20221112151708.175147-1-broonie@kernel.org/
> [3]: https://lore.kernel.org/linux-arm-kernel/Y5c9SLeJacLYHmP7@FVFF77S0Q05N/
> 
> Jinjie Ruan (5):
>   arm64: Move DAIF macros to ptrace.h and use them centrally
>   arm64: Rework exception masking into abstract logical mask
>   arm64: entry: arm64: entry: Move DAIF masking for EL1 exit to C code
>   arm64: entry: Add entry-specific helpers
>   arm64: Introduce helpers for restoring standard exception masks
> 
> Lorenzo Pieralisi (1):
>   irqchip/gic-v3: Implement FEAT_GICv3_NMI support
> 
> Mark Brown (11):
>   arm64/booting: Document boot requirements for FEAT_NMI
>   arm64/sysreg: Add definitions for immediate versions of MSR ALLINT
>   arm64/hyp-stub: Enable access to ALLINT
>   arm64/idreg: Add an override for FEAT_NMI
>   arm64/cpufeature: Detect PE support for FEAT_NMI
>   KVM: arm64: Hide FEAT_NMI from guests
>   arm64/nmi: Manage masking for superpriority interrupts along with DAIF
>   arm64/entry: Don't call preempt_schedule_irq() with NMIs masked
>   arm64/irq: Document handling of FEAT_NMI in irqflags.h
>   arm64/nmi: Add handling of superpriority interrupts as NMIs
>   arm64/nmi: Add Kconfig for NMI
> 
>  Documentation/arch/arm64/booting.rst     |   6 +
>  arch/arm64/Kconfig                       |  17 ++
>  arch/arm64/include/asm/arch_gicv3.h      |   7 +-
>  arch/arm64/include/asm/assembler.h       |  17 +-
>  arch/arm64/include/asm/cpufeature.h      |   5 +
>  arch/arm64/include/asm/cpuidle.h         |  30 ++-
>  arch/arm64/include/asm/daifflags.h       | 144 --------------
>  arch/arm64/include/asm/efi.h             |  22 ++-
>  arch/arm64/include/asm/el2_setup.h       |  13 ++
>  arch/arm64/include/asm/entry-common.h    |  11 +-
>  arch/arm64/include/asm/exception_masks.h | 232 +++++++++++++++++++++++
>  arch/arm64/include/asm/irq.h             |   2 +
>  arch/arm64/include/asm/irqflags.h        |  10 +
>  arch/arm64/include/asm/kvm_host.h        |   2 +-
>  arch/arm64/include/asm/mmu_context.h     |   2 +-
>  arch/arm64/include/asm/nmi.h             |  23 +++
>  arch/arm64/include/asm/ptrace.h          |  11 +-
>  arch/arm64/include/asm/sysreg.h          |   2 +
>  arch/arm64/include/uapi/asm/ptrace.h     |   1 +
>  arch/arm64/kernel/acpi.c                 |  14 +-
>  arch/arm64/kernel/cpufeature.c           |  58 +++++-
>  arch/arm64/kernel/debug-monitors.c       |   9 +-
>  arch/arm64/kernel/entry-common.c         | 167 +++++++++++-----
>  arch/arm64/kernel/entry.S                |   4 -
>  arch/arm64/kernel/hibernate.c            |  10 +-
>  arch/arm64/kernel/idle.c                 |   7 +-
>  arch/arm64/kernel/irq.c                  |  36 +++-
>  arch/arm64/kernel/machine_kexec.c        |   4 +-
>  arch/arm64/kernel/pi/idreg-override.c    |   1 +
>  arch/arm64/kernel/probes/kprobes.c       |   9 +-
>  arch/arm64/kernel/process.c              |   7 +-
>  arch/arm64/kernel/setup.c                |   4 +-
>  arch/arm64/kernel/signal.c               |   2 +-
>  arch/arm64/kernel/smp.c                  |  22 +--
>  arch/arm64/kernel/suspend.c              |  15 +-
>  arch/arm64/kernel/traps.c                |   2 +-
>  arch/arm64/kvm/emulate-nested.c          |   6 +-
>  arch/arm64/kvm/hyp/include/hyp/switch.h  |   6 +
>  arch/arm64/kvm/hyp/nvhe/host.S           |   4 +-
>  arch/arm64/kvm/hyp/nvhe/hyp-init.S       |   3 +-
>  arch/arm64/kvm/hyp/nvhe/hyp-main.c       |   4 +-
>  arch/arm64/kvm/hyp/vgic-v3-sr.c          |   6 +-
>  arch/arm64/kvm/hyp/vhe/switch.c          |   6 +-
>  arch/arm64/kvm/reset.c                   |   6 +-
>  arch/arm64/mm/fault.c                    |   2 +-
>  arch/arm64/mm/mmu.c                      |   6 +-
>  arch/arm64/tools/cpucaps                 |   2 +
>  drivers/firmware/psci/psci.c             |   7 +-
>  drivers/irqchip/irq-gic-v3.c             | 152 +++++++++++++--
>  include/linux/irqchip/arm-gic-v3.h       |   4 +
>  50 files changed, 822 insertions(+), 320 deletions(-)
>  delete mode 100644 arch/arm64/include/asm/daifflags.h
>  create mode 100644 arch/arm64/include/asm/exception_masks.h
>  create mode 100644 arch/arm64/include/asm/nmi.h
> 
> -- 
> 2.34.1
> 

Re: [PATCH 01/17] arm64: Move DAIF macros to ptrace.h and use them centrally

[Breno Leitao]

On Fri, Jul 03, 2026 at 06:01:19PM +0800, Jinjie Ruan wrote:
> So move the definitions of DAIF_PROCCTX, DAIF_PROCCTX_NOIRQ, DAIF_ERRCTX,
> and DAIF_MASK from <asm/daifflags.h> to <asm/ptrace.h>. 

This seems a bit counter intuitive, to have DAIF definitions at
arm/ptrace.h instead of asm/daifflags.h, no?

> diff --git a/arch/arm64/kvm/hyp/nvhe/host.S b/arch/arm64/kvm/hyp/nvhe/host.S
> index 9393fe3ea6a1..cbe2a616c726 100644
> --- a/arch/arm64/kvm/hyp/nvhe/host.S
> +++ b/arch/arm64/kvm/hyp/nvhe/host.S
> @@ -11,6 +11,7 @@
>  #include <asm/kvm_asm.h>
>  #include <asm/kvm_mmu.h>
>  #include <asm/kvm_ptrauth.h>
> +#include <asm/ptrace.h>

..

> diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S b/arch/arm64/kvm/hyp/nvhe/hyp-init.S
> index 89cb553be1e5..26ea02e7f5fd 100644
> --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S
> +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S
> @@ -15,6 +15,7 @@
>  #include <asm/kvm_asm.h>
>  #include <asm/kvm_mmu.h>
>  #include <asm/pgtable-hwdef.h>
> +#include <asm/ptrace.h>

And then you need to do this in many low level files, which sounds less
intuitive to have to have ptrace.h include instead of asm/daifflags.h
(which is clear why we need it).

--breno