[PATCH 0/2] arm64: sdei: Handle exceptions taken inside the event handler

[PATCH 0/2] arm64: sdei: Handle exceptions taken inside the event handler

[Kiryl Shutsemau (Meta)]

An SDEI event can land anywhere, and a synchronous exception taken by its
handler breaks the context it interrupted:

 - With KPTI, an event that lands in user space runs the handler on the
   trampoline vectors, whose current-EL half is empty. The exception
   loops there and the CPU is lost.

 - The exception overwrites ELR, SPSR, ESR and FAR. If the event landed
   in exception entry or return, the interrupted context runs on the
   handler's values: a task gets SIGILL, or the kernel oopses on a fault
   it never took.

Patch 1 runs the handler on the kernel vectors. Patch 2 saves and
restores the exception registers around it.

Handlers take such exceptions in normal operation now that SDEI delivers
NMI backtraces: show_regs() reads through _nofault() accessors that
fault, and a kprobe or WARN anywhere in the path takes a BRK.

Tested under QEMU TCG with TF-A, 4 CPUs, 20 x sysrq-l under load, with a
test-only copy_from_kernel_nofault() of address 0x10 in
sdei_nmi_handler():

 - v7.4-rc1, KPTI on: CPUs stuck in the trampoline vectors, RCU stalls.
 - v7.4-rc1, KPTI off: 60 "unsafe: exception during handler" warnings,
   SIGILL and SIGSEGV in user tasks, and an oops in execve().
 - With the series, KPTI on and off: no warnings, no stalls.

Patch 2 was also run on arm64 hardware with firmware SDEI, where a plain
sysrq-l was enough to trigger the warning without it.

Kiryl Shutsemau (Meta) (2):
  arm64: sdei: Run the event handler on the kernel vectors
  arm64: sdei: Preserve the interrupted context's exception registers

 arch/arm64/kernel/sdei.c | 83 +++++++++++++++++++++++++++++++++++-----
 1 file changed, 74 insertions(+), 9 deletions(-)


base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
-- 
2.54.0

[PATCH 1/2] arm64: sdei: Run the event handler on the kernel vectors

[Kiryl Shutsemau (Meta)]

With KPTI, VBAR_EL1 holds the trampoline vectors from the return to EL0
until the next exception from EL0 maps the kernel and installs the
kernel vectors. The trampoline vectors only handle exceptions from EL0;
their current-EL half is zero-filled.

An SDEI event is dispatched regardless of PSTATE.DAIF, so it lands there
whenever the CPU is in user space. The SDEI entry trampoline maps the
kernel but leaves VBAR_EL1 alone. An exception taken by the handler then
executes the zeros, takes an undefined instruction exception to the same
vector, and loops there forever. The event never completes and the CPU
is lost; RCU stall reports are the first sign of it.

RAS notifications land in user space whenever a task consumes the
error, and NMI backtraces land wherever the CPU happens to be. The GHES
handler takes no exception on its own, but any exception in the handler
path does it: a WARN, a kprobe hit, or a fault fixed up by a _nofault()
accessor.

Install the kernel vectors in do_sdei_event() when VBAR_EL1 holds the
trampoline vectors, and restore it before returning to firmware,
including when the handler fails. The IRQ-like resume into user space
keeps using the interrupted value, as it must go through the trampoline
to map the kernel.

Fixes: 79e9aa59dc29 ("arm64: sdei: Add trampoline code for remapping the kernel")
Cc: stable@vger.kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 arch/arm64/kernel/sdei.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c
index 778f2a1faac8..9fa2a3aeed48 100644
--- a/arch/arm64/kernel/sdei.c
+++ b/arch/arm64/kernel/sdei.c
@@ -18,6 +18,7 @@
 #include <asm/sections.h>
 #include <asm/stacktrace.h>
 #include <asm/sysreg.h>
+#include <asm/vectors.h>
 #include <asm/vmap_stack.h>
 
 unsigned long sdei_exit_mode;
@@ -214,6 +215,20 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 	if (arm64_kernel_unmapped_at_el0())
 		clobbered_registers++;
 
+	/*
+	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
+	 * EL0 until the next exception from EL0 maps the kernel. They only
+	 * handle exceptions from EL0: an exception taken by the handler would
+	 * run their empty current-EL half and never come back. Run the handler
+	 * on the kernel vectors. The IRQ-like resume below keeps using the
+	 * interrupted VBAR, so it still goes through the trampoline.
+	 */
+	if (arm64_kernel_unmapped_at_el0() &&
+	    vbar == (unsigned long)this_cpu_read(this_cpu_vector)) {
+		write_sysreg((unsigned long)vectors, vbar_el1);
+		isb();
+	}
+
 	/* Retrieve the missing registers values */
 	for (i = 0; i < clobbered_registers; i++) {
 		/* from within the handler, this call always succeeds */
@@ -221,8 +236,6 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 	}
 
 	err = sdei_event_handler(regs, arg);
-	if (err)
-		return SDEI_EV_FAILED;
 
 	if (elr != read_sysreg(elr_el1)) {
 		/*
@@ -233,6 +246,11 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 		pr_warn("unsafe: exception during handler\n");
 	}
 
+	write_sysreg(vbar, vbar_el1);
+
+	if (err)
+		return SDEI_EV_FAILED;
+
 	mode = regs->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK);
 
 	/*
-- 
2.54.0

Re: [PATCH 1/2] arm64: sdei: Run the event handler on the kernel vectors

[Usama Arif]

On Wed,  4 Nov 2026 18:09:33 +0000 "Kiryl Shutsemau (Meta)" <kas@kernel.org> wrote:

> With KPTI, VBAR_EL1 holds the trampoline vectors from the return to EL0
> until the next exception from EL0 maps the kernel and installs the
> kernel vectors. The trampoline vectors only handle exceptions from EL0;
> their current-EL half is zero-filled.
> 
> An SDEI event is dispatched regardless of PSTATE.DAIF, so it lands there
> whenever the CPU is in user space. The SDEI entry trampoline maps the
> kernel but leaves VBAR_EL1 alone. An exception taken by the handler then
> executes the zeros, takes an undefined instruction exception to the same
> vector, and loops there forever. The event never completes and the CPU
> is lost; RCU stall reports are the first sign of it.
> 
> RAS notifications land in user space whenever a task consumes the
> error, and NMI backtraces land wherever the CPU happens to be. The GHES
> handler takes no exception on its own, but any exception in the handler
> path does it: a WARN, a kprobe hit, or a fault fixed up by a _nofault()
> accessor.
> 
> Install the kernel vectors in do_sdei_event() when VBAR_EL1 holds the
> trampoline vectors, and restore it before returning to firmware,
> including when the handler fails. The IRQ-like resume into user space
> keeps using the interrupted value, as it must go through the trampoline
> to map the kernel.
> 
> Fixes: 79e9aa59dc29 ("arm64: sdei: Add trampoline code for remapping the kernel")
> Cc: stable@vger.kernel.org
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
>  arch/arm64/kernel/sdei.c | 22 ++++++++++++++++++++--
>  1 file changed, 20 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c
> index 778f2a1faac8..9fa2a3aeed48 100644
> --- a/arch/arm64/kernel/sdei.c
> +++ b/arch/arm64/kernel/sdei.c
> @@ -18,6 +18,7 @@
>  #include <asm/sections.h>
>  #include <asm/stacktrace.h>
>  #include <asm/sysreg.h>
> +#include <asm/vectors.h>
>  #include <asm/vmap_stack.h>
>  
>  unsigned long sdei_exit_mode;
> @@ -214,6 +215,20 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
>  	if (arm64_kernel_unmapped_at_el0())
>  		clobbered_registers++;
>  
> +	/*
> +	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
> +	 * EL0 until the next exception from EL0 maps the kernel. They only
> +	 * handle exceptions from EL0: an exception taken by the handler would
> +	 * run their empty current-EL half and never come back. Run the handler
> +	 * on the kernel vectors. The IRQ-like resume below keeps using the
> +	 * interrupted VBAR, so it still goes through the trampoline.
> +	 */
> +	if (arm64_kernel_unmapped_at_el0() &&
> +	    vbar == (unsigned long)this_cpu_read(this_cpu_vector)) {
> +		write_sysreg((unsigned long)vectors, vbar_el1);
> +		isb();
> +	}
> +

Does this cover the whole path that can take an exception?  The
swap is done inside do_sdei_event(), but __sdei_handler() calls
irqentry_nmi_enter() before it and irqentry_nmi_exit() after it,
and both of those still run with the trampoline vectors in
VBAR_EL1 when the event interrupted user space.

Both can take an exception.  ct_nmi_enter() opens with
WARN_ON_ONCE(ct_nmi_nesting() < 0), and ct_nmi_exit() has
WARN_ON_ONCE(ct_nmi_nesting() <= 0) and
WARN_ON_ONCE(!rcu_is_watching_curr_cpu()).  A WARN is a BRK, so
either would run the zero filled current-EL half and lose the CPU
in the way the commit message describes.

Would doing the swap in __sdei_handler(), around the whole NMI
entry and exit pair, close that window?


>  	/* Retrieve the missing registers values */
>  	for (i = 0; i < clobbered_registers; i++) {
>  		/* from within the handler, this call always succeeds */
> @@ -221,8 +236,6 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
>  	}
>  
>  	err = sdei_event_handler(regs, arg);
> -	if (err)
> -		return SDEI_EV_FAILED;
>  
>  	if (elr != read_sysreg(elr_el1)) {
>  		/*
> @@ -233,6 +246,11 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
>  		pr_warn("unsafe: exception during handler\n");
>  	}
>  
> +	write_sysreg(vbar, vbar_el1);
> +
> +	if (err)
> +		return SDEI_EV_FAILED;
> +
>  	mode = regs->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK);
>  
>  	/*
> -- 
> 2.54.0
> 
> 

[PATCH 2/2] arm64: sdei: Preserve the interrupted context's exception registers

[Kiryl Shutsemau (Meta)]

An SDEI event is dispatched by firmware regardless of PSTATE.DAIF, so it
can interrupt the kernel where the exception registers are live:

 - exception entry, before kernel_entry has saved ELR_EL1/SPSR_EL1 and
   before the C handler has read ESR_EL1/FAR_EL1;
 - exception return, after kernel_exit has loaded ELR_EL1/SPSR_EL1 and
   before the ERET.

An IRQ or pseudo-NMI cannot arrive there with PSTATE.I set. Firmware
saves only x0-x17, PC and PSTATE around an SDEI event, so a synchronous
exception taken by the handler overwrites the interrupted context's
ELR, SPSR, ESR and FAR. do_sdei_event() only notices that ELR changed,
and warns.

The interrupted context then carries on with the handler's state. A
task interrupted on its way into the kernel can be sent SIGILL for the
handler's kernel data abort. A task interrupted on its way back to user
space ERETs to EL1, at the handler's fixup address, with user registers
loaded.

With SDEI delivering NMI backtraces, this happens in normal operation.
The handler runs show_regs(), which reads through possibly-NULL pointers
with the _nofault() accessors. print_worker_info() faults this way for a
kworker that is not running a work item, and a single sysrq-l warned
once per such kworker.

Save ELR, SPSR, ESR and FAR on entry to do_sdei_event() and restore them
before returning to firmware.

Once they are restored, a nested exception is harmless if its handler is
safe in NMI context. That holds for EL1 data aborts on translation,
permission and access-flag faults, which never take the mmap lock in NMI
context, and for BRK. Warn once, with the syndrome, about any other
exception class.

Fixes: f5df26961853 ("arm64: kernel: Add arch-specific SDEI entry code and CPU masking")
Cc: stable@vger.kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 arch/arm64/kernel/sdei.c | 61 +++++++++++++++++++++++++++++++++++-----
 1 file changed, 54 insertions(+), 7 deletions(-)

diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c
index 9fa2a3aeed48..1ea566f30586 100644
--- a/arch/arm64/kernel/sdei.c
+++ b/arch/arm64/kernel/sdei.c
@@ -196,6 +196,31 @@ unsigned long sdei_arch_get_entry_point(int conduit)
 	return 0;
 }
 
+/*
+ * Exceptions an SDEI handler may take without harm once do_sdei_event() has
+ * restored the interrupted context's exception registers. Their handlers are
+ * safe in NMI context:
+ *  - EL1 data aborts on translation, permission and access-flag faults:
+ *    faulthandler_disabled() is true in NMI context, so do_page_fault() never
+ *    takes the mmap lock. This covers the _nofault() accessors used by the
+ *    generic dump_stack()/show_regs() code (e.g. print_worker_info()).
+ *  - BRK: kprobes, WARN/BUG, kCFI and UBSAN traps, all NMI-capable.
+ * Anything else taken at EL1 is fatal on its own, or unexpected here.
+ */
+static bool sdei_nested_exception_ok(unsigned long esr)
+{
+	switch (ESR_ELx_EC(esr)) {
+	case ESR_ELx_EC_DABT_CUR:
+		return esr_fsc_is_translation_fault(esr) ||
+		       esr_fsc_is_permission_fault(esr) ||
+		       esr_fsc_is_access_flag_fault(esr);
+	case ESR_ELx_EC_BRK64:
+		return true;
+	default:
+		return false;
+	}
+}
+
 /*
  * do_sdei_event() returns one of:
  *  SDEI_EV_HANDLED -  success, return to the interrupted context.
@@ -209,6 +234,10 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 	int i, err = 0;
 	int clobbered_registers = 4;
 	u64 elr = read_sysreg(elr_el1);
+	u64 spsr = read_sysreg(spsr_el1);
+	u64 esr = read_sysreg(esr_el1);
+	u64 far = read_sysreg(far_el1);
+	u64 nested_elr, nested_esr, nested_far;
 	u32 kernel_mode = read_sysreg(CurrentEL) | 1;	/* +SPSel */
 	unsigned long vbar = read_sysreg(vbar_el1);
 
@@ -237,15 +266,33 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 
 	err = sdei_event_handler(regs, arg);
 
-	if (elr != read_sysreg(elr_el1)) {
-		/*
-		 * We took a synchronous exception from the SDEI handler.
-		 * This could deadlock, and if you interrupt KVM it will
-		 * hyp-panic instead.
-		 */
-		pr_warn("unsafe: exception during handler\n");
+	/*
+	 * SDEI is not an architectural exception: nothing saved the
+	 * interrupted context's exception registers on the way in, and it may
+	 * be in the middle of exception entry or return (SDEI ignores DAIF).
+	 * Any synchronous exception the handler took has overwritten them.
+	 * Only the last one's syndrome survives; report it if it is not one
+	 * known to be harmless.
+	 *
+	 * If the event interrupted a KVM world switch, VBAR points at the KVM
+	 * hyp vectors and any such exception is a hyp panic instead.
+	 */
+	nested_elr = read_sysreg(elr_el1);
+	nested_esr = read_sysreg(esr_el1);
+	nested_far = read_sysreg(far_el1);
+	if (nested_elr != elr && !sdei_nested_exception_ok(nested_esr)) {
+		pr_warn_once("unexpected exception during handler: ESR %#llx FAR %#llx near %pS\n",
+			     nested_esr, nested_far, (void *)nested_elr);
 	}
 
+	/*
+	 * Put them back unconditionally. An unchanged ELR does not rule out a
+	 * nested exception: one may have returned to the same address.
+	 */
+	write_sysreg(elr, elr_el1);
+	write_sysreg(spsr, spsr_el1);
+	write_sysreg(esr, esr_el1);
+	write_sysreg(far, far_el1);
 	write_sysreg(vbar, vbar_el1);
 
 	if (err)
-- 
2.54.0