[PATCH v2 0/2] arm64: sdei: Handle exceptions taken inside the event handler

[PATCH v2 0/2] arm64: sdei: Handle exceptions taken inside the event handler

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

An SDEI event can land anywhere, and a synchronous exception taken by its
handler breaks the context it interrupted:

 - With KPTI, an event that lands in user space runs the handler on the
   trampoline vectors, whose current-EL half is empty. The exception
   loops there and the CPU is lost.

 - The exception overwrites ELR, SPSR, ESR and FAR. If the event landed
   in exception entry or return, the interrupted context runs on the
   handler's values: a task gets SIGILL, or the kernel oopses on a fault
   it never took.

Patch 1 runs the handler on the kernel vectors. Patch 2 saves and
restores the exception registers around it.

Handlers take such exceptions in normal operation now that SDEI delivers
NMI backtraces: show_regs() reads through _nofault() accessors that
fault, and a kprobe or WARN anywhere in the path takes a BRK.

Tested under QEMU TCG with TF-A, 4 CPUs, 20 x sysrq-l under load, with a
test-only copy_from_kernel_nofault() of address 0x10 in
sdei_nmi_handler():

 - v7.4-rc1, KPTI on: CPUs stuck in the trampoline vectors, RCU stalls.
 - v7.4-rc1, KPTI off: 60 "unsafe: exception during handler" warnings,
   SIGILL and SIGSEGV in user tasks, and an oops in execve().
 - With the series, KPTI on and off: no warnings, no stalls. The same
   with an eprobe on irq_disable that reads address 0x10, which faults
   inside irqentry_nmi_enter() when the event lands in user space.

An earlier revision of patch 2 was also run on arm64 hardware with
firmware SDEI, where a plain sysrq-l was enough to trigger the warning
without it.

Changes since v1
================

  https://lore.kernel.org/all/20261104180936.2981070-1-kas@kernel.org/

 - Switch VBAR and save the exception registers in __sdei_handler(),
   before irqentry_nmi_enter(), and restore them after
   irqentry_nmi_exit(). Both can take an exception too: the eprobe above
   wedged v1 with KPTI on (Usama). do_sdei_event() takes the interrupted
   VBAR for the IRQ-like resume.

Kiryl Shutsemau (Meta) (2):
  arm64: sdei: Run the event handler on the kernel vectors
  arm64: sdei: Preserve the interrupted context's exception registers

 arch/arm64/include/asm/sdei.h    |  3 +-
 arch/arm64/kernel/entry-common.c | 40 +++++++++++++++++++++-
 arch/arm64/kernel/sdei.c         | 59 ++++++++++++++++++++++++++------
 3 files changed, 89 insertions(+), 13 deletions(-)

Range-diff against v1:
1:  f3e1e7fd09f7 ! 1:  93ef4a287a4c arm64: sdei: Run the event handler on the kernel vectors
    @@ Commit message
         path does it: a WARN, a kprobe hit, or a fault fixed up by a _nofault()
         accessor.
     
    -    Install the kernel vectors in do_sdei_event() when VBAR_EL1 holds the
    -    trampoline vectors, and restore it before returning to firmware,
    -    including when the handler fails. The IRQ-like resume into user space
    -    keeps using the interrupted value, as it must go through the trampoline
    -    to map the kernel.
    +    Install the kernel vectors in __sdei_handler() when VBAR_EL1 holds the
    +    trampoline vectors, and restore it before returning to firmware. The
    +    switch covers irqentry_nmi_enter() and irqentry_nmi_exit() too, as Usama
    +    Arif pointed out: their tracepoints and context tracking WARNs can take
    +    an exception as well.
    +
    +    The IRQ-like resume into user space must still go through the
    +    trampoline to map the kernel. do_sdei_event() takes the interrupted
    +    VBAR_EL1 and keeps basing the resume address on it.
     
         Fixes: 79e9aa59dc29 ("arm64: sdei: Add trampoline code for remapping the kernel")
         Cc: stable@vger.kernel.org
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
     
    - ## arch/arm64/kernel/sdei.c ##
    + ## arch/arm64/include/asm/sdei.h ##
    +@@ arch/arm64/include/asm/sdei.h: asmlinkage unsigned long __sdei_handler(struct pt_regs *regs,
    + 					struct sdei_registered_event *arg);
    + 
    + unsigned long do_sdei_event(struct pt_regs *regs,
    +-			    struct sdei_registered_event *arg);
    ++			    struct sdei_registered_event *arg,
    ++			    unsigned long vbar);
    + 
    + unsigned long sdei_arch_get_entry_point(int conduit);
    + #define sdei_arch_get_entry_point(x)	sdei_arch_get_entry_point(x)
    +
    + ## arch/arm64/kernel/entry-common.c ##
     @@
    - #include <asm/sections.h>
      #include <asm/stacktrace.h>
      #include <asm/sysreg.h>
    + #include <asm/system_misc.h>
     +#include <asm/vectors.h>
    - #include <asm/vmap_stack.h>
      
    - unsigned long sdei_exit_mode;
    -@@ arch/arm64/kernel/sdei.c: unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
    - 	if (arm64_kernel_unmapped_at_el0())
    - 		clobbered_registers++;
    + /*
    +  * Where the context being returned to had an active percpu GPR critical
    +@@ arch/arm64/kernel/entry-common.c: asmlinkage void noinstr __noreturn handle_bad_stack(struct pt_regs *regs)
    + asmlinkage noinstr unsigned long
    + __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
    + {
    ++	unsigned long vbar;
    + 	irqentry_state_t state;
    + 	unsigned long ret;
    + 
    +@@ arch/arm64/kernel/entry-common.c: __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
    + 	else if (cpu_has_pan())
    + 		set_pstate_pan(0);
      
     +	/*
     +	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
     +	 * EL0 until the next exception from EL0 maps the kernel. They only
    -+	 * handle exceptions from EL0: an exception taken by the handler would
    -+	 * run their empty current-EL half and never come back. Run the handler
    -+	 * on the kernel vectors. The IRQ-like resume below keeps using the
    -+	 * interrupted VBAR, so it still goes through the trampoline.
    ++	 * handle exceptions from EL0: an exception taken here would run their
    ++	 * empty current-EL half and never come back. Switch to the kernel
    ++	 * vectors. The IRQ-like resume keeps using the interrupted VBAR, so it
    ++	 * still goes through the trampoline.
     +	 */
    ++	vbar = read_sysreg(vbar_el1);
     +	if (arm64_kernel_unmapped_at_el0() &&
    -+	    vbar == (unsigned long)this_cpu_read(this_cpu_vector)) {
    ++	    vbar == (unsigned long)__this_cpu_read(this_cpu_vector)) {
     +		write_sysreg((unsigned long)vectors, vbar_el1);
     +		isb();
     +	}
     +
    - 	/* Retrieve the missing registers values */
    - 	for (i = 0; i < clobbered_registers; i++) {
    - 		/* from within the handler, this call always succeeds */
    -@@ arch/arm64/kernel/sdei.c: unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
    - 	}
    - 
    - 	err = sdei_event_handler(regs, arg);
    --	if (err)
    --		return SDEI_EV_FAILED;
    - 
    - 	if (elr != read_sysreg(elr_el1)) {
    - 		/*
    -@@ arch/arm64/kernel/sdei.c: unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
    - 		pr_warn("unsafe: exception during handler\n");
    - 	}
    + 	state = irqentry_nmi_enter(regs);
    +-	ret = do_sdei_event(regs, arg);
    ++	ret = do_sdei_event(regs, arg, vbar);
    + 	irqentry_nmi_exit(regs, state);
      
     +	write_sysreg(vbar, vbar_el1);
     +
    -+	if (err)
    -+		return SDEI_EV_FAILED;
    -+
    - 	mode = regs->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK);
    + 	return ret;
    + }
    + #endif /* CONFIG_ARM_SDE_INTERFACE */
    +
    + ## arch/arm64/kernel/sdei.c ##
    +@@ arch/arm64/kernel/sdei.c: unsigned long sdei_arch_get_entry_point(int conduit)
    +  *  SDEI_EV_HANDLED -  success, return to the interrupted context.
    +  *  SDEI_EV_FAILED  -  failure, return this error code to firmware.
    +  *  virtual-address -  success, return to this address.
    ++ *
    ++ * vbar is the interrupted context's VBAR_EL1, for the IRQ-like resume.
    +  */
    + unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
    +-				      struct sdei_registered_event *arg)
    ++				      struct sdei_registered_event *arg,
    ++				      unsigned long vbar)
    + {
    + 	u32 mode;
    + 	int i, err = 0;
    + 	int clobbered_registers = 4;
    + 	u64 elr = read_sysreg(elr_el1);
    + 	u32 kernel_mode = read_sysreg(CurrentEL) | 1;	/* +SPSel */
    +-	unsigned long vbar = read_sysreg(vbar_el1);
      
    - 	/*
    + 	if (arm64_kernel_unmapped_at_el0())
    + 		clobbered_registers++;
2:  ba422c8220e5 ! 2:  07d3078373ee arm64: sdei: Preserve the interrupted context's exception registers
    @@ Commit message
         kworker that is not running a work item, and a single sysrq-l warned
         once per such kworker.
     
    -    Save ELR, SPSR, ESR and FAR on entry to do_sdei_event() and restore them
    -    before returning to firmware.
    +    Save ELR, SPSR, ESR and FAR in __sdei_handler() before
    +    irqentry_nmi_enter(), which can take an exception too, and restore them
    +    after irqentry_nmi_exit().
     
         Once they are restored, a nested exception is harmless if its handler is
         safe in NMI context. That holds for EL1 data aborts on translation,
    @@ Commit message
         Cc: stable@vger.kernel.org
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
     
    + ## arch/arm64/kernel/entry-common.c ##
    +@@ arch/arm64/kernel/entry-common.c: asmlinkage void noinstr __noreturn handle_bad_stack(struct pt_regs *regs)
    + asmlinkage noinstr unsigned long
    + __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
    + {
    +-	unsigned long vbar;
    ++	unsigned long elr, spsr, esr, far, vbar;
    + 	irqentry_state_t state;
    + 	unsigned long ret;
    + 
    +@@ arch/arm64/kernel/entry-common.c: __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
    + 	else if (cpu_has_pan())
    + 		set_pstate_pan(0);
    + 
    ++	/*
    ++	 * Nothing saved the interrupted context's exception registers on the
    ++	 * way in, and it may be in the middle of exception entry or return
    ++	 * (SDEI ignores DAIF). Save them before anything can take an exception
    ++	 * and overwrite them.
    ++	 */
    ++	elr = read_sysreg(elr_el1);
    ++	spsr = read_sysreg(spsr_el1);
    ++	esr = read_sysreg(esr_el1);
    ++	far = read_sysreg(far_el1);
    ++
    + 	/*
    + 	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
    + 	 * EL0 until the next exception from EL0 maps the kernel. They only
    +@@ arch/arm64/kernel/entry-common.c: __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
    + 	ret = do_sdei_event(regs, arg, vbar);
    + 	irqentry_nmi_exit(regs, state);
    + 
    ++	/*
    ++	 * Put them back unconditionally. An unchanged ELR does not rule out a
    ++	 * nested exception: one may have returned to the same address.
    ++	 */
    ++	write_sysreg(elr, elr_el1);
    ++	write_sysreg(spsr, spsr_el1);
    ++	write_sysreg(esr, esr_el1);
    ++	write_sysreg(far, far_el1);
    + 	write_sysreg(vbar, vbar_el1);
    + 
    + 	return ret;
    +
      ## arch/arm64/kernel/sdei.c ##
     @@ arch/arm64/kernel/sdei.c: unsigned long sdei_arch_get_entry_point(int conduit)
      	return 0;
      }
      
     +/*
    -+ * Exceptions an SDEI handler may take without harm once do_sdei_event() has
    ++ * Exceptions an SDEI handler may take without harm once __sdei_handler() has
     + * restored the interrupted context's exception registers. Their handlers are
     + * safe in NMI context:
     + *  - EL1 data aborts on translation, permission and access-flag faults:
    @@ arch/arm64/kernel/sdei.c: unsigned long __kprobes do_sdei_event(struct pt_regs *
      	int i, err = 0;
      	int clobbered_registers = 4;
      	u64 elr = read_sysreg(elr_el1);
    -+	u64 spsr = read_sysreg(spsr_el1);
    -+	u64 esr = read_sysreg(esr_el1);
    -+	u64 far = read_sysreg(far_el1);
     +	u64 nested_elr, nested_esr, nested_far;
      	u32 kernel_mode = read_sysreg(CurrentEL) | 1;	/* +SPSel */
    - 	unsigned long vbar = read_sysreg(vbar_el1);
      
    + 	if (arm64_kernel_unmapped_at_el0())
     @@ arch/arm64/kernel/sdei.c: unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
    + 	}
      
      	err = sdei_event_handler(regs, arg);
    - 
    --	if (elr != read_sysreg(elr_el1)) {
    --		/*
    --		 * We took a synchronous exception from the SDEI handler.
    --		 * This could deadlock, and if you interrupt KVM it will
    --		 * hyp-panic instead.
    --		 */
    --		pr_warn("unsafe: exception during handler\n");
    ++
     +	/*
    -+	 * SDEI is not an architectural exception: nothing saved the
    -+	 * interrupted context's exception registers on the way in, and it may
    -+	 * be in the middle of exception entry or return (SDEI ignores DAIF).
    -+	 * Any synchronous exception the handler took has overwritten them.
    -+	 * Only the last one's syndrome survives; report it if it is not one
    -+	 * known to be harmless.
    ++	 * A synchronous exception taken by the handler has overwritten the
    ++	 * exception registers; __sdei_handler() puts them back. Only the last
    ++	 * one's syndrome survives; report it if it is not one known to be
    ++	 * harmless.
     +	 *
     +	 * If the event interrupted a KVM world switch, VBAR points at the KVM
     +	 * hyp vectors and any such exception is a hyp panic instead.
    @@ arch/arm64/kernel/sdei.c: unsigned long __kprobes do_sdei_event(struct pt_regs *
     +	if (nested_elr != elr && !sdei_nested_exception_ok(nested_esr)) {
     +		pr_warn_once("unexpected exception during handler: ESR %#llx FAR %#llx near %pS\n",
     +			     nested_esr, nested_far, (void *)nested_elr);
    - 	}
    - 
    -+	/*
    -+	 * Put them back unconditionally. An unchanged ELR does not rule out a
    -+	 * nested exception: one may have returned to the same address.
    -+	 */
    -+	write_sysreg(elr, elr_el1);
    -+	write_sysreg(spsr, spsr_el1);
    -+	write_sysreg(esr, esr_el1);
    -+	write_sysreg(far, far_el1);
    - 	write_sysreg(vbar, vbar_el1);
    - 
    ++	}
    ++
      	if (err)
    + 		return SDEI_EV_FAILED;
    + 
    +-	if (elr != read_sysreg(elr_el1)) {
    +-		/*
    +-		 * We took a synchronous exception from the SDEI handler.
    +-		 * This could deadlock, and if you interrupt KVM it will
    +-		 * hyp-panic instead.
    +-		 */
    +-		pr_warn("unsafe: exception during handler\n");
    +-	}
    +-
    + 	mode = regs->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK);
    + 
    + 	/*

base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
-- 
2.54.0

[PATCH v2 0/2] arm64: sdei: Handle exceptions taken inside the event handler

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

An SDEI event can land anywhere, and a synchronous exception taken by its
handler breaks the context it interrupted:

 - With KPTI, an event that lands in user space runs the handler on the
   trampoline vectors, whose current-EL half is empty. The exception
   loops there and the CPU is lost.

 - The exception overwrites ELR, SPSR, ESR and FAR. If the event landed
   in exception entry or return, the interrupted context runs on the
   handler's values: a task gets SIGILL, or the kernel oopses on a fault
   it never took.

Patch 1 runs the handler on the kernel vectors. Patch 2 saves and
restores the exception registers around it.

Handlers take such exceptions in normal operation now that SDEI delivers
NMI backtraces: show_regs() reads through _nofault() accessors that
fault, and a kprobe or WARN anywhere in the path takes a BRK.

Tested under QEMU TCG with TF-A, 4 CPUs, 20 x sysrq-l under load, with a
test-only copy_from_kernel_nofault() of address 0x10 in
sdei_nmi_handler():

 - v7.4-rc1, KPTI on: CPUs stuck in the trampoline vectors, RCU stalls.
 - v7.4-rc1, KPTI off: 60 "unsafe: exception during handler" warnings,
   SIGILL and SIGSEGV in user tasks, and an oops in execve().
 - With the series, KPTI on and off: no warnings, no stalls. The same
   with an eprobe on irq_disable that reads address 0x10, which faults
   inside irqentry_nmi_enter() when the event lands in user space.

An earlier revision of patch 2 was also run on arm64 hardware with
firmware SDEI, where a plain sysrq-l was enough to trigger the warning
without it.

Changes since v1
================

  https://lore.kernel.org/all/20261104180936.2981070-1-kas@kernel.org/

 - Switch VBAR and save the exception registers in __sdei_handler(),
   before irqentry_nmi_enter(), and restore them after
   irqentry_nmi_exit(). Both can take an exception too: the eprobe above
   wedged v1 with KPTI on (Usama). do_sdei_event() takes the interrupted
   VBAR for the IRQ-like resume.

Kiryl Shutsemau (Meta) (2):
  arm64: sdei: Run the event handler on the kernel vectors
  arm64: sdei: Preserve the interrupted context's exception registers

 arch/arm64/include/asm/sdei.h    |  3 +-
 arch/arm64/kernel/entry-common.c | 40 +++++++++++++++++++++-
 arch/arm64/kernel/sdei.c         | 59 ++++++++++++++++++++++++++------
 3 files changed, 89 insertions(+), 13 deletions(-)

Range-diff against v1:
1:  f3e1e7fd09f7 ! 1:  93ef4a287a4c arm64: sdei: Run the event handler on the kernel vectors
    @@ Commit message
         path does it: a WARN, a kprobe hit, or a fault fixed up by a _nofault()
         accessor.
     
    -    Install the kernel vectors in do_sdei_event() when VBAR_EL1 holds the
    -    trampoline vectors, and restore it before returning to firmware,
    -    including when the handler fails. The IRQ-like resume into user space
    -    keeps using the interrupted value, as it must go through the trampoline
    -    to map the kernel.
    +    Install the kernel vectors in __sdei_handler() when VBAR_EL1 holds the
    +    trampoline vectors, and restore it before returning to firmware. The
    +    switch covers irqentry_nmi_enter() and irqentry_nmi_exit() too, as Usama
    +    Arif pointed out: their tracepoints and context tracking WARNs can take
    +    an exception as well.
    +
    +    The IRQ-like resume into user space must still go through the
    +    trampoline to map the kernel. do_sdei_event() takes the interrupted
    +    VBAR_EL1 and keeps basing the resume address on it.
     
         Fixes: 79e9aa59dc29 ("arm64: sdei: Add trampoline code for remapping the kernel")
         Cc: stable@vger.kernel.org
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
     
    - ## arch/arm64/kernel/sdei.c ##
    + ## arch/arm64/include/asm/sdei.h ##
    +@@ arch/arm64/include/asm/sdei.h: asmlinkage unsigned long __sdei_handler(struct pt_regs *regs,
    + 					struct sdei_registered_event *arg);
    + 
    + unsigned long do_sdei_event(struct pt_regs *regs,
    +-			    struct sdei_registered_event *arg);
    ++			    struct sdei_registered_event *arg,
    ++			    unsigned long vbar);
    + 
    + unsigned long sdei_arch_get_entry_point(int conduit);
    + #define sdei_arch_get_entry_point(x)	sdei_arch_get_entry_point(x)
    +
    + ## arch/arm64/kernel/entry-common.c ##
     @@
    - #include <asm/sections.h>
      #include <asm/stacktrace.h>
      #include <asm/sysreg.h>
    + #include <asm/system_misc.h>
     +#include <asm/vectors.h>
    - #include <asm/vmap_stack.h>
      
    - unsigned long sdei_exit_mode;
    -@@ arch/arm64/kernel/sdei.c: unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
    - 	if (arm64_kernel_unmapped_at_el0())
    - 		clobbered_registers++;
    + /*
    +  * Where the context being returned to had an active percpu GPR critical
    +@@ arch/arm64/kernel/entry-common.c: asmlinkage void noinstr __noreturn handle_bad_stack(struct pt_regs *regs)
    + asmlinkage noinstr unsigned long
    + __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
    + {
    ++	unsigned long vbar;
    + 	irqentry_state_t state;
    + 	unsigned long ret;
    + 
    +@@ arch/arm64/kernel/entry-common.c: __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
    + 	else if (cpu_has_pan())
    + 		set_pstate_pan(0);
      
     +	/*
     +	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
     +	 * EL0 until the next exception from EL0 maps the kernel. They only
    -+	 * handle exceptions from EL0: an exception taken by the handler would
    -+	 * run their empty current-EL half and never come back. Run the handler
    -+	 * on the kernel vectors. The IRQ-like resume below keeps using the
    -+	 * interrupted VBAR, so it still goes through the trampoline.
    ++	 * handle exceptions from EL0: an exception taken here would run their
    ++	 * empty current-EL half and never come back. Switch to the kernel
    ++	 * vectors. The IRQ-like resume keeps using the interrupted VBAR, so it
    ++	 * still goes through the trampoline.
     +	 */
    ++	vbar = read_sysreg(vbar_el1);
     +	if (arm64_kernel_unmapped_at_el0() &&
    -+	    vbar == (unsigned long)this_cpu_read(this_cpu_vector)) {
    ++	    vbar == (unsigned long)__this_cpu_read(this_cpu_vector)) {
     +		write_sysreg((unsigned long)vectors, vbar_el1);
     +		isb();
     +	}
     +
    - 	/* Retrieve the missing registers values */
    - 	for (i = 0; i < clobbered_registers; i++) {
    - 		/* from within the handler, this call always succeeds */
    -@@ arch/arm64/kernel/sdei.c: unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
    - 	}
    - 
    - 	err = sdei_event_handler(regs, arg);
    --	if (err)
    --		return SDEI_EV_FAILED;
    - 
    - 	if (elr != read_sysreg(elr_el1)) {
    - 		/*
    -@@ arch/arm64/kernel/sdei.c: unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
    - 		pr_warn("unsafe: exception during handler\n");
    - 	}
    + 	state = irqentry_nmi_enter(regs);
    +-	ret = do_sdei_event(regs, arg);
    ++	ret = do_sdei_event(regs, arg, vbar);
    + 	irqentry_nmi_exit(regs, state);
      
     +	write_sysreg(vbar, vbar_el1);
     +
    -+	if (err)
    -+		return SDEI_EV_FAILED;
    -+
    - 	mode = regs->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK);
    + 	return ret;
    + }
    + #endif /* CONFIG_ARM_SDE_INTERFACE */
    +
    + ## arch/arm64/kernel/sdei.c ##
    +@@ arch/arm64/kernel/sdei.c: unsigned long sdei_arch_get_entry_point(int conduit)
    +  *  SDEI_EV_HANDLED -  success, return to the interrupted context.
    +  *  SDEI_EV_FAILED  -  failure, return this error code to firmware.
    +  *  virtual-address -  success, return to this address.
    ++ *
    ++ * vbar is the interrupted context's VBAR_EL1, for the IRQ-like resume.
    +  */
    + unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
    +-				      struct sdei_registered_event *arg)
    ++				      struct sdei_registered_event *arg,
    ++				      unsigned long vbar)
    + {
    + 	u32 mode;
    + 	int i, err = 0;
    + 	int clobbered_registers = 4;
    + 	u64 elr = read_sysreg(elr_el1);
    + 	u32 kernel_mode = read_sysreg(CurrentEL) | 1;	/* +SPSel */
    +-	unsigned long vbar = read_sysreg(vbar_el1);
      
    - 	/*
    + 	if (arm64_kernel_unmapped_at_el0())
    + 		clobbered_registers++;
2:  ba422c8220e5 ! 2:  07d3078373ee arm64: sdei: Preserve the interrupted context's exception registers
    @@ Commit message
         kworker that is not running a work item, and a single sysrq-l warned
         once per such kworker.
     
    -    Save ELR, SPSR, ESR and FAR on entry to do_sdei_event() and restore them
    -    before returning to firmware.
    +    Save ELR, SPSR, ESR and FAR in __sdei_handler() before
    +    irqentry_nmi_enter(), which can take an exception too, and restore them
    +    after irqentry_nmi_exit().
     
         Once they are restored, a nested exception is harmless if its handler is
         safe in NMI context. That holds for EL1 data aborts on translation,
    @@ Commit message
         Cc: stable@vger.kernel.org
         Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
     
    + ## arch/arm64/kernel/entry-common.c ##
    +@@ arch/arm64/kernel/entry-common.c: asmlinkage void noinstr __noreturn handle_bad_stack(struct pt_regs *regs)
    + asmlinkage noinstr unsigned long
    + __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
    + {
    +-	unsigned long vbar;
    ++	unsigned long elr, spsr, esr, far, vbar;
    + 	irqentry_state_t state;
    + 	unsigned long ret;
    + 
    +@@ arch/arm64/kernel/entry-common.c: __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
    + 	else if (cpu_has_pan())
    + 		set_pstate_pan(0);
    + 
    ++	/*
    ++	 * Nothing saved the interrupted context's exception registers on the
    ++	 * way in, and it may be in the middle of exception entry or return
    ++	 * (SDEI ignores DAIF). Save them before anything can take an exception
    ++	 * and overwrite them.
    ++	 */
    ++	elr = read_sysreg(elr_el1);
    ++	spsr = read_sysreg(spsr_el1);
    ++	esr = read_sysreg(esr_el1);
    ++	far = read_sysreg(far_el1);
    ++
    + 	/*
    + 	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
    + 	 * EL0 until the next exception from EL0 maps the kernel. They only
    +@@ arch/arm64/kernel/entry-common.c: __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
    + 	ret = do_sdei_event(regs, arg, vbar);
    + 	irqentry_nmi_exit(regs, state);
    + 
    ++	/*
    ++	 * Put them back unconditionally. An unchanged ELR does not rule out a
    ++	 * nested exception: one may have returned to the same address.
    ++	 */
    ++	write_sysreg(elr, elr_el1);
    ++	write_sysreg(spsr, spsr_el1);
    ++	write_sysreg(esr, esr_el1);
    ++	write_sysreg(far, far_el1);
    + 	write_sysreg(vbar, vbar_el1);
    + 
    + 	return ret;
    +
      ## arch/arm64/kernel/sdei.c ##
     @@ arch/arm64/kernel/sdei.c: unsigned long sdei_arch_get_entry_point(int conduit)
      	return 0;
      }
      
     +/*
    -+ * Exceptions an SDEI handler may take without harm once do_sdei_event() has
    ++ * Exceptions an SDEI handler may take without harm once __sdei_handler() has
     + * restored the interrupted context's exception registers. Their handlers are
     + * safe in NMI context:
     + *  - EL1 data aborts on translation, permission and access-flag faults:
    @@ arch/arm64/kernel/sdei.c: unsigned long __kprobes do_sdei_event(struct pt_regs *
      	int i, err = 0;
      	int clobbered_registers = 4;
      	u64 elr = read_sysreg(elr_el1);
    -+	u64 spsr = read_sysreg(spsr_el1);
    -+	u64 esr = read_sysreg(esr_el1);
    -+	u64 far = read_sysreg(far_el1);
     +	u64 nested_elr, nested_esr, nested_far;
      	u32 kernel_mode = read_sysreg(CurrentEL) | 1;	/* +SPSel */
    - 	unsigned long vbar = read_sysreg(vbar_el1);
      
    + 	if (arm64_kernel_unmapped_at_el0())
     @@ arch/arm64/kernel/sdei.c: unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
    + 	}
      
      	err = sdei_event_handler(regs, arg);
    - 
    --	if (elr != read_sysreg(elr_el1)) {
    --		/*
    --		 * We took a synchronous exception from the SDEI handler.
    --		 * This could deadlock, and if you interrupt KVM it will
    --		 * hyp-panic instead.
    --		 */
    --		pr_warn("unsafe: exception during handler\n");
    ++
     +	/*
    -+	 * SDEI is not an architectural exception: nothing saved the
    -+	 * interrupted context's exception registers on the way in, and it may
    -+	 * be in the middle of exception entry or return (SDEI ignores DAIF).
    -+	 * Any synchronous exception the handler took has overwritten them.
    -+	 * Only the last one's syndrome survives; report it if it is not one
    -+	 * known to be harmless.
    ++	 * A synchronous exception taken by the handler has overwritten the
    ++	 * exception registers; __sdei_handler() puts them back. Only the last
    ++	 * one's syndrome survives; report it if it is not one known to be
    ++	 * harmless.
     +	 *
     +	 * If the event interrupted a KVM world switch, VBAR points at the KVM
     +	 * hyp vectors and any such exception is a hyp panic instead.
    @@ arch/arm64/kernel/sdei.c: unsigned long __kprobes do_sdei_event(struct pt_regs *
     +	if (nested_elr != elr && !sdei_nested_exception_ok(nested_esr)) {
     +		pr_warn_once("unexpected exception during handler: ESR %#llx FAR %#llx near %pS\n",
     +			     nested_esr, nested_far, (void *)nested_elr);
    - 	}
    - 
    -+	/*
    -+	 * Put them back unconditionally. An unchanged ELR does not rule out a
    -+	 * nested exception: one may have returned to the same address.
    -+	 */
    -+	write_sysreg(elr, elr_el1);
    -+	write_sysreg(spsr, spsr_el1);
    -+	write_sysreg(esr, esr_el1);
    -+	write_sysreg(far, far_el1);
    - 	write_sysreg(vbar, vbar_el1);
    - 
    ++	}
    ++
      	if (err)
    + 		return SDEI_EV_FAILED;
    + 
    +-	if (elr != read_sysreg(elr_el1)) {
    +-		/*
    +-		 * We took a synchronous exception from the SDEI handler.
    +-		 * This could deadlock, and if you interrupt KVM it will
    +-		 * hyp-panic instead.
    +-		 */
    +-		pr_warn("unsafe: exception during handler\n");
    +-	}
    +-
    + 	mode = regs->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK);
    + 
    + 	/*

base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
-- 
2.54.0

[PATCH v2 1/2] arm64: sdei: Run the event handler on the kernel vectors

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

With KPTI, VBAR_EL1 holds the trampoline vectors from the return to EL0
until the next exception from EL0 maps the kernel and installs the
kernel vectors. The trampoline vectors only handle exceptions from EL0;
their current-EL half is zero-filled.

An SDEI event is dispatched regardless of PSTATE.DAIF, so it lands there
whenever the CPU is in user space. The SDEI entry trampoline maps the
kernel but leaves VBAR_EL1 alone. An exception taken by the handler then
executes the zeros, takes an undefined instruction exception to the same
vector, and loops there forever. The event never completes and the CPU
is lost; RCU stall reports are the first sign of it.

RAS notifications land in user space whenever a task consumes the
error, and NMI backtraces land wherever the CPU happens to be. The GHES
handler takes no exception on its own, but any exception in the handler
path does it: a WARN, a kprobe hit, or a fault fixed up by a _nofault()
accessor.

Install the kernel vectors in __sdei_handler() when VBAR_EL1 holds the
trampoline vectors, and restore it before returning to firmware. The
switch covers irqentry_nmi_enter() and irqentry_nmi_exit() too, as Usama
Arif pointed out: their tracepoints and context tracking WARNs can take
an exception as well.

The IRQ-like resume into user space must still go through the
trampoline to map the kernel. do_sdei_event() takes the interrupted
VBAR_EL1 and keeps basing the resume address on it.

Fixes: 79e9aa59dc29 ("arm64: sdei: Add trampoline code for remapping the kernel")
Cc: stable@vger.kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 arch/arm64/include/asm/sdei.h    |  3 ++-
 arch/arm64/kernel/entry-common.c | 21 ++++++++++++++++++++-
 arch/arm64/kernel/sdei.c         |  6 ++++--
 3 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/sdei.h b/arch/arm64/include/asm/sdei.h
index b2248bd3cb58..a78a5337ff85 100644
--- a/arch/arm64/include/asm/sdei.h
+++ b/arch/arm64/include/asm/sdei.h
@@ -44,7 +44,8 @@ asmlinkage unsigned long __sdei_handler(struct pt_regs *regs,
 					struct sdei_registered_event *arg);
 
 unsigned long do_sdei_event(struct pt_regs *regs,
-			    struct sdei_registered_event *arg);
+			    struct sdei_registered_event *arg,
+			    unsigned long vbar);
 
 unsigned long sdei_arch_get_entry_point(int conduit);
 #define sdei_arch_get_entry_point(x)	sdei_arch_get_entry_point(x)
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 11d3f888fcd8..8a17e3720f81 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -31,6 +31,7 @@
 #include <asm/stacktrace.h>
 #include <asm/sysreg.h>
 #include <asm/system_misc.h>
+#include <asm/vectors.h>
 
 /*
  * Where the context being returned to had an active percpu GPR critical
@@ -1040,6 +1041,7 @@ asmlinkage void noinstr __noreturn handle_bad_stack(struct pt_regs *regs)
 asmlinkage noinstr unsigned long
 __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
 {
+	unsigned long vbar;
 	irqentry_state_t state;
 	unsigned long ret;
 
@@ -1065,10 +1067,27 @@ __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
 	else if (cpu_has_pan())
 		set_pstate_pan(0);
 
+	/*
+	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
+	 * EL0 until the next exception from EL0 maps the kernel. They only
+	 * handle exceptions from EL0: an exception taken here would run their
+	 * empty current-EL half and never come back. Switch to the kernel
+	 * vectors. The IRQ-like resume keeps using the interrupted VBAR, so it
+	 * still goes through the trampoline.
+	 */
+	vbar = read_sysreg(vbar_el1);
+	if (arm64_kernel_unmapped_at_el0() &&
+	    vbar == (unsigned long)__this_cpu_read(this_cpu_vector)) {
+		write_sysreg((unsigned long)vectors, vbar_el1);
+		isb();
+	}
+
 	state = irqentry_nmi_enter(regs);
-	ret = do_sdei_event(regs, arg);
+	ret = do_sdei_event(regs, arg, vbar);
 	irqentry_nmi_exit(regs, state);
 
+	write_sysreg(vbar, vbar_el1);
+
 	return ret;
 }
 #endif /* CONFIG_ARM_SDE_INTERFACE */
diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c
index 778f2a1faac8..260c13106009 100644
--- a/arch/arm64/kernel/sdei.c
+++ b/arch/arm64/kernel/sdei.c
@@ -200,16 +200,18 @@ unsigned long sdei_arch_get_entry_point(int conduit)
  *  SDEI_EV_HANDLED -  success, return to the interrupted context.
  *  SDEI_EV_FAILED  -  failure, return this error code to firmware.
  *  virtual-address -  success, return to this address.
+ *
+ * vbar is the interrupted context's VBAR_EL1, for the IRQ-like resume.
  */
 unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
-				      struct sdei_registered_event *arg)
+				      struct sdei_registered_event *arg,
+				      unsigned long vbar)
 {
 	u32 mode;
 	int i, err = 0;
 	int clobbered_registers = 4;
 	u64 elr = read_sysreg(elr_el1);
 	u32 kernel_mode = read_sysreg(CurrentEL) | 1;	/* +SPSel */
-	unsigned long vbar = read_sysreg(vbar_el1);
 
 	if (arm64_kernel_unmapped_at_el0())
 		clobbered_registers++;
-- 
2.54.0

[PATCH v2 1/2] arm64: sdei: Run the event handler on the kernel vectors

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

With KPTI, VBAR_EL1 holds the trampoline vectors from the return to EL0
until the next exception from EL0 maps the kernel and installs the
kernel vectors. The trampoline vectors only handle exceptions from EL0;
their current-EL half is zero-filled.

An SDEI event is dispatched regardless of PSTATE.DAIF, so it lands there
whenever the CPU is in user space. The SDEI entry trampoline maps the
kernel but leaves VBAR_EL1 alone. An exception taken by the handler then
executes the zeros, takes an undefined instruction exception to the same
vector, and loops there forever. The event never completes and the CPU
is lost; RCU stall reports are the first sign of it.

RAS notifications land in user space whenever a task consumes the
error, and NMI backtraces land wherever the CPU happens to be. The GHES
handler takes no exception on its own, but any exception in the handler
path does it: a WARN, a kprobe hit, or a fault fixed up by a _nofault()
accessor.

Install the kernel vectors in __sdei_handler() when VBAR_EL1 holds the
trampoline vectors, and restore it before returning to firmware. The
switch covers irqentry_nmi_enter() and irqentry_nmi_exit() too, as Usama
Arif pointed out: their tracepoints and context tracking WARNs can take
an exception as well.

The IRQ-like resume into user space must still go through the
trampoline to map the kernel. do_sdei_event() takes the interrupted
VBAR_EL1 and keeps basing the resume address on it.

Fixes: 79e9aa59dc29 ("arm64: sdei: Add trampoline code for remapping the kernel")
Cc: stable@vger.kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 arch/arm64/include/asm/sdei.h    |  3 ++-
 arch/arm64/kernel/entry-common.c | 21 ++++++++++++++++++++-
 arch/arm64/kernel/sdei.c         |  6 ++++--
 3 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/sdei.h b/arch/arm64/include/asm/sdei.h
index b2248bd3cb58..a78a5337ff85 100644
--- a/arch/arm64/include/asm/sdei.h
+++ b/arch/arm64/include/asm/sdei.h
@@ -44,7 +44,8 @@ asmlinkage unsigned long __sdei_handler(struct pt_regs *regs,
 					struct sdei_registered_event *arg);
 
 unsigned long do_sdei_event(struct pt_regs *regs,
-			    struct sdei_registered_event *arg);
+			    struct sdei_registered_event *arg,
+			    unsigned long vbar);
 
 unsigned long sdei_arch_get_entry_point(int conduit);
 #define sdei_arch_get_entry_point(x)	sdei_arch_get_entry_point(x)
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 11d3f888fcd8..8a17e3720f81 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -31,6 +31,7 @@
 #include <asm/stacktrace.h>
 #include <asm/sysreg.h>
 #include <asm/system_misc.h>
+#include <asm/vectors.h>
 
 /*
  * Where the context being returned to had an active percpu GPR critical
@@ -1040,6 +1041,7 @@ asmlinkage void noinstr __noreturn handle_bad_stack(struct pt_regs *regs)
 asmlinkage noinstr unsigned long
 __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
 {
+	unsigned long vbar;
 	irqentry_state_t state;
 	unsigned long ret;
 
@@ -1065,10 +1067,27 @@ __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
 	else if (cpu_has_pan())
 		set_pstate_pan(0);
 
+	/*
+	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
+	 * EL0 until the next exception from EL0 maps the kernel. They only
+	 * handle exceptions from EL0: an exception taken here would run their
+	 * empty current-EL half and never come back. Switch to the kernel
+	 * vectors. The IRQ-like resume keeps using the interrupted VBAR, so it
+	 * still goes through the trampoline.
+	 */
+	vbar = read_sysreg(vbar_el1);
+	if (arm64_kernel_unmapped_at_el0() &&
+	    vbar == (unsigned long)__this_cpu_read(this_cpu_vector)) {
+		write_sysreg((unsigned long)vectors, vbar_el1);
+		isb();
+	}
+
 	state = irqentry_nmi_enter(regs);
-	ret = do_sdei_event(regs, arg);
+	ret = do_sdei_event(regs, arg, vbar);
 	irqentry_nmi_exit(regs, state);
 
+	write_sysreg(vbar, vbar_el1);
+
 	return ret;
 }
 #endif /* CONFIG_ARM_SDE_INTERFACE */
diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c
index 778f2a1faac8..260c13106009 100644
--- a/arch/arm64/kernel/sdei.c
+++ b/arch/arm64/kernel/sdei.c
@@ -200,16 +200,18 @@ unsigned long sdei_arch_get_entry_point(int conduit)
  *  SDEI_EV_HANDLED -  success, return to the interrupted context.
  *  SDEI_EV_FAILED  -  failure, return this error code to firmware.
  *  virtual-address -  success, return to this address.
+ *
+ * vbar is the interrupted context's VBAR_EL1, for the IRQ-like resume.
  */
 unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
-				      struct sdei_registered_event *arg)
+				      struct sdei_registered_event *arg,
+				      unsigned long vbar)
 {
 	u32 mode;
 	int i, err = 0;
 	int clobbered_registers = 4;
 	u64 elr = read_sysreg(elr_el1);
 	u32 kernel_mode = read_sysreg(CurrentEL) | 1;	/* +SPSel */
-	unsigned long vbar = read_sysreg(vbar_el1);
 
 	if (arm64_kernel_unmapped_at_el0())
 		clobbered_registers++;
-- 
2.54.0

Re: [PATCH v2 1/2] arm64: sdei: Run the event handler on the kernel vectors

[Catalin Marinas]

On Thu, Nov 05, 2026 at 12:08:33PM +0000, Kiryl Shutsemau wrote:
> @@ -1065,10 +1067,27 @@ __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
>  	else if (cpu_has_pan())
>  		set_pstate_pan(0);
>  
> +	/*
> +	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
> +	 * EL0 until the next exception from EL0 maps the kernel. They only
> +	 * handle exceptions from EL0: an exception taken here would run their
> +	 * empty current-EL half and never come back. Switch to the kernel
> +	 * vectors. The IRQ-like resume keeps using the interrupted VBAR, so it
> +	 * still goes through the trampoline.
> +	 */
> +	vbar = read_sysreg(vbar_el1);
> +	if (arm64_kernel_unmapped_at_el0() &&
> +	    vbar == (unsigned long)__this_cpu_read(this_cpu_vector)) {
> +		write_sysreg((unsigned long)vectors, vbar_el1);
> +		isb();
> +	}

Sashiko had a comment that the above should be raw_cpu_read() since with
preempt debugging and pseudo-NMIs, irqs_disabled() is false here and the
check in __this_cpu_read() will warn.

Thinking about it, further down on the irqentry_nmi_enter() ->
lockdep_hardirqs_off() we'll hit the irqs_disabled() issue again. We
could just do a raw_local_irq_save/restore() around this function.
Unnecessary for DAIF but useful with pseudo-NMIs.

Alternatively, __sdei_asm_handler() could save/restore PMR and change
the priorities similar to the kernel_entry code. Actually, this may be
better since do_sdei_event() also checks regs_irqs_disabled() and we
don't have any regs->pmr populated.

Worth a separate patch to fix SDEI handling with pseudo-NMI on and no
changes to __this_cpu_read() above needed. So, for this patch:

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

(I'll probably queue it for -rc but waiting to see if there are more
comments)

-- 
Catalin

Re: [PATCH v2 1/2] arm64: sdei: Run the event handler on the kernel vectors

[Catalin Marinas]

On Thu, Nov 05, 2026 at 12:08:33PM +0000, Kiryl Shutsemau wrote:
> @@ -1065,10 +1067,27 @@ __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
>  	else if (cpu_has_pan())
>  		set_pstate_pan(0);
>  
> +	/*
> +	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
> +	 * EL0 until the next exception from EL0 maps the kernel. They only
> +	 * handle exceptions from EL0: an exception taken here would run their
> +	 * empty current-EL half and never come back. Switch to the kernel
> +	 * vectors. The IRQ-like resume keeps using the interrupted VBAR, so it
> +	 * still goes through the trampoline.
> +	 */
> +	vbar = read_sysreg(vbar_el1);
> +	if (arm64_kernel_unmapped_at_el0() &&
> +	    vbar == (unsigned long)__this_cpu_read(this_cpu_vector)) {
> +		write_sysreg((unsigned long)vectors, vbar_el1);
> +		isb();
> +	}

Sashiko had a comment that the above should be raw_cpu_read() since with
preempt debugging and pseudo-NMIs, irqs_disabled() is false here and the
check in __this_cpu_read() will warn.

Thinking about it, further down on the irqentry_nmi_enter() ->
lockdep_hardirqs_off() we'll hit the irqs_disabled() issue again. We
could just do a raw_local_irq_save/restore() around this function.
Unnecessary for DAIF but useful with pseudo-NMIs.

Alternatively, __sdei_asm_handler() could save/restore PMR and change
the priorities similar to the kernel_entry code. Actually, this may be
better since do_sdei_event() also checks regs_irqs_disabled() and we
don't have any regs->pmr populated.

Worth a separate patch to fix SDEI handling with pseudo-NMI on and no
changes to __this_cpu_read() above needed. So, for this patch:

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

(I'll probably queue it for -rc but waiting to see if there are more
comments)

-- 
Catalin

Re: [PATCH v2 1/2] arm64: sdei: Run the event handler on the kernel vectors

[Kiryl Shutsemau]

On Thu, Nov 05, 2026 at 06:37:46PM +0000, Catalin Marinas wrote:
> Worth a separate patch to fix SDEI handling with pseudo-NMI on and no
> changes to __this_cpu_read() above needed. So, for this patch:
> 
> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

Thanks!

Would it still make sense to switch to raw_cpu_read() in this patch for
stable@? The splat is new with this patch, so without it stable trees
would also need the PMR fix.

Let me know if you want a re-spin with raw_cpu_read().

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH v2 1/2] arm64: sdei: Run the event handler on the kernel vectors

[Kiryl Shutsemau]

On Thu, Nov 05, 2026 at 06:37:46PM +0000, Catalin Marinas wrote:
> Worth a separate patch to fix SDEI handling with pseudo-NMI on and no
> changes to __this_cpu_read() above needed. So, for this patch:
> 
> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

Thanks!

Would it still make sense to switch to raw_cpu_read() in this patch for
stable@? The splat is new with this patch, so without it stable trees
would also need the PMR fix.

Let me know if you want a re-spin with raw_cpu_read().

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH v2 1/2] arm64: sdei: Run the event handler on the kernel vectors

[Catalin Marinas]

On Fri, Nov 06, 2026 at 11:48:00AM +0000, Kiryl Shutsemau wrote:
> On Thu, Nov 05, 2026 at 06:37:46PM +0000, Catalin Marinas wrote:
> > Worth a separate patch to fix SDEI handling with pseudo-NMI on and no
> > changes to __this_cpu_read() above needed. So, for this patch:
> > 
> > Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
> 
> Thanks!
> 
> Would it still make sense to switch to raw_cpu_read() in this patch for
> stable@? The splat is new with this patch, so without it stable trees
> would also need the PMR fix.
> 
> Let me know if you want a re-spin with raw_cpu_read().

No need to respin, I can fix it locally.

-- 
Catalin

Re: [PATCH v2 1/2] arm64: sdei: Run the event handler on the kernel vectors

[Ada Couprie Diaz]

Hello,

On 05/11/2026 12:08, Kiryl Shutsemau wrote:

> From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
>
> With KPTI, VBAR_EL1 holds the trampoline vectors from the return to EL0
> until the next exception from EL0 maps the kernel and installs the
> kernel vectors. The trampoline vectors only handle exceptions from EL0;
> their current-EL half is zero-filled.
>
> An SDEI event is dispatched regardless of PSTATE.DAIF, so it lands there
> whenever the CPU is in user space. The SDEI entry trampoline maps the
> kernel but leaves VBAR_EL1 alone. [...]

This quite broken indeed, thank you for sending a patch !

However, given this and Catalin's comment regarding PMR save/restore,
to me this sounds more like `arch/arm64/entry.S` is not doing its job
properly for SDEI events, rather than the SDEI handler missing something.

In fact, it appears that the SDEI trampoline misses a fair amount of
the "regular" kernel and trampoline entry in general, which might subtly
break more assumptions...

> [...]
>
> Install the kernel vectors in __sdei_handler() when VBAR_EL1 holds the
> trampoline vectors, and restore it before returning to firmware. [...]

Given that this is done in `entry.S` for a regular kernel entry,
I feel like this should be done there too for SDEI : bringing the two
paths closer rather than needing to do some additional special handling
in the handler.

`__sdei_asm_entry_trampoline` and `__sdei_asm_exit_trampoline` already
check if the CPU was running at EL0 and maps/unmaps the kernel accordingly.
We could swap VBAR to the EL1 vectors and back to the EL0 trampoline there...

> The IRQ-like resume into user space must still go through the
> trampoline to map the kernel. do_sdei_event() takes the interrupted
> VBAR_EL1 and keeps basing the resume address on it.

...and to maintain the expected behaviour on return, rather than using
VBAR (which would bypass the trampoline in the EL0 IRQ-like resume case),
we could explicitly choose the correct vector base for KPTI=on ?
Something along those lines in `do_sdei_event()` :

	/* KPTI=off => vectors or the appropriate mitigation, KPTI=on => trampoline */
	unsigned long vbar = __raw_this_cpu_read(this_cpu_vector)
	[...]
	/* KPTI=on => have to bypass unmapped EL1 exceptions of the trampoline vector */
	if (arm64_kernel_unmapped_at_el0() && mode == kernel_mode)
		vbar = vectors
	[...]
	if (mode == kernel_mode)
		return vbar + 0x280;
	else if (mode & PSR_MODE32_BIT)
		return vbar + 0x680;

	return vbar + 0x480;

> Fixes: 79e9aa59dc29 ("arm64: sdei: Add trampoline code for remapping the kernel")
> Cc: stable@vger.kernel.org
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> ---
>   arch/arm64/include/asm/sdei.h    |  3 ++-
>   arch/arm64/kernel/entry-common.c | 21 ++++++++++++++++++++-
>   arch/arm64/kernel/sdei.c         |  6 ++++--
>   3 files changed, 26 insertions(+), 4 deletions(-)
>
> diff --git a/arch/arm64/include/asm/sdei.h b/arch/arm64/include/asm/sdei.h
> index b2248bd3cb58..a78a5337ff85 100644
> --- a/arch/arm64/include/asm/sdei.h
> +++ b/arch/arm64/include/asm/sdei.h
> @@ -44,7 +44,8 @@ asmlinkage unsigned long __sdei_handler(struct pt_regs *regs,
>   					struct sdei_registered_event *arg);
>   
>   unsigned long do_sdei_event(struct pt_regs *regs,
> -			    struct sdei_registered_event *arg);
> +			    struct sdei_registered_event *arg,
> +			    unsigned long vbar);
>   
>   unsigned long sdei_arch_get_entry_point(int conduit);
>   #define sdei_arch_get_entry_point(x)	sdei_arch_get_entry_point(x)
> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> index 11d3f888fcd8..8a17e3720f81 100644
> --- a/arch/arm64/kernel/entry-common.c
> +++ b/arch/arm64/kernel/entry-common.c
> @@ -31,6 +31,7 @@
>   #include <asm/stacktrace.h>
>   #include <asm/sysreg.h>
>   #include <asm/system_misc.h>
> +#include <asm/vectors.h>
>   
>   /*
>    * Where the context being returned to had an active percpu GPR critical
> @@ -1040,6 +1041,7 @@ asmlinkage void noinstr __noreturn handle_bad_stack(struct pt_regs *regs)
>   asmlinkage noinstr unsigned long
>   __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
>   {
> +	unsigned long vbar;
>   	irqentry_state_t state;
>   	unsigned long ret;
>   
> @@ -1065,10 +1067,27 @@ __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
>   	else if (cpu_has_pan())
>   		set_pstate_pan(0);
>   
> +	/*
> +	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
> +	 * EL0 until the next exception from EL0 maps the kernel. They only
> +	 * handle exceptions from EL0: an exception taken here would run their
> +	 * empty current-EL half and never come back. Switch to the kernel
> +	 * vectors. The IRQ-like resume keeps using the interrupted VBAR, so it
> +	 * still goes through the trampoline.
> +	 */
> +	vbar = read_sysreg(vbar_el1);
> +	if (arm64_kernel_unmapped_at_el0() &&
> +	    vbar == (unsigned long)__this_cpu_read(this_cpu_vector)) {
> +		write_sysreg((unsigned long)vectors, vbar_el1);
> +		isb();
> +	}
> +
>   	state = irqentry_nmi_enter(regs);
> -	ret = do_sdei_event(regs, arg);
> +	ret = do_sdei_event(regs, arg, vbar);
>   	irqentry_nmi_exit(regs, state);
>   
> +	write_sysreg(vbar, vbar_el1);
> +
>   	return ret;
>   }

With what I mentioned above, this would be completely replaced by the
changes in `entry.S`, with no change in `__sdei_handler()`.

>   #endif /* CONFIG_ARM_SDE_INTERFACE */
> diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c
> index 778f2a1faac8..260c13106009 100644
> --- a/arch/arm64/kernel/sdei.c
> +++ b/arch/arm64/kernel/sdei.c
> @@ -200,16 +200,18 @@ unsigned long sdei_arch_get_entry_point(int conduit)
>    *  SDEI_EV_HANDLED -  success, return to the interrupted context.
>    *  SDEI_EV_FAILED  -  failure, return this error code to firmware.
>    *  virtual-address -  success, return to this address.
> + *
> + * vbar is the interrupted context's VBAR_EL1, for the IRQ-like resume.
>    */
>   unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
> -				      struct sdei_registered_event *arg)
> +				      struct sdei_registered_event *arg,
> +				      unsigned long vbar)
>   {
>   	u32 mode;
>   	int i, err = 0;
>   	int clobbered_registers = 4;
>   	u64 elr = read_sysreg(elr_el1);
>   	u32 kernel_mode = read_sysreg(CurrentEL) | 1;	/* +SPSel */
> -	unsigned long vbar = read_sysreg(vbar_el1);
>   
>   	if (arm64_kernel_unmapped_at_el0())
>   		clobbered_registers++;

And this part would be made more explicit as to what vector base
we should be jumping back to on return from firmware.


What are your thoughts ?
I understand that this is a completely different approach, but I feel it
would be more in line with what we already do for other entry paths.
This might even allow us to de-duplicate them in the future and reduce
the chance of this happening again.

If my suggestion is not sound, apart from what Catalin already mentioned
the patch looks good to me !

Kind regards,
Ada

Re: [PATCH v2 1/2] arm64: sdei: Run the event handler on the kernel vectors

[Catalin Marinas]

Hi Ada,

On Fri, Nov 06, 2026 at 04:14:56PM +0000, Ada Couprie Diaz wrote:
> What are your thoughts ?
> I understand that this is a completely different approach, but I feel it
> would be more in line with what we already do for other entry paths.

I agree moving this closer to the normal entry path makes sense and the
PMR handling probably belongs there as well. However, I think there are
implications for VHE (the handler runs at EL2 with VBAR_EL2) and for an
event that interrupts a KVM world switch on the hyp vectors. I need to
dig some more into this.

In the meantime, I queued Kiryl's two patches, they're small enough for
stable. Any entry.S rework can come on top of them for future kernels.

Thanks.

-- 
Catalin

[PATCH v2 2/2] arm64: sdei: Preserve the interrupted context's exception registers

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

An SDEI event is dispatched by firmware regardless of PSTATE.DAIF, so it
can interrupt the kernel where the exception registers are live:

 - exception entry, before kernel_entry has saved ELR_EL1/SPSR_EL1 and
   before the C handler has read ESR_EL1/FAR_EL1;
 - exception return, after kernel_exit has loaded ELR_EL1/SPSR_EL1 and
   before the ERET.

An IRQ or pseudo-NMI cannot arrive there with PSTATE.I set. Firmware
saves only x0-x17, PC and PSTATE around an SDEI event, so a synchronous
exception taken by the handler overwrites the interrupted context's
ELR, SPSR, ESR and FAR. do_sdei_event() only notices that ELR changed,
and warns.

The interrupted context then carries on with the handler's state. A
task interrupted on its way into the kernel can be sent SIGILL for the
handler's kernel data abort. A task interrupted on its way back to user
space ERETs to EL1, at the handler's fixup address, with user registers
loaded.

With SDEI delivering NMI backtraces, this happens in normal operation.
The handler runs show_regs(), which reads through possibly-NULL pointers
with the _nofault() accessors. print_worker_info() faults this way for a
kworker that is not running a work item, and a single sysrq-l warned
once per such kworker.

Save ELR, SPSR, ESR and FAR in __sdei_handler() before
irqentry_nmi_enter(), which can take an exception too, and restore them
after irqentry_nmi_exit().

Once they are restored, a nested exception is harmless if its handler is
safe in NMI context. That holds for EL1 data aborts on translation,
permission and access-flag faults, which never take the mmap lock in NMI
context, and for BRK. Warn once, with the syndrome, about any other
exception class.

Fixes: f5df26961853 ("arm64: kernel: Add arch-specific SDEI entry code and CPU masking")
Cc: stable@vger.kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 arch/arm64/kernel/entry-common.c | 21 ++++++++++++-
 arch/arm64/kernel/sdei.c         | 53 ++++++++++++++++++++++++++------
 2 files changed, 64 insertions(+), 10 deletions(-)

diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 8a17e3720f81..64d287e1533b 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -1041,7 +1041,7 @@ asmlinkage void noinstr __noreturn handle_bad_stack(struct pt_regs *regs)
 asmlinkage noinstr unsigned long
 __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
 {
-	unsigned long vbar;
+	unsigned long elr, spsr, esr, far, vbar;
 	irqentry_state_t state;
 	unsigned long ret;
 
@@ -1067,6 +1067,17 @@ __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
 	else if (cpu_has_pan())
 		set_pstate_pan(0);
 
+	/*
+	 * Nothing saved the interrupted context's exception registers on the
+	 * way in, and it may be in the middle of exception entry or return
+	 * (SDEI ignores DAIF). Save them before anything can take an exception
+	 * and overwrite them.
+	 */
+	elr = read_sysreg(elr_el1);
+	spsr = read_sysreg(spsr_el1);
+	esr = read_sysreg(esr_el1);
+	far = read_sysreg(far_el1);
+
 	/*
 	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
 	 * EL0 until the next exception from EL0 maps the kernel. They only
@@ -1086,6 +1097,14 @@ __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
 	ret = do_sdei_event(regs, arg, vbar);
 	irqentry_nmi_exit(regs, state);
 
+	/*
+	 * Put them back unconditionally. An unchanged ELR does not rule out a
+	 * nested exception: one may have returned to the same address.
+	 */
+	write_sysreg(elr, elr_el1);
+	write_sysreg(spsr, spsr_el1);
+	write_sysreg(esr, esr_el1);
+	write_sysreg(far, far_el1);
 	write_sysreg(vbar, vbar_el1);
 
 	return ret;
diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c
index 260c13106009..1b902da8ba43 100644
--- a/arch/arm64/kernel/sdei.c
+++ b/arch/arm64/kernel/sdei.c
@@ -195,6 +195,31 @@ unsigned long sdei_arch_get_entry_point(int conduit)
 	return 0;
 }
 
+/*
+ * Exceptions an SDEI handler may take without harm once __sdei_handler() has
+ * restored the interrupted context's exception registers. Their handlers are
+ * safe in NMI context:
+ *  - EL1 data aborts on translation, permission and access-flag faults:
+ *    faulthandler_disabled() is true in NMI context, so do_page_fault() never
+ *    takes the mmap lock. This covers the _nofault() accessors used by the
+ *    generic dump_stack()/show_regs() code (e.g. print_worker_info()).
+ *  - BRK: kprobes, WARN/BUG, kCFI and UBSAN traps, all NMI-capable.
+ * Anything else taken at EL1 is fatal on its own, or unexpected here.
+ */
+static bool sdei_nested_exception_ok(unsigned long esr)
+{
+	switch (ESR_ELx_EC(esr)) {
+	case ESR_ELx_EC_DABT_CUR:
+		return esr_fsc_is_translation_fault(esr) ||
+		       esr_fsc_is_permission_fault(esr) ||
+		       esr_fsc_is_access_flag_fault(esr);
+	case ESR_ELx_EC_BRK64:
+		return true;
+	default:
+		return false;
+	}
+}
+
 /*
  * do_sdei_event() returns one of:
  *  SDEI_EV_HANDLED -  success, return to the interrupted context.
@@ -211,6 +236,7 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 	int i, err = 0;
 	int clobbered_registers = 4;
 	u64 elr = read_sysreg(elr_el1);
+	u64 nested_elr, nested_esr, nested_far;
 	u32 kernel_mode = read_sysreg(CurrentEL) | 1;	/* +SPSel */
 
 	if (arm64_kernel_unmapped_at_el0())
@@ -223,18 +249,27 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 	}
 
 	err = sdei_event_handler(regs, arg);
+
+	/*
+	 * A synchronous exception taken by the handler has overwritten the
+	 * exception registers; __sdei_handler() puts them back. Only the last
+	 * one's syndrome survives; report it if it is not one known to be
+	 * harmless.
+	 *
+	 * If the event interrupted a KVM world switch, VBAR points at the KVM
+	 * hyp vectors and any such exception is a hyp panic instead.
+	 */
+	nested_elr = read_sysreg(elr_el1);
+	nested_esr = read_sysreg(esr_el1);
+	nested_far = read_sysreg(far_el1);
+	if (nested_elr != elr && !sdei_nested_exception_ok(nested_esr)) {
+		pr_warn_once("unexpected exception during handler: ESR %#llx FAR %#llx near %pS\n",
+			     nested_esr, nested_far, (void *)nested_elr);
+	}
+
 	if (err)
 		return SDEI_EV_FAILED;
 
-	if (elr != read_sysreg(elr_el1)) {
-		/*
-		 * We took a synchronous exception from the SDEI handler.
-		 * This could deadlock, and if you interrupt KVM it will
-		 * hyp-panic instead.
-		 */
-		pr_warn("unsafe: exception during handler\n");
-	}
-
 	mode = regs->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK);
 
 	/*
-- 
2.54.0

[PATCH v2 2/2] arm64: sdei: Preserve the interrupted context's exception registers

[Kiryl Shutsemau]

From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>

An SDEI event is dispatched by firmware regardless of PSTATE.DAIF, so it
can interrupt the kernel where the exception registers are live:

 - exception entry, before kernel_entry has saved ELR_EL1/SPSR_EL1 and
   before the C handler has read ESR_EL1/FAR_EL1;
 - exception return, after kernel_exit has loaded ELR_EL1/SPSR_EL1 and
   before the ERET.

An IRQ or pseudo-NMI cannot arrive there with PSTATE.I set. Firmware
saves only x0-x17, PC and PSTATE around an SDEI event, so a synchronous
exception taken by the handler overwrites the interrupted context's
ELR, SPSR, ESR and FAR. do_sdei_event() only notices that ELR changed,
and warns.

The interrupted context then carries on with the handler's state. A
task interrupted on its way into the kernel can be sent SIGILL for the
handler's kernel data abort. A task interrupted on its way back to user
space ERETs to EL1, at the handler's fixup address, with user registers
loaded.

With SDEI delivering NMI backtraces, this happens in normal operation.
The handler runs show_regs(), which reads through possibly-NULL pointers
with the _nofault() accessors. print_worker_info() faults this way for a
kworker that is not running a work item, and a single sysrq-l warned
once per such kworker.

Save ELR, SPSR, ESR and FAR in __sdei_handler() before
irqentry_nmi_enter(), which can take an exception too, and restore them
after irqentry_nmi_exit().

Once they are restored, a nested exception is harmless if its handler is
safe in NMI context. That holds for EL1 data aborts on translation,
permission and access-flag faults, which never take the mmap lock in NMI
context, and for BRK. Warn once, with the syndrome, about any other
exception class.

Fixes: f5df26961853 ("arm64: kernel: Add arch-specific SDEI entry code and CPU masking")
Cc: stable@vger.kernel.org
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
---
 arch/arm64/kernel/entry-common.c | 21 ++++++++++++-
 arch/arm64/kernel/sdei.c         | 53 ++++++++++++++++++++++++++------
 2 files changed, 64 insertions(+), 10 deletions(-)

diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 8a17e3720f81..64d287e1533b 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -1041,7 +1041,7 @@ asmlinkage void noinstr __noreturn handle_bad_stack(struct pt_regs *regs)
 asmlinkage noinstr unsigned long
 __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
 {
-	unsigned long vbar;
+	unsigned long elr, spsr, esr, far, vbar;
 	irqentry_state_t state;
 	unsigned long ret;
 
@@ -1067,6 +1067,17 @@ __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
 	else if (cpu_has_pan())
 		set_pstate_pan(0);
 
+	/*
+	 * Nothing saved the interrupted context's exception registers on the
+	 * way in, and it may be in the middle of exception entry or return
+	 * (SDEI ignores DAIF). Save them before anything can take an exception
+	 * and overwrite them.
+	 */
+	elr = read_sysreg(elr_el1);
+	spsr = read_sysreg(spsr_el1);
+	esr = read_sysreg(esr_el1);
+	far = read_sysreg(far_el1);
+
 	/*
 	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
 	 * EL0 until the next exception from EL0 maps the kernel. They only
@@ -1086,6 +1097,14 @@ __sdei_handler(struct pt_regs *regs, struct sdei_registered_event *arg)
 	ret = do_sdei_event(regs, arg, vbar);
 	irqentry_nmi_exit(regs, state);
 
+	/*
+	 * Put them back unconditionally. An unchanged ELR does not rule out a
+	 * nested exception: one may have returned to the same address.
+	 */
+	write_sysreg(elr, elr_el1);
+	write_sysreg(spsr, spsr_el1);
+	write_sysreg(esr, esr_el1);
+	write_sysreg(far, far_el1);
 	write_sysreg(vbar, vbar_el1);
 
 	return ret;
diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c
index 260c13106009..1b902da8ba43 100644
--- a/arch/arm64/kernel/sdei.c
+++ b/arch/arm64/kernel/sdei.c
@@ -195,6 +195,31 @@ unsigned long sdei_arch_get_entry_point(int conduit)
 	return 0;
 }
 
+/*
+ * Exceptions an SDEI handler may take without harm once __sdei_handler() has
+ * restored the interrupted context's exception registers. Their handlers are
+ * safe in NMI context:
+ *  - EL1 data aborts on translation, permission and access-flag faults:
+ *    faulthandler_disabled() is true in NMI context, so do_page_fault() never
+ *    takes the mmap lock. This covers the _nofault() accessors used by the
+ *    generic dump_stack()/show_regs() code (e.g. print_worker_info()).
+ *  - BRK: kprobes, WARN/BUG, kCFI and UBSAN traps, all NMI-capable.
+ * Anything else taken at EL1 is fatal on its own, or unexpected here.
+ */
+static bool sdei_nested_exception_ok(unsigned long esr)
+{
+	switch (ESR_ELx_EC(esr)) {
+	case ESR_ELx_EC_DABT_CUR:
+		return esr_fsc_is_translation_fault(esr) ||
+		       esr_fsc_is_permission_fault(esr) ||
+		       esr_fsc_is_access_flag_fault(esr);
+	case ESR_ELx_EC_BRK64:
+		return true;
+	default:
+		return false;
+	}
+}
+
 /*
  * do_sdei_event() returns one of:
  *  SDEI_EV_HANDLED -  success, return to the interrupted context.
@@ -211,6 +236,7 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 	int i, err = 0;
 	int clobbered_registers = 4;
 	u64 elr = read_sysreg(elr_el1);
+	u64 nested_elr, nested_esr, nested_far;
 	u32 kernel_mode = read_sysreg(CurrentEL) | 1;	/* +SPSel */
 
 	if (arm64_kernel_unmapped_at_el0())
@@ -223,18 +249,27 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 	}
 
 	err = sdei_event_handler(regs, arg);
+
+	/*
+	 * A synchronous exception taken by the handler has overwritten the
+	 * exception registers; __sdei_handler() puts them back. Only the last
+	 * one's syndrome survives; report it if it is not one known to be
+	 * harmless.
+	 *
+	 * If the event interrupted a KVM world switch, VBAR points at the KVM
+	 * hyp vectors and any such exception is a hyp panic instead.
+	 */
+	nested_elr = read_sysreg(elr_el1);
+	nested_esr = read_sysreg(esr_el1);
+	nested_far = read_sysreg(far_el1);
+	if (nested_elr != elr && !sdei_nested_exception_ok(nested_esr)) {
+		pr_warn_once("unexpected exception during handler: ESR %#llx FAR %#llx near %pS\n",
+			     nested_esr, nested_far, (void *)nested_elr);
+	}
+
 	if (err)
 		return SDEI_EV_FAILED;
 
-	if (elr != read_sysreg(elr_el1)) {
-		/*
-		 * We took a synchronous exception from the SDEI handler.
-		 * This could deadlock, and if you interrupt KVM it will
-		 * hyp-panic instead.
-		 */
-		pr_warn("unsafe: exception during handler\n");
-	}
-
 	mode = regs->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK);
 
 	/*
-- 
2.54.0

Re: [PATCH v2 2/2] arm64: sdei: Preserve the interrupted context's exception registers

[Breno Leitao]

On Thu, Nov 05, 2026 at 12:08:34PM +0000, Kiryl Shutsemau wrote:
> From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
> 
> An SDEI event is dispatched by firmware regardless of PSTATE.DAIF, so it
> can interrupt the kernel where the exception registers are live:
> 
>  - exception entry, before kernel_entry has saved ELR_EL1/SPSR_EL1 and
>    before the C handler has read ESR_EL1/FAR_EL1;
>  - exception return, after kernel_exit has loaded ELR_EL1/SPSR_EL1 and
>    before the ERET.
> 
> An IRQ or pseudo-NMI cannot arrive there with PSTATE.I set. Firmware
> saves only x0-x17, PC and PSTATE around an SDEI event, so a synchronous
> exception taken by the handler overwrites the interrupted context's
> ELR, SPSR, ESR and FAR. do_sdei_event() only notices that ELR changed,
> and warns.
> 
> The interrupted context then carries on with the handler's state. A
> task interrupted on its way into the kernel can be sent SIGILL for the
> handler's kernel data abort. A task interrupted on its way back to user
> space ERETs to EL1, at the handler's fixup address, with user registers
> loaded.
> 
> With SDEI delivering NMI backtraces, this happens in normal operation.
> The handler runs show_regs(), which reads through possibly-NULL pointers
> with the _nofault() accessors. print_worker_info() faults this way for a
> kworker that is not running a work item, and a single sysrq-l warned
> once per such kworker.
> 
> Save ELR, SPSR, ESR and FAR in __sdei_handler() before
> irqentry_nmi_enter(), which can take an exception too, and restore them
> after irqentry_nmi_exit().
> 
> Once they are restored, a nested exception is harmless if its handler is
> safe in NMI context. That holds for EL1 data aborts on translation,
> permission and access-flag faults, which never take the mmap lock in NMI
> context, and for BRK. Warn once, with the syndrome, about any other
> exception class.
> 
> Fixes: f5df26961853 ("arm64: kernel: Add arch-specific SDEI entry code and CPU masking")
> Cc: stable@vger.kernel.org
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>

Tested-by: Breno Leitao <leitao@debian.org>

Tested on NVIDIA Grace (firmware SDEI) with next-20261104, KASAN + lockdep,
ARM_SDEI_NMI=y: sysrq-l under stress-ng, with a kprobe on show_regs() and
eprobes on irq_disable/irq_enable that fault on purpose.

Without the series the box crashed (a task interrupted in el0t_64_sync
got the handler's BRK, then lockdep/KASAN fallout and a hard lockup)

With it, KPTI off and kpti=1, ~4000 kprobe hits and ~5500 faults in
irqentry_nmi_enter()/exit() produced no warnings or stalls.

Re: [PATCH v2 2/2] arm64: sdei: Preserve the interrupted context's exception registers

[Breno Leitao]

On Thu, Nov 05, 2026 at 12:08:34PM +0000, Kiryl Shutsemau wrote:
> From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
> 
> An SDEI event is dispatched by firmware regardless of PSTATE.DAIF, so it
> can interrupt the kernel where the exception registers are live:
> 
>  - exception entry, before kernel_entry has saved ELR_EL1/SPSR_EL1 and
>    before the C handler has read ESR_EL1/FAR_EL1;
>  - exception return, after kernel_exit has loaded ELR_EL1/SPSR_EL1 and
>    before the ERET.
> 
> An IRQ or pseudo-NMI cannot arrive there with PSTATE.I set. Firmware
> saves only x0-x17, PC and PSTATE around an SDEI event, so a synchronous
> exception taken by the handler overwrites the interrupted context's
> ELR, SPSR, ESR and FAR. do_sdei_event() only notices that ELR changed,
> and warns.
> 
> The interrupted context then carries on with the handler's state. A
> task interrupted on its way into the kernel can be sent SIGILL for the
> handler's kernel data abort. A task interrupted on its way back to user
> space ERETs to EL1, at the handler's fixup address, with user registers
> loaded.
> 
> With SDEI delivering NMI backtraces, this happens in normal operation.
> The handler runs show_regs(), which reads through possibly-NULL pointers
> with the _nofault() accessors. print_worker_info() faults this way for a
> kworker that is not running a work item, and a single sysrq-l warned
> once per such kworker.
> 
> Save ELR, SPSR, ESR and FAR in __sdei_handler() before
> irqentry_nmi_enter(), which can take an exception too, and restore them
> after irqentry_nmi_exit().
> 
> Once they are restored, a nested exception is harmless if its handler is
> safe in NMI context. That holds for EL1 data aborts on translation,
> permission and access-flag faults, which never take the mmap lock in NMI
> context, and for BRK. Warn once, with the syndrome, about any other
> exception class.
> 
> Fixes: f5df26961853 ("arm64: kernel: Add arch-specific SDEI entry code and CPU masking")
> Cc: stable@vger.kernel.org
> Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>

Tested-by: Breno Leitao <leitao@debian.org>

Tested on NVIDIA Grace (firmware SDEI) with next-20261104, KASAN + lockdep,
ARM_SDEI_NMI=y: sysrq-l under stress-ng, with a kprobe on show_regs() and
eprobes on irq_disable/irq_enable that fault on purpose.

Without the series the box crashed (a task interrupted in el0t_64_sync
got the handler's BRK, then lockdep/KASAN fallout and a hard lockup)

With it, KPTI off and kpti=1, ~4000 kprobe hits and ~5500 faults in
irqentry_nmi_enter()/exit() produced no warnings or stalls.

Re: [PATCH v2 2/2] arm64: sdei: Preserve the interrupted context's exception registers

[Kiryl Shutsemau]

On Thu, Nov 05, 2026 at 07:41:12AM -0800, Breno Leitao wrote:
> On Thu, Nov 05, 2026 at 12:08:34PM +0000, Kiryl Shutsemau wrote:
> > From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
> > 
> > An SDEI event is dispatched by firmware regardless of PSTATE.DAIF, so it
> > can interrupt the kernel where the exception registers are live:
> > 
> >  - exception entry, before kernel_entry has saved ELR_EL1/SPSR_EL1 and
> >    before the C handler has read ESR_EL1/FAR_EL1;
> >  - exception return, after kernel_exit has loaded ELR_EL1/SPSR_EL1 and
> >    before the ERET.
> > 
> > An IRQ or pseudo-NMI cannot arrive there with PSTATE.I set. Firmware
> > saves only x0-x17, PC and PSTATE around an SDEI event, so a synchronous
> > exception taken by the handler overwrites the interrupted context's
> > ELR, SPSR, ESR and FAR. do_sdei_event() only notices that ELR changed,
> > and warns.
> > 
> > The interrupted context then carries on with the handler's state. A
> > task interrupted on its way into the kernel can be sent SIGILL for the
> > handler's kernel data abort. A task interrupted on its way back to user
> > space ERETs to EL1, at the handler's fixup address, with user registers
> > loaded.
> > 
> > With SDEI delivering NMI backtraces, this happens in normal operation.
> > The handler runs show_regs(), which reads through possibly-NULL pointers
> > with the _nofault() accessors. print_worker_info() faults this way for a
> > kworker that is not running a work item, and a single sysrq-l warned
> > once per such kworker.
> > 
> > Save ELR, SPSR, ESR and FAR in __sdei_handler() before
> > irqentry_nmi_enter(), which can take an exception too, and restore them
> > after irqentry_nmi_exit().
> > 
> > Once they are restored, a nested exception is harmless if its handler is
> > safe in NMI context. That holds for EL1 data aborts on translation,
> > permission and access-flag faults, which never take the mmap lock in NMI
> > context, and for BRK. Warn once, with the syndrome, about any other
> > exception class.
> > 
> > Fixes: f5df26961853 ("arm64: kernel: Add arch-specific SDEI entry code and CPU masking")
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> 
> Tested-by: Breno Leitao <leitao@debian.org>
> 
> Tested on NVIDIA Grace (firmware SDEI) with next-20261104, KASAN + lockdep,
> ARM_SDEI_NMI=y: sysrq-l under stress-ng, with a kprobe on show_regs() and
> eprobes on irq_disable/irq_enable that fault on purpose.
> 
> Without the series the box crashed (a task interrupted in el0t_64_sync
> got the handler's BRK, then lockdep/KASAN fallout and a hard lockup)
> 
> With it, KPTI off and kpti=1, ~4000 kprobe hits and ~5500 faults in
> irqentry_nmi_enter()/exit() produced no warnings or stalls.

Thanks for testing!

Just to clarify, does your Tested-by apply for whole series, right?

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH v2 2/2] arm64: sdei: Preserve the interrupted context's exception registers

[Kiryl Shutsemau]

On Thu, Nov 05, 2026 at 07:41:12AM -0800, Breno Leitao wrote:
> On Thu, Nov 05, 2026 at 12:08:34PM +0000, Kiryl Shutsemau wrote:
> > From: "Kiryl Shutsemau (Meta)" <kas@kernel.org>
> > 
> > An SDEI event is dispatched by firmware regardless of PSTATE.DAIF, so it
> > can interrupt the kernel where the exception registers are live:
> > 
> >  - exception entry, before kernel_entry has saved ELR_EL1/SPSR_EL1 and
> >    before the C handler has read ESR_EL1/FAR_EL1;
> >  - exception return, after kernel_exit has loaded ELR_EL1/SPSR_EL1 and
> >    before the ERET.
> > 
> > An IRQ or pseudo-NMI cannot arrive there with PSTATE.I set. Firmware
> > saves only x0-x17, PC and PSTATE around an SDEI event, so a synchronous
> > exception taken by the handler overwrites the interrupted context's
> > ELR, SPSR, ESR and FAR. do_sdei_event() only notices that ELR changed,
> > and warns.
> > 
> > The interrupted context then carries on with the handler's state. A
> > task interrupted on its way into the kernel can be sent SIGILL for the
> > handler's kernel data abort. A task interrupted on its way back to user
> > space ERETs to EL1, at the handler's fixup address, with user registers
> > loaded.
> > 
> > With SDEI delivering NMI backtraces, this happens in normal operation.
> > The handler runs show_regs(), which reads through possibly-NULL pointers
> > with the _nofault() accessors. print_worker_info() faults this way for a
> > kworker that is not running a work item, and a single sysrq-l warned
> > once per such kworker.
> > 
> > Save ELR, SPSR, ESR and FAR in __sdei_handler() before
> > irqentry_nmi_enter(), which can take an exception too, and restore them
> > after irqentry_nmi_exit().
> > 
> > Once they are restored, a nested exception is harmless if its handler is
> > safe in NMI context. That holds for EL1 data aborts on translation,
> > permission and access-flag faults, which never take the mmap lock in NMI
> > context, and for BRK. Warn once, with the syndrome, about any other
> > exception class.
> > 
> > Fixes: f5df26961853 ("arm64: kernel: Add arch-specific SDEI entry code and CPU masking")
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> 
> Tested-by: Breno Leitao <leitao@debian.org>
> 
> Tested on NVIDIA Grace (firmware SDEI) with next-20261104, KASAN + lockdep,
> ARM_SDEI_NMI=y: sysrq-l under stress-ng, with a kprobe on show_regs() and
> eprobes on irq_disable/irq_enable that fault on purpose.
> 
> Without the series the box crashed (a task interrupted in el0t_64_sync
> got the handler's BRK, then lockdep/KASAN fallout and a hard lockup)
> 
> With it, KPTI off and kpti=1, ~4000 kprobe hits and ~5500 faults in
> irqentry_nmi_enter()/exit() produced no warnings or stalls.

Thanks for testing!

Just to clarify, does your Tested-by apply for whole series, right?

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH v2 2/2] arm64: sdei: Preserve the interrupted context's exception registers

[Breno Leitao]

On Fri, Nov 06, 2026 at 11:34:19AM +0000, Kiryl Shutsemau wrote:
> On Thu, Nov 05, 2026 at 07:41:12AM -0800, Breno Leitao wrote:
> > On Thu, Nov 05, 2026 at 12:08:34PM +0000, Kiryl Shutsemau wrote:
> > Tested-by: Breno Leitao <leitao@debian.org>
> > 
> > Tested on NVIDIA Grace (firmware SDEI) with next-20261104, KASAN + lockdep,
> > ARM_SDEI_NMI=y: sysrq-l under stress-ng, with a kprobe on show_regs() and
> > eprobes on irq_disable/irq_enable that fault on purpose.
> > 
> > Without the series the box crashed (a task interrupted in el0t_64_sync
> > got the handler's BRK, then lockdep/KASAN fallout and a hard lockup)
> > 
> > With it, KPTI off and kpti=1, ~4000 kprobe hits and ~5500 faults in
> > irqentry_nmi_enter()/exit() produced no warnings or stalls.
> 
> Thanks for testing!
> 
> Just to clarify, does your Tested-by apply for whole series, right?

Right. I was planning to do it in the cover letter, but, there was no
tags there (such as Signed-off-by:) so, I did it on the second one.

Re: [PATCH v2 2/2] arm64: sdei: Preserve the interrupted context's exception registers

[Breno Leitao]

On Fri, Nov 06, 2026 at 11:34:19AM +0000, Kiryl Shutsemau wrote:
> On Thu, Nov 05, 2026 at 07:41:12AM -0800, Breno Leitao wrote:
> > On Thu, Nov 05, 2026 at 12:08:34PM +0000, Kiryl Shutsemau wrote:
> > Tested-by: Breno Leitao <leitao@debian.org>
> > 
> > Tested on NVIDIA Grace (firmware SDEI) with next-20261104, KASAN + lockdep,
> > ARM_SDEI_NMI=y: sysrq-l under stress-ng, with a kprobe on show_regs() and
> > eprobes on irq_disable/irq_enable that fault on purpose.
> > 
> > Without the series the box crashed (a task interrupted in el0t_64_sync
> > got the handler's BRK, then lockdep/KASAN fallout and a hard lockup)
> > 
> > With it, KPTI off and kpti=1, ~4000 kprobe hits and ~5500 faults in
> > irqentry_nmi_enter()/exit() produced no warnings or stalls.
> 
> Thanks for testing!
> 
> Just to clarify, does your Tested-by apply for whole series, right?

Right. I was planning to do it in the cover letter, but, there was no
tags there (such as Signed-off-by:) so, I did it on the second one.

Re: [PATCH v2 0/2] arm64: sdei: Handle exceptions taken inside the event handler

[Catalin Marinas]

On Thu, 05 Nov 2026 12:08:32 +0000, Kiryl Shutsemau wrote:
> An SDEI event can land anywhere, and a synchronous exception taken by its
> handler breaks the context it interrupted:
> 
>  - With KPTI, an event that lands in user space runs the handler on the
>    trampoline vectors, whose current-EL half is empty. The exception
>    loops there and the CPU is lost.
> 
> [...]

Applied to arm64 (for-next/fixes), thanks!

[1/2] arm64: sdei: Run the event handler on the kernel vectors
      https://git.kernel.org/arm64/c/9bf6ee7952c7
[2/2] arm64: sdei: Preserve the interrupted context's exception registers
      https://git.kernel.org/arm64/c/58c1eb84e977