Re: [PATCHv2] arm64/entry: Fix arm64-specific rseq brokenness

[Jinjie Ruan <ruanjinjie@huawei.com>]

On 5/8/2026 10:20 PM, Mark Rutland wrote:
> Hi,
> 
> In all the confusion with the rseq fixes, it looks like the arm64 patch
> didn't get picked up into the tip tree. That was queued up in Thomas's
> devel/core/rseq branch, but it didn't get reposted and picked up into
> tip.
> 
> Thomas, Peter, are you happy to pick the below into the tip sched/urgent
> branch? It's the same as what what in Thomas's devel/core/rseq branch:
> 
>   https://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git/commit/?h=core/rseq&id=04db289d5633dad546a0778537f0f0e52ebfc88b
> 
> ... which was taken from my arm64/rseq branch:
> 
>   https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/commit/?h=arm64/rseq&id=05bce437d4f2fd14a7be4706c684a618e2fcc82f
> 
> It applies cleanly and passes all tests.
> 
> Catalin, Will, you both mentioned off-list that you were happy for this
> to go via tip with the rest of the rseq fixes. Can either of you please
> confirm with an ack?
> 
> Mark.
> 
> ---->8----
> Mathias Stearn reports that since v6.19, there are two big issues
> affecting rseq:
> 
> (1) On arm64 specifically, rseq critical sections aren't aborted when
>     they should be.
> 
> (2) The 'cpu_id_start' field is no longer written by the kernel in all
>     cases it used to be, including some cases where TCMalloc depends on
>     the kernel clobbering the field.
> 
> This patch fixes issue #1. This patch DOES NOT fix issue #2, which will
> need to be addressed by other patches.
> 
> The arm64-specific brokenness is a result of commits:
> 
>   2fc0e4b4126c ("rseq: Record interrupt from user space")
>   39a167560a61 ("rseq: Optimize event setting")
> 
> The first commit failed to add a call to rseq_note_user_irq_entry() on
> arm64. Thus arm64 never sets rseq_event::user_irq to record that it may
> be necessary to abort an active rseq critical section upon return to
> userspace. On its own, this commit had no functional impact as the value
> of rseq_event::user_irq was not consumed.
> 
> The second commit relied upon rseq_event::user_irq to determine whether
> or not to bother to perform rseq work when returning to userspace. As
> rseq_event::user_irq wasn't set on arm64, this work would be skipped,
> and consequently an active rseq critical section would not be aborted.
> 
> Fix this by giving arm64 syscall-specific entry/exit paths, and
> performing the relevant logic in syscall and non-syscall paths,
> including calling rseq_note_user_irq_entry() for non-syscall entry.
> 
> Currently arm64 cannot use syscall_enter_from_user_mode(),
> syscall_exit_to_user_mode(), and irqentry_exit_to_user_mode(), due to
> ordering constraints with exception masking, and risk of ABI breakage
> for syscall tracing/audit/etc. For the moment the entry/exit logic is
> left as arm64-specific, directly using enter_from_user_mode() and
> exit_to_user_mode(), but mirroring the generic code.
> 
> I intend to follow up with refactoring/cleanup, as we did for kernel
> mode entry paths in commit:
> 
>   041aa7a85390 ("entry: Split preemption from irqentry_exit_to_kernel_mode()")
> 
> ... which will allow arm64 to use the GENERIC_IRQ_ENTRY functions directly.
> 
> Fixes: 39a167560a61 ("rseq: Optimize event setting")
> Reported-by: Mathias Stearn <mathias@mongodb.com>
> Link: https://lore.kernel.org/regressions/CAHnCjA25b+nO2n5CeifknSKHssJpPrjnf+dtr7UgzRw4Zgu=oA@mail.gmail.com/
> Signed-off-by: Mark Rutland <mark.rutland@arm.com>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Chris Kennelly <ckennelly@google.com>
> Cc: Dmitry Vyukov <dvyukov@google.com>
> Cc: Jinjie Ruan <ruanjinjie@huawei.com>
> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Will Deacon <will@kernel.org>
> ---
>  arch/arm64/kernel/entry-common.c | 31 ++++++++++++++++++++++++-------
>  include/linux/irq-entry-common.h |  8 --------
>  include/linux/rseq_entry.h       | 19 -------------------
>  3 files changed, 24 insertions(+), 34 deletions(-)
> 
> Since v1 [1]:
> * Restore sme_{enter_from,exit_to}_user_mode()
> * Describe use of enter_from_user_mode() in commit message

Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>

> 
> [1] https://lore.kernel.org/linux-arm-kernel/aeueE1I1OuVkOcEZ@J2N7QTR9R3/
> 
> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> index cb54335465f66..c7a23f7c22122 100644
> --- a/arch/arm64/kernel/entry-common.c
> +++ b/arch/arm64/kernel/entry-common.c
> @@ -62,6 +62,13 @@ static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
>  	irqentry_exit_to_kernel_mode_after_preempt(regs, state);
>  }
>  
> +static __always_inline void arm64_syscall_enter_from_user_mode(struct pt_regs *regs)
> +{
> +	enter_from_user_mode(regs);
> +	mte_disable_tco_entry(current);
> +	sme_enter_from_user_mode();
> +}
> +
>  /*
>   * Handle IRQ/context state management when entering from user mode.
>   * Before this function is called it is not safe to call regular kernel code,
> @@ -70,20 +77,30 @@ static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
>  static __always_inline void arm64_enter_from_user_mode(struct pt_regs *regs)
>  {
>  	enter_from_user_mode(regs);
> +	rseq_note_user_irq_entry();
>  	mte_disable_tco_entry(current);
>  	sme_enter_from_user_mode();
>  }
>  
> +static __always_inline void arm64_syscall_exit_to_user_mode(struct pt_regs *regs)
> +{
> +	local_irq_disable();
> +	syscall_exit_to_user_mode_prepare(regs);
> +	local_daif_mask();
> +	sme_exit_to_user_mode();
> +	mte_check_tfsr_exit();
> +	exit_to_user_mode();
> +}
> +
>  /*
>   * Handle IRQ/context state management when exiting to user mode.
>   * After this function returns it is not safe to call regular kernel code,
>   * instrumentable code, or any code which may trigger an exception.
>   */
> -
>  static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
>  {
>  	local_irq_disable();
> -	exit_to_user_mode_prepare_legacy(regs);
> +	irqentry_exit_to_user_mode_prepare(regs);
>  	local_daif_mask();
>  	sme_exit_to_user_mode();
>  	mte_check_tfsr_exit();
> @@ -92,7 +109,7 @@ static __always_inline void arm64_exit_to_user_mode(struct pt_regs *regs)
>  
>  asmlinkage void noinstr asm_exit_to_user_mode(struct pt_regs *regs)
>  {
> -	arm64_exit_to_user_mode(regs);
> +	arm64_syscall_exit_to_user_mode(regs);
>  }
>  
>  /*
> @@ -716,12 +733,12 @@ static void noinstr el0_brk64(struct pt_regs *regs, unsigned long esr)
>  
>  static void noinstr el0_svc(struct pt_regs *regs)
>  {
> -	arm64_enter_from_user_mode(regs);
> +	arm64_syscall_enter_from_user_mode(regs);
>  	cortex_a76_erratum_1463225_svc_handler();
>  	fpsimd_syscall_enter();
>  	local_daif_restore(DAIF_PROCCTX);
>  	do_el0_svc(regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_syscall_exit_to_user_mode(regs);
>  	fpsimd_syscall_exit();
>  }
>  
> @@ -868,11 +885,11 @@ static void noinstr el0_cp15(struct pt_regs *regs, unsigned long esr)
>  
>  static void noinstr el0_svc_compat(struct pt_regs *regs)
>  {
> -	arm64_enter_from_user_mode(regs);
> +	arm64_syscall_enter_from_user_mode(regs);
>  	cortex_a76_erratum_1463225_svc_handler();
>  	local_daif_restore(DAIF_PROCCTX);
>  	do_el0_svc_compat(regs);
> -	arm64_exit_to_user_mode(regs);
> +	arm64_syscall_exit_to_user_mode(regs);
>  }
>  
>  static void noinstr el0_bkpt32(struct pt_regs *regs, unsigned long esr)
> diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-common.h
> index 167fba7dbf043..1fabf0f5ea8e7 100644
> --- a/include/linux/irq-entry-common.h
> +++ b/include/linux/irq-entry-common.h
> @@ -218,14 +218,6 @@ static __always_inline void __exit_to_user_mode_validate(void)
>  	lockdep_sys_exit();
>  }
>  
> -/* Temporary workaround to keep ARM64 alive */
> -static __always_inline void exit_to_user_mode_prepare_legacy(struct pt_regs *regs)
> -{
> -	__exit_to_user_mode_prepare(regs, EXIT_TO_USER_MODE_WORK);
> -	rseq_exit_to_user_mode_legacy();
> -	__exit_to_user_mode_validate();
> -}
> -
>  /**
>   * syscall_exit_to_user_mode_prepare - call exit_to_user_mode_loop() if required
>   * @regs:	Pointer to pt_regs on entry stack
> diff --git a/include/linux/rseq_entry.h b/include/linux/rseq_entry.h
> index 2d0295df5107c..63bc72086e75b 100644
> --- a/include/linux/rseq_entry.h
> +++ b/include/linux/rseq_entry.h
> @@ -749,24 +749,6 @@ static __always_inline void rseq_irqentry_exit_to_user_mode(void)
>  	ev->events = 0;
>  }
>  
> -/* Required to keep ARM64 working */
> -static __always_inline void rseq_exit_to_user_mode_legacy(void)
> -{
> -	struct rseq_event *ev = &current->rseq.event;
> -
> -	rseq_stat_inc(rseq_stats.exit);
> -
> -	if (static_branch_unlikely(&rseq_debug_enabled))
> -		WARN_ON_ONCE(ev->sched_switch);
> -
> -	/*
> -	 * Ensure that event (especially user_irq) is cleared when the
> -	 * interrupt did not result in a schedule and therefore the
> -	 * rseq processing did not clear it.
> -	 */
> -	ev->events = 0;
> -}
> -
>  void __rseq_debug_syscall_return(struct pt_regs *regs);
>  
>  static __always_inline void rseq_debug_syscall_return(struct pt_regs *regs)
> @@ -782,7 +764,6 @@ static inline bool rseq_exit_to_user_mode_restart(struct pt_regs *regs, unsigned
>  }
>  static inline void rseq_syscall_exit_to_user_mode(void) { }
>  static inline void rseq_irqentry_exit_to_user_mode(void) { }
> -static inline void rseq_exit_to_user_mode_legacy(void) { }
>  static inline void rseq_debug_syscall_return(struct pt_regs *regs) { }
>  static inline bool rseq_grant_slice_extension(unsigned long ti_work, unsigned long mask) { return false; }
>  #endif /* !CONFIG_RSEQ */