Re: [RFC v2 PATCH 0/16] Optimize this_cpu_*() ops for non-x86 (ARM64 for this series)

[Mark Rutland <mark.rutland@arm.com>]

On Thu, Jul 16, 2026 at 02:23:33PM +0100, Ryan Roberts wrote:
> We have observed a few performance regressions recently, for which the
> root cause is increased use of this_cpu_*. We have somebody at Arm
> about to start an investigation into whether in-kernel rseq can solve
> the problem.

FWIW, I had a look into something rseq like (resetting the PC upon
preemption or any exception), and I ended up with a GPR fixup scheme.
That avoids the need to disable/enable preemption, and only requires a
read of current and a couple of stores, which *should* be cheap.

The approach is similar to Peter Zijlstra's in-kernel rseq scheme [1],
and Heiko Carsten's approach for s390 [2]:

  [1] https://lore.kernel.org/lkml/20260223163843.GR1282955@noisy.programming.kicks-ass.net/
  [2] https://lore.kernel.org/lkml/20260526055702.1429061-1-hca@linux.ibm.com/

A key difference is that I ignore the PC entirely, and *only* track whether
GPRs are in use by a critical section. The GPR fixup is idempotent, and
can safely be applied anywhere in the critical section. Ignoring the PC
means that (in theory at least) this should work with kprobes, etc.

I've rebased and cleaned that up a bit, and pushed a WIP version to my
arm64/percpu-fixup branch:

  git://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git

I'll aim to have a more complete/polished version out in the near
future.

The key idea is that every percpu op has a critical section during which
it maintains three distinct GPRs:

  <pcp> : The original __percpu pointer.
  <off> : The percpu offset.
  <addr>: The final pointer (<pcp> + <off>).

Whenever the kernel performs an exception return back into a critical
section, it can fix up <off> and <addr>, as described below. The
critical section is roughly as follows:

	/*
	 * Prologue.
	 *
	 * At the start, <pcp> is already set, but <off> and <addr> are
	 * uninitialised.
	 *
	 * After this STR, upon an exception return into this critical
	 * section, the kernel will:
	 * 1. Update <off> to the current CPU's offset.
	 * 2. Update <addr> to be <pcp> + <off>.
	 */
	mrs	<tsk>, sp_el0
	mov	<tmp>, ENCODE_REGISTER_NUMBERS(<pcp>, <off>, <addr>)
	strh	<tmp>, [<tsk>, #TSK_TI_PCPU_GPRS]

	/*
	 * Generate the address. Any preemption within this critical
	 * section will result in <off> and <addr> being updated to
	 * match the CPU that this is resumed upon.
	 */
	mrs	<off>, TPIDR_ELx
	add	<addr>, <pcp>, <off>

	/* The actual operation, e.g. read/write/cmpxchg */
	DO_SOMETHING_WITH(<addr>)

	/*
	 * Epilogue.
	 *
	 * After this STR, the kernel will no longer apply the fixups.
	 */
	strh	wzr, [<tsk>, #TSK_TI_PCPU_GPRS]

To handle nesting, the exception code is updated to save/restore the
register numbers in pt_regs (and clearing the active value at entry) so
that this safely nests.

With that, a simple test case such as:

  u64 outline_this_cpu_read_u64(u64 __percpu *p)
  {
          return this_cpu_read(*p);
  }

... is reduced from:

  <outline_this_cpu_read_u64>:
         paciasp
         stp     x29, x30, [sp, #-32]!
         mrs     x1, sp_el0
         mov     x29, sp
         ldr     w2, [x1, #8]
         add     w2, w2, #0x1
         str     w2, [x1, #8]
         mrs     x2, tpidr_el1
         ldr     x0, [x0, x2]
         ldr     x2, [x1, #8]
         sub     x2, x2, #0x1
         str     w2, [x1, #8]
         cbz     x2, 1f
         ldr     x1, [x1, #8]
         cbnz    x1, 2f
  1:
         str     x0, [sp, #24]
         bl      0 <preempt_schedule_notrace>
         ldr     x0, [sp, #24]
  2:
         ldp     x29, x30, [sp], #32
         autiasp
         ret

... down to:

  <outline_this_cpu_read_u64>:
         mrs     x2, sp_el0        // Prologue
         mov     x4, #0xc80        // Prologue
         strh    w4, [x2, #20]     // Prologue
         mrs     x4, tpidr_el1
         add     x3, x0, x4
         ldr     x1, [x3]
         strh    wzr, [x2, #20]    // Epilogue
         mov     x0, x1
         ret

... which is clearly much better.

As noted above, the branch is a WIP. There are a bunch of things to do
(in particular, factoring out the xchg and cmpxchg assembly), but I
don't currently see a major technical blocker for the fixup approach.

Mark.