Re: [RFC v2 PATCH 0/16] Optimize this_cpu_*() ops for non-x86 (ARM64 for this series)

[Mark Rutland <mark.rutland@arm.com>]

On Tue, Jul 21, 2026 at 04:20:50PM -0700, Yang Shi wrote:
> On 7/16/26 6:23 AM, Ryan Roberts wrote:
> > On 15/07/2026 19:04, Yang Shi wrote:
> > But I think that there are other issues with this concept that likely block it
> > from being accepted upstream; see below...

> > We have observed a few performance regressions recently, for which the root
> > cause is increased use of this_cpu_*. We have somebody at Arm about to start an
> > investigation into whether in-kernel rseq can solve the problem.
> 
> It is good to know someone is exploring the alternative path. But I doubt
> restart sequence can really boost the performance. It needs check whether
> preemption (or cpu migration) happens in the middle of percpu operations
> then manipulate the ip to restart the operation.

With the rseq approach, that check happens out-of-line in exception
handling code, *only* when preemption occurs, so in the fast path there
is *no code whatsoever* to check for preemption/migration, and no
branches to cause a restart.

Please note that with the approach I've prototyped, the exception
handling code doesn't even adjust the PC; we just adjust a GPR
containing the address. The forward progress guarantees (and behaviour
under contention) should be the same as a single instruction. The fixup
logic itself is trivial.

Please see my message where I outlined that approach in detail:

  https://lore.kernel.org/linux-arm-kernel/al_DpFJFcmVhxpvW@J2N7QTR9R3/

I expect that should come with a reasonable benefit, but I don't have
benchmark figures yet as I haven't finished converting the xchg and
cmpxchg implementations.

> It sounds like it just moved the cost from one place to the other
> place and it also seems hacky TBH.

The existing (high) cost is removed from the fast path. A new (low) cost
is added to the rare path where we perform an exception return into the
middle of a critical section.

I don't think this is any more hacky than percpu page tables. It's far
more contained to the relevant pieces of code (only the this_cpu_*() ops
and exception entry/return).

> It may reduce the preempt_disable/preempt_enable cost somehow for some
> less contended cases, but it may be worse for high contended cases.
> The percpu page table can work well for both.

It entirely removes the preempt_disable/preempt_enable cost.

I don't follow your concern with contention. Any contention on the
memory location is going to be independent of preemption, and if a
thread is preempted mid-operation, the impact is going to be the same
with either the adress fixup or percpu page tables -- I don't think you
avoid that.

> S390 implemented restart sequence. Heiko didn't share too much benchmark
> data, just confirmed there is no regression and I had a lengthy discussion
> with him
> (https://lore.kernel.org/lkml/20260520092243.264847-1-hca@linux.ibm.com/).

As above, for s390 Heiko didn't implement a restart sequence; he
implemented an address fixup sequence. There is no restart.

The discussion you had there seemed to focus on the s390 implementation
details (e.g. why he used an address fixup rather than a restart), and
the benefits of removing the preempt_count manipulation (which all the
approaches, including mine, do).

Am I missing some key detail?

> Improving this_cpu performance is just one of the usecase of percpu page
> table. The other potential usecase is kernel text replication I mentioned at
> LSFMM. The idea is not new, someone else has explored it, see https://lore.kernel.org/linux-arm-kernel/ZMKNYEkM7YnrDtOt@shell.armlinux.org.uk/.

That was looked at in the past, but people weren't keen given the
complexity it would lead to, which is why it never saw much traction.

> Percpu page table can make kernel support it more naturally.

Perhaps, but the only difference is that you've moving the support code
into the core kernel mm code. I think the general concerns people have
with percpu page tables still apply.

[...]

> > > Known Issues
> > > ============

> > > 3. Shared TLB machines
> > > ----------------------
> > > Some machines may share TLB between CPUs, for example, SMT machines may share
> > > TLB between the two hardware threads in one core.
> > > The per cpu page table just can't work with it. Maybe we need a new
> > > cpufeature to indicate whether per cpu page table is allowed or not. Then
> > > just enable it for not-shared-TLB machines.
> > The architectural feature you're referring to here is FEAT_TTCNP (common not
> > private). There are many CPUs out there that use this feature (not just those
> > that support SMT).
> > 
> > FEAT_TTCNP is mandatory from Armv8.2 and the presence of the feature only tells
> > you whether you can legally set the CnP bit - it doesn't actually tell you if
> > the HW does any sharing - there is no way to detect this. So making per-cpu
> > pgtables mutually exclusive with CNP is a non-starter as all CPUs from v8.2
> > onwards advertise CNP.
> > 
> > Even ignoring the CNP problem, you'll end up installing the per-cpu TLB entries
> > tagged with whatever user ASID happens to be installed at the time. Which means
> > you could end up with lots of duplicated TLB entries all differing only by ASID.
> > It's probably not the end of the world, but it seems... inefficient.
> 
> Really? I thought ASID will be ignored if nG == 0.

The nG bit only applies to leaf entries (Page or Block descriptors). All
intermediate entries (Table descriptors) are ASID tagged.

> > The only way I can see to make this work fully is to rely on FEAT_ASID2, which
> > allows specifying separate ASIDs for TTBR0 and TTBR1. Then each CPU can have
> > it's own (permanently installed) ASID for local mappings in TTBR1. That would
> > solve the CNP issue. FEAT_ASID2 is optional from v9.4 and mandatory from v9.5.
> 
> I agree it is safer to handle CNP problem by depending on FEAT_ASID2.

I think that depending on FEAT_ASID2 is a non-starter.

There is no extant hardware with FEAT_ASID2, and I don't think we want
to maintain distinct versions of hte percpu code accross architecture
versions.

Mark.