Re: [LSF/MM/BPF TOPIC] Improve this_cpu_ops performance for ARM64 (and potentially other architectures)

[Catalin Marinas <catalin.marinas@arm.com>]

On Wed, Feb 11, 2026 at 03:58:50PM -0800, Yang Shi wrote:
> On Wed, Feb 11, 2026 at 3:29 PM Tejun Heo <tj@kernel.org> wrote:
> > On Wed, Feb 11, 2026 at 03:14:57PM -0800, Yang Shi wrote:
> > ...
> > > Overhead
> > > ========
> > > 1. Some extra virtual memory space. But it shouldn’t be too much. I
> > > saw 960K with Fedora default kernel config. Given terabytes virtual
> > > memory space on 64 bit machine, 960K is negligible.
> > > 2. Some extra physical memory for percpu kernel page table. 4K *
> > > (nr_cpus – 1) for PGD pages, plus the page tables used by percpu local
> > > mapping area. A couple of megabytes with Fedora default kernel config
> > > on AmpereOne with 160 cores.
> > > 3. Percpu allocation and free will be slower due to extra virtual
> > > memory allocation and page table manipulation. However, percpu is
> > > allocated by chunk. One chunk typically holds a lot percpu variables.
> > > So the slowdown should be negligible. The test result below also
> > > proved it.
[...]
> > One property that this breaks is per_cpu_ptr() of a given CPU disagreeing
> > with this_cpu_ptr(). e.g. If there are users that take this_cpu_ptr() and
> > uses that outside preempt disable block (which is a bit odd but allowed),
> > the end result would be surprising. Hmm... I wonder whether it'd be
> > worthwhile to keep this_cpu_ptr() returning the global address - ie. make it
> > access global offset from local mapping and then return the computed global
> > address. This should still be pretty cheap and gets rid of surprising and
> > potentially extremely subtle corner cases.
> 
> Yes, this is going to be a problem. So we don't change how
> this_cpu_ptr() works and keep it returning the global address. Because
> I noticed this may cause confusion for list APIs too. For example,
> when initializing a list embedded into a percpu variable, the ->next
> and ->prev will be initialized to global addresses by using
> per_cpu_ptr(), but if the list is accessed via this_cpu_ptr(), list
> head will be dereferenced by using local address, then list_empty()
> will complain, which compare the list head pointer and ->next pointer.
> This will cause some problems.
> 
> So we just use the local address for this_cpu_add/sub/inc/dec and so
> on, which just manipulate a scalar counter.

I wonder how much overhead is caused by calling into the scheduler on
preempt_enable(). It would be good to get some numbers for something
like the patch below (also removing the preempt disabling for
this_cpu_read() as I don't think it matters - a thread cannot
distinguish whether it was preempted between TPIDR read and variable
read or immediately after the variable read; we can't do this for writes
as other threads may notice unexpected updates).

Another wild hack could be to read the kernel instruction at
(current_pt_regs()->pc - 4) in arch_irqentry_exit_need_resched() and
return false if it's a read from TPIDR_EL1/2, together with removing the
preempt disabling. Or some other lighter way of detecting this_cpu_*
constructs without full preemption disabling.

-----------------8<------------------------------------
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index b57b2bb00967..7194cc997293 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -153,11 +153,17 @@ PERCPU_RET_OP(add, add, ldadd)
  * disabled.
  */
 
+#ifdef preempt_enable_no_resched_notrace
+#define _pcp_preempt_enable_notrace	preempt_enable_no_resched_notrace
+#else
+#define _pcp_preempt_enable_notrace	preempt_enable_notrace
+#endif
+
 #define _pcp_protect(op, pcp, ...)					\
 ({									\
 	preempt_disable_notrace();					\
 	op(raw_cpu_ptr(&(pcp)), __VA_ARGS__);				\
-	preempt_enable_notrace();					\
+	_pcp_preempt_enable_notrace();					\
 })
 
 #define _pcp_protect_return(op, pcp, args...)				\
@@ -165,18 +171,21 @@ PERCPU_RET_OP(add, add, ldadd)
 	typeof(pcp) __retval;						\
 	preempt_disable_notrace();					\
 	__retval = (typeof(pcp))op(raw_cpu_ptr(&(pcp)), ##args);	\
-	preempt_enable_notrace();					\
+	_pcp_preempt_enable_notrace();					\
 	__retval;							\
 })
 
+#define _pcp_return(op, pcp, args...)					\
+	((typeof(pcp))op(raw_cpu_ptr(&(pcp)), ##args))
+
 #define this_cpu_read_1(pcp)		\
-	_pcp_protect_return(__percpu_read_8, pcp)
+	_pcp_return(__percpu_read_8, pcp)
 #define this_cpu_read_2(pcp)		\
-	_pcp_protect_return(__percpu_read_16, pcp)
+	_pcp_return(__percpu_read_16, pcp)
 #define this_cpu_read_4(pcp)		\
-	_pcp_protect_return(__percpu_read_32, pcp)
+	_pcp_return(__percpu_read_32, pcp)
 #define this_cpu_read_8(pcp)		\
-	_pcp_protect_return(__percpu_read_64, pcp)
+	_pcp_return(__percpu_read_64, pcp)
 
 #define this_cpu_write_1(pcp, val)	\
 	_pcp_protect(__percpu_write_8, pcp, (unsigned long)val)
@@ -253,7 +262,7 @@ PERCPU_RET_OP(add, add, ldadd)
 	preempt_disable_notrace();					\
 	ptr__ = raw_cpu_ptr(&(pcp));					\
 	ret__ = cmpxchg128_local((void *)ptr__, old__, new__);		\
-	preempt_enable_notrace();					\
+	_pcp_preempt_enable_notrace();					\
 	ret__;								\
 })