* [PATCH v2 01/20] arm64: percpu: Fix this_cpu_write() casting
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-05 8:37 ` David Laight
2026-08-04 17:04 ` [PATCH v2 02/20] arm64: percpu: Fix this_cpu_and() mask generation Mark Rutland
` (18 subsequent siblings)
19 siblings, 1 reply; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
The arm64 implementation of this_cpu_write() casts 'val' to unsigned
long. This is necessary to handle cases where 'val' is a pointer type,
and to avoid spurious compiler warnings for the (unreachable!) cases
where the pointer type would be cast to a smaller integer type.
Unfortunately, the cast is applied to 'val' rather than '(val)', which
won't always generate the expected value when 'val' is an expression.
For example, for this_cpu_write(pcp, zero - 1), where 'pcp' is a u64 and
'zero' is a u32:
* 'zero' ===> (u32) 0x00000000
* 'zero - 1' ===> (u32) 0xffffffff
* '(unsigned long)zero - 1' ===> (u64) 0xffffffffffffffff
* '(unsigned long)(zero - 1)' ===> (u64) 0x00000000ffffffff
Fix this by adding brackets around 'val'
The bug described above can be seen from the disassembly of the
following test code:
| void this_cpu_write_zero_minus_1(u64 __percpu *pcp)
| {
| u32 zero = 0;
| this_cpu_write(*pcp, zero - 1);
| }
|
| void this_cpu_write_zero_minus_1_brackets(u64 __percpu *pcp)
| {
| u32 zero = 0;
| this_cpu_write(*pcp, (zero - 1));
| }
Generated code before this patch:
| <this_cpu_write_zero_minus_1>:
| paciasp
| stp x29, x30, [sp, #-16]!
| mrs x1, sp_el0
| mov x29, sp
| ldr w2, [x1, #8]
| add w2, w2, #0x1
| str w2, [x1, #8]
| mov x3, #0xffffffffffffffff
| mrs x2, tpidr_el1
| str x3, [x0, x2]
| ldr x0, [x1, #8]
| add x0, x0, x3
| str w0, [x1, #8]
| cbz x0, 1f
| ldr x0, [x1, #8]
| cbnz x0, 2f
| 1: bl preempt_schedule_notrace
| 2: ldp x29, x30, [sp], #16
| autiasp
| ret
|
| <this_cpu_write_zero_minus_1_brackets>:
| paciasp
| stp x29, x30, [sp, #-16]!
| mrs x1, sp_el0
| mov x29, sp
| ldr w2, [x1, #8]
| add w2, w2, #0x1
| str w2, [x1, #8]
| mov x3, #0xffffffff
| mrs x2, tpidr_el1
| str x3, [x0, x2]
| ldr x0, [x1, #8]
| sub x0, x0, #0x1
| str w0, [x1, #8]
| cbz x0, 1f
| ldr x0, [x1, #8]
| cbnz x0, 2f
| 1: bl preempt_schedule_notrace
| 2: ldp x29, x30, [sp], #16
| autiasp
| ret
Generated code after this patch:
| <this_cpu_write_zero_minus_1>:
| paciasp
| stp x29, x30, [sp, #-16]!
| mrs x1, sp_el0
| mov x29, sp
| ldr w2, [x1, #8]
| add w2, w2, #0x1
| str w2, [x1, #8]
| mov x3, #0xffffffff
| mrs x2, tpidr_el1
| str x3, [x0, x2]
| ldr x0, [x1, #8]
| sub x0, x0, #0x1
| str w0, [x1, #8]
| cbz x0, 1f
| ldr x0, [x1, #8]
| cbnz x0, 2f
| 1: bl preempt_schedule_notrace
| 2: ldp x29, x30, [sp], #16
| autiasp
| ret
|
| <this_cpu_write_zero_minus_1_brackets>:
| b this_cpu_write_zero_minus_1
Fixes: 959bf2fd03b5 ("arm64: percpu: Rewrite per-cpu ops to allow use of LSE atomics")
Reported-by: David Laight <david.laight.linux@gmail.com>
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
Cc: stable@vger.kernel.org
---
arch/arm64/include/asm/percpu.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index b57b2bb009677..63bbfd4944a37 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -179,13 +179,13 @@ PERCPU_RET_OP(add, add, ldadd)
_pcp_protect_return(__percpu_read_64, pcp)
#define this_cpu_write_1(pcp, val) \
- _pcp_protect(__percpu_write_8, pcp, (unsigned long)val)
+ _pcp_protect(__percpu_write_8, pcp, (unsigned long)(val))
#define this_cpu_write_2(pcp, val) \
- _pcp_protect(__percpu_write_16, pcp, (unsigned long)val)
+ _pcp_protect(__percpu_write_16, pcp, (unsigned long)(val))
#define this_cpu_write_4(pcp, val) \
- _pcp_protect(__percpu_write_32, pcp, (unsigned long)val)
+ _pcp_protect(__percpu_write_32, pcp, (unsigned long)(val))
#define this_cpu_write_8(pcp, val) \
- _pcp_protect(__percpu_write_64, pcp, (unsigned long)val)
+ _pcp_protect(__percpu_write_64, pcp, (unsigned long)(val))
#define this_cpu_add_1(pcp, val) \
_pcp_protect(__percpu_add_case_8, pcp, val)
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* Re: [PATCH v2 01/20] arm64: percpu: Fix this_cpu_write() casting
2026-08-04 17:04 ` [PATCH v2 01/20] arm64: percpu: Fix this_cpu_write() casting Mark Rutland
@ 2026-08-05 8:37 ` David Laight
0 siblings, 0 replies; 39+ messages in thread
From: David Laight @ 2026-08-05 8:37 UTC (permalink / raw)
To: Mark Rutland
Cc: vladimir.murzin, ryan.roberts, peterz, catalin.marinas,
ruanjinjie, stable, james.morse, yang, cl, maz, david, ljs, will,
ardb, linux-arm-kernel
On Tue, 4 Aug 2026 18:04:44 +0100
Mark Rutland <mark.rutland@arm.com> wrote:
> The arm64 implementation of this_cpu_write() casts 'val' to unsigned
> long. This is necessary to handle cases where 'val' is a pointer type,
> and to avoid spurious compiler warnings for the (unreachable!) cases
> where the pointer type would be cast to a smaller integer type.
>
> Unfortunately, the cast is applied to 'val' rather than '(val)', which
> won't always generate the expected value when 'val' is an expression.
>
> For example, for this_cpu_write(pcp, zero - 1), where 'pcp' is a u64 and
> 'zero' is a u32:
>
> * 'zero' ===> (u32) 0x00000000
> * 'zero - 1' ===> (u32) 0xffffffff
> * '(unsigned long)zero - 1' ===> (u64) 0xffffffffffffffff
> * '(unsigned long)(zero - 1)' ===> (u64) 0x00000000ffffffff
>
> Fix this by adding brackets around 'val'
>
> The bug described above can be seen from the disassembly of the
> following test code:
>
> | void this_cpu_write_zero_minus_1(u64 __percpu *pcp)
> | {
> | u32 zero = 0;
> | this_cpu_write(*pcp, zero - 1);
> | }
> |
> | void this_cpu_write_zero_minus_1_brackets(u64 __percpu *pcp)
> | {
> | u32 zero = 0;
> | this_cpu_write(*pcp, (zero - 1));
> | }
>
> Generated code before this patch:
>
> | <this_cpu_write_zero_minus_1>:
> | paciasp
> | stp x29, x30, [sp, #-16]!
> | mrs x1, sp_el0
> | mov x29, sp
> | ldr w2, [x1, #8]
> | add w2, w2, #0x1
> | str w2, [x1, #8]
> | mov x3, #0xffffffffffffffff
> | mrs x2, tpidr_el1
> | str x3, [x0, x2]
> | ldr x0, [x1, #8]
> | add x0, x0, x3
> | str w0, [x1, #8]
> | cbz x0, 1f
> | ldr x0, [x1, #8]
> | cbnz x0, 2f
> | 1: bl preempt_schedule_notrace
> | 2: ldp x29, x30, [sp], #16
> | autiasp
> | ret
> |
> | <this_cpu_write_zero_minus_1_brackets>:
> | paciasp
> | stp x29, x30, [sp, #-16]!
> | mrs x1, sp_el0
> | mov x29, sp
> | ldr w2, [x1, #8]
> | add w2, w2, #0x1
> | str w2, [x1, #8]
> | mov x3, #0xffffffff
> | mrs x2, tpidr_el1
> | str x3, [x0, x2]
> | ldr x0, [x1, #8]
> | sub x0, x0, #0x1
> | str w0, [x1, #8]
> | cbz x0, 1f
> | ldr x0, [x1, #8]
> | cbnz x0, 2f
> | 1: bl preempt_schedule_notrace
> | 2: ldp x29, x30, [sp], #16
> | autiasp
> | ret
>
> Generated code after this patch:
>
> | <this_cpu_write_zero_minus_1>:
> | paciasp
> | stp x29, x30, [sp, #-16]!
> | mrs x1, sp_el0
> | mov x29, sp
> | ldr w2, [x1, #8]
> | add w2, w2, #0x1
> | str w2, [x1, #8]
> | mov x3, #0xffffffff
> | mrs x2, tpidr_el1
> | str x3, [x0, x2]
> | ldr x0, [x1, #8]
> | sub x0, x0, #0x1
> | str w0, [x1, #8]
> | cbz x0, 1f
> | ldr x0, [x1, #8]
> | cbnz x0, 2f
> | 1: bl preempt_schedule_notrace
> | 2: ldp x29, x30, [sp], #16
> | autiasp
> | ret
> |
> | <this_cpu_write_zero_minus_1_brackets>:
> | b this_cpu_write_zero_minus_1
>
> Fixes: 959bf2fd03b5 ("arm64: percpu: Rewrite per-cpu ops to allow use of LSE atomics")
> Reported-by: David Laight <david.laight.linux@gmail.com>
> Signed-off-by: Mark Rutland <mark.rutland@arm.com>
> Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Cc: Ard Biesheuvel <ardb@kernel.org>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: James Morse <james.morse@arm.com>
> Cc: Jinjie Ruan <ruanjinjie@huawei.com>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Vladimir Murzin <vladimir.murzin@arm.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Yang Shi <yang@os.amperecomputing.com>
> Cc: stable@vger.kernel.org
I suspect there is no affected code - largely testable by comparing the
output of two arm64 allmodconfig builds.
Reviewed-by: David Laight <david.laight.linux@gmail.com>
> ---
> arch/arm64/include/asm/percpu.h | 8 ++++----
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
> index b57b2bb009677..63bbfd4944a37 100644
> --- a/arch/arm64/include/asm/percpu.h
> +++ b/arch/arm64/include/asm/percpu.h
> @@ -179,13 +179,13 @@ PERCPU_RET_OP(add, add, ldadd)
> _pcp_protect_return(__percpu_read_64, pcp)
>
> #define this_cpu_write_1(pcp, val) \
> - _pcp_protect(__percpu_write_8, pcp, (unsigned long)val)
> + _pcp_protect(__percpu_write_8, pcp, (unsigned long)(val))
> #define this_cpu_write_2(pcp, val) \
> - _pcp_protect(__percpu_write_16, pcp, (unsigned long)val)
> + _pcp_protect(__percpu_write_16, pcp, (unsigned long)(val))
> #define this_cpu_write_4(pcp, val) \
> - _pcp_protect(__percpu_write_32, pcp, (unsigned long)val)
> + _pcp_protect(__percpu_write_32, pcp, (unsigned long)(val))
> #define this_cpu_write_8(pcp, val) \
> - _pcp_protect(__percpu_write_64, pcp, (unsigned long)val)
> + _pcp_protect(__percpu_write_64, pcp, (unsigned long)(val))
>
> #define this_cpu_add_1(pcp, val) \
> _pcp_protect(__percpu_add_case_8, pcp, val)
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v2 02/20] arm64: percpu: Fix this_cpu_and() mask generation
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
2026-08-04 17:04 ` [PATCH v2 01/20] arm64: percpu: Fix this_cpu_write() casting Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-05 9:14 ` David Laight
2026-08-04 17:04 ` [PATCH v2 03/20] arm64: cmpxchg: LL/SC: Avoid redundant extension Mark Rutland
` (17 subsequent siblings)
19 siblings, 1 reply; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
The arm64 implementation of this_cpu_and(pcp, val) is built in terms of
ANDNOT operations, which requires the 'val' argument to be bitwise
negated. The bitwise negation is not implemented correctly, with two
bugs described below.
(1) The bitwise negation is performed as '~val' rather than '~(val)'.
This won't always generate the expected value when 'val' is an
expression.
For example, for this_cpu_and(pcp, 1 - 1):
* 'val' is '1 - 1' ===> (int) 0x00000000
* '~val' is '~1 - 1' ===> (int) 0xfffffffd
* '~(val)' is '~(1 - 1)' ===> (int) 0xffffffff
... and thus bit[1] of 'pcp' would be preserved unexpectedly by the
ANDNOT operation.
(2) The bitwise negation is performed on 'val' before it has been cast
to (at least) the width of 'pcp'. This won't always generate the
expected value for the upper bits.
For example, for this_cpu_and(pcp, zero), where 'pcp' is a u64 and
'zero' is a u32:
* 'zero' ===> (u32) 0x00000000
* '~(zero)' ===> (u32) 0xffffffff
* '(u64)~(zero)' ===> (u64) 0x00000000ffffffff
* '~((u64)(zero))' ===> (u64) 0xffffffffffffffff
... and thus bits[63:32] of 'pcp' would be preserved unexpectedly by
the ANDNOT operation.
Fix these issues by adding brackets around 'val', and by casting 'val'
to an appropriately-sized type before bitwise negation.
The bugs described above can be seen from the disassembly of the
following test code:
| void this_cpu_and_u64_zero(u64 __percpu *pcp)
| {
| u64 zero = 0;
| this_cpu_and(*pcp, zero);
| }
|
| void this_cpu_and_u32_zero(u64 __percpu *pcp)
| {
| u32 zero = 0;
| this_cpu_and(*pcp, zero);
| }
|
| void this_cpu_and_expr_zero(u64 __percpu *pcp)
| {
| this_cpu_and(*pcp, 1 - 1);
| }
|
| void this_cpu_and_expr_zero_brackets(u64 __percpu *pcp)
| {
| this_cpu_and(*pcp, (1 - 1));
| }
Before this patch:
| <this_cpu_and_u64_zero>:
| paciasp
| stp x29, x30, [sp, #-16]!
| mrs x1, sp_el0
| mov x29, sp
| ldr w2, [x1, #8]
| add w2, w2, #0x1
| str w2, [x1, #8]
| mov x3, #0xffffffffffffffff // #-1
| mrs x2, tpidr_el1
| add x0, x0, x2
| 1: ldxr x5, [x0]
| bic x5, x5, x3
| stxr w4, x5, [x0]
| cbnz w4, 1b
| ldr x0, [x1, #8]
| add x0, x0, x3
| str w0, [x1, #8]
| cbz x0, 2f
| ldr x0, [x1, #8]
| cbnz x0, 3f
| 2: bl preempt_schedule_notrace
| 3: ldp x29, x30, [sp], #16
| autiasp
| ret
|
| <this_cpu_and_u32_zero>:
| paciasp
| stp x29, x30, [sp, #-16]!
| mrs x1, sp_el0
| mov x29, sp
| ldr w2, [x1, #8]
| add w2, w2, #0x1
| str w2, [x1, #8]
| mov x3, #0xffffffff // #4294967295
| mrs x2, tpidr_el1
| add x0, x0, x2
| 1: ldxr x5, [x0]
| bic x5, x5, x3
| stxr w4, x5, [x0]
| cbnz w4, 1b
| ldr x0, [x1, #8]
| sub x0, x0, #0x1
| str w0, [x1, #8]
| cbz x0, 2f
| ldr x0, [x1, #8]
| cbnz x0, 3f
| 2: bl preempt_schedule_notrace
| 3: ldp x29, x30, [sp], #16
| autiasp
| ret
|
| <this_cpu_and_expr_zero>:
| paciasp
| stp x29, x30, [sp, #-16]!
| mrs x1, sp_el0
| mov x29, sp
| ldr w2, [x1, #8]
| add w2, w2, #0x1
| str w2, [x1, #8]
| mov x3, #0xfffffffffffffffd // #-3
| mrs x2, tpidr_el1
| add x0, x0, x2
| 1: ldxr x5, [x0]
| bic x5, x5, x3
| stxr w4, x5, [x0]
| cbnz w4, 1b
| ldr x0, [x1, #8]
| sub x0, x0, #0x1
| str w0, [x1, #8]
| cbz x0, 2f
| ldr x0, [x1, #8]
| cbnz x0, 3f
| 2: bl preempt_schedule_notrace
| 3: ldp x29, x30, [sp], #16
| autiasp
| ret
After this patch:
| <this_cpu_and_u64_zero>:
| paciasp
| stp x29, x30, [sp, #-16]!
| mrs x1, sp_el0
| mov x29, sp
| ldr w2, [x1, #8]
| add w2, w2, #0x1
| str w2, [x1, #8]
| mov x3, #0xffffffffffffffff // #-1
| mrs x2, tpidr_el1
| add x0, x0, x2
| 1: ldxr x5, [x0]
| bic x5, x5, x3
| stxr w4, x5, [x0]
| cbnz w4, 1b
| ldr x0, [x1, #8]
| add x0, x0, x3
| str w0, [x1, #8]
| cbz x0, 2f
| ldr x0, [x1, #8]
| cbnz x0, 3f
| 2: bl 0 <preempt_schedule_notrace>
| 3: ldp x29, x30, [sp], #16
| autiasp
| ret
|
| <this_cpu_and_u32_zero>:
| b this_cpu_and_u64_zero
|
| <this_cpu_and_expr_zero>:
| b this_cpu_and_u64_zero
|
| <this_cpu_and_expr_zero_brackets>:
| b this_cpu_and_u64_zero
Fixes: 959bf2fd03b5 ("arm64: percpu: Rewrite per-cpu ops to allow use of LSE atomics")
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
Cc: stable@vger.kernel.org
---
arch/arm64/include/asm/percpu.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 63bbfd4944a37..31193bcf89a2b 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -206,13 +206,13 @@ PERCPU_RET_OP(add, add, ldadd)
_pcp_protect_return(__percpu_add_return_case_64, pcp, val)
#define this_cpu_and_1(pcp, val) \
- _pcp_protect(__percpu_andnot_case_8, pcp, ~val)
+ _pcp_protect(__percpu_andnot_case_8, pcp, ~(u8)(val))
#define this_cpu_and_2(pcp, val) \
- _pcp_protect(__percpu_andnot_case_16, pcp, ~val)
+ _pcp_protect(__percpu_andnot_case_16, pcp, ~(u16)(val))
#define this_cpu_and_4(pcp, val) \
- _pcp_protect(__percpu_andnot_case_32, pcp, ~val)
+ _pcp_protect(__percpu_andnot_case_32, pcp, ~(u32)(val))
#define this_cpu_and_8(pcp, val) \
- _pcp_protect(__percpu_andnot_case_64, pcp, ~val)
+ _pcp_protect(__percpu_andnot_case_64, pcp, ~(u64)(val))
#define this_cpu_or_1(pcp, val) \
_pcp_protect(__percpu_or_case_8, pcp, val)
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* Re: [PATCH v2 02/20] arm64: percpu: Fix this_cpu_and() mask generation
2026-08-04 17:04 ` [PATCH v2 02/20] arm64: percpu: Fix this_cpu_and() mask generation Mark Rutland
@ 2026-08-05 9:14 ` David Laight
2026-08-05 13:02 ` Mark Rutland
0 siblings, 1 reply; 39+ messages in thread
From: David Laight @ 2026-08-05 9:14 UTC (permalink / raw)
To: Mark Rutland
Cc: vladimir.murzin, ryan.roberts, peterz, catalin.marinas,
ruanjinjie, stable, james.morse, yang, cl, maz, david, ljs, will,
ardb, linux-arm-kernel
On Tue, 4 Aug 2026 18:04:45 +0100
Mark Rutland <mark.rutland@arm.com> wrote:
> The arm64 implementation of this_cpu_and(pcp, val) is built in terms of
> ANDNOT operations, which requires the 'val' argument to be bitwise
> negated. The bitwise negation is not implemented correctly, with two
> bugs described below.
>
> (1) The bitwise negation is performed as '~val' rather than '~(val)'.
> This won't always generate the expected value when 'val' is an
> expression.
>
> For example, for this_cpu_and(pcp, 1 - 1):
>
> * 'val' is '1 - 1' ===> (int) 0x00000000
> * '~val' is '~1 - 1' ===> (int) 0xfffffffd
> * '~(val)' is '~(1 - 1)' ===> (int) 0xffffffff
>
> ... and thus bit[1] of 'pcp' would be preserved unexpectedly by the
> ANDNOT operation.
>
> (2) The bitwise negation is performed on 'val' before it has been cast
> to (at least) the width of 'pcp'. This won't always generate the
> expected value for the upper bits.
>
> For example, for this_cpu_and(pcp, zero), where 'pcp' is a u64 and
> 'zero' is a u32:
>
> * 'zero' ===> (u32) 0x00000000
> * '~(zero)' ===> (u32) 0xffffffff
> * '(u64)~(zero)' ===> (u64) 0x00000000ffffffff
> * '~((u64)(zero))' ===> (u64) 0xffffffffffffffff
>
> ... and thus bits[63:32] of 'pcp' would be preserved unexpectedly by
> the ANDNOT operation.
>
> Fix these issues by adding brackets around 'val', and by casting 'val'
> to an appropriately-sized type before bitwise negation.
>
I'm not sure the arm asm output is really needed.
>
> Fixes: 959bf2fd03b5 ("arm64: percpu: Rewrite per-cpu ops to allow use of LSE atomics")
> Signed-off-by: Mark Rutland <mark.rutland@arm.com>
> Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Cc: Ard Biesheuvel <ardb@kernel.org>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: James Morse <james.morse@arm.com>
> Cc: Jinjie Ruan <ruanjinjie@huawei.com>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Vladimir Murzin <vladimir.murzin@arm.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Yang Shi <yang@os.amperecomputing.com>
> Cc: stable@vger.kernel.org
> ---
> arch/arm64/include/asm/percpu.h | 8 ++++----
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
> index 63bbfd4944a37..31193bcf89a2b 100644
> --- a/arch/arm64/include/asm/percpu.h
> +++ b/arch/arm64/include/asm/percpu.h
> @@ -206,13 +206,13 @@ PERCPU_RET_OP(add, add, ldadd)
> _pcp_protect_return(__percpu_add_return_case_64, pcp, val)
>
> #define this_cpu_and_1(pcp, val) \
> - _pcp_protect(__percpu_andnot_case_8, pcp, ~val)
> + _pcp_protect(__percpu_andnot_case_8, pcp, ~(u8)(val))
> #define this_cpu_and_2(pcp, val) \
> - _pcp_protect(__percpu_andnot_case_16, pcp, ~val)
> + _pcp_protect(__percpu_andnot_case_16, pcp, ~(u16)(val))
I don't think the (u8) or (u16) casts are needed.
They force the high 24/16 bits to be ones, but the asm should
ignore those bits (or possible even prefer they be zeros).
They might also force the compiler to emit code to mask the high bits.
Actually the (u16) cast is wrong for (s8)128.
That is tricky to fix, maybe:
~(sizeof(val) == 1 ? (u8)(val) : (val))
(Remember ?: promotes its operands to int.)
> #define this_cpu_and_4(pcp, val) \
> - _pcp_protect(__percpu_andnot_case_32, pcp, ~val)
> + _pcp_protect(__percpu_andnot_case_32, pcp, ~(u32)(val))
The (u32) cast isn't needed (and has pretty much no effect).
> #define this_cpu_and_8(pcp, val) \
> - _pcp_protect(__percpu_andnot_case_64, pcp, ~val)
> + _pcp_protect(__percpu_andnot_case_64, pcp, ~(u64)(val))
This one still isn't right.
If val is a signed int with a negative value then it is sign extended
before being inverted.
val (int)0x80000000
(u64)(val) 0xffffffff80000000
~(u64)(val) 0x000000007fffffff
Something like ~(u64)((val) + 0u) will DTRT.
David
>
> #define this_cpu_or_1(pcp, val) \
> _pcp_protect(__percpu_or_case_8, pcp, val)
^ permalink raw reply [flat|nested] 39+ messages in thread* Re: [PATCH v2 02/20] arm64: percpu: Fix this_cpu_and() mask generation
2026-08-05 9:14 ` David Laight
@ 2026-08-05 13:02 ` Mark Rutland
2026-08-06 8:28 ` David Laight
0 siblings, 1 reply; 39+ messages in thread
From: Mark Rutland @ 2026-08-05 13:02 UTC (permalink / raw)
To: David Laight
Cc: vladimir.murzin, ryan.roberts, peterz, catalin.marinas,
ruanjinjie, stable, james.morse, yang, cl, maz, david, ljs, will,
ardb, linux-arm-kernel
On Wed, Aug 05, 2026 at 10:14:16AM +0100, David Laight wrote:
> On Tue, 4 Aug 2026 18:04:45 +0100
> Mark Rutland <mark.rutland@arm.com> wrote:
>
> > The arm64 implementation of this_cpu_and(pcp, val) is built in terms of
> > ANDNOT operations, which requires the 'val' argument to be bitwise
> > negated. The bitwise negation is not implemented correctly, with two
> > bugs described below.
> >
> > (1) The bitwise negation is performed as '~val' rather than '~(val)'.
> > This won't always generate the expected value when 'val' is an
> > expression.
> >
> > For example, for this_cpu_and(pcp, 1 - 1):
> >
> > * 'val' is '1 - 1' ===> (int) 0x00000000
> > * '~val' is '~1 - 1' ===> (int) 0xfffffffd
> > * '~(val)' is '~(1 - 1)' ===> (int) 0xffffffff
> >
> > ... and thus bit[1] of 'pcp' would be preserved unexpectedly by the
> > ANDNOT operation.
> >
> > (2) The bitwise negation is performed on 'val' before it has been cast
> > to (at least) the width of 'pcp'. This won't always generate the
> > expected value for the upper bits.
> >
> > For example, for this_cpu_and(pcp, zero), where 'pcp' is a u64 and
> > 'zero' is a u32:
> >
> > * 'zero' ===> (u32) 0x00000000
> > * '~(zero)' ===> (u32) 0xffffffff
> > * '(u64)~(zero)' ===> (u64) 0x00000000ffffffff
> > * '~((u64)(zero))' ===> (u64) 0xffffffffffffffff
> >
> > ... and thus bits[63:32] of 'pcp' would be preserved unexpectedly by
> > the ANDNOT operation.
> >
> > Fix these issues by adding brackets around 'val', and by casting 'val'
> > to an appropriately-sized type before bitwise negation.
> > diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
> > index 63bbfd4944a37..31193bcf89a2b 100644
> > --- a/arch/arm64/include/asm/percpu.h
> > +++ b/arch/arm64/include/asm/percpu.h
> > @@ -206,13 +206,13 @@ PERCPU_RET_OP(add, add, ldadd)
> > _pcp_protect_return(__percpu_add_return_case_64, pcp, val)
> >
> > #define this_cpu_and_1(pcp, val) \
> > - _pcp_protect(__percpu_andnot_case_8, pcp, ~val)
> > + _pcp_protect(__percpu_andnot_case_8, pcp, ~(u8)(val))
> > #define this_cpu_and_2(pcp, val) \
> > - _pcp_protect(__percpu_andnot_case_16, pcp, ~val)
> > + _pcp_protect(__percpu_andnot_case_16, pcp, ~(u16)(val))
>
> I don't think the (u8) or (u16) casts are needed.
They're not strictly needed, but I added them for consistency with the
other cases.
> They force the high 24/16 bits to be ones, but the asm should
> ignore those bits (or possible even prefer they be zeros).
For 'sz' bits, the asm for this op only cares about val[sz-1:0], and
val[63:sz] is immaterial. There's no preference.
> They might also force the compiler to emit code to mask the high bits.
If __percpu_andnot_case_##sz() gets outlined, sure. When
__percpu_andnot_case_##sz() is inlined (which we expect in almost all cases
today), the compiler has visibility that bits [63:sz] are unused, and won't
generate redundant code.
That's a minor redundancy, not a functional issue. If we're worried about that,
we can have __percpu_andnot_case_##sz() take its argument as a u##sz (which TBH
we probably should anyway).
For example, see the code generated for:
| void this_cpu_and_u8__0xf0(u8 __percpu *p)
| {
| this_cpu_and(*p, 0xf0);
| }
At this point in the series, GCC 15.2.0 generates:
| <this_cpu_and_u8__0xf0>:
| paciasp
| stp x29, x30, [sp, #-16]!
| mrs x1, sp_el0
| mov x29, sp
| ldr w2, [x1, #8]
| add w2, w2, #0x1
| str w2, [x1, #8]
| mov w3, #0xf // <------ Low 8 bits only!
| mrs x2, tpidr_el1
| add x0, x0, x2
| 1: ldxrb w5, [x0]
| bic w5, w5, w3
| stxrb w4, w5, [x0]
| cbnz w4, 1b
| ldr x0, [x1, #8]
| sub x0, x0, #0x1
| str w0, [x1, #8]
| cbz x0, 2f
| ldr x0, [x1, #8]
| cbnz x0, 3f
| 2: bl preempt_schedule_notrace
| 3: ldp x29, x30, [sp], #16
| autiasp
| ret
> Actually the (u16) cast is wrong for (s8)128.
> That is tricky to fix, maybe:
> ~(sizeof(val) == 1 ? (u8)(val) : (val))
> (Remember ?: promotes its operands to int.)
I do not follow, and I think you are wrong.
My understanding is that the value arguments to a this_cpu_*() operation
should be subject to the usual type promotion rules. For a u16 'pcp' and
an s8 'v', 'pcp & v' should result in sign-extension of v, and
this_cpu_and(pcp, v) should do the same.
What makes you believe the semantic you propose is correct, and the
semantic I've implemented is wrong? Is there some documentation?
Tvhe semantic youe propose doesn't match what __this_cpu_and() does, and
it doesn't match what this_cpu_and() does on x86_64.
Note how __this_cpu_and() behaves. For the following test case:
| void outline_and__u16__s8_128(u16 __percpu *p)
| {
| s8 v = 128;
| __this_cpu_and(*p, v);
| }
|
| void outline_and__s16__s8_128(s16 __percpu *p)
| {
| s8 v = 128;
| __this_cpu_and(*p, v);
| }
For arm64 this generates:
| <outline_and__u16__s8_128>:
| mrs x2, tpidr_el1
| ldrh w1, [x0, x2]
| and w1, w1, #0xffffff80
| strh w1, [x0, x2]
|
| <outline_and__s16__s8_128>:
| mrs x2, tpidr_el1
| ldrh w1, [x0, x2]
| and w1, w1, #0xffffff80
| strh w1, [x0, x2]
| ret
... where '(s8)128' is sign-extended to (at least) 16 bits.
Note that LDRH and STRH only use the low 16 bits of the register, the
upper bits of the AND are irrelevant.
Note that for x86, those tests generate:
| andw $0xff80,%gs:(%rdi)
... which is clearly sign-extending to 16 bits.
> > #define this_cpu_and_4(pcp, val) \
> > - _pcp_protect(__percpu_andnot_case_32, pcp, ~val)
> > + _pcp_protect(__percpu_andnot_case_32, pcp, ~(u32)(val))
>
> The (u32) cast isn't needed (and has pretty much no effect).
As above, this is for consistency. I agree it happens to do nothing.
> > #define this_cpu_and_8(pcp, val) \
> > - _pcp_protect(__percpu_andnot_case_64, pcp, ~val)
> > + _pcp_protect(__percpu_andnot_case_64, pcp, ~(u64)(val))
>
> This one still isn't right.
> If val is a signed int with a negative value then it is sign extended
> before being inverted.
> val (int)0x80000000
> (u64)(val) 0xffffffff80000000
> ~(u64)(val) 0x000000007fffffff
> Something like ~(u64)((val) + 0u) will DTRT.
As above, where have you got that idea from?
AFAICT, a smaller signed type *should* be sign extended, and that must
happen before bitwise negation, since that bitwise negation is to cancel
out the NOT part of the ANDNOT operation.
Think:
'pcp' is (u64) 0x0123456789abcdef
'val' is (int) 0x800000000
'(u64)(val)' is (u64) 0xffffffff80000000
'pcp & (u64)(val)' is (u64) 0x0123456780000000
'~(u64)(val)' is (u64) 0x000000007fffffff
'pcp ANDNOT ~(u64)(val)' is (u64) 0x0123456780000000
See:
| void outline_and__u64__int_0x80000000(u64 __percpu *p)
| {
| int v = 0x80000000;
| __this_cpu_and(*p, v);
| }
|
| void outline_and__s64__int_0x80000000(s64 __percpu *p)
| {
| int v = 0x80000000;
| __this_cpu_and(*p, v);
| }
For which GCC 15.2.0 generates the following:
| <outline_and__u64__int_0x80000000>:
| mrs x2, tpidr_el1
| ldr x1, [x0, x2]
| and x1, x1, #0xffffffff80000000
| str x1, [x0, x2]
| ret
|
| <outline_and__s64__int_0x80000000>:
| mrs x2, tpidr_el1
| ldr x1, [x0, x2]
| and x1, x1, #0xffffffff80000000
| str x1, [x0, x2]
| ret
Likewise on x86 this generates:
| andq $0xffffffff80000000,%gs:(%rdi)
Mark.
^ permalink raw reply [flat|nested] 39+ messages in thread* Re: [PATCH v2 02/20] arm64: percpu: Fix this_cpu_and() mask generation
2026-08-05 13:02 ` Mark Rutland
@ 2026-08-06 8:28 ` David Laight
2026-08-06 10:23 ` Mark Rutland
0 siblings, 1 reply; 39+ messages in thread
From: David Laight @ 2026-08-06 8:28 UTC (permalink / raw)
To: Mark Rutland
Cc: vladimir.murzin, ryan.roberts, peterz, catalin.marinas,
ruanjinjie, stable, james.morse, yang, cl, maz, david, ljs, will,
ardb, linux-arm-kernel
On Wed, 5 Aug 2026 14:02:03 +0100
Mark Rutland <mark.rutland@arm.com> wrote:
> On Wed, Aug 05, 2026 at 10:14:16AM +0100, David Laight wrote:
> > On Tue, 4 Aug 2026 18:04:45 +0100
> > Mark Rutland <mark.rutland@arm.com> wrote:
> >
> > > The arm64 implementation of this_cpu_and(pcp, val) is built in terms of
> > > ANDNOT operations, which requires the 'val' argument to be bitwise
> > > negated. The bitwise negation is not implemented correctly, with two
> > > bugs described below.
> > >
> > > (1) The bitwise negation is performed as '~val' rather than '~(val)'.
> > > This won't always generate the expected value when 'val' is an
> > > expression.
> > >
> > > For example, for this_cpu_and(pcp, 1 - 1):
> > >
> > > * 'val' is '1 - 1' ===> (int) 0x00000000
> > > * '~val' is '~1 - 1' ===> (int) 0xfffffffd
> > > * '~(val)' is '~(1 - 1)' ===> (int) 0xffffffff
> > >
> > > ... and thus bit[1] of 'pcp' would be preserved unexpectedly by the
> > > ANDNOT operation.
> > >
> > > (2) The bitwise negation is performed on 'val' before it has been cast
> > > to (at least) the width of 'pcp'. This won't always generate the
> > > expected value for the upper bits.
> > >
> > > For example, for this_cpu_and(pcp, zero), where 'pcp' is a u64 and
> > > 'zero' is a u32:
> > >
> > > * 'zero' ===> (u32) 0x00000000
> > > * '~(zero)' ===> (u32) 0xffffffff
> > > * '(u64)~(zero)' ===> (u64) 0x00000000ffffffff
> > > * '~((u64)(zero))' ===> (u64) 0xffffffffffffffff
> > >
> > > ... and thus bits[63:32] of 'pcp' would be preserved unexpectedly by
> > > the ANDNOT operation.
> > >
> > > Fix these issues by adding brackets around 'val', and by casting 'val'
> > > to an appropriately-sized type before bitwise negation.
...
> > > #define this_cpu_and_8(pcp, val) \
> > > - _pcp_protect(__percpu_andnot_case_64, pcp, ~val)
> > > + _pcp_protect(__percpu_andnot_case_64, pcp, ~(u64)(val))
> >
> > This one still isn't right.
> > If val is a signed int with a negative value then it is sign extended
> > before being inverted.
> > val (int)0x80000000
> > (u64)(val) 0xffffffff80000000
> > ~(u64)(val) 0x000000007fffffff
> > Something like ~(u64)((val) + 0u) will DTRT.
>
> As above, where have you got that idea from?
>
> AFAICT, a smaller signed type *should* be sign extended, and that must
> happen before bitwise negation, since that bitwise negation is to cancel
> out the NOT part of the ANDNOT operation.
>
> Think:
>
> 'pcp' is (u64) 0x0123456789abcdef
> 'val' is (int) 0x800000000
> '(u64)(val)' is (u64) 0xffffffff80000000
> 'pcp & (u64)(val)' is (u64) 0x0123456780000000
>
> '~(u64)(val)' is (u64) 0x000000007fffffff
> 'pcp ANDNOT ~(u64)(val)' is (u64) 0x0123456780000000
The problem tends to arise with (u8)128 << 24 which is signed even
though that is never intended.
To my mind sign extension prior to and/or operations is almost
certainly unexpected.
David
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [PATCH v2 02/20] arm64: percpu: Fix this_cpu_and() mask generation
2026-08-06 8:28 ` David Laight
@ 2026-08-06 10:23 ` Mark Rutland
0 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-06 10:23 UTC (permalink / raw)
To: David Laight
Cc: vladimir.murzin, ryan.roberts, peterz, catalin.marinas,
ruanjinjie, stable, james.morse, yang, cl, maz, david, ljs, will,
ardb, linux-arm-kernel
On Thu, Aug 06, 2026 at 09:28:15AM +0100, David Laight wrote:
> On Wed, 5 Aug 2026 14:02:03 +0100
> Mark Rutland <mark.rutland@arm.com> wrote:
> > On Wed, Aug 05, 2026 at 10:14:16AM +0100, David Laight wrote:
> > > On Tue, 4 Aug 2026 18:04:45 +0100
> > > Mark Rutland <mark.rutland@arm.com> wrote:
> > > > #define this_cpu_and_8(pcp, val) \
> > > > - _pcp_protect(__percpu_andnot_case_64, pcp, ~val)
> > > > + _pcp_protect(__percpu_andnot_case_64, pcp, ~(u64)(val))
> > >
> > > This one still isn't right.
> > > If val is a signed int with a negative value then it is sign extended
> > > before being inverted.
> > > val (int)0x80000000
> > > (u64)(val) 0xffffffff80000000
> > > ~(u64)(val) 0x000000007fffffff
> > > Something like ~(u64)((val) + 0u) will DTRT.
> >
> > As above, where have you got that idea from?
> >
> > AFAICT, a smaller signed type *should* be sign extended, and that must
> > happen before bitwise negation, since that bitwise negation is to cancel
> > out the NOT part of the ANDNOT operation.
> >
> > Think:
> >
> > 'pcp' is (u64) 0x0123456789abcdef
> > 'val' is (int) 0x800000000
> > '(u64)(val)' is (u64) 0xffffffff80000000
> > 'pcp & (u64)(val)' is (u64) 0x0123456780000000
> >
> > '~(u64)(val)' is (u64) 0x000000007fffffff
> > 'pcp ANDNOT ~(u64)(val)' is (u64) 0x0123456780000000
>
> The problem tends to arise with (u8)128 << 24 which is signed even
> though that is never intended.
Never intended by whom?
The expression '(u8)128 << 24' has int type. If an int is passed into a
u64 function parameter, it will be sign extended to 64 bits. If that's
passed into a binary expression against a u64, it will be sign extended.
That's the way __this_cpu_and() evaluates its argument too.
This is no different from calling any other function. If that caller
doesn't want the int value sign-extended to 64 bits to match the 64-bit
argument type, the caller needs to cast to an unsigned type.
See:
| void some_callee(u64 arg);
|
| void some_caller(u8 arg)
| {
| some_caller(arg << 24).
| }
|
| u64 some_inline_and(u8 arg)
| {
| u64 lhs = 0x0123456789abcdef;
| return lhs & (arg << 24);
| }
|
| void outline___this_cpu_and_expr(u64 __percpu *p)
| {
| __this_cpu_and(*p, (u8)128 << 24);
| }
Generated code:
| <some_caller>:
| paciasp
| stp x29, x30, [sp, #-16]!
| lsl w0, w0, #24 // left-shift 24
| mov x29, sp
| sxtw x0, w0 // sign-extend from 32-bit to 64-bit
| bl some_callee
| ldp x29, x30, [sp], #16
| autiasp
| ret
|
| <some_inline_and>:
| lsl w0, w0, #24 // left-shift 24
| mov x1, #0xcdef
| movk x1, #0x89ab, lsl #16
| sxtw x0, w0 // sign-extend from 32-bit to 64-bit
| movk x1, #0x4567, lsl #32
| movk x1, #0x123, lsl #48
| and x0, x0, x1
| ret
|
| <outline___this_cpu_and_expr>:
| mrs x2, tpidr_el1
| ldr x1, [x0, x2]
| and x1, x1, #0xffffffff80000000
| str x1, [x0, x2]
| ret
> To my mind sign extension prior to and/or operations is almost
> certainly unexpected.
As above, I disagree. I think you're starting from an incorrect
assumption about how these should work.
Having sign extension here is is entirely consistent with the usual way
type promotions are applied. It's the semantic of the __this_cpu_*()
operations that should match the this_cpu_*() operations.
Applying an arbitrarily different rule here doesn't help, and would be a
bug. I am not going to change this.
Mark.
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v2 03/20] arm64: cmpxchg: LL/SC: Avoid redundant extension
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
2026-08-04 17:04 ` [PATCH v2 01/20] arm64: percpu: Fix this_cpu_write() casting Mark Rutland
2026-08-04 17:04 ` [PATCH v2 02/20] arm64: percpu: Fix this_cpu_and() mask generation Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-04 17:04 ` [PATCH v2 04/20] arm64: cmpxchg128: LSE: Remove redundant operands Mark Rutland
` (16 subsequent siblings)
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
The __ll_sc__cmpxchg_case_##name##sz() template always takes 'old' as an
unsigned long. When 'sz' is less than 32 bits, the 'old' is explicitly
truncated so that old[31:sz-1] is zero, which is necessary so that the
W-register EOR+CBNZ sequence doesn't fail spuriously.
For 32-bit types specifically, no extension is necessary, but as 'old'
is 64 bits, the caller must extend 'old' to 64 bits. This leads to a
redundant instruction (typically a MOV) to extend the value, as can be
seen from disassembly of the test case below.
Avoid this by always taking 'old' as a 'u##sz' type, and explicitly
zero-extend this to a 64-bit or 32-bit type matching the X or W register
used by the assembly. To make this work, the table of cases now
explicitly lists 'x' for the cases where an X register is used.
For 32-bit types this removes the redundant instruction. For 64-bit
types there is no truncation, and hence no change. For {16,8}-bit types,
the existing zero-extension to 64-bit is equivalent to the new
zero-extension to 32-bit, and compilers happen to use the same
instructions for this.
I've placed the logic for zero extension into a new <asm/xwreg.h> header
as it will be used by other logic in subsequent patches.
Test case:
| u32 outline_cmpxchg_u32(u32 *p, u32 o, u32 n)
| {
| return cmpxchg(p, o, n);
| }
Before this patch:
| <outline_cmpxchg_u32>:
| b 1f
| casal w1, w2, [x0]
| mov w0, w1
| ret
| 1: mov w3, w1
| prfm pstl1strm, [x0]
| 2: ldxr w1, [x0]
| eor w4, w1, w3
| cbnz w4, 3f
| stlxr w4, w2, [x0]
| cbnz w4, 2b
| dmb ish
| 3: mov w0, w1
| ret
After this patch:
| <outline_cmpxchg_u32>:
| b 1f
| casal w1, w2, [x0]
| mov w0, w1
| ret
| 1: prfm pstl1strm, [x0]
| 2: ldxr w3, [x0]
| eor w4, w3, w1
| cbnz w4, 3f
| stlxr w4, w2, [x0]
| cbnz w4, 2b
| dmb ish
| 3: mov w0, w3
| ret
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/atomic_ll_sc.h | 27 +++++++++++++--------------
arch/arm64/include/asm/xwreg.h | 16 ++++++++++++++++
2 files changed, 29 insertions(+), 14 deletions(-)
create mode 100644 arch/arm64/include/asm/xwreg.h
diff --git a/arch/arm64/include/asm/atomic_ll_sc.h b/arch/arm64/include/asm/atomic_ll_sc.h
index 89d2ba2723590..160c1251f0ec9 100644
--- a/arch/arm64/include/asm/atomic_ll_sc.h
+++ b/arch/arm64/include/asm/atomic_ll_sc.h
@@ -11,6 +11,7 @@
#define __ASM_ATOMIC_LL_SC_H
#include <linux/stringify.h>
+#include <asm/xwreg.h>
#ifndef CONFIG_CC_HAS_K_CONSTRAINT
#define K
@@ -239,20 +240,17 @@ __ll_sc_atomic64_dec_if_positive(atomic64_t *v)
#define __CMPXCHG_CASE(w, sfx, name, sz, mb, acq, rel, cl, constraint) \
static __always_inline u##sz \
__ll_sc__cmpxchg_case_##name##sz(volatile void *ptr, \
- unsigned long old, \
+ u##sz old, \
u##sz new) \
{ \
+ /* \
+ * Sub-word sizes require zero extension so that EOR+CBNZ won't \
+ * consume non-zero upper bits of the register containing "old".\
+ */ \
+ xwreg_t(w) cmpval = xwreg_zero_extend(old, w, sz); \
unsigned long tmp; \
u##sz oldval; \
\
- /* \
- * Sub-word sizes require explicit casting so that the compare \
- * part of the cmpxchg doesn't end up interpreting non-zero \
- * upper bits of the register containing "old". \
- */ \
- if (sz < 32) \
- old = (u##sz)old; \
- \
asm volatile( \
" prfm pstl1strm, %[v]\n" \
"1: ld" #acq "xr" #sfx "\t%" #w "[oldval], %[v]\n" \
@@ -264,7 +262,8 @@ __ll_sc__cmpxchg_case_##name##sz(volatile void *ptr, \
"2:" \
: [tmp] "=&r" (tmp), [oldval] "=&r" (oldval), \
[v] "+Q" (*(u##sz *)ptr) \
- : [old] __stringify(constraint) "r" (old), [new] "r" (new) \
+ : [old] __stringify(constraint) "r" (cmpval), \
+ [new] "r" (new) \
: cl); \
\
return oldval; \
@@ -278,19 +277,19 @@ __ll_sc__cmpxchg_case_##name##sz(volatile void *ptr, \
__CMPXCHG_CASE(w, b, , 8, , , , , K)
__CMPXCHG_CASE(w, h, , 16, , , , , K)
__CMPXCHG_CASE(w, , , 32, , , , , K)
-__CMPXCHG_CASE( , , , 64, , , , , L)
+__CMPXCHG_CASE(x, , , 64, , , , , L)
__CMPXCHG_CASE(w, b, acq_, 8, , a, , "memory", K)
__CMPXCHG_CASE(w, h, acq_, 16, , a, , "memory", K)
__CMPXCHG_CASE(w, , acq_, 32, , a, , "memory", K)
-__CMPXCHG_CASE( , , acq_, 64, , a, , "memory", L)
+__CMPXCHG_CASE(x, , acq_, 64, , a, , "memory", L)
__CMPXCHG_CASE(w, b, rel_, 8, , , l, "memory", K)
__CMPXCHG_CASE(w, h, rel_, 16, , , l, "memory", K)
__CMPXCHG_CASE(w, , rel_, 32, , , l, "memory", K)
-__CMPXCHG_CASE( , , rel_, 64, , , l, "memory", L)
+__CMPXCHG_CASE(x, , rel_, 64, , , l, "memory", L)
__CMPXCHG_CASE(w, b, mb_, 8, dmb ish, , l, "memory", K)
__CMPXCHG_CASE(w, h, mb_, 16, dmb ish, , l, "memory", K)
__CMPXCHG_CASE(w, , mb_, 32, dmb ish, , l, "memory", K)
-__CMPXCHG_CASE( , , mb_, 64, dmb ish, , l, "memory", L)
+__CMPXCHG_CASE(x, , mb_, 64, dmb ish, , l, "memory", L)
#undef __CMPXCHG_CASE
diff --git a/arch/arm64/include/asm/xwreg.h b/arch/arm64/include/asm/xwreg.h
new file mode 100644
index 0000000000000..d55e3ac68f7eb
--- /dev/null
+++ b/arch/arm64/include/asm/xwreg.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __ASM_XWREG_H
+#define __ASM_XWREG_H
+
+#include <asm/types.h>
+
+#define __xwreg_t_x u64
+#define __xwreg_t_w u32
+#define xwreg_t(xw) __xwreg_t_##xw
+
+/*
+ * Zero extend 'v' from 'sz' bits (8/16/32/64) to fill an X or W register.
+ */
+#define xwreg_zero_extend(v, xw, sz) ((xwreg_t(xw))(u##sz)(v))
+
+#endif /* __ASM_XWREG_H */
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 04/20] arm64: cmpxchg128: LSE: Remove redundant operands
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (2 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 03/20] arm64: cmpxchg: LL/SC: Avoid redundant extension Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-04 17:04 ` [PATCH v2 05/20] arm64: preempt: Simplify and optimize __preempt_count_dec_and_test() Mark Rutland
` (15 subsequent siblings)
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
The LSE assembly for cmpxhg128*() has redundant operands which cause
unnecessary register pressure. These operands can be removed without
adverse effects, as described below.
It is necessary to force specific registers for the 64-bit halves of
'old' and 'new', as the encoding of CASP[A][L] requires that these
halves are allocated into even/odd register pairs, and contemporary
versions of LLVM don't support a mechanism to allocate a value into an
even/odd register pair for inline assembly. We manually allocate these
into x0/x1/x2/x3.
It is not necessary to force the address into a specific register, as
the encoding of CASP[A][L] can accept this in any GPR or SP. Hence, it
is not necessary to manually allocate the address into x4 for the
'[ptr]' operand. The assembly uses the '[v]' operand, and contemporary
compilers happen to allocate '[v]' into a separate register from
'[ptr]', meaning that '[ptr]' only serves to create register pressure.
It is not necessary to allocate the '[oldval1]' and '[oldval2]'
operands, as these are not used by the assembly. The assembly uses the
'[old1]' and '[old2]' operands for both input and output. The
'[oldval1]' and '[oldval2]' operands only serve to create register
pressure.
Remove the redundant operands. This saves on register pressure, as
demonstrated by the test case below. There's still some unfortunate
register shuffling due to the manual allocation of 'old' and 'new', but
this should be less prominent within a larger function.
Test case:
| u128 outline_cmpxchg128(u128 *p, u128 old, u128 new)
| {
| return cmpxchg128(p, old, new);
| }
Generated code before this patch:
| <outline_cmpxchg128>:
| mov x6, x0
| mov x1, x3
| mov x0, x2
| b 1f
| mov x2, x4
| mov x3, x5
| mov x4, x6
| mov x5, x0
| mov x7, x1
| caspal x0, x1, x2, x3, [x6]
| ret
| 1: prfm pstl1strm, [x6]
| 2: ldxp x8, x7, [x6]
| cmp x8, x0
| ccmp x7, x3, #0x0, eq
| b.ne 3f
| stlxp w2, x4, x5, [x6]
| cbnz w2, 2b
| dmb ish
| 3: mov x0, x8
| mov x1, x7
| ret
Generated code after this patch:
| <outline_cmpxchg128>:
| mov x6, x0
| mov x0, x2
| b 1f
| mov x1, x3
| mov x2, x4
| mov x3, x5
| caspal x0, x1, x2, x3, [x6]
| ret
| 1: prfm pstl1strm, [x6]
| 2: ldxp x8, x7, [x6]
| cmp x8, x0
| ccmp x7, x3, #0x0, eq
| 3f
| stlxp w2, x4, x5, [x6]
| cbnz w2, 2b
| dmb ish
| 3: mov x0, x8
| mov x1, x7
| ret
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/atomic_lse.h | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/arch/arm64/include/asm/atomic_lse.h b/arch/arm64/include/asm/atomic_lse.h
index afad1849c4cf5..d588af0565331 100644
--- a/arch/arm64/include/asm/atomic_lse.h
+++ b/arch/arm64/include/asm/atomic_lse.h
@@ -291,15 +291,13 @@ __lse__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \
register unsigned long x1 asm ("x1") = o.high; \
register unsigned long x2 asm ("x2") = n.low; \
register unsigned long x3 asm ("x3") = n.high; \
- register unsigned long x4 asm ("x4") = (unsigned long)ptr; \
\
asm volatile( \
__LSE_PREAMBLE \
" casp" #mb "\t%[old1], %[old2], %[new1], %[new2], %[v]\n"\
: [old1] "+&r" (x0), [old2] "+&r" (x1), \
[v] "+Q" (*(u128 *)ptr) \
- : [new1] "r" (x2), [new2] "r" (x3), [ptr] "r" (x4), \
- [oldval1] "r" (o.low), [oldval2] "r" (o.high) \
+ : [new1] "r" (x2), [new2] "r" (x3) \
: cl); \
\
r.low = x0; r.high = x1; \
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 05/20] arm64: preempt: Simplify and optimize __preempt_count_dec_and_test()
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (3 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 04/20] arm64: cmpxchg128: LSE: Remove redundant operands Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-04 17:04 ` [PATCH v2 06/20] arm64: preempt: Treat should_resched() as unlikely Mark Rutland
` (14 subsequent siblings)
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
In arm64's __preempt_count_dec_and_test(), the final conditional load of
'ti->preempt_count' is always executed in the common case. The
conditional load leads to unfortunate code generation for
preempt_enable[_notrace](), and it would be better to unconditionally
load 'ti->preempt_count', as described below.
On arm64, struct thread_info contains the following union:
| union {
| u64 preempt_count;
| struct {
| u32 count;
| u32 need_resched;
| } preempt;
| };
Note: 'need_resched' is encoded so that '0' means a reschedule is
needed, and '1' means a reschedule is NOT needed.
The core logic of __preempt_count_dec_and_test() is:
| static inline bool __preempt_count_dec_and_test(void)
| {
| struct thread_info *ti = current_thread_info();
| u64 pc = READ_ONCE(ti->preempt_count);
|
| WRITE_ONCE(ti->preempt.count, --pc);
|
| return !pc || !READ_ONCE(ti->preempt_count);
| }
The '!pc' condition can only be true when both:
* The initial value of 'need_resched' was 0, meaning that a reschedule
is needed. This should be rare.
* The initial value of 'count' was exactly 1. This cannot be true for a
nested preempt_disable() ... preempt_enable() sequence.
Hence in common cases, '!pc' will be false, and it's necessary to
execute the final READ_ONCE(ti->preempt_count).
This results in a conditional branch in the common case, as can be seen
when __preempt_count_dec_and_test() is outlined:
| <outline___preempt_count_dec_and_test>:
| mrs x2, sp_el0
| ldr x1, [x2, #8]
| mov w0, #0x1
| sub x1, x1, #0x1
| str w1, [x2, #8]
| cbz x1, 1f
| ldr x0, [x2, #8]
| cmp x0, #0x0
| cset w0, eq // eq = none
| 1: ret
It would be better to avoid the special case for 'pc == 0', and to
always load 'ti->preempt_count' after decrementing 'ti->preempt.count'.
For the common cases this will remove a conditional branch. For the rare
cases where preemption is needed initially, this only adds a single
load, whose cost should be dominated by other factors.
Remove the special case for 'pc == 0', and always load the combined
'ti->preempt_count' after decrementing 'ti->preempt.count'.
The removal of the conditional branch helps with code generation, as the
compiler can more easily move a dependent slow path out-of-line, as
demonstrated with the following compiled with GCC 15.2.0:
| void outline_preempt_enable_notrace(void)
| {
| preempt_enable_notrace();
| }
Before this patch:
| <outline_preempt_enable_notrace>:
| mrs x1, sp_el0
| ldr x0, [x1, #8]
| sub x0, x0, #0x1
| str w0, [x1, #8]
| cbz x0, 1f
| ldr x0, [x1, #8]
| cbnz x0, 2f
| 1: paciasp
| stp x29, x30, [sp, #-16]!
| mov x29, sp
| bl preempt_schedule_notrace
| ldp x29, x30, [sp], #16
| autiasp
| ret
| 2: ret
After this patch:
| <outline_preempt_enable_notrace>:
| mrs x0, sp_el0
| ldr w1, [x0, #8]
| sub w1, w1, #0x1
| str w1, [x0, #8]
| ldr x0, [x0, #8]
| cbz x0, 1f
| ret
| 1: paciasp
| stp x29, x30, [sp, #-16]!
| mov x29, sp
| bl preempt_schedule_notrace
| ldp x29, x30, [sp], #16
| autiasp
| ret
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/preempt.h | 29 +++++++++++------------------
1 file changed, 11 insertions(+), 18 deletions(-)
diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h
index 932ea4b620428..ca2ad1a8db095 100644
--- a/arch/arm64/include/asm/preempt.h
+++ b/arch/arm64/include/asm/preempt.h
@@ -55,30 +55,23 @@ static inline void __preempt_count_sub(int val)
WRITE_ONCE(current_thread_info()->preempt.count, pc);
}
-static inline bool __preempt_count_dec_and_test(void)
-{
- struct thread_info *ti = current_thread_info();
- u64 pc = READ_ONCE(ti->preempt_count);
-
- /* Update only the count field, leaving need_resched unchanged */
- WRITE_ONCE(ti->preempt.count, --pc);
-
- /*
- * If we wrote back all zeroes, then we're preemptible and in
- * need of a reschedule. Otherwise, we need to reload the
- * preempt_count in case the need_resched flag was cleared by an
- * interrupt occurring between the non-atomic READ_ONCE/WRITE_ONCE
- * pair.
- */
- return !pc || !READ_ONCE(ti->preempt_count);
-}
-
static inline bool should_resched(int preempt_offset)
{
u64 pc = READ_ONCE(current_thread_info()->preempt_count);
return pc == preempt_offset;
}
+static inline bool __preempt_count_dec_and_test(void)
+{
+ /*
+ * We must load the combined 'prempt_count' after decrementing
+ * 'preempt.count' as an interrupt could modify 'need_resched' before
+ * __preempt_count_sub() writes back to 'preempt.count'.
+ */
+ __preempt_count_sub(1);
+ return should_resched(0);
+}
+
#ifdef CONFIG_PREEMPTION
void preempt_schedule(void);
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 06/20] arm64: preempt: Treat should_resched() as unlikely
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (4 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 05/20] arm64: preempt: Simplify and optimize __preempt_count_dec_and_test() Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-04 17:04 ` [PATCH v2 07/20] arm64: ptrace: Always inline pt_regs_[read,write}_reg() Mark Rutland
` (13 subsequent siblings)
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
The arm64 implementation of should_resched() doesn't treat the need to
resched as unlikely, while all other implementations do, e.g.
* asm-generic since:
bdb43806589096ac ("sched: Extract the basic add/sub preempt_count modifiers")
* x86 since:
c2daa3bed53a8117 ("sched, x86: Provide a per-cpu preempt_count implementation")
* s390 since:
c360192bf4a8dc72 ("s390/preempt: move preempt_count to the lowcore")
Do the same for arm64.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/preempt.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h
index ca2ad1a8db095..3ef24ab8f3399 100644
--- a/arch/arm64/include/asm/preempt.h
+++ b/arch/arm64/include/asm/preempt.h
@@ -58,7 +58,7 @@ static inline void __preempt_count_sub(int val)
static inline bool should_resched(int preempt_offset)
{
u64 pc = READ_ONCE(current_thread_info()->preempt_count);
- return pc == preempt_offset;
+ return unlikely(pc == preempt_offset);
}
static inline bool __preempt_count_dec_and_test(void)
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 07/20] arm64: ptrace: Always inline pt_regs_[read,write}_reg()
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (5 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 06/20] arm64: preempt: Treat should_resched() as unlikely Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-04 17:04 ` [PATCH v2 08/20] arm64: percpu: Factor out percpu offset asm Mark Rutland
` (12 subsequent siblings)
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
In subsequent patches we'll want to use pt_regs_read_reg() and
pt_regs_write_reg() in noinstr code.
To ensure noinstr safety, both functions must be inlined into their
caller. As both functions are marked as 'inline' rather than
'__always_inline', that's not strictly guaranteed, and the compiler is
permitted to generate out-of-line copies.
Mark both functions as '__always_inline' to ensure this.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/ptrace.h | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index 39582511ad72f..a1aa668aad50e 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -268,7 +268,8 @@ static inline u64 regs_get_register(struct pt_regs *regs, unsigned int offset)
* Read a register given an architectural register index r.
* This handles the common case where 31 means XZR, not SP.
*/
-static inline unsigned long pt_regs_read_reg(const struct pt_regs *regs, int r)
+static __always_inline unsigned long
+pt_regs_read_reg(const struct pt_regs *regs, int r)
{
return (r == 31) ? 0 : regs->regs[r];
}
@@ -277,8 +278,8 @@ static inline unsigned long pt_regs_read_reg(const struct pt_regs *regs, int r)
* Write a register given an architectural register index r.
* This handles the common case where 31 means XZR, not SP.
*/
-static inline void pt_regs_write_reg(struct pt_regs *regs, int r,
- unsigned long val)
+static __always_inline void
+pt_regs_write_reg(struct pt_regs *regs, int r, unsigned long val)
{
if (r != 31)
regs->regs[r] = val;
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 08/20] arm64: percpu: Factor out percpu offset asm
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (6 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 07/20] arm64: ptrace: Always inline pt_regs_[read,write}_reg() Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-04 17:04 ` [PATCH v2 09/20] arm64: gpr-num: Add wxN aliases for wN registers Mark Rutland
` (11 subsequent siblings)
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
Depending on whether the kernel runs at EL1 or EL2, the percpu offset is
stored in either TPIDR_EL1 or TPIDR_EL2, and __kern_my_cpu_offset() uses
an ALTERNATIVE() sequence to read the relevant TPIDR_ELx.
In subsequent patches we'll need to read the percpu offset in other
assembly sequences. Factor the ALTERNATIVE sequence out of
__kern_my_cpu_offset() into a new __KERN_ASM_CPU_OFFSET() macro so that
we can share the code sequence.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/percpu.h | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 31193bcf89a2b..29f20c8748fe1 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -29,6 +29,11 @@ static inline unsigned long __hyp_my_cpu_offset(void)
return read_sysreg(tpidr_el2);
}
+#define __KERN_ASM_CPU_OFFSET(dst) \
+ ALTERNATIVE("mrs " dst ", tpidr_el1", \
+ "mrs " dst ", tpidr_el2", \
+ ARM64_HAS_VIRT_HOST_EXTN)
+
static inline unsigned long __kern_my_cpu_offset(void)
{
unsigned long off;
@@ -37,11 +42,11 @@ static inline unsigned long __kern_my_cpu_offset(void)
* We want to allow caching the value, so avoid using volatile and
* instead use a fake stack read to hazard against barrier().
*/
- asm(ALTERNATIVE("mrs %0, tpidr_el1",
- "mrs %0, tpidr_el2",
- ARM64_HAS_VIRT_HOST_EXTN)
- : "=r" (off) :
- "Q" (*(const unsigned long *)current_stack_pointer));
+ asm(
+ __KERN_ASM_CPU_OFFSET("%0")
+ : "=r" (off)
+ : "Q" (*(const unsigned long *)current_stack_pointer)
+ );
return off;
}
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 09/20] arm64: gpr-num: Add wxN aliases for wN registers
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (7 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 08/20] arm64: percpu: Factor out percpu offset asm Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-04 17:04 ` [PATCH v2 10/20] arm64: gpr-num: add __GPR_NUM() helper Mark Rutland
` (10 subsequent siblings)
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
Some A64 instructions require a W register name for specific register
arguments, and will not accept an X register name. For example:
STRH <Wt>, [<Xn|SP>, (<Wm>|<Xm>){, <extend> {<amount>}}]
This is painful when writing macros for inline assembly, especially
where a single register is used as both an X register and a W register.
We solved this for plain assembly by adding a wxN alias for each xN
register in commit:
4c4dcd3541f83d21 ("arm64: assembler: introduce wxN aliases for wN registers")
Add the same facility for inline assembly, with a new
__DEFINE_ASM_GPR_ALIASES macro.
As with the other macros in gpr-num.h, different escaping is necessary
for inline vs non-inline assembly, so we can't share a single definition
that we stringify().
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/gpr-num.h | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/arch/arm64/include/asm/gpr-num.h b/arch/arm64/include/asm/gpr-num.h
index a114e4f8209b0..240cd25d55c5a 100644
--- a/arch/arm64/include/asm/gpr-num.h
+++ b/arch/arm64/include/asm/gpr-num.h
@@ -21,6 +21,11 @@
" .equ .L__gpr_num_xzr, 31\n" \
" .equ .L__gpr_num_wzr, 31\n"
+#define __DEFINE_ASM_GPR_ALIASES \
+" .irp n,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30\n" \
+" wx\\n .req w\\n\n" \
+" .endr\n"
+
#endif /* __ASSEMBLER__ */
#endif /* __ASM_GPR_NUM_H */
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 10/20] arm64: gpr-num: add __GPR_NUM() helper
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (8 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 09/20] arm64: gpr-num: Add wxN aliases for wN registers Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-04 17:04 ` [PATCH v2 11/20] arm64: entry: sdei: Restore all clobberable GPRs Mark Rutland
` (9 subsequent siblings)
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
The gpr-num infrastructure requires users to concatenate the register
name with a long prefix in order to get a symbol that evaluates to a
compile-time constant. Doing this manually is error-prone and painful to
read.
Add a new __GPR_NUM() helper which wraps the concatenation.
For the moment I've left existing open-coded concatenation as-is. In
particular, the DEFINE_MRS_S() and DEFINE_MSR_S() macros have awkward
stringification requirements and would require invasive changes that
outweigh the benefit. I have plans to rework those in the near future to
remove the need to use gpr-num values.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/gpr-num.h | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/arm64/include/asm/gpr-num.h b/arch/arm64/include/asm/gpr-num.h
index 240cd25d55c5a..30fb8a67b19f0 100644
--- a/arch/arm64/include/asm/gpr-num.h
+++ b/arch/arm64/include/asm/gpr-num.h
@@ -11,6 +11,8 @@
.equ .L__gpr_num_xzr, 31
.equ .L__gpr_num_wzr, 31
+#define __GPR_NUM(gpr) (.L__gpr_num_##gpr)
+
#else /* __ASSEMBLER__ */
#define __DEFINE_ASM_GPR_NUMS \
@@ -26,6 +28,8 @@
" wx\\n .req w\\n\n" \
" .endr\n"
+#define __GPR_NUM(gpr) "(.L__gpr_num_" gpr ")"
+
#endif /* __ASSEMBLER__ */
#endif /* __ASM_GPR_NUM_H */
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 11/20] arm64: entry: sdei: Restore all clobberable GPRs
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (9 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 10/20] arm64: gpr-num: add __GPR_NUM() helper Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-04 17:04 ` [PATCH v2 12/20] arm64: entry: sdei: Make 'tsk' available Mark Rutland
` (8 subsequent siblings)
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
Currently __sdei_asm_handler() avoids restoring x20 to x27, as these are
not be clobbered by existing code. This is correct today, but it makes
it awkward to rework __sdei_asm_handler(). Given the rarity of SDEI
events, restoring x20 to x27 should not have a measurable impact.
Restore x20 to x27 when returning from an SDEI event handler. This will
free up those GPRs for usage in __sdei_asm_handler() prior to return.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/kernel/entry.S | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index e0db14e9c843a..eadf4e74e72a8 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -1048,10 +1048,14 @@ SYM_CODE_START(__sdei_asm_handler)
bl __sdei_handler
msr sp_el0, x28
- /* restore regs >x17 that we clobbered */
+ /* restore regs >x17 that firmware won't restore */
mov x4, x19 // keep x4 for __sdei_asm_exit_trampoline
- ldp x28, x29, [x4, #SDEI_EVENT_INTREGS + 16 * 14]
ldp x18, x19, [x4, #SDEI_EVENT_INTREGS + 16 * 9]
+ ldp x20, x21, [x4, #SDEI_EVENT_INTREGS + 16 * 10]
+ ldp x22, x23, [x4, #SDEI_EVENT_INTREGS + 16 * 11]
+ ldp x24, x25, [x4, #SDEI_EVENT_INTREGS + 16 * 12]
+ ldp x22, x27, [x4, #SDEI_EVENT_INTREGS + 16 * 13]
+ ldp x28, x29, [x4, #SDEI_EVENT_INTREGS + 16 * 14]
ldp lr, x1, [x4, #SDEI_EVENT_INTREGS + S_LR]
mov sp, x1
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 12/20] arm64: entry: sdei: Make 'tsk' available
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (10 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 11/20] arm64: entry: sdei: Restore all clobberable GPRs Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-04 17:04 ` [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops Mark Rutland
` (7 subsequent siblings)
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
For regular entry/exit sequences, we use 'tsk' (x28) to hold the current
task pointer, allowing this to be used by various assembly macros. The
SDEI entry/exit sequence uses x28 to hold the value of the interrupted
sp_el0, and doesn't retain the current task pointer in a register.
These differences makes it awkward to share assembly macros across
regular entry/exit and SDEI entry/exit, and risk surprises.
Align the SDEI entry/exit sequence with regular entry/exit, keeping the
current task pointer in 'tsk', and preserving the interrupted sp_el0 in
another (callee-saved) register. I've used x20 as x19 was already in
used for the relevant sdei_registered_event, and these are the
lowest-numbered registers above x18 (which we must preserve for shadow
call stack).
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/kernel/entry.S | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index eadf4e74e72a8..1b4cd70515550 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -1027,11 +1027,12 @@ SYM_CODE_START(__sdei_asm_handler)
/*
* We may have interrupted userspace, or a guest, or exit-from or
- * return-to either of these. We can't trust sp_el0, restore it.
+ * return-to either of these. Preserve the interrupted sp_el0, and
+ * initialize sp_el0 to the current task.
*/
- mrs x28, sp_el0
- ldr_this_cpu dst=x0, sym=__entry_task, tmp=x1
- msr sp_el0, x0
+ mrs x20, sp_el0
+ ldr_this_cpu dst=tsk, sym=__entry_task, tmp=x1
+ msr sp_el0, tsk
/* If we interrupted the kernel point to the previous stack/frame. */
and x0, x3, #0xc
@@ -1047,7 +1048,7 @@ SYM_CODE_START(__sdei_asm_handler)
mov x1, x19
bl __sdei_handler
- msr sp_el0, x28
+ msr sp_el0, x20
/* restore regs >x17 that firmware won't restore */
mov x4, x19 // keep x4 for __sdei_asm_exit_trampoline
ldp x18, x19, [x4, #SDEI_EVENT_INTREGS + 16 * 9]
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (11 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 12/20] arm64: entry: sdei: Make 'tsk' available Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-04 22:45 ` Pedro Falcato
2026-08-05 6:45 ` David Hildenbrand (Arm)
2026-08-04 17:04 ` [PATCH v2 14/20] arm64: percpu: Implement preemptible read/write ops Mark Rutland
` (6 subsequent siblings)
19 siblings, 2 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
Currently arm64's this_cpu_*() ops transiently disable preemption in
order to guarantee that the address generation and memory access(es)
occur on the same CPU.
Transiently disabling preemption can be expensive. When re-enabling
preemption it is necessary to make a conditional function call to
preempt_schedule[_notrace]() in order to handle the rare case that the
task needs to be rescheduled. The potential function call has a number
of negative effects on code generation (e.g. due to the need to create a
stack frame and spill registers), and the conditionality can result in
poor code generation and/or poor branch prediction.
This patch adds infrastructure for a scheme where this_cpu_*() ops do
not need to transiently disable preemption, avoiding the negative
impacts described above. Individual operations will be converted in
subsequent patches.
Each operation registers a critical section during which the exception
return code will adjust the offset and addresses if preemption occurs
mid-sequence. The critical section is registered/unregistered with a
small prologue and epilogue which encodes three distinct GPRRs (<pcp>,
<off>, <addr>) into a new thread_info::pcp_gprs field:
// Prologue. Enable fixups for <off> and <addr>.
mrs <tsk>, sp_el0
mov <tmp>, #__VAL_PCPU_GPRS(<pcp>, <off>, <addr>)
strh <tmp>, [<tsk>, #TSK_TI_PCPU_GPRS]
// Generate cpu-specific address
mrs <off>, TPIDR_ELx
add <addr>, <pcp>, <off>
// Perform access sequence
ldr <val>, [<addr>]
// Epilogue. Disable fixups
strh wzr, [<tsk>, #TSK_TI_PCPU_GPRS]
If an exception is taken from within the critical section, the exception
return code will adjust <off> to be the current CPU's offset, and will
adjust <addr> to be (<pcp> + <off>). Distinct registers are used for
<pcp>, <off>, and <addr>, so that the fixup can be applied safely at any
point during the critical section.
To ensure that this_cpu_*() operations within exception handlers work
correctly and do not corrupt state, thread_info::pcpu_gprs is saved
into a new pt_regs::pcpu_gprs field upon exception entry, and restored
upon exception return.
Looking at a simple this_cpu_operation:
| void outline_this_cpu_add_u64(u64 __percpu *p, u64 v)
| {
| this_cpu_add(*p, v);
| }
Atop v7.2-rc4, with GCC 15.2.0 and defconfig, this is compiled as:
| <outline_this_cpu_add_u64>:
| paciasp
| stp x29, x30, [sp, #-16]!
| mrs x2, sp_el0
| mov x29, sp
| ldr w3, [x2, #8]
| add w3, w3, #0x1
| str w3, [x2, #8]
| mrs x3, tpidr_el1
| add x0, x0, x3
| 1: ldxr x5, [x0]
| add x5, x5, x1
| stxr w4, x5, [x0]
| cbnz w4, 1b
| ldr x0, [x2, #8]
| sub x0, x0, #0x1
| str w0, [x2, #8]
| cbz x0, 2f
| ldr x0, [x2, #8]
| cbnz x0, 3f
| 2: bl preempt_schedule_notrace
| 3: ldp x29, x30, [sp], #16
| autiasp
| ret
With the scheme added in this patch, this can be compiled as:
| <outline_this_cpu_add_u64>:
| mrs x2, sp_el0
| mov x4, #0xc80 // __VAL_PCPU_GPRS(x0, x4, x3)
| strh w4, [x2, #20]
| mrs x4, tpidr_el1
| add x3, x0, x4
| 1: ldxr x6, [x3]
| add x6, x6, x1
| stxr w5, x6, [x3]
| cbnz w5, 1b
| strh wzr, [x2, #20]
| ret
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/percpu.h | 71 ++++++++++++++++++++++++++++
arch/arm64/include/asm/ptrace.h | 5 ++
arch/arm64/include/asm/thread_info.h | 1 +
arch/arm64/kernel/asm-offsets.c | 2 +
arch/arm64/kernel/entry-common.c | 38 +++++++++++++++
arch/arm64/kernel/entry.S | 19 ++++++++
6 files changed, 136 insertions(+)
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 29f20c8748fe1..8b4c9ea05d4e6 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -5,10 +5,13 @@
#ifndef __ASM_PERCPU_H
#define __ASM_PERCPU_H
+#include <linux/bits.h>
#include <linux/preempt.h>
+#include <linux/stringify.h>
#include <asm/alternative.h>
#include <asm/cmpxchg.h>
+#include <asm/gpr-num.h>
#include <asm/stack_pointer.h>
#include <asm/sysreg.h>
@@ -51,6 +54,74 @@ static inline unsigned long __kern_my_cpu_offset(void)
return off;
}
+#define PCPU_GPR_PCP_SHIFT 0
+#define PCPU_GPR_PCP GENMASK(4, 0)
+#define PCPU_GPR_OFF_SHIFT 5
+#define PCPU_GPR_OFF GENMASK(9, 5)
+#define PCPU_GPR_ADDR_SHIFT 10
+#define PCPU_GPR_ADDR GENMASK(14, 10)
+
+#define __VAL_PCPU_GPRS(pcp, off, addr) \
+ "(" \
+ "(" __GPR_NUM(pcp) " << " __stringify(PCPU_GPR_PCP_SHIFT) ") | " \
+ "(" __GPR_NUM(off) " << " __stringify(PCPU_GPR_OFF_SHIFT) ") | " \
+ "(" __GPR_NUM(addr) " << " __stringify(PCPU_GPR_ADDR_SHIFT) ")" \
+ ")"
+
+#define __ASSERT_PCPU_GPRS_DISTINCT(pcp, off, addr) \
+ ".if (" \
+ "(" __GPR_NUM(pcp) " == " __GPR_NUM(off) ") || " \
+ "(" __GPR_NUM(pcp) " == " __GPR_NUM(addr) ") || " \
+ "(" __GPR_NUM(off) " == " __GPR_NUM(addr) ")" \
+ " )\n" \
+ ".error \"PCPU GPRS overlap: {" pcp "," off "," addr "}\"\n" \
+ ".endif\n"
+
+#define ____PCPU_GPRS_BEGIN(gprs, pcp, off, addr) \
+ "// ____PCPU_GPRS_BEGIN(" gprs ", " pcp ", " off ", " addr")\n" \
+ __DEFINE_ASM_GPR_NUMS \
+ __DEFINE_ASM_GPR_ALIASES \
+ __ASSERT_PCPU_GPRS_DISTINCT(pcp, off, addr) \
+ " mov w" off ", #" __VAL_PCPU_GPRS(pcp, off, addr) "\n" \
+ " strh w" off ", " gprs "\n" \
+ __KERN_ASM_CPU_OFFSET(off) "\n"
+
+/*
+ * Begin a PCPU GPR critical section which requires <addr> (and <off>).
+ *
+ * At the start of the critical section, and upon any (preemptible) exception
+ * until __PCPU_GPRS_END():
+ * - <off> will be set to the current CPU's percpu offset.
+ * - <addr> will be set to <pcp> + <off>.
+ *
+ * The <pcp>, <off>, and <addr> registers must be distinct GPRs.
+ *
+ * <gprs> must be '¤t_thread_info()->gprs', as a memory operand which can
+ * be written both at the start and end of the critical section (e.g. using
+ * "=Qo" constraints).
+ */
+#define __PCPU_GPRS_BEGIN(gprs, pcp, off, addr) \
+ ____PCPU_GPRS_BEGIN(gprs, pcp, off, addr) \
+ " add " addr ", " pcp ", " off "\n"
+
+/*
+ * Begin a PCPU GPR critical section which only requires <off> and does not
+ * require <addr>.
+ *
+ * This is only for operations that can use register-offset addressing,
+ * e.g. STR <Xt>, [<Xn>, <Xm].
+ *
+ * All other details are the same as __PCPU_GPRS_BEGIN().
+ */
+#define __PCPU_GPRS_BEGIN_OFFSET(gprs, pcp, off) \
+ ____PCPU_GPRS_BEGIN(gprs, pcp, off, "xzr")
+
+/*
+ * End a PCU GPR critical section.
+ */
+#define __PCPU_GPRS_END(gprs) \
+ " strh wzr, " gprs "\n"
+
#ifdef __KVM_NVHE_HYPERVISOR__
#define __my_cpu_offset __hyp_my_cpu_offset()
#else
diff --git a/arch/arm64/include/asm/ptrace.h b/arch/arm64/include/asm/ptrace.h
index a1aa668aad50e..962013df20c7a 100644
--- a/arch/arm64/include/asm/ptrace.h
+++ b/arch/arm64/include/asm/ptrace.h
@@ -169,6 +169,11 @@ struct pt_regs {
u64 sdei_ttbr1;
struct frame_record_meta stackframe;
+
+ u16 pcpu_gprs;
+ u16 __unused1;
+ u32 __unused2;
+ u64 __unused3;
};
/* For correct stack alignment, pt_regs has to be a multiple of 16 bytes. */
diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
index 5d7fe3e153c85..6db5fa72211d1 100644
--- a/arch/arm64/include/asm/thread_info.h
+++ b/arch/arm64/include/asm/thread_info.h
@@ -46,6 +46,7 @@ struct thread_info {
u64 mpam_partid_pmg;
#endif
u32 cpu;
+ u16 pcpu_gprs;
};
#define thread_saved_pc(tsk) \
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index b6367ff3a49ca..bde09e9fe3e64 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -36,6 +36,7 @@ int main(void)
DEFINE(TSK_TI_SCS_BASE, offsetof(struct task_struct, thread_info.scs_base));
DEFINE(TSK_TI_SCS_SP, offsetof(struct task_struct, thread_info.scs_sp));
#endif
+ DEFINE(TSK_TI_PCPU_GPRS, offsetof(struct task_struct, thread_info.pcpu_gprs));
DEFINE(TSK_STACK, offsetof(struct task_struct, stack));
#ifdef CONFIG_STACKPROTECTOR
DEFINE(TSK_STACK_CANARY, offsetof(struct task_struct, stack_canary));
@@ -78,6 +79,7 @@ int main(void)
DEFINE(S_PMR, offsetof(struct pt_regs, pmr));
DEFINE(S_STACKFRAME, offsetof(struct pt_regs, stackframe));
DEFINE(S_STACKFRAME_TYPE, offsetof(struct pt_regs, stackframe.type));
+ DEFINE(S_PCPU_GPRS, offsetof(struct pt_regs, pcpu_gprs));
DEFINE(PT_REGS_SIZE, sizeof(struct pt_regs));
BLANK();
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index ceb4eb11232a6..5bba1359280c3 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -25,12 +25,49 @@
#include <asm/irq_regs.h>
#include <asm/kprobes.h>
#include <asm/mmu.h>
+#include <asm/percpu.h>
#include <asm/processor.h>
#include <asm/sdei.h>
#include <asm/stacktrace.h>
#include <asm/sysreg.h>
#include <asm/system_misc.h>
+/*
+ * Where the context being returned to had an active percpu GPR critical
+ * section, ensure that the offset and address GPRs are updated to match the
+ * current CPU.
+ *
+ * For simplicity we always update the GPRs when a critical section is active
+ * and preemption was *possible*, regardless of whether preemption actually
+ * occurred. Where preemption did not occur, the updates are redundant but not
+ * harmful.
+ */
+static __always_inline void irqentry_exit_pcpu_adjust(struct pt_regs *regs)
+{
+ int reg_pcp, reg_off, reg_addr;
+ unsigned long pcp, off, addr;
+ u16 gprs = regs->pcpu_gprs;
+
+ /*
+ * Zero means no active PCPU GPRs. As the PCPU GPRs must be distinct,
+ * a PCPU critical section cannot possibly use {x0,x0,x0}.
+ */
+ if (likely(!gprs))
+ return;
+
+ reg_pcp = FIELD_GET(PCPU_GPR_PCP, gprs);
+ reg_off = FIELD_GET(PCPU_GPR_OFF, gprs);
+ reg_addr = FIELD_GET(PCPU_GPR_ADDR, gprs);
+
+ pcp = pt_regs_read_reg(regs, reg_pcp);
+
+ off = __kern_my_cpu_offset();
+ pt_regs_write_reg(regs, reg_off, off);
+
+ addr = pcp + off;
+ pt_regs_write_reg(regs, reg_addr, addr);
+}
+
/*
* Handle IRQ/context state management when entering from kernel mode.
* Before this function is called it is not safe to call regular kernel code,
@@ -58,6 +95,7 @@ static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs,
local_irq_disable();
irqentry_exit_to_kernel_mode_preempt(regs, state);
local_daif_mask();
+ irqentry_exit_pcpu_adjust(regs);
mte_check_tfsr_exit();
irqentry_exit_to_kernel_mode_after_preempt(regs, state);
}
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index 1b4cd70515550..567d8096b87d9 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -194,6 +194,17 @@ alternative_cb_end
#endif
.endm
+ .macro pcpu_gprs_entry, tsk:req, regs:req, tmp:req
+ ldrh w\tmp, [\tsk, #TSK_TI_PCPU_GPRS]
+ strh w\tmp, [\regs, #S_PCPU_GPRS]
+ strh wzr, [\tsk, #TSK_TI_PCPU_GPRS]
+ .endm
+
+ .macro pcpu_gprs_exit, tsk:req, regs:req, tmp:req
+ ldrh w\tmp, [\regs, #S_PCPU_GPRS]
+ strh w\tmp, [\tsk, #TSK_TI_PCPU_GPRS]
+ .endm
+
.macro kernel_entry, el, regsize = 64
.if \el == 0
alternative_insn nop, SET_PSTATE_DIT(1), ARM64_HAS_DIT
@@ -277,6 +288,7 @@ alternative_else_nop_endif
.else
add x21, sp, #PT_REGS_SIZE
get_current_task tsk
+ pcpu_gprs_entry tsk, sp, x0
.endif /* \el == 0 */
mrs x22, elr_el1
mrs x23, spsr_el1
@@ -335,6 +347,7 @@ alternative_else_nop_endif
.macro kernel_exit, el
.if \el != 0
disable_daif
+ pcpu_gprs_exit tsk, sp, x0
.endif
#ifdef CONFIG_ARM64_PSEUDO_NMI
@@ -1044,10 +1057,16 @@ SYM_CODE_START(__sdei_asm_handler)
stp x29, x4, [sp, #-16]!
mov x29, sp
+ add x16, x19, #SDEI_EVENT_INTREGS
+ pcpu_gprs_entry tsk, x16, x17
+
add x0, x19, #SDEI_EVENT_INTREGS
mov x1, x19
bl __sdei_handler
+ add x16, x19, #SDEI_EVENT_INTREGS
+ pcpu_gprs_exit tsk, x16, x17
+
msr sp_el0, x20
/* restore regs >x17 that firmware won't restore */
mov x4, x19 // keep x4 for __sdei_asm_exit_trampoline
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* Re: [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops
2026-08-04 17:04 ` [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops Mark Rutland
@ 2026-08-04 22:45 ` Pedro Falcato
2026-08-05 10:27 ` David Laight
2026-08-05 6:45 ` David Hildenbrand (Arm)
1 sibling, 1 reply; 39+ messages in thread
From: Pedro Falcato @ 2026-08-04 22:45 UTC (permalink / raw)
To: Mark Rutland
Cc: vladimir.murzin, ryan.roberts, peterz, catalin.marinas,
david.laight.linux, stable, ruanjinjie, james.morse, yang, cl,
maz, david, ljs, will, ardb, linux-arm-kernel
On Tue, Aug 04, 2026 at 06:04:56PM +0100, Mark Rutland wrote:
> Currently arm64's this_cpu_*() ops transiently disable preemption in
> order to guarantee that the address generation and memory access(es)
> occur on the same CPU.
>
> Transiently disabling preemption can be expensive. When re-enabling
> preemption it is necessary to make a conditional function call to
> preempt_schedule[_notrace]() in order to handle the rare case that the
> task needs to be rescheduled. The potential function call has a number
> of negative effects on code generation (e.g. due to the need to create a
> stack frame and spill registers), and the conditionality can result in
> poor code generation and/or poor branch prediction.
>
> This patch adds infrastructure for a scheme where this_cpu_*() ops do
> not need to transiently disable preemption, avoiding the negative
> impacts described above. Individual operations will be converted in
> subsequent patches.
>
> Each operation registers a critical section during which the exception
> return code will adjust the offset and addresses if preemption occurs
> mid-sequence. The critical section is registered/unregistered with a
> small prologue and epilogue which encodes three distinct GPRRs (<pcp>,
> <off>, <addr>) into a new thread_info::pcp_gprs field:
>
> // Prologue. Enable fixups for <off> and <addr>.
> mrs <tsk>, sp_el0
> mov <tmp>, #__VAL_PCPU_GPRS(<pcp>, <off>, <addr>)
> strh <tmp>, [<tsk>, #TSK_TI_PCPU_GPRS]
>
> // Generate cpu-specific address
> mrs <off>, TPIDR_ELx
> add <addr>, <pcp>, <off>
>
> // Perform access sequence
> ldr <val>, [<addr>]
>
> // Epilogue. Disable fixups
> strh wzr, [<tsk>, #TSK_TI_PCPU_GPRS]
>
> If an exception is taken from within the critical section, the exception
> return code will adjust <off> to be the current CPU's offset, and will
> adjust <addr> to be (<pcp> + <off>). Distinct registers are used for
> <pcp>, <off>, and <addr>, so that the fixup can be applied safely at any
> point during the critical section.
>
> To ensure that this_cpu_*() operations within exception handlers work
> correctly and do not corrupt state, thread_info::pcpu_gprs is saved
> into a new pt_regs::pcpu_gprs field upon exception entry, and restored
> upon exception return.
>
> Looking at a simple this_cpu_operation:
>
> | void outline_this_cpu_add_u64(u64 __percpu *p, u64 v)
> | {
> | this_cpu_add(*p, v);
> | }
>
> Atop v7.2-rc4, with GCC 15.2.0 and defconfig, this is compiled as:
>
> | <outline_this_cpu_add_u64>:
> | paciasp
> | stp x29, x30, [sp, #-16]!
> | mrs x2, sp_el0
> | mov x29, sp
> | ldr w3, [x2, #8]
> | add w3, w3, #0x1
> | str w3, [x2, #8]
> | mrs x3, tpidr_el1
> | add x0, x0, x3
> | 1: ldxr x5, [x0]
> | add x5, x5, x1
> | stxr w4, x5, [x0]
> | cbnz w4, 1b
> | ldr x0, [x2, #8]
> | sub x0, x0, #0x1
> | str w0, [x2, #8]
> | cbz x0, 2f
> | ldr x0, [x2, #8]
> | cbnz x0, 3f
> | 2: bl preempt_schedule_notrace
> | 3: ldp x29, x30, [sp], #16
> | autiasp
> | ret
>
> With the scheme added in this patch, this can be compiled as:
>
> | <outline_this_cpu_add_u64>:
> | mrs x2, sp_el0
> | mov x4, #0xc80 // __VAL_PCPU_GPRS(x0, x4, x3)
> | strh w4, [x2, #20]
> | mrs x4, tpidr_el1
> | add x3, x0, x4
> | 1: ldxr x6, [x3]
> | add x6, x6, x1
> | stxr w5, x6, [x3]
> | cbnz w5, 1b
> | strh wzr, [x2, #20]
> | ret
I think I had an Interesting Idea(tm) while reading the per-cpu discussion
in linux-mm. In case the 3 instruction preamble is too expensive:
1) Pass -ffixed-x18 (this natively conflicts with SHADOW_CALL_STACK.
SHADOW_CALL_STACK is already not-optimal codegen wise, so maybe not a big deal).
2) arm64 kernel bits will use x18 as a cheap task flags register
3) #define TASK_KRSEQ (1 << 0)
4) Switching into the krseq mode is just a matter of toggling the bit in x18, so
orr x18, x18, #TASK_KRSEQ
a single instruction.
5) Switching off is just a matter of clearing the bit in x18, so:
and x18, x18, #~TASK_KRSEQ
6) On the preempt side we keep the krseq tables in memory, and do a sort of lookup
(binary search sounds easiest?) on them. But _only_ if x18 TASK_KRSEQ is set.
This penalises unlucky preempts but keeps fast paths maximally fast.
7) entry points of course get to clear it after saving it
The end result would look something like:
| <outline_this_cpu_add_u64>:
| orr x18, x18, #TASK_KRSEQ
| mrs x4, tpidr_el1
| add x3, x0, x4
| 1: ldxr x6, [x3]
| add x6, x6, x1
| stxr w5, x6, [x3]
| cbnz w5, 1b
| 2:
| and x18, x18, #~TASK_KRSEQ
| ret
| .pushsection .data.krseq
| .word 1b
| .word 2b
| .word whateverelse
| .popsection
This of course precludes the use of x18 for the compiler, so it would
require careful benchmarking in case it negatively affects codegen too much.
But it avoids any sort of extraneous stores in the fast path.
Other architectures could do similar as long as they have interesting ways of
signaling this using solely the register set.
Anyway, just throwing it out there in case this can actually make a
difference & rings some bells on people smarter than me :)
--
Pedro
^ permalink raw reply [flat|nested] 39+ messages in thread* Re: [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops
2026-08-04 22:45 ` Pedro Falcato
@ 2026-08-05 10:27 ` David Laight
2026-08-05 12:50 ` Pedro Falcato
0 siblings, 1 reply; 39+ messages in thread
From: David Laight @ 2026-08-05 10:27 UTC (permalink / raw)
To: Pedro Falcato
Cc: Mark Rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, ruanjinjie, stable, james.morse, yang, cl, maz,
david, ljs, will, ardb, linux-arm-kernel
On Tue, 4 Aug 2026 23:45:56 +0100
Pedro Falcato <pfalcato@suse.de> wrote:
> On Tue, Aug 04, 2026 at 06:04:56PM +0100, Mark Rutland wrote:
> > Currently arm64's this_cpu_*() ops transiently disable preemption in
> > order to guarantee that the address generation and memory access(es)
> > occur on the same CPU.
> >
> > Transiently disabling preemption can be expensive. When re-enabling
> > preemption it is necessary to make a conditional function call to
> > preempt_schedule[_notrace]() in order to handle the rare case that the
> > task needs to be rescheduled. The potential function call has a number
> > of negative effects on code generation (e.g. due to the need to create a
> > stack frame and spill registers), and the conditionality can result in
> > poor code generation and/or poor branch prediction.
> >
> > This patch adds infrastructure for a scheme where this_cpu_*() ops do
> > not need to transiently disable preemption, avoiding the negative
> > impacts described above. Individual operations will be converted in
> > subsequent patches.
> >
> > Each operation registers a critical section during which the exception
> > return code will adjust the offset and addresses if preemption occurs
> > mid-sequence. The critical section is registered/unregistered with a
> > small prologue and epilogue which encodes three distinct GPRRs (<pcp>,
> > <off>, <addr>) into a new thread_info::pcp_gprs field:
> >
> > // Prologue. Enable fixups for <off> and <addr>.
> > mrs <tsk>, sp_el0
> > mov <tmp>, #__VAL_PCPU_GPRS(<pcp>, <off>, <addr>)
> > strh <tmp>, [<tsk>, #TSK_TI_PCPU_GPRS]
> >
> > // Generate cpu-specific address
> > mrs <off>, TPIDR_ELx
> > add <addr>, <pcp>, <off>
> >
> > // Perform access sequence
> > ldr <val>, [<addr>]
> >
> > // Epilogue. Disable fixups
> > strh wzr, [<tsk>, #TSK_TI_PCPU_GPRS]
> >
> > If an exception is taken from within the critical section, the exception
> > return code will adjust <off> to be the current CPU's offset, and will
> > adjust <addr> to be (<pcp> + <off>). Distinct registers are used for
> > <pcp>, <off>, and <addr>, so that the fixup can be applied safely at any
> > point during the critical section.
> >
> > To ensure that this_cpu_*() operations within exception handlers work
> > correctly and do not corrupt state, thread_info::pcpu_gprs is saved
> > into a new pt_regs::pcpu_gprs field upon exception entry, and restored
> > upon exception return.
> >
> > Looking at a simple this_cpu_operation:
> >
> > | void outline_this_cpu_add_u64(u64 __percpu *p, u64 v)
> > | {
> > | this_cpu_add(*p, v);
> > | }
...
> I think I had an Interesting Idea(tm) while reading the per-cpu discussion
> in linux-mm. In case the 3 instruction preamble is too expensive:
>
> 1) Pass -ffixed-x18 (this natively conflicts with SHADOW_CALL_STACK.
> SHADOW_CALL_STACK is already not-optimal codegen wise, so maybe not a big deal).
> 2) arm64 kernel bits will use x18 as a cheap task flags register
> 3) #define TASK_KRSEQ (1 << 0)
> 4) Switching into the krseq mode is just a matter of toggling the bit in x18, so
> orr x18, x18, #TASK_KRSEQ
> a single instruction.
> 5) Switching off is just a matter of clearing the bit in x18, so:
> and x18, x18, #~TASK_KRSEQ
> 6) On the preempt side we keep the krseq tables in memory, and do a sort of lookup
> (binary search sounds easiest?) on them. But _only_ if x18 TASK_KRSEQ is set.
> This penalises unlucky preempts but keeps fast paths maximally fast.
> 7) entry points of course get to clear it after saving it
>
> The end result would look something like:
> | <outline_this_cpu_add_u64>:
> | orr x18, x18, #TASK_KRSEQ
> | mrs x4, tpidr_el1
> | add x3, x0, x4
> | 1: ldxr x6, [x3]
> | add x6, x6, x1
> | stxr w5, x6, [x3]
> | cbnz w5, 1b
> | 2:
> | and x18, x18, #~TASK_KRSEQ
> | ret
> | .pushsection .data.krseq
> | .word 1b
> | .word 2b
> | .word whateverelse
> | .popsection
>
> This of course precludes the use of x18 for the compiler, so it would
> require careful benchmarking in case it negatively affects codegen too much.
> But it avoids any sort of extraneous stores in the fast path.
The extra stores are independent of the main instruction flow.
On a multi-issue (and especially out-of-order) cpu they are pretty much
likely to be noise.
The biggest cost is likely to be in the I-cache and instruction decoders.
Put a memory read in the 'main' path and the few clocks needed for the
D-cache read are likely to dominate - so the writes to the pcp_gprs
are actually likely to be free.
OTOH stealing a gpr for some flags will cost everwhere.
David
>
> Other architectures could do similar as long as they have interesting ways of
> signaling this using solely the register set.
>
> Anyway, just throwing it out there in case this can actually make a
> difference & rings some bells on people smarter than me :)
>
^ permalink raw reply [flat|nested] 39+ messages in thread* Re: [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops
2026-08-05 10:27 ` David Laight
@ 2026-08-05 12:50 ` Pedro Falcato
0 siblings, 0 replies; 39+ messages in thread
From: Pedro Falcato @ 2026-08-05 12:50 UTC (permalink / raw)
To: David Laight
Cc: Mark Rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, ruanjinjie, stable, james.morse, yang, cl, maz,
david, ljs, will, ardb, linux-arm-kernel
On Wed, Aug 05, 2026 at 11:27:07AM +0100, David Laight wrote:
> On Tue, 4 Aug 2026 23:45:56 +0100
> Pedro Falcato <pfalcato@suse.de> wrote:
>
> > On Tue, Aug 04, 2026 at 06:04:56PM +0100, Mark Rutland wrote:
> > > Currently arm64's this_cpu_*() ops transiently disable preemption in
> > > order to guarantee that the address generation and memory access(es)
> > > occur on the same CPU.
> > >
> > > Transiently disabling preemption can be expensive. When re-enabling
> > > preemption it is necessary to make a conditional function call to
> > > preempt_schedule[_notrace]() in order to handle the rare case that the
> > > task needs to be rescheduled. The potential function call has a number
> > > of negative effects on code generation (e.g. due to the need to create a
> > > stack frame and spill registers), and the conditionality can result in
> > > poor code generation and/or poor branch prediction.
> > >
> > > This patch adds infrastructure for a scheme where this_cpu_*() ops do
> > > not need to transiently disable preemption, avoiding the negative
> > > impacts described above. Individual operations will be converted in
> > > subsequent patches.
> > >
> > > Each operation registers a critical section during which the exception
> > > return code will adjust the offset and addresses if preemption occurs
> > > mid-sequence. The critical section is registered/unregistered with a
> > > small prologue and epilogue which encodes three distinct GPRRs (<pcp>,
> > > <off>, <addr>) into a new thread_info::pcp_gprs field:
> > >
> > > // Prologue. Enable fixups for <off> and <addr>.
> > > mrs <tsk>, sp_el0
> > > mov <tmp>, #__VAL_PCPU_GPRS(<pcp>, <off>, <addr>)
> > > strh <tmp>, [<tsk>, #TSK_TI_PCPU_GPRS]
> > >
> > > // Generate cpu-specific address
> > > mrs <off>, TPIDR_ELx
> > > add <addr>, <pcp>, <off>
> > >
> > > // Perform access sequence
> > > ldr <val>, [<addr>]
> > >
> > > // Epilogue. Disable fixups
> > > strh wzr, [<tsk>, #TSK_TI_PCPU_GPRS]
> > >
> > > If an exception is taken from within the critical section, the exception
> > > return code will adjust <off> to be the current CPU's offset, and will
> > > adjust <addr> to be (<pcp> + <off>). Distinct registers are used for
> > > <pcp>, <off>, and <addr>, so that the fixup can be applied safely at any
> > > point during the critical section.
> > >
> > > To ensure that this_cpu_*() operations within exception handlers work
> > > correctly and do not corrupt state, thread_info::pcpu_gprs is saved
> > > into a new pt_regs::pcpu_gprs field upon exception entry, and restored
> > > upon exception return.
> > >
> > > Looking at a simple this_cpu_operation:
> > >
> > > | void outline_this_cpu_add_u64(u64 __percpu *p, u64 v)
> > > | {
> > > | this_cpu_add(*p, v);
> > > | }
> ...
> > I think I had an Interesting Idea(tm) while reading the per-cpu discussion
> > in linux-mm. In case the 3 instruction preamble is too expensive:
> >
> > 1) Pass -ffixed-x18 (this natively conflicts with SHADOW_CALL_STACK.
> > SHADOW_CALL_STACK is already not-optimal codegen wise, so maybe not a big deal).
> > 2) arm64 kernel bits will use x18 as a cheap task flags register
> > 3) #define TASK_KRSEQ (1 << 0)
> > 4) Switching into the krseq mode is just a matter of toggling the bit in x18, so
> > orr x18, x18, #TASK_KRSEQ
> > a single instruction.
> > 5) Switching off is just a matter of clearing the bit in x18, so:
> > and x18, x18, #~TASK_KRSEQ
> > 6) On the preempt side we keep the krseq tables in memory, and do a sort of lookup
> > (binary search sounds easiest?) on them. But _only_ if x18 TASK_KRSEQ is set.
> > This penalises unlucky preempts but keeps fast paths maximally fast.
> > 7) entry points of course get to clear it after saving it
> >
> > The end result would look something like:
> > | <outline_this_cpu_add_u64>:
> > | orr x18, x18, #TASK_KRSEQ
> > | mrs x4, tpidr_el1
> > | add x3, x0, x4
> > | 1: ldxr x6, [x3]
> > | add x6, x6, x1
> > | stxr w5, x6, [x3]
> > | cbnz w5, 1b
> > | 2:
> > | and x18, x18, #~TASK_KRSEQ
> > | ret
> > | .pushsection .data.krseq
> > | .word 1b
> > | .word 2b
> > | .word whateverelse
> > | .popsection
... Something I overlooked is that, since the solution shifted from
"literally kernel rseq" to "funky kernel rseq but pcpu-specific" you don't
probably don't need the table at all, as long as you dedicate a good few of
those x18 bits to stash the pcp_gprs value.
> >
> > This of course precludes the use of x18 for the compiler, so it would
> > require careful benchmarking in case it negatively affects codegen too much.
> > But it avoids any sort of extraneous stores in the fast path.
>
> The extra stores are independent of the main instruction flow.
> On a multi-issue (and especially out-of-order) cpu they are pretty much
> likely to be noise.
Oh, I agree, it is probably in the noise, modern uarchs are awesome :)
> The biggest cost is likely to be in the I-cache and instruction decoders.
> Put a memory read in the 'main' path and the few clocks needed for the
> D-cache read are likely to dominate - so the writes to the pcp_gprs
> are actually likely to be free.
While I do like theorycrafting, I think we need numbers to know (numbers
which I do not have, and seemingly no one seems to have for now).
>
> OTOH stealing a gpr for some flags will cost everwhere.
Perhaps, but arm64 isn't exactly short on registers :) In any case,
I generally agree with your take that it quite possibly doesn't matter,
I was just throwing this out there in case it can help.
--
Pedro
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops
2026-08-04 17:04 ` [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops Mark Rutland
2026-08-04 22:45 ` Pedro Falcato
@ 2026-08-05 6:45 ` David Hildenbrand (Arm)
2026-08-05 6:47 ` David Hildenbrand (Arm)
1 sibling, 1 reply; 39+ messages in thread
From: David Hildenbrand (Arm) @ 2026-08-05 6:45 UTC (permalink / raw)
To: Mark Rutland, linux-arm-kernel
Cc: vladimir.murzin, ryan.roberts, peterz, catalin.marinas,
david.laight.linux, stable, ruanjinjie, james.morse, yang, cl,
maz, ljs, will, ardb
> Atop v7.2-rc4, with GCC 15.2.0 and defconfig, this is compiled as:
>
> | <outline_this_cpu_add_u64>:
> | paciasp
> | stp x29, x30, [sp, #-16]!
> | mrs x2, sp_el0
> | mov x29, sp
> | ldr w3, [x2, #8]
> | add w3, w3, #0x1
> | str w3, [x2, #8]
> | mrs x3, tpidr_el1
> | add x0, x0, x3
> | 1: ldxr x5, [x0]
> | add x5, x5, x1
> | stxr w4, x5, [x0]
> | cbnz w4, 1b
> | ldr x0, [x2, #8]
> | sub x0, x0, #0x1
> | str w0, [x2, #8]
> | cbz x0, 2f
> | ldr x0, [x2, #8]
> | cbnz x0, 3f
> | 2: bl preempt_schedule_notrace
> | 3: ldp x29, x30, [sp], #16
> | autiasp
> | ret
FWIW, in a recent discussion on some prototype hacking [1] we saw some overhead
in micro-benchmarks that would really hammer on a path that would now do a
preempt_disable()+preempt_enable().
Switching from preempt_disable() to preempt_enable_no_resched() made it turn to
noise. Of course, that has other undesirable impacts, and I am not sure if we
are in the territory of code layout changes affecting the numbers.
Just mentioning it as some data point.
--
Cheers,
David
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops
2026-08-05 6:45 ` David Hildenbrand (Arm)
@ 2026-08-05 6:47 ` David Hildenbrand (Arm)
2026-08-06 11:21 ` Mark Rutland
0 siblings, 1 reply; 39+ messages in thread
From: David Hildenbrand (Arm) @ 2026-08-05 6:47 UTC (permalink / raw)
To: Mark Rutland, linux-arm-kernel
Cc: vladimir.murzin, ryan.roberts, peterz, catalin.marinas,
david.laight.linux, stable, ruanjinjie, james.morse, yang, cl,
maz, ljs, will, ardb
On 8/5/26 08:45, David Hildenbrand (Arm) wrote:
>> Atop v7.2-rc4, with GCC 15.2.0 and defconfig, this is compiled as:
>>
>> | <outline_this_cpu_add_u64>:
>> | paciasp
>> | stp x29, x30, [sp, #-16]!
>> | mrs x2, sp_el0
>> | mov x29, sp
>> | ldr w3, [x2, #8]
>> | add w3, w3, #0x1
>> | str w3, [x2, #8]
>> | mrs x3, tpidr_el1
>> | add x0, x0, x3
>> | 1: ldxr x5, [x0]
>> | add x5, x5, x1
>> | stxr w4, x5, [x0]
>> | cbnz w4, 1b
>> | ldr x0, [x2, #8]
>> | sub x0, x0, #0x1
>> | str w0, [x2, #8]
>> | cbz x0, 2f
>> | ldr x0, [x2, #8]
>> | cbnz x0, 3f
>> | 2: bl preempt_schedule_notrace
>> | 3: ldp x29, x30, [sp], #16
>> | autiasp
>> | ret
>
> FWIW, in a recent discussion on some prototype hacking [1] we saw some overhead
> in micro-benchmarks that would really hammer on a path that would now do a
> preempt_disable()+preempt_enable().
>
> Switching from preempt_disable() to preempt_enable_no_resched() made it turn to
> noise. Of course, that has other undesirable impacts, and I am not sure if we
> are in the territory of code layout changes affecting the numbers.
>
> Just mentioning it as some data point.
>
[1] https://lore.kernel.org/linux-mm/20260630174852-mutt-send-email-mst@kernel.org/
--
Cheers,
David
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops
2026-08-05 6:47 ` David Hildenbrand (Arm)
@ 2026-08-06 11:21 ` Mark Rutland
2026-08-06 11:32 ` David Hildenbrand (Arm)
0 siblings, 1 reply; 39+ messages in thread
From: Mark Rutland @ 2026-08-06 11:21 UTC (permalink / raw)
To: David Hildenbrand (Arm)
Cc: vladimir.murzin, ryan.roberts, peterz, catalin.marinas,
david.laight.linux, stable, ruanjinjie, james.morse, yang, cl,
maz, ljs, will, ardb, linux-arm-kernel
On Wed, Aug 05, 2026 at 08:47:08AM +0200, David Hildenbrand (Arm) wrote:
> On 8/5/26 08:45, David Hildenbrand (Arm) wrote:
> >> Atop v7.2-rc4, with GCC 15.2.0 and defconfig, this is compiled as:
> >>
> >> | <outline_this_cpu_add_u64>:
> >> | paciasp
> >> | stp x29, x30, [sp, #-16]!
> >> | mrs x2, sp_el0
> >> | mov x29, sp
> >> | ldr w3, [x2, #8]
> >> | add w3, w3, #0x1
> >> | str w3, [x2, #8]
> >> | mrs x3, tpidr_el1
> >> | add x0, x0, x3
> >> | 1: ldxr x5, [x0]
> >> | add x5, x5, x1
> >> | stxr w4, x5, [x0]
> >> | cbnz w4, 1b
> >> | ldr x0, [x2, #8]
> >> | sub x0, x0, #0x1
> >> | str w0, [x2, #8]
> >> | cbz x0, 2f
> >> | ldr x0, [x2, #8]
> >> | cbnz x0, 3f
> >> | 2: bl preempt_schedule_notrace
> >> | 3: ldp x29, x30, [sp], #16
> >> | autiasp
> >> | ret
> >
> > FWIW, in a recent discussion on some prototype hacking [1] we saw some overhead
> > in micro-benchmarks that would really hammer on a path that would now do a
> > preempt_disable()+preempt_enable().
> >
> > Switching from preempt_disable() to preempt_enable_no_resched() made it turn to
> > noise. Of course, that has other undesirable impacts, and I am not sure if we
> > are in the territory of code layout changes affecting the numbers.
> >
> > Just mentioning it as some data point.
>
> [1] https://lore.kernel.org/linux-mm/20260630174852-mutt-send-email-mst@kernel.org/
Thanks for the pointer.
IIUC in those cases you're using preempt_disable() .. preempt_enable()
directly, not this_cpu_*(), right?
If so, patches 5 and 6 of this series [2,3] might have an impact, but I
wouldn't expect a significant change unless you're calling
preempt_enable a lot.
Please beware that it's not safe to use preempt_enable_no_resched()
UNLESS it is immediately followed by a call to schedule(). That's not
documented today (and I couldn't find a good reference), so more folk
are likely to be tempted to use it...
I'll send a patch to clarify that (or at least start the discussion with
scheduler/preempt folk).
Mark.
[2] https://lore.kernel.org/linux-arm-kernel/20260804170503.3513916-6-mark.rutland@arm.com/
[3] https://lore.kernel.org/linux-arm-kernel/20260804170503.3513916-7-mark.rutland@arm.com/
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops
2026-08-06 11:21 ` Mark Rutland
@ 2026-08-06 11:32 ` David Hildenbrand (Arm)
2026-08-06 12:02 ` Mark Rutland
2026-08-06 13:25 ` David Laight
0 siblings, 2 replies; 39+ messages in thread
From: David Hildenbrand (Arm) @ 2026-08-06 11:32 UTC (permalink / raw)
To: Mark Rutland
Cc: vladimir.murzin, ryan.roberts, peterz, catalin.marinas,
david.laight.linux, stable, ruanjinjie, james.morse, yang, cl,
maz, ljs, will, ardb, linux-arm-kernel
On 8/6/26 13:21, Mark Rutland wrote:
> On Wed, Aug 05, 2026 at 08:47:08AM +0200, David Hildenbrand (Arm) wrote:
>> On 8/5/26 08:45, David Hildenbrand (Arm) wrote:
>>>
>>> FWIW, in a recent discussion on some prototype hacking [1] we saw some overhead
>>> in micro-benchmarks that would really hammer on a path that would now do a
>>> preempt_disable()+preempt_enable().
>>>
>>> Switching from preempt_disable() to preempt_enable_no_resched() made it turn to
>>> noise. Of course, that has other undesirable impacts, and I am not sure if we
>>> are in the territory of code layout changes affecting the numbers.
>>>
>>> Just mentioning it as some data point.
>>
>> [1] https://lore.kernel.org/linux-mm/20260630174852-mutt-send-email-mst@kernel.org/
>
> Thanks for the pointer.
>
> IIUC in those cases you're using preempt_disable() .. preempt_enable()
> directly, not this_cpu_*(), right?
It was purely preempt_disable/preempt_enable experiments without any percpu stuff.
>
> If so, patches 5 and 6 of this series [2,3] might have an impact, but I
> wouldn't expect a significant change unless you're calling
> preempt_enable a lot.
>
> Please beware that it's not safe to use preempt_enable_no_resched()
> UNLESS it is immediately followed by a call to schedule(). That's not
> documented today (and I couldn't find a good reference), so more folk
> are likely to be tempted to use it...
Yes, that's also why we abandoned that (including for various other reasons :) ).
preempt_enable_no_resched() helped to identify that the preempt_enable() was
really causing the noticeable overhead, not the other minor stuff we added on
some hot paths.
--
Cheers,
David
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops
2026-08-06 11:32 ` David Hildenbrand (Arm)
@ 2026-08-06 12:02 ` Mark Rutland
2026-08-06 13:25 ` David Laight
1 sibling, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-06 12:02 UTC (permalink / raw)
To: David Hildenbrand (Arm)
Cc: vladimir.murzin, ryan.roberts, peterz, catalin.marinas,
david.laight.linux, stable, ruanjinjie, james.morse, yang, cl,
maz, ljs, will, ardb, linux-arm-kernel
On Thu, Aug 06, 2026 at 01:32:52PM +0200, David Hildenbrand (Arm) wrote:
> On 8/6/26 13:21, Mark Rutland wrote:
> > On Wed, Aug 05, 2026 at 08:47:08AM +0200, David Hildenbrand (Arm) wrote:
> >> On 8/5/26 08:45, David Hildenbrand (Arm) wrote:
> >>>
> >>> FWIW, in a recent discussion on some prototype hacking [1] we saw some overhead
> >>> in micro-benchmarks that would really hammer on a path that would now do a
> >>> preempt_disable()+preempt_enable().
> >>>
> >>> Switching from preempt_disable() to preempt_enable_no_resched() made it turn to
> >>> noise. Of course, that has other undesirable impacts, and I am not sure if we
> >>> are in the territory of code layout changes affecting the numbers.
> >>>
> >>> Just mentioning it as some data point.
> >>
> >> [1] https://lore.kernel.org/linux-mm/20260630174852-mutt-send-email-mst@kernel.org/
> >
> > Thanks for the pointer.
> >
> > IIUC in those cases you're using preempt_disable() .. preempt_enable()
> > directly, not this_cpu_*(), right?
>
> It was purely preempt_disable/preempt_enable experiments without any percpu stuff.
>
> > If so, patches 5 and 6 of this series [2,3] might have an impact, but I
> > wouldn't expect a significant change unless you're calling
> > preempt_enable a lot.
> >
> > Please beware that it's not safe to use preempt_enable_no_resched()
> > UNLESS it is immediately followed by a call to schedule(). That's not
> > documented today (and I couldn't find a good reference), so more folk
> > are likely to be tempted to use it...
> Yes, that's also why we abandoned that (including for various other reasons :) ).
:)
> preempt_enable_no_resched() helped to identify that the preempt_enable() was
> really causing the noticeable overhead, not the other minor stuff we added on
> some hot paths.
Understood!
If we seeeing particularly noticeable overhead from preempt_enable() in
some workloads, there are some options we could investigate to reduce
that impact (e.g. using __preserve_most or a trampoline like x86's
preempt_schedule_thunk to reduce necessary spills and register
pressure).
Please let me know if you see anything that stands out, as any examples
would be useful for investigation. We'd want to figure out how much of
the overhead comes from register pressure, and how much of it comes from
the conditional call itself.
If you're testing with PREEMPT_DYNAMIC=y, on some architectures
(including arm64) you might see overhead reduced by:
https://lore.kernel.org/lkml/20260803191731.3244294-1-mark.rutland@arm.com/
... but IIUC on x86 that won't change the cost of
preempt_enable[_notrace](). Today that makes a static call to
preempt_schedule[_notrace]_thunk, and a plain call will be the same
cost.
Mark.
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops
2026-08-06 11:32 ` David Hildenbrand (Arm)
2026-08-06 12:02 ` Mark Rutland
@ 2026-08-06 13:25 ` David Laight
2026-08-06 13:30 ` David Hildenbrand (Arm)
1 sibling, 1 reply; 39+ messages in thread
From: David Laight @ 2026-08-06 13:25 UTC (permalink / raw)
To: David Hildenbrand (Arm)
Cc: Mark Rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, ruanjinjie, stable, james.morse, yang, cl, maz,
ljs, will, ardb, linux-arm-kernel
On Thu, 6 Aug 2026 13:32:52 +0200
"David Hildenbrand (Arm)" <david@kernel.org> wrote:
> On 8/6/26 13:21, Mark Rutland wrote:
> > On Wed, Aug 05, 2026 at 08:47:08AM +0200, David Hildenbrand (Arm) wrote:
> >> On 8/5/26 08:45, David Hildenbrand (Arm) wrote:
> >>>
> >>> FWIW, in a recent discussion on some prototype hacking [1] we saw some overhead
> >>> in micro-benchmarks that would really hammer on a path that would now do a
> >>> preempt_disable()+preempt_enable().
> >>>
> >>> Switching from preempt_disable() to preempt_enable_no_resched() made it turn to
> >>> noise. Of course, that has other undesirable impacts, and I am not sure if we
> >>> are in the territory of code layout changes affecting the numbers.
> >>>
> >>> Just mentioning it as some data point.
> >>
> >> [1] https://lore.kernel.org/linux-mm/20260630174852-mutt-send-email-mst@kernel.org/
> >
> > Thanks for the pointer.
> >
> > IIUC in those cases you're using preempt_disable() .. preempt_enable()
> > directly, not this_cpu_*(), right?
>
> It was purely preempt_disable/preempt_enable experiments without any percpu stuff.
Did you check that preempt_enable() isn't likely to speculatively execute the
schedule() call.
Even if you write:
if (unlikely(a == b))
function();
the compiler tends to generate a forwards branch around the function call.
Since the branch is likely to be assumed 'not taken' the cpu will
speculatively execute the function.
Adding a non-empty else clause (eg an asm() comment) should get the function
call out of line and hopefully not speculatively called.
This is likely made worse because the condition is reading the full 64bits
of a location that has just had 32bits written.
This almost certainly has to wait for the write to 'drain' from the store
buffer before the read can be done from the D-cache.
(A read of the same/smaller size might be snooped from the store buffer.)
I'm not sure of the mis-predict penalty for a typical arm cpu.
I see ~20 clocks on a Zen-5 for a simple (value in register) one, here
I suspect an extra 5-10 clocks get added because of the memory accesses.
David
>
> >
> > If so, patches 5 and 6 of this series [2,3] might have an impact, but I
> > wouldn't expect a significant change unless you're calling
> > preempt_enable a lot.
> >
> > Please beware that it's not safe to use preempt_enable_no_resched()
> > UNLESS it is immediately followed by a call to schedule(). That's not
> > documented today (and I couldn't find a good reference), so more folk
> > are likely to be tempted to use it...
> Yes, that's also why we abandoned that (including for various other reasons :) ).
>
> preempt_enable_no_resched() helped to identify that the preempt_enable() was
> really causing the noticeable overhead, not the other minor stuff we added on
> some hot paths.
>
^ permalink raw reply [flat|nested] 39+ messages in thread
* Re: [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops
2026-08-06 13:25 ` David Laight
@ 2026-08-06 13:30 ` David Hildenbrand (Arm)
0 siblings, 0 replies; 39+ messages in thread
From: David Hildenbrand (Arm) @ 2026-08-06 13:30 UTC (permalink / raw)
To: David Laight
Cc: Mark Rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, ruanjinjie, stable, james.morse, yang, cl, maz,
ljs, will, ardb, linux-arm-kernel
On 8/6/26 15:25, David Laight wrote:
> On Thu, 6 Aug 2026 13:32:52 +0200
> "David Hildenbrand (Arm)" <david@kernel.org> wrote:
>
>> On 8/6/26 13:21, Mark Rutland wrote:
>>>
>>> Thanks for the pointer.
>>>
>>> IIUC in those cases you're using preempt_disable() .. preempt_enable()
>>> directly, not this_cpu_*(), right?
>>
>> It was purely preempt_disable/preempt_enable experiments without any percpu stuff.
>
> Did you check that preempt_enable() isn't likely to speculatively execute the
> schedule() call.
> Even if you write:
> if (unlikely(a == b))
> function();
> the compiler tends to generate a forwards branch around the function call.
I didn't look closer (Michael ran the experiments), in particular, why it added
overhead. I was assuming that it's just suboptimal code generation as you and
Mark hinted. It was sufficient for us to understand that the overhead we saw can
be removed somehow, but as we dropped the approach entirely, we didn't look
closer into that.
--
Cheers,
David
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v2 14/20] arm64: percpu: Implement preemptible read/write ops
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (12 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-05 9:24 ` Ryan Roberts
2026-08-04 17:04 ` [PATCH v2 15/20] arm64: percpu: Implement preemptible void RMW ops Mark Rutland
` (5 subsequent siblings)
19 siblings, 1 reply; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
Use the PCPU GPR infrastructure to implement preemptible this_cpu_read()
and this_cpu_write().
This change means that this_cpu_read() will always use a plain LDR, even
in LTO configurations where READ_ONCE() will use LDA[P]R. Using plain
LDR is preferable, given that the rationale for using LDA[P]R in
READ_ONCE() was to retain address dependencies against values written by
other CPUs, which isn't expected usage for this_cpu_read(). Using plain
LDR will enforce fewer ordering constraints.
Test case:
| void outline_this_cpu_write_u64(u64 __percpu *p, u64 v)
| {
| this_cpu_write(*p, v);
| }
Generated code before this patch (v7.2-rc4):
| <outline_this_cpu_write_u64>:
| paciasp
| stp x29, x30, [sp, #-16]!
| mrs x2, sp_el0
| mov x29, sp
| ldr w3, [x2, #8]
| add w3, w3, #0x1
| str w3, [x2, #8]
| mrs x3, tpidr_el1
| str x1, [x0, x3]
| ldr x0, [x2, #8]
| sub x0, x0, #0x1
| str w0, [x2, #8]
| cbz x0, 1f
| ldr x0, [x2, #8]
| cbnz x0, 2f
| 1: bl preempt_schedule_notrace
| 2: ldp x29, x30, [sp], #16
| autiasp
| ret
Generated code after this patch:
| <outline_this_cpu_write_u64>:
| mrs x2, sp_el0
| mov w3, #0x7c60
| strh w3, [x2, #20]
| mrs x3, tpidr_el1
| str x1, [x0, x3]
| strh wzr, [x2, #20]
| ret
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/percpu.h | 79 ++++++++++++++++++++++++---------
1 file changed, 58 insertions(+), 21 deletions(-)
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 8b4c9ea05d4e6..a094c137d797f 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -128,15 +128,42 @@ static inline unsigned long __kern_my_cpu_offset(void)
#define __my_cpu_offset __kern_my_cpu_offset()
#endif
-#define PERCPU_RW_OPS(sz) \
-static inline unsigned long __percpu_read_##sz(void *ptr) \
-{ \
- return READ_ONCE(*(u##sz *)ptr); \
-} \
- \
-static inline void __percpu_write_##sz(void *ptr, unsigned long val) \
-{ \
- WRITE_ONCE(*(u##sz *)ptr, (u##sz)val); \
+#define PERCPU_RW_OPS(w, sfx, sz) \
+static inline unsigned long __percpu_read_##sz(void __percpu *pcp) \
+{ \
+ u16 *gprs = ¤t_thread_info()->pcpu_gprs; \
+ unsigned long off; \
+ unsigned long val; \
+ \
+ asm volatile( \
+ __PCPU_GPRS_BEGIN_OFFSET("%[gprs]", "%[pcp]", "%[off]") \
+ " ldr" #sfx "\t%" #w "[val], [%[pcp], %[off]]\n" \
+ __PCPU_GPRS_END("%[gprs]") \
+ : [gprs] "=Qo" (*gprs), \
+ [off] "=&r" (off), \
+ [val] "=&r" (val) \
+ : [pcp] "r" (pcp) \
+ : "memory" \
+ ); \
+ \
+ return val; \
+} \
+ \
+static inline void __percpu_write_##sz(void __percpu *pcp, u##sz val) \
+{ \
+ u16 *gprs = ¤t_thread_info()->pcpu_gprs; \
+ unsigned long off; \
+ \
+ asm volatile( \
+ __PCPU_GPRS_BEGIN_OFFSET("%[gprs]", "%[pcp]", "%[off]") \
+ " str" #sfx "\t%" #w "[val], [%[pcp], %[off]]\n" \
+ __PCPU_GPRS_END("%[gprs]") \
+ : [gprs] "=Qo" (*gprs), \
+ [off] "=&r" (off) \
+ : [pcp] "r" (pcp), \
+ [val] "r" (val) \
+ : "memory" \
+ ); \
}
#define __PERCPU_OP_CASE(w, sfx, name, sz, op_llsc, op_lse) \
@@ -196,10 +223,10 @@ __percpu_##name##_return_case_##sz(void *ptr, unsigned long val) \
__PERCPU_RET_OP_CASE(w, , name, 32, op_llsc, op_lse) \
__PERCPU_RET_OP_CASE( , , name, 64, op_llsc, op_lse)
-PERCPU_RW_OPS(8)
-PERCPU_RW_OPS(16)
-PERCPU_RW_OPS(32)
-PERCPU_RW_OPS(64)
+PERCPU_RW_OPS(w, b, 8)
+PERCPU_RW_OPS(w, h, 16)
+PERCPU_RW_OPS(w, , 32)
+PERCPU_RW_OPS( , , 64)
/*
* Use value-returning atomics for CPU-local ops as they are more likely
@@ -245,23 +272,33 @@ PERCPU_RET_OP(add, add, ldadd)
__retval; \
})
+#define _pcp_wrap(op, pcp, ...) \
+({ \
+ op(&(pcp), __VA_ARGS__); \
+})
+
+#define _pcp_wrap_return(op, pcp, args...) \
+({ \
+ (typeof(pcp))op(&(pcp), ##args); \
+})
+
#define this_cpu_read_1(pcp) \
- _pcp_protect_return(__percpu_read_8, pcp)
+ _pcp_wrap_return(__percpu_read_8, pcp)
#define this_cpu_read_2(pcp) \
- _pcp_protect_return(__percpu_read_16, pcp)
+ _pcp_wrap_return(__percpu_read_16, pcp)
#define this_cpu_read_4(pcp) \
- _pcp_protect_return(__percpu_read_32, pcp)
+ _pcp_wrap_return(__percpu_read_32, pcp)
#define this_cpu_read_8(pcp) \
- _pcp_protect_return(__percpu_read_64, pcp)
+ _pcp_wrap_return(__percpu_read_64, pcp)
#define this_cpu_write_1(pcp, val) \
- _pcp_protect(__percpu_write_8, pcp, (unsigned long)(val))
+ _pcp_wrap(__percpu_write_8, pcp, (unsigned long)(val))
#define this_cpu_write_2(pcp, val) \
- _pcp_protect(__percpu_write_16, pcp, (unsigned long)(val))
+ _pcp_wrap(__percpu_write_16, pcp, (unsigned long)(val))
#define this_cpu_write_4(pcp, val) \
- _pcp_protect(__percpu_write_32, pcp, (unsigned long)(val))
+ _pcp_wrap(__percpu_write_32, pcp, (unsigned long)(val))
#define this_cpu_write_8(pcp, val) \
- _pcp_protect(__percpu_write_64, pcp, (unsigned long)(val))
+ _pcp_wrap(__percpu_write_64, pcp, (unsigned long)(val))
#define this_cpu_add_1(pcp, val) \
_pcp_protect(__percpu_add_case_8, pcp, val)
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* Re: [PATCH v2 14/20] arm64: percpu: Implement preemptible read/write ops
2026-08-04 17:04 ` [PATCH v2 14/20] arm64: percpu: Implement preemptible read/write ops Mark Rutland
@ 2026-08-05 9:24 ` Ryan Roberts
2026-08-05 12:08 ` David Laight
2026-08-05 13:34 ` Mark Rutland
0 siblings, 2 replies; 39+ messages in thread
From: Ryan Roberts @ 2026-08-05 9:24 UTC (permalink / raw)
To: Mark Rutland, linux-arm-kernel
Cc: vladimir.murzin, peterz, catalin.marinas, david.laight.linux,
stable, ruanjinjie, james.morse, yang, cl, maz, david, ljs, will,
ardb
On 04/08/2026 18:04, Mark Rutland wrote:
> Use the PCPU GPR infrastructure to implement preemptible this_cpu_read()
> and this_cpu_write().
>
> This change means that this_cpu_read() will always use a plain LDR, even
> in LTO configurations where READ_ONCE() will use LDA[P]R. Using plain
> LDR is preferable, given that the rationale for using LDA[P]R in
> READ_ONCE() was to retain address dependencies against values written by
> other CPUs, which isn't expected usage for this_cpu_read(). Using plain
> LDR will enforce fewer ordering constraints.
>
> Test case:
>
> | void outline_this_cpu_write_u64(u64 __percpu *p, u64 v)
> | {
> | this_cpu_write(*p, v);
> | }
>
> Generated code before this patch (v7.2-rc4):
>
> | <outline_this_cpu_write_u64>:
> | paciasp
> | stp x29, x30, [sp, #-16]!
> | mrs x2, sp_el0
> | mov x29, sp
> | ldr w3, [x2, #8]
> | add w3, w3, #0x1
> | str w3, [x2, #8]
> | mrs x3, tpidr_el1
> | str x1, [x0, x3]
> | ldr x0, [x2, #8]
> | sub x0, x0, #0x1
> | str w0, [x2, #8]
> | cbz x0, 1f
> | ldr x0, [x2, #8]
> | cbnz x0, 2f
> | 1: bl preempt_schedule_notrace
> | 2: ldp x29, x30, [sp], #16
> | autiasp
> | ret
>
> Generated code after this patch:
>
> | <outline_this_cpu_write_u64>:
> | mrs x2, sp_el0
> | mov w3, #0x7c60
> | strh w3, [x2, #20]
> | mrs x3, tpidr_el1
> | str x1, [x0, x3]
> | strh wzr, [x2, #20]
> | ret
>
> Signed-off-by: Mark Rutland <mark.rutland@arm.com>
> Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> Cc: Ard Biesheuvel <ardb@kernel.org>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: James Morse <james.morse@arm.com>
> Cc: Jinjie Ruan <ruanjinjie@huawei.com>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Vladimir Murzin <vladimir.murzin@arm.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Yang Shi <yang@os.amperecomputing.com>
> ---
FYI I'm seeing build warnings caused by this patch (with ftrace enabled - based
on the warnings, I'm guessing that's the key bit), using:
aarch64-linux-gnu-gcc (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44
I haven't investigated the cause.
Thanks,
Ryan
---8<---
In file included from linux/arch/arm64/include/asm/spectre.h:17,
from linux/arch/arm64/include/asm/processor.h:47,
from linux/include/linux/sched.h:13,
from linux/include/linux/ratelimit.h:6,
from linux/include/linux/dev_printk.h:16,
from linux/include/linux/device.h:15,
from linux/include/linux/node.h:18,
from linux/include/linux/cpu.h:17,
from linux/include/linux/stop_machine.h:5,
from linux/kernel/trace/ftrace.c:17:
linux/kernel/trace/ftrace.c: In function 'ftrace_filter_pid_sched_switch_probe':
linux/arch/arm64/include/asm/percpu.h:295:42: warning: conversion from 'long unsigned int' to 'u8' {aka 'unsigned char'} changes value from '18446744073709551615' to '255' [-Woverflow]
295 | _pcp_wrap(__percpu_write_8, pcp, (unsigned long)(val))
| ^~~~~~~~~~~~~~~~~~~~
linux/arch/arm64/include/asm/percpu.h:277:20: note: in definition of macro '_pcp_wrap'
277 | op(&(pcp), __VA_ARGS__); \
| ^~~~~~~~~~~
linux/include/linux/percpu-defs.h:369:25: note: in expansion of macro 'this_cpu_write_1'
369 | case 1: stem##1(variable, __VA_ARGS__);break; \
| ^~~~
linux/include/linux/percpu-defs.h:500:41: note: in expansion of macro '__pcpu_size_call'
500 | #define this_cpu_write(pcp, val) __pcpu_size_call(this_cpu_write_, pcp, val)
| ^~~~~~~~~~~~~~~~
linux/kernel/trace/ftrace.c:8628:17: note: in expansion of macro 'this_cpu_write'
8628 | this_cpu_write(tr->array_buffer.data->ftrace_ignore_pid,
| ^~~~~~~~~~~~~~
linux/arch/arm64/include/asm/percpu.h:297:43: warning: conversion from 'long unsigned int' to 'u16' {aka 'short unsigned int'} changes value from '18446744073709551615' to '65535' [-Woverflow]
297 | _pcp_wrap(__percpu_write_16, pcp, (unsigned long)(val))
| ^~~~~~~~~~~~~~~~~~~~
linux/arch/arm64/include/asm/percpu.h:277:20: note: in definition of macro '_pcp_wrap'
277 | op(&(pcp), __VA_ARGS__); \
| ^~~~~~~~~~~
linux/include/linux/percpu-defs.h:370:25: note: in expansion of macro 'this_cpu_write_2'
370 | case 2: stem##2(variable, __VA_ARGS__);break; \
| ^~~~
linux/include/linux/percpu-defs.h:500:41: note: in expansion of macro '__pcpu_size_call'
500 | #define this_cpu_write(pcp, val) __pcpu_size_call(this_cpu_write_, pcp, val)
| ^~~~~~~~~~~~~~~~
linux/kernel/trace/ftrace.c:8628:17: note: in expansion of macro 'this_cpu_write'
8628 | this_cpu_write(tr->array_buffer.data->ftrace_ignore_pid,
| ^~~~~~~~~~~~~~
linux/arch/arm64/include/asm/percpu.h:299:43: warning: conversion from 'long unsigned int' to 'u32' {aka 'unsigned int'} changes value from '18446744073709551615' to '4294967295' [-Woverflow]
299 | _pcp_wrap(__percpu_write_32, pcp, (unsigned long)(val))
| ^~~~~~~~~~~~~~~~~~~~
linux/arch/arm64/include/asm/percpu.h:277:20: note: in definition of macro '_pcp_wrap'
277 | op(&(pcp), __VA_ARGS__); \
| ^~~~~~~~~~~
linux/include/linux/percpu-defs.h:371:25: note: in expansion of macro 'this_cpu_write_4'
371 | case 4: stem##4(variable, __VA_ARGS__);break; \
| ^~~~
linux/include/linux/percpu-defs.h:500:41: note: in expansion of macro '__pcpu_size_call'
500 | #define this_cpu_write(pcp, val) __pcpu_size_call(this_cpu_write_, pcp, val)
| ^~~~~~~~~~~~~~~~
linux/kernel/trace/ftrace.c:8628:17: note: in expansion of macro 'this_cpu_write'
8628 | this_cpu_write(tr->array_buffer.data->ftrace_ignore_pid,
| ^~~~~~~~~~~~~~
linux/kernel/trace/ftrace.c: In function 'ignore_task_cpu':
linux/arch/arm64/include/asm/percpu.h:295:42: warning: conversion from 'long unsigned int' to 'u8' {aka 'unsigned char'} changes value from '18446744073709551615' to '255' [-Woverflow]
295 | _pcp_wrap(__percpu_write_8, pcp, (unsigned long)(val))
| ^~~~~~~~~~~~~~~~~~~~
linux/arch/arm64/include/asm/percpu.h:277:20: note: in definition of macro '_pcp_wrap'
277 | op(&(pcp), __VA_ARGS__); \
| ^~~~~~~~~~~
linux/include/linux/percpu-defs.h:369:25: note: in expansion of macro 'this_cpu_write_1'
369 | case 1: stem##1(variable, __VA_ARGS__);break; \
| ^~~~
linux/include/linux/percpu-defs.h:500:41: note: in expansion of macro '__pcpu_size_call'
500 | #define this_cpu_write(pcp, val) __pcpu_size_call(this_cpu_write_, pcp, val)
| ^~~~~~~~~~~~~~~~
linux/kernel/trace/ftrace.c:8898:17: note: in expansion of macro 'this_cpu_write'
8898 | this_cpu_write(tr->array_buffer.data->ftrace_ignore_pid,
| ^~~~~~~~~~~~~~
linux/arch/arm64/include/asm/percpu.h:297:43: warning: conversion from 'long unsigned int' to 'u16' {aka 'short unsigned int'} changes value from '18446744073709551615' to '65535' [-Woverflow]
297 | _pcp_wrap(__percpu_write_16, pcp, (unsigned long)(val))
| ^~~~~~~~~~~~~~~~~~~~
linux/arch/arm64/include/asm/percpu.h:277:20: note: in definition of macro '_pcp_wrap'
277 | op(&(pcp), __VA_ARGS__); \
| ^~~~~~~~~~~
linux/include/linux/percpu-defs.h:370:25: note: in expansion of macro 'this_cpu_write_2'
370 | case 2: stem##2(variable, __VA_ARGS__);break; \
| ^~~~
linux/include/linux/percpu-defs.h:500:41: note: in expansion of macro '__pcpu_size_call'
500 | #define this_cpu_write(pcp, val) __pcpu_size_call(this_cpu_write_, pcp, val)
| ^~~~~~~~~~~~~~~~
linux/kernel/trace/ftrace.c:8898:17: note: in expansion of macro 'this_cpu_write'
8898 | this_cpu_write(tr->array_buffer.data->ftrace_ignore_pid,
| ^~~~~~~~~~~~~~
linux/arch/arm64/include/asm/percpu.h:299:43: warning: conversion from 'long unsigned int' to 'u32' {aka 'unsigned int'} changes value from '18446744073709551615' to '4294967295' [-Woverflow]
299 | _pcp_wrap(__percpu_write_32, pcp, (unsigned long)(val))
| ^~~~~~~~~~~~~~~~~~~~
linux/arch/arm64/include/asm/percpu.h:277:20: note: in definition of macro '_pcp_wrap'
277 | op(&(pcp), __VA_ARGS__); \
| ^~~~~~~~~~~
linux/include/linux/percpu-defs.h:371:25: note: in expansion of macro 'this_cpu_write_4'
371 | case 4: stem##4(variable, __VA_ARGS__);break; \
| ^~~~
linux/include/linux/percpu-defs.h:500:41: note: in expansion of macro '__pcpu_size_call'
500 | #define this_cpu_write(pcp, val) __pcpu_size_call(this_cpu_write_, pcp, val)
| ^~~~~~~~~~~~~~~~
linux/kernel/trace/ftrace.c:8898:17: note: in expansion of macro 'this_cpu_write'
8898 | this_cpu_write(tr->array_buffer.data->ftrace_ignore_pid,
| ^~~~~~~~~~~~~~
---8<---
^ permalink raw reply [flat|nested] 39+ messages in thread* Re: [PATCH v2 14/20] arm64: percpu: Implement preemptible read/write ops
2026-08-05 9:24 ` Ryan Roberts
@ 2026-08-05 12:08 ` David Laight
2026-08-05 13:34 ` Mark Rutland
1 sibling, 0 replies; 39+ messages in thread
From: David Laight @ 2026-08-05 12:08 UTC (permalink / raw)
To: Ryan Roberts
Cc: Mark Rutland, vladimir.murzin, peterz, catalin.marinas,
ruanjinjie, stable, james.morse, yang, cl, maz, david, ljs, will,
ardb, linux-arm-kernel
On Wed, 5 Aug 2026 10:24:04 +0100
Ryan Roberts <ryan.roberts@arm.com> wrote:
> On 04/08/2026 18:04, Mark Rutland wrote:
> > Use the PCPU GPR infrastructure to implement preemptible this_cpu_read()
> > and this_cpu_write().
> >
> > This change means that this_cpu_read() will always use a plain LDR, even
> > in LTO configurations where READ_ONCE() will use LDA[P]R. Using plain
> > LDR is preferable, given that the rationale for using LDA[P]R in
> > READ_ONCE() was to retain address dependencies against values written by
> > other CPUs, which isn't expected usage for this_cpu_read(). Using plain
> > LDR will enforce fewer ordering constraints.
> >
> > Test case:
> >
> > | void outline_this_cpu_write_u64(u64 __percpu *p, u64 v)
> > | {
> > | this_cpu_write(*p, v);
> > | }
> >
> > Generated code before this patch (v7.2-rc4):
> >
> > | <outline_this_cpu_write_u64>:
> > | paciasp
> > | stp x29, x30, [sp, #-16]!
> > | mrs x2, sp_el0
> > | mov x29, sp
> > | ldr w3, [x2, #8]
> > | add w3, w3, #0x1
> > | str w3, [x2, #8]
> > | mrs x3, tpidr_el1
> > | str x1, [x0, x3]
> > | ldr x0, [x2, #8]
> > | sub x0, x0, #0x1
> > | str w0, [x2, #8]
> > | cbz x0, 1f
> > | ldr x0, [x2, #8]
> > | cbnz x0, 2f
> > | 1: bl preempt_schedule_notrace
> > | 2: ldp x29, x30, [sp], #16
> > | autiasp
> > | ret
> >
> > Generated code after this patch:
> >
> > | <outline_this_cpu_write_u64>:
> > | mrs x2, sp_el0
> > | mov w3, #0x7c60
> > | strh w3, [x2, #20]
> > | mrs x3, tpidr_el1
> > | str x1, [x0, x3]
> > | strh wzr, [x2, #20]
> > | ret
> >
> > Signed-off-by: Mark Rutland <mark.rutland@arm.com>
> > Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
> > Cc: Ard Biesheuvel <ardb@kernel.org>
> > Cc: Catalin Marinas <catalin.marinas@arm.com>
> > Cc: James Morse <james.morse@arm.com>
> > Cc: Jinjie Ruan <ruanjinjie@huawei.com>
> > Cc: Marc Zyngier <maz@kernel.org>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: Vladimir Murzin <vladimir.murzin@arm.com>
> > Cc: Will Deacon <will@kernel.org>
> > Cc: Yang Shi <yang@os.amperecomputing.com>
> > ---
> FYI I'm seeing build warnings caused by this patch (with ftrace enabled - based
> on the warnings, I'm guessing that's the key bit), using:
>
> aarch64-linux-gnu-gcc (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44
>
> I haven't investigated the cause.
It'll be the earlier patch that changed the casts.
The code is doing:
this_cpu_write(tr->array_buffer.data->ftrace_ignore_pid,
FTRACE_PID_IGNORE);
where the constant is -1.
I suspect the warning messages are coming from the switch case that are
optimised away because the size if wrong (ftrace_ignore_pid seems to
be a pid number to ignore and is 32 bits.
That does seem a long-winded way to access per-cpu data.
To make any sense the code must be running with preemption disabled and
be caching info in per-cpu memory.
In which case it can just access it directly.
OTOH it could save the 'global' address of the per-cpu data and then access
it using the pointer so that access would be a normal one.
David
>
> Thanks,
> Ryan
>
> ---8<---
> In file included from linux/arch/arm64/include/asm/spectre.h:17,
> from linux/arch/arm64/include/asm/processor.h:47,
> from linux/include/linux/sched.h:13,
> from linux/include/linux/ratelimit.h:6,
> from linux/include/linux/dev_printk.h:16,
> from linux/include/linux/device.h:15,
> from linux/include/linux/node.h:18,
> from linux/include/linux/cpu.h:17,
> from linux/include/linux/stop_machine.h:5,
> from linux/kernel/trace/ftrace.c:17:
> linux/kernel/trace/ftrace.c: In function 'ftrace_filter_pid_sched_switch_probe':
> linux/arch/arm64/include/asm/percpu.h:295:42: warning: conversion from 'long unsigned int' to 'u8' {aka 'unsigned char'} changes value from '18446744073709551615' to '255' [-Woverflow]
> 295 | _pcp_wrap(__percpu_write_8, pcp, (unsigned long)(val))
> | ^~~~~~~~~~~~~~~~~~~~
> linux/arch/arm64/include/asm/percpu.h:277:20: note: in definition of macro '_pcp_wrap'
> 277 | op(&(pcp), __VA_ARGS__); \
> | ^~~~~~~~~~~
> linux/include/linux/percpu-defs.h:369:25: note: in expansion of macro 'this_cpu_write_1'
> 369 | case 1: stem##1(variable, __VA_ARGS__);break; \
> | ^~~~
> linux/include/linux/percpu-defs.h:500:41: note: in expansion of macro '__pcpu_size_call'
> 500 | #define this_cpu_write(pcp, val) __pcpu_size_call(this_cpu_write_, pcp, val)
> | ^~~~~~~~~~~~~~~~
> linux/kernel/trace/ftrace.c:8628:17: note: in expansion of macro 'this_cpu_write'
> 8628 | this_cpu_write(tr->array_buffer.data->ftrace_ignore_pid,
> | ^~~~~~~~~~~~~~
> linux/arch/arm64/include/asm/percpu.h:297:43: warning: conversion from 'long unsigned int' to 'u16' {aka 'short unsigned int'} changes value from '18446744073709551615' to '65535' [-Woverflow]
> 297 | _pcp_wrap(__percpu_write_16, pcp, (unsigned long)(val))
> | ^~~~~~~~~~~~~~~~~~~~
> linux/arch/arm64/include/asm/percpu.h:277:20: note: in definition of macro '_pcp_wrap'
> 277 | op(&(pcp), __VA_ARGS__); \
> | ^~~~~~~~~~~
> linux/include/linux/percpu-defs.h:370:25: note: in expansion of macro 'this_cpu_write_2'
> 370 | case 2: stem##2(variable, __VA_ARGS__);break; \
> | ^~~~
> linux/include/linux/percpu-defs.h:500:41: note: in expansion of macro '__pcpu_size_call'
> 500 | #define this_cpu_write(pcp, val) __pcpu_size_call(this_cpu_write_, pcp, val)
> | ^~~~~~~~~~~~~~~~
> linux/kernel/trace/ftrace.c:8628:17: note: in expansion of macro 'this_cpu_write'
> 8628 | this_cpu_write(tr->array_buffer.data->ftrace_ignore_pid,
> | ^~~~~~~~~~~~~~
> linux/arch/arm64/include/asm/percpu.h:299:43: warning: conversion from 'long unsigned int' to 'u32' {aka 'unsigned int'} changes value from '18446744073709551615' to '4294967295' [-Woverflow]
> 299 | _pcp_wrap(__percpu_write_32, pcp, (unsigned long)(val))
> | ^~~~~~~~~~~~~~~~~~~~
> linux/arch/arm64/include/asm/percpu.h:277:20: note: in definition of macro '_pcp_wrap'
> 277 | op(&(pcp), __VA_ARGS__); \
> | ^~~~~~~~~~~
> linux/include/linux/percpu-defs.h:371:25: note: in expansion of macro 'this_cpu_write_4'
> 371 | case 4: stem##4(variable, __VA_ARGS__);break; \
> | ^~~~
> linux/include/linux/percpu-defs.h:500:41: note: in expansion of macro '__pcpu_size_call'
> 500 | #define this_cpu_write(pcp, val) __pcpu_size_call(this_cpu_write_, pcp, val)
> | ^~~~~~~~~~~~~~~~
> linux/kernel/trace/ftrace.c:8628:17: note: in expansion of macro 'this_cpu_write'
> 8628 | this_cpu_write(tr->array_buffer.data->ftrace_ignore_pid,
> | ^~~~~~~~~~~~~~
> linux/kernel/trace/ftrace.c: In function 'ignore_task_cpu':
> linux/arch/arm64/include/asm/percpu.h:295:42: warning: conversion from 'long unsigned int' to 'u8' {aka 'unsigned char'} changes value from '18446744073709551615' to '255' [-Woverflow]
> 295 | _pcp_wrap(__percpu_write_8, pcp, (unsigned long)(val))
> | ^~~~~~~~~~~~~~~~~~~~
> linux/arch/arm64/include/asm/percpu.h:277:20: note: in definition of macro '_pcp_wrap'
> 277 | op(&(pcp), __VA_ARGS__); \
> | ^~~~~~~~~~~
> linux/include/linux/percpu-defs.h:369:25: note: in expansion of macro 'this_cpu_write_1'
> 369 | case 1: stem##1(variable, __VA_ARGS__);break; \
> | ^~~~
> linux/include/linux/percpu-defs.h:500:41: note: in expansion of macro '__pcpu_size_call'
> 500 | #define this_cpu_write(pcp, val) __pcpu_size_call(this_cpu_write_, pcp, val)
> | ^~~~~~~~~~~~~~~~
> linux/kernel/trace/ftrace.c:8898:17: note: in expansion of macro 'this_cpu_write'
> 8898 | this_cpu_write(tr->array_buffer.data->ftrace_ignore_pid,
> | ^~~~~~~~~~~~~~
> linux/arch/arm64/include/asm/percpu.h:297:43: warning: conversion from 'long unsigned int' to 'u16' {aka 'short unsigned int'} changes value from '18446744073709551615' to '65535' [-Woverflow]
> 297 | _pcp_wrap(__percpu_write_16, pcp, (unsigned long)(val))
> | ^~~~~~~~~~~~~~~~~~~~
> linux/arch/arm64/include/asm/percpu.h:277:20: note: in definition of macro '_pcp_wrap'
> 277 | op(&(pcp), __VA_ARGS__); \
> | ^~~~~~~~~~~
> linux/include/linux/percpu-defs.h:370:25: note: in expansion of macro 'this_cpu_write_2'
> 370 | case 2: stem##2(variable, __VA_ARGS__);break; \
> | ^~~~
> linux/include/linux/percpu-defs.h:500:41: note: in expansion of macro '__pcpu_size_call'
> 500 | #define this_cpu_write(pcp, val) __pcpu_size_call(this_cpu_write_, pcp, val)
> | ^~~~~~~~~~~~~~~~
> linux/kernel/trace/ftrace.c:8898:17: note: in expansion of macro 'this_cpu_write'
> 8898 | this_cpu_write(tr->array_buffer.data->ftrace_ignore_pid,
> | ^~~~~~~~~~~~~~
> linux/arch/arm64/include/asm/percpu.h:299:43: warning: conversion from 'long unsigned int' to 'u32' {aka 'unsigned int'} changes value from '18446744073709551615' to '4294967295' [-Woverflow]
> 299 | _pcp_wrap(__percpu_write_32, pcp, (unsigned long)(val))
> | ^~~~~~~~~~~~~~~~~~~~
> linux/arch/arm64/include/asm/percpu.h:277:20: note: in definition of macro '_pcp_wrap'
> 277 | op(&(pcp), __VA_ARGS__); \
> | ^~~~~~~~~~~
> linux/include/linux/percpu-defs.h:371:25: note: in expansion of macro 'this_cpu_write_4'
> 371 | case 4: stem##4(variable, __VA_ARGS__);break; \
> | ^~~~
> linux/include/linux/percpu-defs.h:500:41: note: in expansion of macro '__pcpu_size_call'
> 500 | #define this_cpu_write(pcp, val) __pcpu_size_call(this_cpu_write_, pcp, val)
> | ^~~~~~~~~~~~~~~~
> linux/kernel/trace/ftrace.c:8898:17: note: in expansion of macro 'this_cpu_write'
> 8898 | this_cpu_write(tr->array_buffer.data->ftrace_ignore_pid,
> | ^~~~~~~~~~~~~~
> ---8<---
>
>
^ permalink raw reply [flat|nested] 39+ messages in thread* Re: [PATCH v2 14/20] arm64: percpu: Implement preemptible read/write ops
2026-08-05 9:24 ` Ryan Roberts
2026-08-05 12:08 ` David Laight
@ 2026-08-05 13:34 ` Mark Rutland
1 sibling, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-05 13:34 UTC (permalink / raw)
To: Ryan Roberts
Cc: vladimir.murzin, peterz, catalin.marinas, david.laight.linux,
stable, ruanjinjie, james.morse, yang, cl, maz, david, ljs, will,
ardb, linux-arm-kernel
On Wed, Aug 05, 2026 at 10:24:04AM +0100, Ryan Roberts wrote:
> On 04/08/2026 18:04, Mark Rutland wrote:
> > ---
> FYI I'm seeing build warnings caused by this patch (with ftrace enabled - based
> on the warnings, I'm guessing that's the key bit), using:
>
> aarch64-linux-gnu-gcc (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44
>
> I haven't investigated the cause.
Thanks for the heads-up.
This is an unfortunate effect of the casting in this_cpu_write_##sz
wrappers:
(1) For cases where pcp is itself a pointer, compilers warn for the
unreachable (sz != 8) cases, unless we add a cast to unsigned long
in all of the cases.
(2) With the cast, compilers warn about implicit truncation (as you're
seeing here), where we've performed redundant extension in the first
place.
To workaround that I believe we need an explicit cast somewhere, unless we can
get the compilers to realise the other cases are obviously unreachable. Two
possible workarounds:
(a) We make __percpu_write_##sz() take an unsigned long, and cast it
down to u##sz in the asm arguments.
(b) We update this_cpu_write_{1,2,4,8} with {u8,u16,u32,u64} casts,
applied after the unsigned long cast.
For now, (a) is the simplest, as below. I'll see if there's a better option.q
Mark.
---->8----
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index d868a88960123..dd49d14d9cd91 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -149,7 +149,7 @@ static inline unsigned long __percpu_read_##sz(void __percpu *pcp) \
return val; \
} \
\
-static inline void __percpu_write_##sz(void __percpu *pcp, u##sz val) \
+static inline void __percpu_write_##sz(void __percpu *pcp, unsigned long val) \
{ \
u16 *gprs = ¤t_thread_info()->pcpu_gprs; \
unsigned long off; \
@@ -161,7 +161,7 @@ static inline void __percpu_write_##sz(void __percpu *pcp, u##sz val) \
: [gprs] "=Qo" (*gprs), \
[off] "=&r" (off) \
: [pcp] "r" (pcp), \
- [val] "r" (val) \
+ [val] "r" ((u##sz)val) \
: "memory" \
); \
}
^ permalink raw reply related [flat|nested] 39+ messages in thread
* [PATCH v2 15/20] arm64: percpu: Implement preemptible void RMW ops
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (13 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 14/20] arm64: percpu: Implement preemptible read/write ops Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-04 17:04 ` [PATCH v2 16/20] arm64: percpu: Implement preemptible return " Mark Rutland
` (4 subsequent siblings)
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
Use the PCPU GPR infrastructure to implement all of the RMW ops which do
not return a value.
Test case:
| void outline_this_cpu_add_u64(u64 __percpu *p, u64 v)
| {
| this_cpu_add(*p, v);
| }
Generated code before this patch (v7.2-rc4):
| <outline_this_cpu_add_u64>:
| paciasp
| stp x29, x30, [sp, #-16]!
| mrs x2, sp_el0
| mov x29, sp
| ldr w3, [x2, #8]
| add w3, w3, #0x1
| str w3, [x2, #8]
| mrs x3, tpidr_el1
| add x0, x0, x3
| 1: ldxr x5, [x0]
| add x5, x5, x1
| stxr w4, x5, [x0]
| cbnz w4, 1b
| ldr x0, [x2, #8]
| sub x0, x0, #0x1
| str w0, [x2, #8]
| cbz x0, 2f
| ldr x0, [x2, #8]
| cbnz x0, 3f
| 2: bl preempt_schedule_notrace
| 3: ldp x29, x30, [sp], #16
| autiasp
| ret
Generated code after this patch:
| <outline_this_cpu_add_u64>:
| mrs x2, sp_el0
| mov w4, #0xc80
| strh w4, [x2, #20]
| mrs x4, tpidr_el1
| add x3, x0, x4
| 1: ldxr x6, [x3]
| add x6, x6, x1
| stxr w5, x6, [x3]
| cbnz w5, 1b
| strh wzr, [x2, #20]
| ret
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/percpu.h | 52 ++++++++++++++++++++-------------
1 file changed, 32 insertions(+), 20 deletions(-)
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index a094c137d797f..505926363d754 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -168,23 +168,35 @@ static inline void __percpu_write_##sz(void __percpu *pcp, u##sz val) \
#define __PERCPU_OP_CASE(w, sfx, name, sz, op_llsc, op_lse) \
static inline void \
-__percpu_##name##_case_##sz(void *ptr, unsigned long val) \
+__percpu_##name##_case_##sz(void __percpu *pcp, unsigned long val) \
{ \
+ u16 *gprs = ¤t_thread_info()->pcpu_gprs; \
+ unsigned long addr; \
+ unsigned long off; \
unsigned int loop; \
u##sz tmp; \
\
- asm volatile (ARM64_LSE_ATOMIC_INSN( \
+ asm volatile ( \
+ __PCPU_GPRS_BEGIN("%[gprs]", "%[pcp]", "%[off]", "%[addr]") \
+ ARM64_LSE_ATOMIC_INSN( \
/* LL/SC */ \
- "1: ldxr" #sfx "\t%" #w "[tmp], %[ptr]\n" \
+ "1: ldxr" #sfx "\t%" #w "[tmp], [%[addr]]\n" \
#op_llsc "\t%" #w "[tmp], %" #w "[tmp], %" #w "[val]\n" \
- " stxr" #sfx "\t%w[loop], %" #w "[tmp], %[ptr]\n" \
+ " stxr" #sfx "\t%w[loop], %" #w "[tmp], [%[addr]]\n" \
" cbnz %w[loop], 1b", \
/* LSE atomics */ \
- #op_lse "\t%" #w "[val], %" #w "[tmp], %[ptr]\n" \
+ #op_lse "\t%" #w "[val], %" #w "[tmp], [%[addr]]\n" \
__nops(3)) \
- : [loop] "=&r" (loop), [tmp] "=&r" (tmp), \
- [ptr] "+Q"(*(u##sz *)ptr) \
- : [val] "r" ((u##sz)(val))); \
+ __PCPU_GPRS_END("%[gprs]") \
+ : [gprs] "=Qo" (*gprs), \
+ [addr] "=&r" (addr), \
+ [off] "=&r" (off), \
+ [loop] "=&r" (loop), \
+ [tmp] "=&r" (tmp) \
+ : [pcp] "r" (pcp), \
+ [val] "r" ((u##sz)(val)) \
+ : "memory" \
+ ); \
}
#define __PERCPU_RET_OP_CASE(w, sfx, name, sz, op_llsc, op_lse) \
@@ -301,13 +313,13 @@ PERCPU_RET_OP(add, add, ldadd)
_pcp_wrap(__percpu_write_64, pcp, (unsigned long)(val))
#define this_cpu_add_1(pcp, val) \
- _pcp_protect(__percpu_add_case_8, pcp, val)
+ _pcp_wrap(__percpu_add_case_8, pcp, val)
#define this_cpu_add_2(pcp, val) \
- _pcp_protect(__percpu_add_case_16, pcp, val)
+ _pcp_wrap(__percpu_add_case_16, pcp, val)
#define this_cpu_add_4(pcp, val) \
- _pcp_protect(__percpu_add_case_32, pcp, val)
+ _pcp_wrap(__percpu_add_case_32, pcp, val)
#define this_cpu_add_8(pcp, val) \
- _pcp_protect(__percpu_add_case_64, pcp, val)
+ _pcp_wrap(__percpu_add_case_64, pcp, val)
#define this_cpu_add_return_1(pcp, val) \
_pcp_protect_return(__percpu_add_return_case_8, pcp, val)
@@ -319,22 +331,22 @@ PERCPU_RET_OP(add, add, ldadd)
_pcp_protect_return(__percpu_add_return_case_64, pcp, val)
#define this_cpu_and_1(pcp, val) \
- _pcp_protect(__percpu_andnot_case_8, pcp, ~(u8)(val))
+ _pcp_wrap(__percpu_andnot_case_8, pcp, ~(u8)(val))
#define this_cpu_and_2(pcp, val) \
- _pcp_protect(__percpu_andnot_case_16, pcp, ~(u16)(val))
+ _pcp_wrap(__percpu_andnot_case_16, pcp, ~(u16)(val))
#define this_cpu_and_4(pcp, val) \
- _pcp_protect(__percpu_andnot_case_32, pcp, ~(u32)(val))
+ _pcp_wrap(__percpu_andnot_case_32, pcp, ~(u32)(val))
#define this_cpu_and_8(pcp, val) \
- _pcp_protect(__percpu_andnot_case_64, pcp, ~(u64)(val))
+ _pcp_wrap(__percpu_andnot_case_64, pcp, ~(u64)(val))
#define this_cpu_or_1(pcp, val) \
- _pcp_protect(__percpu_or_case_8, pcp, val)
+ _pcp_wrap(__percpu_or_case_8, pcp, val)
#define this_cpu_or_2(pcp, val) \
- _pcp_protect(__percpu_or_case_16, pcp, val)
+ _pcp_wrap(__percpu_or_case_16, pcp, val)
#define this_cpu_or_4(pcp, val) \
- _pcp_protect(__percpu_or_case_32, pcp, val)
+ _pcp_wrap(__percpu_or_case_32, pcp, val)
#define this_cpu_or_8(pcp, val) \
- _pcp_protect(__percpu_or_case_64, pcp, val)
+ _pcp_wrap(__percpu_or_case_64, pcp, val)
#define this_cpu_xchg_1(pcp, val) \
_pcp_protect_return(xchg_relaxed, pcp, val)
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 16/20] arm64: percpu: Implement preemptible return RMW ops
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (14 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 15/20] arm64: percpu: Implement preemptible void RMW ops Mark Rutland
@ 2026-08-04 17:04 ` Mark Rutland
2026-08-04 17:05 ` [PATCH v2 17/20] arm64: percpu: Implement preemptible XCHG ops Mark Rutland
` (3 subsequent siblings)
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
Use the PCPU GPR infrastructure to implement all of the RMW ops which
return a value.
Test case:
| u64 outline_this_cpu_add_return_u64(u64 __percpu *p, u64 v)
| {
| return this_cpu_add_return(*p, v);
| }
Generated code before this patch (v7.2-rc4):
| <outline_this_cpu_add_return_u64>:
| paciasp
| stp x29, x30, [sp, #-32]!
| mrs x3, sp_el0
| mov x29, sp
| ldr w2, [x3, #8]
| add w2, w2, #0x1
| str w2, [x3, #8]
| mov x2, x0
| mrs x4, tpidr_el1
| add x2, x2, x4
| 1: ldxr x0, [x2]
| add x0, x0, x1
| stxr w5, x0, [x2]
| cbnz w5, 1b
| ldr x1, [x3, #8]
| sub x1, x1, #0x1
| str w1, [x3, #8]
| cbz x1, 2f
| ldr x1, [x3, #8]
| cbnz x1, 3f
| 2: str x0, [sp, #24]
| bl 0 <preempt_schedule_notrace>
| ldr x0, [sp, #24]
| 3: ldp x29, x30, [sp], #32
| autiasp
| ret
Generated code after this patch:
| <outline_this_cpu_add_return_u64>:
| mrs x3, sp_el0
| mov w5, #0x10a0
| strh w5, [x3, #20]
| mrs x5, tpidr_el1
| add x4, x0, x5
| 1: ldxr x2, [x4]
| add x2, x2, x1
| stxr w6, x2, [x4]
| cbnz w6, 1b
| strh wzr, [x3, #20]
| mov x0, x2
| ret
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/percpu.h | 36 ++++++++++++++++++++++-----------
1 file changed, 24 insertions(+), 12 deletions(-)
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 505926363d754..8e0862704095e 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -201,24 +201,36 @@ __percpu_##name##_case_##sz(void __percpu *pcp, unsigned long val) \
#define __PERCPU_RET_OP_CASE(w, sfx, name, sz, op_llsc, op_lse) \
static inline u##sz \
-__percpu_##name##_return_case_##sz(void *ptr, unsigned long val) \
+__percpu_##name##_return_case_##sz(void __percpu *pcp, unsigned long val) \
{ \
+ u16 *gprs = ¤t_thread_info()->pcpu_gprs; \
+ unsigned long addr; \
+ unsigned long off; \
unsigned int loop; \
u##sz ret; \
\
- asm volatile (ARM64_LSE_ATOMIC_INSN( \
+ asm volatile ( \
+ __PCPU_GPRS_BEGIN("%[gprs]", "%[pcp]", "%[off]", "%[addr]") \
+ ARM64_LSE_ATOMIC_INSN( \
/* LL/SC */ \
- "1: ldxr" #sfx "\t%" #w "[ret], %[ptr]\n" \
+ "1: ldxr" #sfx "\t%" #w "[ret], [%[addr]]\n" \
#op_llsc "\t%" #w "[ret], %" #w "[ret], %" #w "[val]\n" \
- " stxr" #sfx "\t%w[loop], %" #w "[ret], %[ptr]\n" \
+ " stxr" #sfx "\t%w[loop], %" #w "[ret], [%[addr]]\n" \
" cbnz %w[loop], 1b", \
/* LSE atomics */ \
- #op_lse "\t%" #w "[val], %" #w "[ret], %[ptr]\n" \
+ #op_lse "\t%" #w "[val], %" #w "[ret], [%[addr]]\n" \
#op_llsc "\t%" #w "[ret], %" #w "[ret], %" #w "[val]\n" \
__nops(2)) \
- : [loop] "=&r" (loop), [ret] "=&r" (ret), \
- [ptr] "+Q"(*(u##sz *)ptr) \
- : [val] "r" ((u##sz)(val))); \
+ __PCPU_GPRS_END("%[gprs]") \
+ : [gprs] "=Qo" (*gprs), \
+ [addr] "=&r" (addr), \
+ [off] "=&r" (off), \
+ [loop] "=&r" (loop), \
+ [ret] "=&r" (ret) \
+ : [pcp] "r" (pcp), \
+ [val] "r" ((u##sz)(val)) \
+ : "memory" \
+ ); \
\
return ret; \
}
@@ -322,13 +334,13 @@ PERCPU_RET_OP(add, add, ldadd)
_pcp_wrap(__percpu_add_case_64, pcp, val)
#define this_cpu_add_return_1(pcp, val) \
- _pcp_protect_return(__percpu_add_return_case_8, pcp, val)
+ _pcp_wrap_return(__percpu_add_return_case_8, pcp, val)
#define this_cpu_add_return_2(pcp, val) \
- _pcp_protect_return(__percpu_add_return_case_16, pcp, val)
+ _pcp_wrap_return(__percpu_add_return_case_16, pcp, val)
#define this_cpu_add_return_4(pcp, val) \
- _pcp_protect_return(__percpu_add_return_case_32, pcp, val)
+ _pcp_wrap_return(__percpu_add_return_case_32, pcp, val)
#define this_cpu_add_return_8(pcp, val) \
- _pcp_protect_return(__percpu_add_return_case_64, pcp, val)
+ _pcp_wrap_return(__percpu_add_return_case_64, pcp, val)
#define this_cpu_and_1(pcp, val) \
_pcp_wrap(__percpu_andnot_case_8, pcp, ~(u8)(val))
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 17/20] arm64: percpu: Implement preemptible XCHG ops
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (15 preceding siblings ...)
2026-08-04 17:04 ` [PATCH v2 16/20] arm64: percpu: Implement preemptible return " Mark Rutland
@ 2026-08-04 17:05 ` Mark Rutland
2026-08-04 17:05 ` [PATCH v2 18/20] arm64: percpu: Implement preemptible CMPXCHG ops Mark Rutland
` (2 subsequent siblings)
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:05 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
Use the PCPU GPR infrastructure to implement all of the xchg ops
Test case:
| u64 outline_xchg_name(u64 *p, u64 v)
| {
| return xchg(p, v);
| }
Generated code before this patch (v7.2-rc4):
| <outline_this_cpu_xchg_u64>:
| paciasp
| stp x29, x30, [sp, #-32]!
| mrs x3, sp_el0
| mov x29, sp
| ldr w2, [x3, #8]
| add w2, w2, #0x1
| str w2, [x3, #8]
| mov x2, x0
| mrs x4, tpidr_el1
| add x2, x2, x4
| prfm pstl1strm, [x2]
| 1: ldxr x0, [x2]
| stxr w5, x1, [x2]
| cbnz w5, 1b
| ldr x1, [x3, #8]
| sub x1, x1, #0x1
| str w1, [x3, #8]
| cbz x1, 2f
| ldr x1, [x3, #8]
| cbnz x1, 3f
| 2: str x0, [sp, #24]
| bl preempt_schedule_notrace
| ldr x0, [sp, #24]
| 3: ldp x29, x30, [sp], #32
| autiasp
| ret
Generated code after this patch:
| <outline_this_cpu_xchg_u64>:
| mrs x3, sp_el0
| mov w5, #0x10a0
| strh w5, [x3, #20]
| mrs x5, tpidr_el1
| add x4, x0, x5
| prfm pstl1strm, [x4]
| 1: ldxr x2, [x4]
| stxr w6, x1, [x4]
| cbnz w6, 1b
| strh wzr, [x3, #20]
| mov x0, x2
| ret
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/percpu.h | 57 ++++++++++++++++++++++++++++++---
1 file changed, 53 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 8e0862704095e..dfae0aecdb895 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -269,6 +269,50 @@ PERCPU_RET_OP(add, add, ldadd)
#undef PERCPU_OP
#undef PERCPU_RET_OP
+#define PERCPU_XCHG_OP(w, sfx, sz) \
+static inline unsigned long \
+__percpu_xchg_case_##sz(void __percpu *pcp, u##sz val) \
+{ \
+ u16 *gprs = ¤t_thread_info()->pcpu_gprs; \
+ unsigned long addr; \
+ unsigned long off; \
+ unsigned int loop; \
+ unsigned long ret; \
+ \
+ asm volatile ( \
+ __PCPU_GPRS_BEGIN("%[gprs]", "%[pcp]", "%[off]", "%[addr]") \
+ ARM64_LSE_ATOMIC_INSN( \
+ /* LL/SC */ \
+ " prfm pstl1strm, [%[addr]]\n" \
+ "1: ldxr" #sfx "\t%" #w "[ret], [%[addr]]\n" \
+ " stxr" #sfx "\t%w[loop], %" #w "[val], [%[addr]]\n" \
+ " cbnz %w[loop], 1b\n" \
+ , \
+ /* LSE atomics */ \
+ " swp" #sfx "\t%" #w "[val], %" #w "[ret], [%[addr]]\n" \
+ __nops(3) \
+ ) \
+ __PCPU_GPRS_END("%[gprs]") \
+ : [gprs] "=Qo" (*gprs), \
+ [addr] "=&r" (addr), \
+ [off] "=&r" (off), \
+ [loop] "=&r" (loop), \
+ [ret] "=&r" (ret) \
+ : [pcp] "r" (pcp), \
+ [val] "r" (val) \
+ : "memory" \
+ ); \
+ \
+ return ret; \
+}
+
+PERCPU_XCHG_OP(w, b, 8)
+PERCPU_XCHG_OP(w, h, 16)
+PERCPU_XCHG_OP(w, , 32)
+PERCPU_XCHG_OP(x, , 64)
+
+#undef PERCPU_XCHG_OP
+
/*
* It would be nice to avoid the conditional call into the scheduler when
* re-enabling preemption for preemptible kernels, but doing that in a way
@@ -306,6 +350,11 @@ PERCPU_RET_OP(add, add, ldadd)
(typeof(pcp))op(&(pcp), ##args); \
})
+#define _pcp_wrap_xchg(op, pcp, val) \
+({ \
+ (typeof(pcp))op(&(pcp), (unsigned long)(val)); \
+})
+
#define this_cpu_read_1(pcp) \
_pcp_wrap_return(__percpu_read_8, pcp)
#define this_cpu_read_2(pcp) \
@@ -361,13 +410,13 @@ PERCPU_RET_OP(add, add, ldadd)
_pcp_wrap(__percpu_or_case_64, pcp, val)
#define this_cpu_xchg_1(pcp, val) \
- _pcp_protect_return(xchg_relaxed, pcp, val)
+ _pcp_wrap_xchg(__percpu_xchg_case_8, pcp, val)
#define this_cpu_xchg_2(pcp, val) \
- _pcp_protect_return(xchg_relaxed, pcp, val)
+ _pcp_wrap_xchg(__percpu_xchg_case_16, pcp, val)
#define this_cpu_xchg_4(pcp, val) \
- _pcp_protect_return(xchg_relaxed, pcp, val)
+ _pcp_wrap_xchg(__percpu_xchg_case_32, pcp, val)
#define this_cpu_xchg_8(pcp, val) \
- _pcp_protect_return(xchg_relaxed, pcp, val)
+ _pcp_wrap_xchg(__percpu_xchg_case_64, pcp, val)
#define this_cpu_cmpxchg_1(pcp, o, n) \
_pcp_protect_return(cmpxchg_relaxed, pcp, o, n)
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 18/20] arm64: percpu: Implement preemptible CMPXCHG ops
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (16 preceding siblings ...)
2026-08-04 17:05 ` [PATCH v2 17/20] arm64: percpu: Implement preemptible XCHG ops Mark Rutland
@ 2026-08-04 17:05 ` Mark Rutland
2026-08-04 17:05 ` [PATCH v2 19/20] arm64: percpu: Implement preemptible CMPXCHG128 ops Mark Rutland
2026-08-04 17:05 ` [PATCH v2 20/20] arm64: percpu: Remove _pcp_protect*() wrappers Mark Rutland
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:05 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
Use the PCPU GPR infrastructure to implement all of the {8,16,32,64}-bit
cmpxchg ops.
Note that before this patch, the LL/SC and LSE implementation was chosen
with an alternative branch. After this patch the implementations are
patched inline, matching the style of the other percpu ops.
Test case:
| u64 outline_this_cpu_cmpxchg_u64(u64 __percpu *p, u64 o, u64 n)
| {
| return this_cpu_cmpxchg(*p, o, n);
| }
Generated code before this patch (v7.2-rc4):
| <outline_this_cpu_cmpxchg_u64>:
| paciasp
| stp x29, x30, [sp, #-32]!
| mrs x4, sp_el0
| mov x29, sp
| ldr w3, [x4, #8]
| add w3, w3, #0x1
| str w3, [x4, #8]
| mrs x3, tpidr_el1
| add x3, x0, x3
| b 4f // alternative branch
| cas x1, x2, [x3]
| mov x0, x1
| 1: mrs x2, sp_el0
| ldr x1, [x2, #8]
| sub x1, x1, #0x1
| str w1, [x2, #8]
| cbz x1, 2f
| ldr x1, [x2, #8]
| cbnz x1, 3f
| 2: str x0, [sp, #24]
| bl preempt_schedule_notrace
| ldr x0, [sp, #24]
| 3: ldp x29, x30, [sp], #32
| autiasp
| ret
| 4: prfm pstl1strm, [x3]
| 5: ldxr x0, [x3]
| eor x4, x0, x1
| cbnz x4, 6f
| stxr w4, x2, [x3]
| cbnz w4, 5b
| 6: b 1b
Generated code after this patch:
| <outline_this_cpu_cmpxchg_u64>:
| mrs x4, sp_el0
| mov w6, #0x14c0
| strh w6, [x4, #20]
| mrs x6, tpidr_el1
| add x5, x0, x6
| prfm pstl1strm, [x5]
| 1: ldxr x3, [x5]
| eor x7, x3, x1
| cbnz x7, 2f
| stxr w7, x2, [x5]
| cbnz w7, 1b
| 2: strh wzr, [x4, #20]
| mov x0, x3
| ret
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
WIP: improve extension in this_cpu_cmpxchg*()
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
---
arch/arm64/include/asm/percpu.h | 69 +++++++++++++++++++++++++++++++--
1 file changed, 65 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index dfae0aecdb895..af1d4ca5c85ef 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -306,12 +306,67 @@ __percpu_xchg_case_##sz(void __percpu *pcp, u##sz val) \
return ret; \
}
+#define PERCPU_CMPXCHG_OP(w, sfx, sz) \
+static inline unsigned long \
+__percpu_cmpxchg_case_##sz(void __percpu *pcp, \
+ u##sz old, \
+ u##sz new) \
+{ \
+ /* \
+ * Sub-word sizes require zero extension so that EOR+CBNZ don't \
+ * consume non-zero upper bits of the register containing "old".\
+ */ \
+ xwreg_t(w) cmpval = xwreg_zero_extend(old, w, sz); \
+ u16 *gprs = ¤t_thread_info()->pcpu_gprs; \
+ unsigned long addr; \
+ unsigned long off; \
+ unsigned long tmp; \
+ unsigned long oldval; \
+ \
+ asm volatile ( \
+ __PCPU_GPRS_BEGIN("%[gprs]", "%[pcp]", "%[off]", "%[addr]") \
+ ARM64_LSE_ATOMIC_INSN( \
+ /* LL/SC */ \
+ " prfm pstl1strm, [%[addr]]\n" \
+ "1: ldxr" #sfx "\t%" #w "[oldval], [%[addr]]\n" \
+ " eor %" #w "[tmp], %" #w "[oldval], %" #w "[old]\n" \
+ " cbnz %" #w "[tmp], 2f\n" \
+ " stxr" #sfx "\t%w[tmp], %" #w "[new], [%[addr]]\n" \
+ " cbnz %w[tmp], 1b\n" \
+ "2:\n" \
+ , \
+ /* LSE atomics */ \
+ " mov %" #w "[oldval], %" #w "[old]\n" \
+ " cas" #sfx "\t%" #w "[oldval], %" #w "[new], [%[addr]]\n"\
+ __nops(4) \
+ ) \
+ __PCPU_GPRS_END("%[gprs]") \
+ : [gprs] "=Qo" (*gprs), \
+ [addr] "=&r" (addr), \
+ [off] "=&r" (off), \
+ [tmp] "=&r" (tmp), \
+ [oldval] "=&r" (oldval) \
+ : [pcp] "r" (pcp), \
+ [old] "r" (cmpval), \
+ [new] "r" (new) \
+ : "memory" \
+ ); \
+ \
+ return oldval; \
+}
+
PERCPU_XCHG_OP(w, b, 8)
PERCPU_XCHG_OP(w, h, 16)
PERCPU_XCHG_OP(w, , 32)
PERCPU_XCHG_OP(x, , 64)
+PERCPU_CMPXCHG_OP(w, b, 8)
+PERCPU_CMPXCHG_OP(w, h, 16)
+PERCPU_CMPXCHG_OP(w, , 32)
+PERCPU_CMPXCHG_OP(x, , 64)
+
#undef PERCPU_XCHG_OP
+#undef PERCPU_CMPXCHG_OP
/*
* It would be nice to avoid the conditional call into the scheduler when
@@ -355,6 +410,12 @@ PERCPU_XCHG_OP(x, , 64)
(typeof(pcp))op(&(pcp), (unsigned long)(val)); \
})
+#define _pcp_wrap_cmpxchg(op, pcp, old, new) \
+({ \
+ (typeof(pcp))op(&(pcp), (unsigned long)(old), \
+ (unsigned long)(new)); \
+})
+
#define this_cpu_read_1(pcp) \
_pcp_wrap_return(__percpu_read_8, pcp)
#define this_cpu_read_2(pcp) \
@@ -419,13 +480,13 @@ PERCPU_XCHG_OP(x, , 64)
_pcp_wrap_xchg(__percpu_xchg_case_64, pcp, val)
#define this_cpu_cmpxchg_1(pcp, o, n) \
- _pcp_protect_return(cmpxchg_relaxed, pcp, o, n)
+ _pcp_wrap_cmpxchg(__percpu_cmpxchg_case_8, pcp, o, n)
#define this_cpu_cmpxchg_2(pcp, o, n) \
- _pcp_protect_return(cmpxchg_relaxed, pcp, o, n)
+ _pcp_wrap_cmpxchg(__percpu_cmpxchg_case_16, pcp, o, n)
#define this_cpu_cmpxchg_4(pcp, o, n) \
- _pcp_protect_return(cmpxchg_relaxed, pcp, o, n)
+ _pcp_wrap_cmpxchg(__percpu_cmpxchg_case_32, pcp, o, n)
#define this_cpu_cmpxchg_8(pcp, o, n) \
- _pcp_protect_return(cmpxchg_relaxed, pcp, o, n)
+ _pcp_wrap_cmpxchg(__percpu_cmpxchg_case_64, pcp, o, n)
#define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n)
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 19/20] arm64: percpu: Implement preemptible CMPXCHG128 ops
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (17 preceding siblings ...)
2026-08-04 17:05 ` [PATCH v2 18/20] arm64: percpu: Implement preemptible CMPXCHG ops Mark Rutland
@ 2026-08-04 17:05 ` Mark Rutland
2026-08-04 17:05 ` [PATCH v2 20/20] arm64: percpu: Remove _pcp_protect*() wrappers Mark Rutland
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:05 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
Use the PCPU GPR infrastructure to implement this_cpu_cmpxchg128().
Note that before this patch, the LL/SC and LSE implementation was chosen
with an alternative branch. After this patch the implementations are
patched inline, matching the style of the other percpu ops.
The sub-optimal register shuffling before and after this patch occurs
because we hard-code specific registers, as LLVM (currently) lacks a way
to place a 128-bit type in a compiler-allocated even-odd pair of
registers for inline assembly. We should be able to improve that in
future, given compiler support.
Test case:
| u128 outline_this_cpu_cmpxchg128(u128 __percpu *p, u128 old, u128 new)
| {
| return this_cpu_cmpxchg128(*p, old, new);
| }
Generated code before this patch (v7.2-rc4):
| <outline_this_cpu_cmpxchg128>:
| paciasp
| stp x29, x30, [sp, #-32]!
| mov x6, x0
| mov x0, x2
| mov x29, sp
| mov x1, x3
| mrs x2, sp_el0
| ldr w7, [x2, #8]
| add w7, w7, #0x1
| str w7, [x2, #8]
| mrs x2, tpidr_el1
| add x6, x6, x2
| b 4f
| mov x2, x4
| mov x3, x5
| mov x4, x6
| mov x5, x0
| mov x7, x1
| casp x0, x1, x2, x3, [x6]
| 1: mrs x3, sp_el0
| ldr x2, [x3, #8]
| sub x2, x2, #0x1
| str w2, [x3, #8]
| cbz x2, 2f
| ldr x2, [x3, #8]
| cbnz x2, 3f
| 2: stp x0, x1, [sp, #16]
| bl preempt_schedule_notrace
| ldp x0, x1, [sp, #16]
| 3: ldp x29, x30, [sp], #32
| autiasp
| ret
| 4: prfm pstl1strm, [x6]
| 5: ldxp x8, x7, [x6]
| cmp x8, x0
| ccmp x7, x3, #0x0, eq
| b.ne 6f
| stxp w2, x4, x5, [x6]
| cbnz w2, 5b
| 6: mov x0, x8
| mov x1, x7
| b 1b
Generated code after this patch:
| <outline_this_cpu_cmpxchg128>:
| mov x6, x0
| mov x1, x3
| mov x0, x2
| mov x3, x5
| mrs x7, sp_el0
| mov x2, x4
| mov w5, #0x10a6
| strh w5, [x7, #20]
| mrs x5, tpidr_el1
| add x4, x6, x5
| prfm pstl1strm, [x4]
| 1: ldxp x9, x10, [x4]
| cmp x9, x0
| ccmp x10, x1, #0x0, eq
| b.ne 2f
| stxp w8, x2, x3, [x4]
| cbnz w8, 1b
| 2: strh wzr, [x7, #20]
| mov x0, x9
| mov x1, x10
| ret
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/percpu.h | 70 +++++++++++++++++++++++++++------
1 file changed, 57 insertions(+), 13 deletions(-)
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index af1d4ca5c85ef..8087ed6e12241 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -490,19 +490,63 @@ PERCPU_CMPXCHG_OP(x, , 64)
#define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n)
-#define this_cpu_cmpxchg128(pcp, o, n) \
-({ \
- typedef typeof(pcp) pcp_op_T__; \
- u128 old__, new__, ret__; \
- pcp_op_T__ *ptr__; \
- old__ = o; \
- new__ = n; \
- preempt_disable_notrace(); \
- ptr__ = raw_cpu_ptr(&(pcp)); \
- ret__ = cmpxchg128_local((void *)ptr__, old__, new__); \
- preempt_enable_notrace(); \
- ret__; \
-})
+static inline u128
+__percpu_cmpxchg_128(void __percpu *pcp, u128 old, u128 new)
+{
+ u16 *gprs = ¤t_thread_info()->pcpu_gprs;
+ unsigned long addr;
+ unsigned long off;
+ union __u128_halves r, o = { .full = (old) },
+ n = { .full = (new) };
+ register unsigned long ol asm ("x0") = o.low;
+ register unsigned long oh asm ("x1") = o.high;
+ register unsigned long nl asm ("x2") = n.low;
+ register unsigned long nh asm ("x3") = n.high;
+ unsigned long rl, rh;
+ unsigned long tmp;
+
+ asm volatile (
+ __PCPU_GPRS_BEGIN("%[gprs]", "%[pcp]", "%[off]", "%[addr]")
+ ARM64_LSE_ATOMIC_INSN(
+ /* LL/SC */
+ " prfm pstl1strm, [%[addr]]\n"
+ "1: ldxp %[rl], %[rh], [%[addr]]\n"
+ " cmp %[rl], %[ol]\n"
+ " ccmp %[rh], %[oh], 0, eq\n"
+ " b.ne 2f\n"
+ " stxp %w[tmp], %[nl], %[nh], [%[addr]]\n"
+ " cbnz %w[tmp], 1b\n"
+ "2:\n"
+ ,
+ /* LSE atomics */
+ " casp %[ol], %[oh], %[nl], %[nh], [%[addr]]\n"
+ " mov %[rl], %[ol]\n"
+ " mov %[rh], %[oh]\n"
+ __nops(4)
+ )
+ __PCPU_GPRS_END("%[gprs]")
+ : [gprs] "=Qo" (*gprs),
+ [addr] "=&r" (addr),
+ [off] "=&r" (off),
+ [tmp] "=&r" (tmp),
+ [ol] "+&r" (ol),
+ [oh] "+&r" (oh),
+ [rl] "=&r" (rl),
+ [rh] "=&r" (rh)
+ : [pcp] "r" (pcp),
+ [nl] "r" (nl),
+ [nh] "r" (nh)
+ : "memory", "cc"
+ );
+
+ r.low = rl;
+ r.high = rh;
+
+ return r.full;
+}
+
+#define this_cpu_cmpxchg128(pcp, o, n) \
+ _pcp_wrap_return(__percpu_cmpxchg_128, pcp, o, n)
#ifdef __KVM_NVHE_HYPERVISOR__
extern unsigned long __hyp_per_cpu_offset(unsigned int cpu);
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v2 20/20] arm64: percpu: Remove _pcp_protect*() wrappers
2026-08-04 17:04 [PATCH v2 00/20] arm64: Preemptible this_cpu_*() operations Mark Rutland
` (18 preceding siblings ...)
2026-08-04 17:05 ` [PATCH v2 19/20] arm64: percpu: Implement preemptible CMPXCHG128 ops Mark Rutland
@ 2026-08-04 17:05 ` Mark Rutland
19 siblings, 0 replies; 39+ messages in thread
From: Mark Rutland @ 2026-08-04 17:05 UTC (permalink / raw)
To: linux-arm-kernel
Cc: mark.rutland, vladimir.murzin, ryan.roberts, peterz,
catalin.marinas, david.laight.linux, stable, ruanjinjie,
james.morse, yang, cl, maz, david, ljs, will, ardb
Now that all this_cpu_*() operations are preemptible, there are no users
of _pcp_protect() or _pcp_protect_return().
Remove them both, along with the associated comment regarding
preemption.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Ada Couprie Diaz <ada.coupriediaz@arm.com>
Cc: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Jinjie Ruan <ruanjinjie@huawei.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Vladimir Murzin <vladimir.murzin@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yang Shi <yang@os.amperecomputing.com>
---
arch/arm64/include/asm/percpu.h | 27 ---------------------------
1 file changed, 27 deletions(-)
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 8087ed6e12241..d868a88960123 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -368,33 +368,6 @@ PERCPU_CMPXCHG_OP(x, , 64)
#undef PERCPU_XCHG_OP
#undef PERCPU_CMPXCHG_OP
-/*
- * It would be nice to avoid the conditional call into the scheduler when
- * re-enabling preemption for preemptible kernels, but doing that in a way
- * which builds inside a module would mean messing directly with the preempt
- * count. If you do this, peterz and tglx will hunt you down.
- *
- * Not to mention it'll break the actual preemption model for missing a
- * preemption point when TIF_NEED_RESCHED gets set while preemption is
- * disabled.
- */
-
-#define _pcp_protect(op, pcp, ...) \
-({ \
- preempt_disable_notrace(); \
- op(raw_cpu_ptr(&(pcp)), __VA_ARGS__); \
- preempt_enable_notrace(); \
-})
-
-#define _pcp_protect_return(op, pcp, args...) \
-({ \
- typeof(pcp) __retval; \
- preempt_disable_notrace(); \
- __retval = (typeof(pcp))op(raw_cpu_ptr(&(pcp)), ##args); \
- preempt_enable_notrace(); \
- __retval; \
-})
-
#define _pcp_wrap(op, pcp, ...) \
({ \
op(&(pcp), __VA_ARGS__); \
--
2.30.2
^ permalink raw reply related [flat|nested] 39+ messages in thread