From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9D427C55184 for ; Tue, 4 Aug 2026 17:06:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-Id:Date :Subject:To:From:Reply-To:Content-Type:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3YqmqGKIPkphFV2jpv6a6xSKr4jecc8pLFqqUVC6B/4=; b=JDiIkH99w3i0CX x9K1Z/9JzNWGWXhPjCjOPVr7GOy9hbGlgi2uOoR/AlgdMSMY20WRVMzQ8jcM7+FzqyP/KEKrwE8f6 6HpLLDxedfGAcdamFBrBUpBhz9LOH4yICNa3RJqcQ+O0tTT1uyx+ySmh1cKrWq7TEFiaDCKXTcarn T5OanJSJN+rwcNtSvUxhyHrpxYGfnz3fhA4eyI6xonRAaSUv2gCwp5YVM3BUO84wGEAAaq85WcVgC dV5iQHxU+Er47MYocdhdHqm70Ma1HNcbg8nQiPyuZ/b4tueVpOkE/fzeVmJI0gnV5vcqfKESqUp5L HN5sSssoi0CuqXXZdOrg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1wrIam-00000002QVL-1bh3; Tue, 04 Aug 2026 17:06:16 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1wrIah-00000002QPp-2C3O for linux-arm-kernel@lists.infradead.org; Tue, 04 Aug 2026 17:06:12 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id F121C1684; Tue, 4 Aug 2026 10:06:06 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 293303F632; Tue, 4 Aug 2026 10:06:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1785863171; bh=XZjvGkTQofdBvvBwSuBBvXySbzPZJO4hBi/MD5ui/Nk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=C3HdOizzKiGOwIKH+hZzDcVkHqzhmhmHvBYB3DWy4UWFurOF6YXL++L+zFPeVavel 6xnXc0dgdXfe+aqLL+s8NA8ki+PtkXYW1WO+uqV+sX9ibyxRl4t9FnLTCa4TqMbDd0 JOUDd1eTxA0MPYFevtL8g1AO2QI3DSMrOBa+89JQ= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 13/20] arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops Date: Tue, 4 Aug 2026 18:04:56 +0100 Message-Id: <20260804170503.3513916-14-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260804170503.3513916-1-mark.rutland@arm.com> References: <20260804170503.3513916-1-mark.rutland@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.9.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260804_100611_669348_FCFE2EE8 X-CRM114-Status: GOOD ( 27.74 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, vladimir.murzin@arm.com, ryan.roberts@arm.com, peterz@infradead.org, catalin.marinas@arm.com, david.laight.linux@gmail.com, stable@vger.kernel.org, ruanjinjie@huawei.com, james.morse@arm.com, yang@os.amperecomputing.com, cl@gentwo.org, maz@kernel.org, david@kernel.org, ljs@kernel.org, will@kernel.org, ardb@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently arm64's this_cpu_*() ops transiently disable preemption in order to guarantee that the address generation and memory access(es) occur on the same CPU. Transiently disabling preemption can be expensive. When re-enabling preemption it is necessary to make a conditional function call to preempt_schedule[_notrace]() in order to handle the rare case that the task needs to be rescheduled. The potential function call has a number of negative effects on code generation (e.g. due to the need to create a stack frame and spill registers), and the conditionality can result in poor code generation and/or poor branch prediction. This patch adds infrastructure for a scheme where this_cpu_*() ops do not need to transiently disable preemption, avoiding the negative impacts described above. Individual operations will be converted in subsequent patches. Each operation registers a critical section during which the exception return code will adjust the offset and addresses if preemption occurs mid-sequence. The critical section is registered/unregistered with a small prologue and epilogue which encodes three distinct GPRRs (, , ) into a new thread_info::pcp_gprs field: // Prologue. Enable fixups for and . mrs , sp_el0 mov , #__VAL_PCPU_GPRS(, , ) strh , [, #TSK_TI_PCPU_GPRS] // Generate cpu-specific address mrs , TPIDR_ELx add , , // Perform access sequence ldr , [] // Epilogue. Disable fixups strh wzr, [, #TSK_TI_PCPU_GPRS] If an exception is taken from within the critical section, the exception return code will adjust to be the current CPU's offset, and will adjust to be ( + ). Distinct registers are used for , , and , so that the fixup can be applied safely at any point during the critical section. To ensure that this_cpu_*() operations within exception handlers work correctly and do not corrupt state, thread_info::pcpu_gprs is saved into a new pt_regs::pcpu_gprs field upon exception entry, and restored upon exception return. Looking at a simple this_cpu_operation: | void outline_this_cpu_add_u64(u64 __percpu *p, u64 v) | { | this_cpu_add(*p, v); | } Atop v7.2-rc4, with GCC 15.2.0 and defconfig, this is compiled as: | : | paciasp | stp x29, x30, [sp, #-16]! | mrs x2, sp_el0 | mov x29, sp | ldr w3, [x2, #8] | add w3, w3, #0x1 | str w3, [x2, #8] | mrs x3, tpidr_el1 | add x0, x0, x3 | 1: ldxr x5, [x0] | add x5, x5, x1 | stxr w4, x5, [x0] | cbnz w4, 1b | ldr x0, [x2, #8] | sub x0, x0, #0x1 | str w0, [x2, #8] | cbz x0, 2f | ldr x0, [x2, #8] | cbnz x0, 3f | 2: bl preempt_schedule_notrace | 3: ldp x29, x30, [sp], #16 | autiasp | ret With the scheme added in this patch, this can be compiled as: | : | mrs x2, sp_el0 | mov x4, #0xc80 // __VAL_PCPU_GPRS(x0, x4, x3) | strh w4, [x2, #20] | mrs x4, tpidr_el1 | add x3, x0, x4 | 1: ldxr x6, [x3] | add x6, x6, x1 | stxr w5, x6, [x3] | cbnz w5, 1b | strh wzr, [x2, #20] | ret Signed-off-by: Mark Rutland Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Jinjie Ruan Cc: Marc Zyngier Cc: Peter Zijlstra Cc: Vladimir Murzin Cc: Will Deacon Cc: Yang Shi --- arch/arm64/include/asm/percpu.h | 71 ++++++++++++++++++++++++++++ arch/arm64/include/asm/ptrace.h | 5 ++ arch/arm64/include/asm/thread_info.h | 1 + arch/arm64/kernel/asm-offsets.c | 2 + arch/arm64/kernel/entry-common.c | 38 +++++++++++++++ arch/arm64/kernel/entry.S | 19 ++++++++ 6 files changed, 136 insertions(+) diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h index 29f20c8748fe1..8b4c9ea05d4e6 100644 --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -5,10 +5,13 @@ #ifndef __ASM_PERCPU_H #define __ASM_PERCPU_H +#include #include +#include #include #include +#include #include #include @@ -51,6 +54,74 @@ static inline unsigned long __kern_my_cpu_offset(void) return off; } +#define PCPU_GPR_PCP_SHIFT 0 +#define PCPU_GPR_PCP GENMASK(4, 0) +#define PCPU_GPR_OFF_SHIFT 5 +#define PCPU_GPR_OFF GENMASK(9, 5) +#define PCPU_GPR_ADDR_SHIFT 10 +#define PCPU_GPR_ADDR GENMASK(14, 10) + +#define __VAL_PCPU_GPRS(pcp, off, addr) \ + "(" \ + "(" __GPR_NUM(pcp) " << " __stringify(PCPU_GPR_PCP_SHIFT) ") | " \ + "(" __GPR_NUM(off) " << " __stringify(PCPU_GPR_OFF_SHIFT) ") | " \ + "(" __GPR_NUM(addr) " << " __stringify(PCPU_GPR_ADDR_SHIFT) ")" \ + ")" + +#define __ASSERT_PCPU_GPRS_DISTINCT(pcp, off, addr) \ + ".if (" \ + "(" __GPR_NUM(pcp) " == " __GPR_NUM(off) ") || " \ + "(" __GPR_NUM(pcp) " == " __GPR_NUM(addr) ") || " \ + "(" __GPR_NUM(off) " == " __GPR_NUM(addr) ")" \ + " )\n" \ + ".error \"PCPU GPRS overlap: {" pcp "," off "," addr "}\"\n" \ + ".endif\n" + +#define ____PCPU_GPRS_BEGIN(gprs, pcp, off, addr) \ + "// ____PCPU_GPRS_BEGIN(" gprs ", " pcp ", " off ", " addr")\n" \ + __DEFINE_ASM_GPR_NUMS \ + __DEFINE_ASM_GPR_ALIASES \ + __ASSERT_PCPU_GPRS_DISTINCT(pcp, off, addr) \ + " mov w" off ", #" __VAL_PCPU_GPRS(pcp, off, addr) "\n" \ + " strh w" off ", " gprs "\n" \ + __KERN_ASM_CPU_OFFSET(off) "\n" + +/* + * Begin a PCPU GPR critical section which requires (and ). + * + * At the start of the critical section, and upon any (preemptible) exception + * until __PCPU_GPRS_END(): + * - will be set to the current CPU's percpu offset. + * - will be set to + . + * + * The , , and registers must be distinct GPRs. + * + * must be '¤t_thread_info()->gprs', as a memory operand which can + * be written both at the start and end of the critical section (e.g. using + * "=Qo" constraints). + */ +#define __PCPU_GPRS_BEGIN(gprs, pcp, off, addr) \ + ____PCPU_GPRS_BEGIN(gprs, pcp, off, addr) \ + " add " addr ", " pcp ", " off "\n" + +/* + * Begin a PCPU GPR critical section which only requires and does not + * require . + * + * This is only for operations that can use register-offset addressing, + * e.g. STR , [, #include #include +#include #include #include #include #include #include +/* + * Where the context being returned to had an active percpu GPR critical + * section, ensure that the offset and address GPRs are updated to match the + * current CPU. + * + * For simplicity we always update the GPRs when a critical section is active + * and preemption was *possible*, regardless of whether preemption actually + * occurred. Where preemption did not occur, the updates are redundant but not + * harmful. + */ +static __always_inline void irqentry_exit_pcpu_adjust(struct pt_regs *regs) +{ + int reg_pcp, reg_off, reg_addr; + unsigned long pcp, off, addr; + u16 gprs = regs->pcpu_gprs; + + /* + * Zero means no active PCPU GPRs. As the PCPU GPRs must be distinct, + * a PCPU critical section cannot possibly use {x0,x0,x0}. + */ + if (likely(!gprs)) + return; + + reg_pcp = FIELD_GET(PCPU_GPR_PCP, gprs); + reg_off = FIELD_GET(PCPU_GPR_OFF, gprs); + reg_addr = FIELD_GET(PCPU_GPR_ADDR, gprs); + + pcp = pt_regs_read_reg(regs, reg_pcp); + + off = __kern_my_cpu_offset(); + pt_regs_write_reg(regs, reg_off, off); + + addr = pcp + off; + pt_regs_write_reg(regs, reg_addr, addr); +} + /* * Handle IRQ/context state management when entering from kernel mode. * Before this function is called it is not safe to call regular kernel code, @@ -58,6 +95,7 @@ static void noinstr arm64_exit_to_kernel_mode(struct pt_regs *regs, local_irq_disable(); irqentry_exit_to_kernel_mode_preempt(regs, state); local_daif_mask(); + irqentry_exit_pcpu_adjust(regs); mte_check_tfsr_exit(); irqentry_exit_to_kernel_mode_after_preempt(regs, state); } diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 1b4cd70515550..567d8096b87d9 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -194,6 +194,17 @@ alternative_cb_end #endif .endm + .macro pcpu_gprs_entry, tsk:req, regs:req, tmp:req + ldrh w\tmp, [\tsk, #TSK_TI_PCPU_GPRS] + strh w\tmp, [\regs, #S_PCPU_GPRS] + strh wzr, [\tsk, #TSK_TI_PCPU_GPRS] + .endm + + .macro pcpu_gprs_exit, tsk:req, regs:req, tmp:req + ldrh w\tmp, [\regs, #S_PCPU_GPRS] + strh w\tmp, [\tsk, #TSK_TI_PCPU_GPRS] + .endm + .macro kernel_entry, el, regsize = 64 .if \el == 0 alternative_insn nop, SET_PSTATE_DIT(1), ARM64_HAS_DIT @@ -277,6 +288,7 @@ alternative_else_nop_endif .else add x21, sp, #PT_REGS_SIZE get_current_task tsk + pcpu_gprs_entry tsk, sp, x0 .endif /* \el == 0 */ mrs x22, elr_el1 mrs x23, spsr_el1 @@ -335,6 +347,7 @@ alternative_else_nop_endif .macro kernel_exit, el .if \el != 0 disable_daif + pcpu_gprs_exit tsk, sp, x0 .endif #ifdef CONFIG_ARM64_PSEUDO_NMI @@ -1044,10 +1057,16 @@ SYM_CODE_START(__sdei_asm_handler) stp x29, x4, [sp, #-16]! mov x29, sp + add x16, x19, #SDEI_EVENT_INTREGS + pcpu_gprs_entry tsk, x16, x17 + add x0, x19, #SDEI_EVENT_INTREGS mov x1, x19 bl __sdei_handler + add x16, x19, #SDEI_EVENT_INTREGS + pcpu_gprs_exit tsk, x16, x17 + msr sp_el0, x20 /* restore regs >x17 that firmware won't restore */ mov x4, x19 // keep x4 for __sdei_asm_exit_trampoline -- 2.30.2