From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1795044158; b=lU0cEkzCxgJCvbRMPsLK2IErT6sX9RO2/zHGjuxjlHHXaZ6gbDAQQdix5xcLPSLGospm rOBXawFtIzkaBj2lP0SaD29DL7InNOmA64Hr8jB/x+mUQ8nbchi+na7aGmzJTA46e8skG acLDVaitjwO/MFmglblBSXdAogZNMgQ+ftFPXC2AX4RTtt63yRXW0HLdQDUt0PXUbklH6 lkgwKhoMGJLl1pGIdEaEqxp1VPwYprGOJyw94EV8Po1baqy+JuIaIwun98SMMT+pJZveL 8ZA5T8fRucg1Laa2XRsmHe5m/NmfkY9YPSy87U/vhMRlssssuajbSymO0jmIwCcBPtQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1795044158; h=Received:DKIM-Signature:From:To:Cc:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:User-Agent:X-stable:X-Patchwork-Hint: MIME-Version:Content-Transfer-Encoding; bh=vsQyqBNfekSva9UgdBc0F0S0upWYgmuiP5YKjTir2jA=; b=3Xdaez737CPSqo3879wVChCYSmEOI0A6cXCJHydUfcqcxGVDCuOcu9lMT4sUEfrS/cZY iADJl6LDiH+uSp8IAcrBkaDXTdu14O9T87rIIfgJ7/d+OZP4q3p38ed427dgHpitbOfUZ iGTMVMpFz/TaUtj3eVLtvGfhAL24UZfgnHbmCy+RLj2LPTm/1uB3A4HutazvKlHNPtH2+ SS6RMc8+KmyxanPlDrl3kZCMt5oF4PYPH3QptONK50VzKPnV1gljYP4nzV8HqWHn3jxnJ MliGUvtTFiEky46ncEyBUEQKgiSYux/Hfn1hcMT583WR3214LxoGstSfUrNkTNGqNHw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b="fpz/+g2A"; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6AC8606A31C; Wed, 18 Nov 2026 23:22:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=fpz/+g2A Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3830E1F000C4; Wed, 18 Nov 2026 23:22:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=korg; t=1795044158; bh=vsQyqBNfekSva9UgdBc0F0S0upWYgmuiP5YKjTir2jA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=fpz/+g2AMpS12PE0bRAZ5aPM9GPsQiqWx4B6jo1QhNWCzDg6wG5oBjMC2JJAXo4fd kotnECfSGJ5HdGGd7R0hS4wVJoBTf52V4tz1eIG3PQXwLMsMwzFq87HCjGBpc7wEBF 9l58Rze2w3wdmbTpP1ycQVWtfDjWKAh/whmctBeA= From: Greg Kroah-Hartman To: stable@vger.kernel.org Cc: Greg Kroah-Hartman , patches@lists.linux.dev, Catalin Marinas , "Paul E. McKenney" , Will Deacon , Palmer Dabbelt , Sasha Levin Subject: [PATCH 6.1 0419/2857] arm64: Use load LSE atomics for the non-return per-CPU atomic operations Date: Wed, 18 Nov 2026 15:24:13 +0100 Message-ID: <20261118150216.467316559@linuxfoundation.org> X-Mailer: git-send-email 2.56.0 In-Reply-To: <20261118150201.293410965@linuxfoundation.org> References: <20261118150201.293410965@linuxfoundation.org> User-Agent: quilt/0.69 X-stable: review X-Patchwork-Hint: ignore X-Mailing-List: patches@lists.linux.dev List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit 6.1-stable review patch. If anyone has any objections, please let me know. ------------------ From: Catalin Marinas [ Upstream commit 535fdfc5a228524552ee8810c9175e877e127c27 ] The non-return per-CPU this_cpu_*() atomic operations are implemented as STADD/STCLR/STSET when FEAT_LSE is available. On many microarchitecture implementations, these instructions tend to be executed "far" in the interconnect or memory subsystem (unless the data is already in the L1 cache). This is in general more efficient when there is contention as it avoids bouncing cache lines between CPUs. The load atomics (e.g. LDADD without XZR as destination), OTOH, tend to be executed "near" with the data loaded into the L1 cache. STADD executed back to back as in srcu_read_{lock,unlock}*() incur an additional overhead due to the default posting behaviour on several CPU implementations. Since the per-CPU atomics are unlikely to be used concurrently on the same memory location, encourage the hardware to to execute them "near" by issuing load atomics - LDADD/LDCLR/LDSET - with the destination register unused (but not XZR). Signed-off-by: Catalin Marinas Link: https://lore.kernel.org/r/e7d539ed-ced0-4b96-8ecd-048a5b803b85@paulmck-laptop Reported-by: Paul E. McKenney Tested-by: Paul E. McKenney Cc: Will Deacon Reviewed-by: Palmer Dabbelt [will: Add comment and link to the discussion thread] Signed-off-by: Will Deacon Stable-dep-of: 8cf2093f5372 ("arm64: percpu: Fix LSE operations on {8,16}-bit types") Signed-off-by: Sasha Levin --- arch/arm64/include/asm/percpu.h | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h index 3b57b36161412..f4b57b911139d 100644 --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -77,7 +77,7 @@ __percpu_##name##_case_##sz(void *ptr, unsigned long val) \ " stxr" #sfx "\t%w[loop], %" #w "[tmp], %[ptr]\n" \ " cbnz %w[loop], 1b", \ /* LSE atomics */ \ - #op_lse "\t%" #w "[val], %[ptr]\n" \ + #op_lse "\t%" #w "[val], %" #w "[tmp], %[ptr]\n" \ __nops(3)) \ : [loop] "=&r" (loop), [tmp] "=&r" (tmp), \ [ptr] "+Q"(*(u##sz *)ptr) \ @@ -124,9 +124,16 @@ PERCPU_RW_OPS(8) PERCPU_RW_OPS(16) PERCPU_RW_OPS(32) PERCPU_RW_OPS(64) -PERCPU_OP(add, add, stadd) -PERCPU_OP(andnot, bic, stclr) -PERCPU_OP(or, orr, stset) + +/* + * Use value-returning atomics for CPU-local ops as they are more likely + * to execute "near" to the CPU (e.g. in L1$). + * + * https://lore.kernel.org/r/e7d539ed-ced0-4b96-8ecd-048a5b803b85@paulmck-laptop + */ +PERCPU_OP(add, add, ldadd) +PERCPU_OP(andnot, bic, ldclr) +PERCPU_OP(or, orr, ldset) PERCPU_RET_OP(add, add, ldadd) #undef PERCPU_RW_OPS -- 2.53.0