From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1795034722; b=phrArIBpdcTgJ/8oKdm82R9/U0Ip8dvLBwQmR7PjClAYRxusZ9hyIEwm5+zOl4GizbL4 YhBbz3S29NS+W/L66RH/FOVtpof/BxoIKC2YeKfh31UvO7WxIm9TqaDOtW2StHst7ZQWR SkewVfJbB7cSgbW2B0Q2+of/wmWv6tMaYMUeEm6PJ4joBwlVc6GFWO6utqmveeNPMfeHw PtOxbaDIzIwkyOpVILyRM4LL1CWb7PfBvXWxwv9CU7WEBkiJrg/XY0QpLj/U6SGZFvgPj GyXFrUGlJylZK45RHm8TSNDutnGaIpygGpMx+ILkibGtBfQkZbR+JarLsWkDcXPGn9Q== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1795034722; h=Received:DKIM-Signature:From:To:Cc:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:User-Agent:X-stable:X-Patchwork-Hint: MIME-Version:Content-Transfer-Encoding; bh=G9COZYt+8p7wUfOaMfnj10WInUYt2RydzZVrNgEt0fQ=; b=is5UB0ZfTgvbXiEftbpXYRPrGJgGVqzPO1xMfvZPnFjKIllGKoK4L4qsWbVtz7TRDStP 0+jC/UagDe3/jVzK40L9WRf4N3RwULNbTG4jQLpQjDswMfzy3u4qIpxyUS+dyOoXXwBue M3sRYXPP+vYhAnjn1HLBiEn9KWGoe4M9CdruCkJONRlezEH3dcRMruMiMujT6HqEMGVIE zS2+foF/b52nSu1tHFF2R9HsnSqY2nVC+iYlfObmMYwvoZP8D68SV/GC9PYQKOqmBDbUJ azDPSMfCy9GE3emygX17ss/3l6FmE6gjPL8whaZWSohVWkCQd1bSuQ3Oqn2DPlp0kbw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b="H/eL4hap"; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7477A606A32F; Wed, 18 Nov 2026 20:45:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=H/eL4hap Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7B9ED1F000C4; Wed, 18 Nov 2026 20:45:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=korg; t=1795034722; bh=G9COZYt+8p7wUfOaMfnj10WInUYt2RydzZVrNgEt0fQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=H/eL4hapcYww4IDow8wq7hFBfTLvKc+/539zmKwmhXQbbYBaG9nGcTHlBcXM4MdUx iS2jicayT0qbWpBZ7cBLJvQSq9J+RQYjG1PGFVEq42PrGPyRc+Al4baGc2VJ41s8Qw xqgNQDcrRmOAV6VN9iTbFytru9MAx1m0v+cBsvdc= From: Greg Kroah-Hartman To: stable@vger.kernel.org Cc: Greg Kroah-Hartman , patches@lists.linux.dev, Catalin Marinas , "Paul E. McKenney" , Will Deacon , Palmer Dabbelt , Sasha Levin Subject: [PATCH 6.6 0469/3318] arm64: Use load LSE atomics for the non-return per-CPU atomic operations Date: Wed, 18 Nov 2026 15:18:11 +0100 Message-ID: <20261118150330.949177773@linuxfoundation.org> X-Mailer: git-send-email 2.56.0 In-Reply-To: <20261118150313.527275416@linuxfoundation.org> References: <20261118150313.527275416@linuxfoundation.org> User-Agent: quilt/0.69 X-stable: review X-Patchwork-Hint: ignore X-Mailing-List: stable@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit 6.6-stable review patch. If anyone has any objections, please let me know. ------------------ From: Catalin Marinas [ Upstream commit 535fdfc5a228524552ee8810c9175e877e127c27 ] The non-return per-CPU this_cpu_*() atomic operations are implemented as STADD/STCLR/STSET when FEAT_LSE is available. On many microarchitecture implementations, these instructions tend to be executed "far" in the interconnect or memory subsystem (unless the data is already in the L1 cache). This is in general more efficient when there is contention as it avoids bouncing cache lines between CPUs. The load atomics (e.g. LDADD without XZR as destination), OTOH, tend to be executed "near" with the data loaded into the L1 cache. STADD executed back to back as in srcu_read_{lock,unlock}*() incur an additional overhead due to the default posting behaviour on several CPU implementations. Since the per-CPU atomics are unlikely to be used concurrently on the same memory location, encourage the hardware to to execute them "near" by issuing load atomics - LDADD/LDCLR/LDSET - with the destination register unused (but not XZR). Signed-off-by: Catalin Marinas Link: https://lore.kernel.org/r/e7d539ed-ced0-4b96-8ecd-048a5b803b85@paulmck-laptop Reported-by: Paul E. McKenney Tested-by: Paul E. McKenney Cc: Will Deacon Reviewed-by: Palmer Dabbelt [will: Add comment and link to the discussion thread] Signed-off-by: Will Deacon Stable-dep-of: 8cf2093f5372 ("arm64: percpu: Fix LSE operations on {8,16}-bit types") Signed-off-by: Sasha Levin --- arch/arm64/include/asm/percpu.h | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h index 939dbaa7c90cb..31193bcf89a2b 100644 --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -77,7 +77,7 @@ __percpu_##name##_case_##sz(void *ptr, unsigned long val) \ " stxr" #sfx "\t%w[loop], %" #w "[tmp], %[ptr]\n" \ " cbnz %w[loop], 1b", \ /* LSE atomics */ \ - #op_lse "\t%" #w "[val], %[ptr]\n" \ + #op_lse "\t%" #w "[val], %" #w "[tmp], %[ptr]\n" \ __nops(3)) \ : [loop] "=&r" (loop), [tmp] "=&r" (tmp), \ [ptr] "+Q"(*(u##sz *)ptr) \ @@ -124,9 +124,16 @@ PERCPU_RW_OPS(8) PERCPU_RW_OPS(16) PERCPU_RW_OPS(32) PERCPU_RW_OPS(64) -PERCPU_OP(add, add, stadd) -PERCPU_OP(andnot, bic, stclr) -PERCPU_OP(or, orr, stset) + +/* + * Use value-returning atomics for CPU-local ops as they are more likely + * to execute "near" to the CPU (e.g. in L1$). + * + * https://lore.kernel.org/r/e7d539ed-ced0-4b96-8ecd-048a5b803b85@paulmck-laptop + */ +PERCPU_OP(add, add, ldadd) +PERCPU_OP(andnot, bic, ldclr) +PERCPU_OP(or, orr, ldset) PERCPU_RET_OP(add, add, ldadd) #undef PERCPU_RW_OPS -- 2.53.0