From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 98E14C88E53 for ; Tue, 15 Sep 2026 14:22:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id: Content-Transfer-Encoding:Content-Type:In-Reply-To:From:References:To:Subject :MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=gGCsQm9YpUdiq3eXOgbPOdt7uVL3ILyfl1uN8ytp8/I=; b=vuDQ9/w0GBerVk m2qpzQr+ThviajLSZ+p3/Hazj1pBonXxs2P7kXNL423QKQ1JH3zcSFyW1cnVxVslSTjjhOBaLvoE2 Cze4PEDpDnhtsykOrHBjnLarXLSu05E1a1VRp6BiRy/H+9MF/p4ySywgdPLdlq9alazrTmwwHMmY/ FW8sr30Es1dHhSXnyFdPhmNYwRv8yI2wIx3ID0X5l1UYAuTwp5aupOKLw82qswOdMO24ou8QEhLdT pucf7gnQYpX4mYK8lqQSwxCTL9FZS0a/okvW6dwf9cUxsY4nbclzQ+oOWbl6bNW5+xuEKGEjNJcqy lFxN02itL3pFX2cllkHQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1x6U2l-00000006uul-0YEI; Tue, 15 Sep 2026 14:21:55 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.99.1 #2 (Red Hat Linux)) id 1x6U2k-00000006uuG-0SGI for linux-arm-kernel@bombadil.infradead.org; Tue, 15 Sep 2026 14:21:54 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:Content-Type :In-Reply-To:From:References:Cc:To:Subject:MIME-Version:Date:Message-ID: Sender:Reply-To:Content-ID:Content-Description; bh=gGCsQm9YpUdiq3eXOgbPOdt7uVL3ILyfl1uN8ytp8/I=; b=eCl7wUf1iJs4A0DQ+3iC1NWE+U P5DkoLd/MkxbvIo2REHG2K/ZFcClbGWN7Z7lDKwI83brouiXcn13j+R0aEzLq1zgz1fSK/gBJIDcT j76ngsxTG8mudksPtNpeI9lXgEtITg9D2v4vO1ULOcnKm2NYoq/2SbUAEU+x6XYgb7uuwx4pPmFvu YEqfqDHnm2EvEablsB57f4EZzdB5zsH4NweUwhldfS4n+BHnP88am49lQdqOMBJoH5+vDf2fQeYih uGDCuql3jmCsg4vXhB1dtKnAJYRTTX0jrFhDgc55mLGgCLEcwDeaaqjseMfvrHwwcsPO+hDSykLov D4NkSBhg==; Received: from foss.arm.com ([217.140.110.172]) by desiato.infradead.org with esmtp (Exim 4.99.2 #2 (Red Hat Linux)) id 1x6U2g-00000006sSi-2kFg for linux-arm-kernel@lists.infradead.org; Tue, 15 Sep 2026 14:21:53 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id F36A4169E; Tue, 15 Sep 2026 07:21:43 -0700 (PDT) Received: from [10.0.152.207] (unknown [10.0.152.207]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0B5613F7B4; Tue, 15 Sep 2026 07:21:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1789482107; bh=yVvsPUAJ9nUMcv66V+HAQRZUzYsJTC6bXwI/hW9rHV0=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From; b=d1oBEWt+SH+MTNntFusAamd7R1ArBNQCfDNXJ2e3GzdRPl2Bj2D5Ful3+lEgcw6Zw 6nMImcWQwoEGnQmz8dl6zaR2EQCYxNp6uVnMb0I7fDT1C09bQs+QapYrded2yyqpUn Ex/hSLnHk0PYlnrx0l5/1u850ZCbMc+Joeb2NrYc= Message-ID: Date: Tue, 15 Sep 2026 15:21:42 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 04/21] arm64: cmpxchg: LL/SC: Avoid redundant extension To: Mark Rutland , linux-arm-kernel@lists.infradead.org References: <20260908151741.394589-1-mark.rutland@arm.com> <20260908151741.394589-5-mark.rutland@arm.com> Content-Language: en-GB From: Vladimir Murzin In-Reply-To: <20260908151741.394589-5-mark.rutland@arm.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.9.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260915_152151_063512_925CC1D5 X-CRM114-Status: GOOD ( 28.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ryan.roberts@arm.com, usama.anjum@arm.com, peterz@infradead.org, catalin.marinas@arm.com, david.laight.linux@gmail.com, stable@vger.kernel.org, ruanjinjie@huawei.com, james.morse@arm.com, yang@os.amperecomputing.com, cl@gentwo.org, maz@kernel.org, david@kernel.org, ljs@kernel.org, will@kernel.org, ardb@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 9/8/26 16:17, Mark Rutland wrote: > The __ll_sc__cmpxchg_case_##name##sz() template always takes 'old' as an > unsigned long. When 'sz' is less than 32 bits, the 'old' is explicitly > truncated so that old[31:sz-1] is zero, which is necessary so that the > W-register EOR+CBNZ sequence doesn't fail spuriously. > > For 32-bit types specifically, no extension is necessary, but as 'old' > is 64 bits, the caller must extend 'old' to 64 bits. This leads to a > redundant instruction (typically a MOV) to extend the value, as can be > seen from disassembly of the test case below. > > Avoid this by always taking 'old' as a 'u##sz' type, and explicitly > zero-extend this to a 64-bit or 32-bit type matching the X or W register > used by the assembly. To make this work, the table of cases now > explicitly lists 'x' for the cases where an X register is used. > > For 32-bit types this removes the redundant instruction. For 64-bit > types there is no truncation, and hence no change. For {16,8}-bit types, > the existing zero-extension to 64-bit is equivalent to the new > zero-extension to 32-bit, and compilers happen to use the same > instructions for this. > > I've placed the logic for zero extension into a new header > as it will be used by other logic in subsequent patches. > > Test case: > > | u32 outline_cmpxchg_u32(u32 *p, u32 o, u32 n) > | { > | return cmpxchg(p, o, n); > | } > > Before this patch: > > | : > | b 1f > | casal w1, w2, [x0] > | mov w0, w1 > | ret > | 1: mov w3, w1 > | prfm pstl1strm, [x0] > | 2: ldxr w1, [x0] > | eor w4, w1, w3 > | cbnz w4, 3f > | stlxr w4, w2, [x0] > | cbnz w4, 2b > | dmb ish > | 3: mov w0, w1 > | ret > > After this patch: > > | : > | b 1f > | casal w1, w2, [x0] > | mov w0, w1 > | ret > | 1: prfm pstl1strm, [x0] > | 2: ldxr w3, [x0] > | eor w4, w3, w1 > | cbnz w4, 3f > | stlxr w4, w2, [x0] > | cbnz w4, 2b > | dmb ish > | 3: mov w0, w3 > | ret > > Signed-off-by: Mark Rutland > Tested-by: Muhammad Usama Anjum > Cc: Ada Couprie Diaz > Cc: Ard Biesheuvel > Cc: Catalin Marinas > Cc: James Morse > Cc: Jinjie Ruan > Cc: Marc Zyngier > Cc: Peter Zijlstra > Cc: Vladimir Murzin > Cc: Will Deacon > Cc: Yang Shi > --- > arch/arm64/include/asm/atomic_ll_sc.h | 27 +++++++++++++-------------- > arch/arm64/include/asm/xwreg.h | 16 ++++++++++++++++ > 2 files changed, 29 insertions(+), 14 deletions(-) > create mode 100644 arch/arm64/include/asm/xwreg.h > > diff --git a/arch/arm64/include/asm/atomic_ll_sc.h b/arch/arm64/include/asm/atomic_ll_sc.h > index 89d2ba2723590..160c1251f0ec9 100644 > --- a/arch/arm64/include/asm/atomic_ll_sc.h > +++ b/arch/arm64/include/asm/atomic_ll_sc.h > @@ -11,6 +11,7 @@ > #define __ASM_ATOMIC_LL_SC_H > > #include > +#include > > #ifndef CONFIG_CC_HAS_K_CONSTRAINT > #define K > @@ -239,20 +240,17 @@ __ll_sc_atomic64_dec_if_positive(atomic64_t *v) > #define __CMPXCHG_CASE(w, sfx, name, sz, mb, acq, rel, cl, constraint) \ > static __always_inline u##sz \ > __ll_sc__cmpxchg_case_##name##sz(volatile void *ptr, \ > - unsigned long old, \ > + u##sz old, \ > u##sz new) \ > { \ > + /* \ > + * Sub-word sizes require zero extension so that EOR+CBNZ won't \ > + * consume non-zero upper bits of the register containing "old".\ > + */ \ > + xwreg_t(w) cmpval = xwreg_zero_extend(old, w, sz); \ > unsigned long tmp; \ > u##sz oldval; \ > \ > - /* \ > - * Sub-word sizes require explicit casting so that the compare \ > - * part of the cmpxchg doesn't end up interpreting non-zero \ > - * upper bits of the register containing "old". \ > - */ \ > - if (sz < 32) \ > - old = (u##sz)old; \ > - \ > asm volatile( \ > " prfm pstl1strm, %[v]\n" \ > "1: ld" #acq "xr" #sfx "\t%" #w "[oldval], %[v]\n" \ > @@ -264,7 +262,8 @@ __ll_sc__cmpxchg_case_##name##sz(volatile void *ptr, \ > "2:" \ > : [tmp] "=&r" (tmp), [oldval] "=&r" (oldval), \ > [v] "+Q" (*(u##sz *)ptr) \ > - : [old] __stringify(constraint) "r" (old), [new] "r" (new) \ > + : [old] __stringify(constraint) "r" (cmpval), \ > + [new] "r" (new) \ > : cl); \ > \ > return oldval; \ > @@ -278,19 +277,19 @@ __ll_sc__cmpxchg_case_##name##sz(volatile void *ptr, \ > __CMPXCHG_CASE(w, b, , 8, , , , , K) > __CMPXCHG_CASE(w, h, , 16, , , , , K) > __CMPXCHG_CASE(w, , , 32, , , , , K) > -__CMPXCHG_CASE( , , , 64, , , , , L) > +__CMPXCHG_CASE(x, , , 64, , , , , L) > __CMPXCHG_CASE(w, b, acq_, 8, , a, , "memory", K) > __CMPXCHG_CASE(w, h, acq_, 16, , a, , "memory", K) > __CMPXCHG_CASE(w, , acq_, 32, , a, , "memory", K) > -__CMPXCHG_CASE( , , acq_, 64, , a, , "memory", L) > +__CMPXCHG_CASE(x, , acq_, 64, , a, , "memory", L) > __CMPXCHG_CASE(w, b, rel_, 8, , , l, "memory", K) > __CMPXCHG_CASE(w, h, rel_, 16, , , l, "memory", K) > __CMPXCHG_CASE(w, , rel_, 32, , , l, "memory", K) > -__CMPXCHG_CASE( , , rel_, 64, , , l, "memory", L) > +__CMPXCHG_CASE(x, , rel_, 64, , , l, "memory", L) > __CMPXCHG_CASE(w, b, mb_, 8, dmb ish, , l, "memory", K) > __CMPXCHG_CASE(w, h, mb_, 16, dmb ish, , l, "memory", K) > __CMPXCHG_CASE(w, , mb_, 32, dmb ish, , l, "memory", K) > -__CMPXCHG_CASE( , , mb_, 64, dmb ish, , l, "memory", L) > +__CMPXCHG_CASE(x, , mb_, 64, dmb ish, , l, "memory", L) > > #undef __CMPXCHG_CASE > > diff --git a/arch/arm64/include/asm/xwreg.h b/arch/arm64/include/asm/xwreg.h > new file mode 100644 > index 0000000000000..d55e3ac68f7eb > --- /dev/null > +++ b/arch/arm64/include/asm/xwreg.h > @@ -0,0 +1,16 @@ > +/* SPDX-License-Identifier: GPL-2.0-only */ > +#ifndef __ASM_XWREG_H > +#define __ASM_XWREG_H > + > +#include > + > +#define __xwreg_t_x u64 > +#define __xwreg_t_w u32 > +#define xwreg_t(xw) __xwreg_t_##xw > + > +/* > + * Zero extend 'v' from 'sz' bits (8/16/32/64) to fill an X or W register. > + */ > +#define xwreg_zero_extend(v, xw, sz) ((xwreg_t(xw))(u##sz)(v)) > + > +#endif /* __ASM_XWREG_H */ > -- 2.30.2 > FWIW, Reviewed-by: Vladimir Murzin