From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 54CEEC55184 for ; Tue, 4 Aug 2026 17:05:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-Id:Date :Subject:To:From:Reply-To:Content-Type:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Jxuiu4IBbnfhJ+O4j2h0m19X6B2mNwUZ06fB82uY7uM=; b=ztgLE0IvsBMqJC +sFTzemWrjXHnw/5n2rKphbwBsQpxKrKlUoEaYOu1WksZgtWqG5cumMpJ32MTZZwBXCXFaZY/Vj+l d7HzPwM+oWt/pzFKwiUL/zczVjwvSrPouiTPY0dNqzb5t/eUOcjl6wJQLBqo9mobjORlvxQdGM8jm Io8BqRM3ayvmAiAmhbqTKzX3R9VFv6jwvyH0rVG1JX1iOS9RXgtuXFhKPUMktazi8UWdHPVlCtEbq uKRnYL7h+JyRjafq9f2xoFlci49fDcDguhqA65qr5RrtAF0ol9Rou8y9XC4NE+3VSotnPDsJaA/RC e5QzZ09Igst+p7nSIRPQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1wrIZy-00000002PnS-2y0I; Tue, 04 Aug 2026 17:05:26 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1wrIZw-00000002PlV-1O4k for linux-arm-kernel@lists.infradead.org; Tue, 04 Aug 2026 17:05:25 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3518D165C; Tue, 4 Aug 2026 10:05:19 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B024F3F632; Tue, 4 Aug 2026 10:05:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1785863123; bh=WDsn/mHahpvx62D9TO6yFmeHtgjl099mMZSfB+gdWx8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B4iI2hXmltZNoYxh61wFs+EWimdCWGTamsf0wpEgVEMKfTEgOXEPyfrZwrHh0f6+G vgrL0enQ6ykpPJkrkuQK/IUQ9zpmlecN+p8eN6GvG7n/u9ycuCClHkQNiop7ORYgzy FPXrTImNT1nmeUXQ4FFJPFUSy4pZGGZCXeQOlGHc= From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 02/20] arm64: percpu: Fix this_cpu_and() mask generation Date: Tue, 4 Aug 2026 18:04:45 +0100 Message-Id: <20260804170503.3513916-3-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20260804170503.3513916-1-mark.rutland@arm.com> References: <20260804170503.3513916-1-mark.rutland@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.9.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260804_100524_497687_7FEC55F8 X-CRM114-Status: GOOD ( 14.28 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, vladimir.murzin@arm.com, ryan.roberts@arm.com, peterz@infradead.org, catalin.marinas@arm.com, david.laight.linux@gmail.com, stable@vger.kernel.org, ruanjinjie@huawei.com, james.morse@arm.com, yang@os.amperecomputing.com, cl@gentwo.org, maz@kernel.org, david@kernel.org, ljs@kernel.org, will@kernel.org, ardb@kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The arm64 implementation of this_cpu_and(pcp, val) is built in terms of ANDNOT operations, which requires the 'val' argument to be bitwise negated. The bitwise negation is not implemented correctly, with two bugs described below. (1) The bitwise negation is performed as '~val' rather than '~(val)'. This won't always generate the expected value when 'val' is an expression. For example, for this_cpu_and(pcp, 1 - 1): * 'val' is '1 - 1' ===> (int) 0x00000000 * '~val' is '~1 - 1' ===> (int) 0xfffffffd * '~(val)' is '~(1 - 1)' ===> (int) 0xffffffff ... and thus bit[1] of 'pcp' would be preserved unexpectedly by the ANDNOT operation. (2) The bitwise negation is performed on 'val' before it has been cast to (at least) the width of 'pcp'. This won't always generate the expected value for the upper bits. For example, for this_cpu_and(pcp, zero), where 'pcp' is a u64 and 'zero' is a u32: * 'zero' ===> (u32) 0x00000000 * '~(zero)' ===> (u32) 0xffffffff * '(u64)~(zero)' ===> (u64) 0x00000000ffffffff * '~((u64)(zero))' ===> (u64) 0xffffffffffffffff ... and thus bits[63:32] of 'pcp' would be preserved unexpectedly by the ANDNOT operation. Fix these issues by adding brackets around 'val', and by casting 'val' to an appropriately-sized type before bitwise negation. The bugs described above can be seen from the disassembly of the following test code: | void this_cpu_and_u64_zero(u64 __percpu *pcp) | { | u64 zero = 0; | this_cpu_and(*pcp, zero); | } | | void this_cpu_and_u32_zero(u64 __percpu *pcp) | { | u32 zero = 0; | this_cpu_and(*pcp, zero); | } | | void this_cpu_and_expr_zero(u64 __percpu *pcp) | { | this_cpu_and(*pcp, 1 - 1); | } | | void this_cpu_and_expr_zero_brackets(u64 __percpu *pcp) | { | this_cpu_and(*pcp, (1 - 1)); | } Before this patch: | : | paciasp | stp x29, x30, [sp, #-16]! | mrs x1, sp_el0 | mov x29, sp | ldr w2, [x1, #8] | add w2, w2, #0x1 | str w2, [x1, #8] | mov x3, #0xffffffffffffffff // #-1 | mrs x2, tpidr_el1 | add x0, x0, x2 | 1: ldxr x5, [x0] | bic x5, x5, x3 | stxr w4, x5, [x0] | cbnz w4, 1b | ldr x0, [x1, #8] | add x0, x0, x3 | str w0, [x1, #8] | cbz x0, 2f | ldr x0, [x1, #8] | cbnz x0, 3f | 2: bl preempt_schedule_notrace | 3: ldp x29, x30, [sp], #16 | autiasp | ret | | : | paciasp | stp x29, x30, [sp, #-16]! | mrs x1, sp_el0 | mov x29, sp | ldr w2, [x1, #8] | add w2, w2, #0x1 | str w2, [x1, #8] | mov x3, #0xffffffff // #4294967295 | mrs x2, tpidr_el1 | add x0, x0, x2 | 1: ldxr x5, [x0] | bic x5, x5, x3 | stxr w4, x5, [x0] | cbnz w4, 1b | ldr x0, [x1, #8] | sub x0, x0, #0x1 | str w0, [x1, #8] | cbz x0, 2f | ldr x0, [x1, #8] | cbnz x0, 3f | 2: bl preempt_schedule_notrace | 3: ldp x29, x30, [sp], #16 | autiasp | ret | | : | paciasp | stp x29, x30, [sp, #-16]! | mrs x1, sp_el0 | mov x29, sp | ldr w2, [x1, #8] | add w2, w2, #0x1 | str w2, [x1, #8] | mov x3, #0xfffffffffffffffd // #-3 | mrs x2, tpidr_el1 | add x0, x0, x2 | 1: ldxr x5, [x0] | bic x5, x5, x3 | stxr w4, x5, [x0] | cbnz w4, 1b | ldr x0, [x1, #8] | sub x0, x0, #0x1 | str w0, [x1, #8] | cbz x0, 2f | ldr x0, [x1, #8] | cbnz x0, 3f | 2: bl preempt_schedule_notrace | 3: ldp x29, x30, [sp], #16 | autiasp | ret After this patch: | : | paciasp | stp x29, x30, [sp, #-16]! | mrs x1, sp_el0 | mov x29, sp | ldr w2, [x1, #8] | add w2, w2, #0x1 | str w2, [x1, #8] | mov x3, #0xffffffffffffffff // #-1 | mrs x2, tpidr_el1 | add x0, x0, x2 | 1: ldxr x5, [x0] | bic x5, x5, x3 | stxr w4, x5, [x0] | cbnz w4, 1b | ldr x0, [x1, #8] | add x0, x0, x3 | str w0, [x1, #8] | cbz x0, 2f | ldr x0, [x1, #8] | cbnz x0, 3f | 2: bl 0 | 3: ldp x29, x30, [sp], #16 | autiasp | ret | | : | b this_cpu_and_u64_zero | | : | b this_cpu_and_u64_zero | | : | b this_cpu_and_u64_zero Fixes: 959bf2fd03b5 ("arm64: percpu: Rewrite per-cpu ops to allow use of LSE atomics") Signed-off-by: Mark Rutland Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Jinjie Ruan Cc: Marc Zyngier Cc: Peter Zijlstra Cc: Vladimir Murzin Cc: Will Deacon Cc: Yang Shi Cc: stable@vger.kernel.org --- arch/arm64/include/asm/percpu.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h index 63bbfd4944a37..31193bcf89a2b 100644 --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -206,13 +206,13 @@ PERCPU_RET_OP(add, add, ldadd) _pcp_protect_return(__percpu_add_return_case_64, pcp, val) #define this_cpu_and_1(pcp, val) \ - _pcp_protect(__percpu_andnot_case_8, pcp, ~val) + _pcp_protect(__percpu_andnot_case_8, pcp, ~(u8)(val)) #define this_cpu_and_2(pcp, val) \ - _pcp_protect(__percpu_andnot_case_16, pcp, ~val) + _pcp_protect(__percpu_andnot_case_16, pcp, ~(u16)(val)) #define this_cpu_and_4(pcp, val) \ - _pcp_protect(__percpu_andnot_case_32, pcp, ~val) + _pcp_protect(__percpu_andnot_case_32, pcp, ~(u32)(val)) #define this_cpu_and_8(pcp, val) \ - _pcp_protect(__percpu_andnot_case_64, pcp, ~val) + _pcp_protect(__percpu_andnot_case_64, pcp, ~(u64)(val)) #define this_cpu_or_1(pcp, val) \ _pcp_protect(__percpu_or_case_8, pcp, val) -- 2.30.2