From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pg1-f172.google.com (mail-pg1-f172.google.com [209.85.215.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9CC3F286A9 for ; Sun, 15 Mar 2026 00:50:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773535807; cv=none; b=GQxfXoj2l9IN7SEL+jLdIZBvdqS9DdEvYeFkhyOxHnq1MYw640c1KBSXnccD+P3BdZ6DlR99CxYOArp7sV0lnXIXGKB4pa9c7d3ZlVQeyw6RtL2AHUl12UpI6+tXjM6Bl65RG8AvSv0ItBcIW22o2KNWUL3zedISKIrCP0esAzk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773535807; c=relaxed/simple; bh=0C7kqFzQL1mCdVnf0+mlCBvN4N36107hmCJ3qd5gXlM=; h=Date:From:To:Cc:Subject:Message-ID:MIME-Version:Content-Type: Content-Disposition; b=UmkPmIneYN16NLjDPdYlxp3PY40oWDo9+Dr7ss6rgHypMCco/bCO/nJCuYhEeZBwIcW99VC2KTm5TwtQFq9ZxHH+av9mh2ic/wWeTF7FhRijHYqdfvr+LfaOvlPW6qiOGiHUezX2/ILF6dCMQkVo/pdbtXLFqZlZJIMYEWWh4O8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Eqry9V1X; arc=none smtp.client-ip=209.85.215.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Eqry9V1X" Received: by mail-pg1-f172.google.com with SMTP id 41be03b00d2f7-c73aabd620bso2297106a12.1 for ; Sat, 14 Mar 2026 17:50:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1773535806; x=1774140606; darn=vger.kernel.org; h=user-agent:content-transfer-encoding:content-disposition :mime-version:mail-followup-to:message-id:subject:cc:to:from:date :from:to:cc:subject:date:message-id:reply-to; bh=XpbOikVku5YR5vNSA6pCrMtta1WFwEavqnarNyqEQ2k=; b=Eqry9V1X3aiQ+GzLT9sqSof1Uwy7jKI/TViJmc37Wy9WGiS+CAmL5Xseo8BLnDfAYv o5J4BLgnWDA4Tyzuz7ALl8x1E5YFm3bk8pfaOKr+jRSASst6qatnhgKdUsVXx9tvFX1w 0Gys4QNyoxiDOLKZLDzpJAcouq2l9c1IK3zV5SAkjl4GbZQU/jq1fon+VzZHzeLAYPaM 0tpm5xdWDSUGUsOX1MfNrwNEa+oQiItorcyl43Ah4Ld//93rg/+htDq0hqjoZOyLvdQo qbce6ZlGzWPS9w4nDMoAHyY03RWA9JgO9aG0MytUnVIpdLgzhB1yXaOxT56ZxLE/7db/ HyqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1773535806; x=1774140606; h=user-agent:content-transfer-encoding:content-disposition :mime-version:mail-followup-to:message-id:subject:cc:to:from:date :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=XpbOikVku5YR5vNSA6pCrMtta1WFwEavqnarNyqEQ2k=; b=Jvny67n4eFXkUe9c85/KVjrFUSZ5lbJeu881LubyusPnRRUcSFn8JoxVVx51IYCr/N xZFWJW2xVdlsX72lxoXw0eO2otA7tWM0E76LNtK1PDQq8ONON8JIRP52Z85ZGOTIgv5I JmkVlyzA2EfT9grKY5k6r7cZMdaxyarydudA23zu9qFwWiuDKx35bpph+LBwZEOXXlcp 7zv+CSZYFrHeLsDT1yo/uA4kkHF7OX1eWcrKgK1nFDU8D2JZu1AjZloOq+xCCt8wCgl4 B/qe4ak5pW9Ve+gsrtuB7irQDoi5mY9M/TtyxmWTFquVM3hUdY7IuHsNJS0EnxJh7B5O PaYg== X-Forwarded-Encrypted: i=1; AJvYcCVGT1uxnJ8Xmf63Oj5UIyTNRBZPDKP5JT/wbz/Jhko3WeOG3VmhzTx306dZU92ShQ/ddfBgql23UaRRcT0=@vger.kernel.org X-Gm-Message-State: AOJu0Yyk04HMms0qtqFOXfJATEpUyGiCE85kZuVD3qfeJcrAZonwlXyG h0J2ikbAQKkspGsXqn3o1S6aXAJOR9XvhrdSZ3Ry8ssR1xw/UwjB+hkPvGCr++A1Tu8= X-Gm-Gg: ATEYQzx6YEkPilwohvHXcrAtsrzxRDMY7dw6/fetOze2QU/7bD0KmT+2hjQbzpPZTRz 1/S0XAnkWVJjSMgxtjvY3TyeXMK39S9ttPlgJ5CK26pMkU+euPwIKHfE5BilbT+RvWo5gNWZPTV hGJMdlznI1lsltL9cHDL68cmvivWuFT6w5CVRgCT/W9INjEGZqlonTZMl9irUAESY3hgWat/QuR u4fpgeJO1NSdlVTXbFpIUFl+dgHHsLy/T/F5LdMSTQv8tjL1Laca5i1aHPd4w5bsY5HehDFyfnq h5ZLcYqJbmwSLF0F4H73N0R04wgnZG7mKgfKABmbPexIoPLIddGcWpkEc5ak2WgBBQVPhh7Ui0Z 6hemebayG6dNc3fE91J/bnHgqtcze4UykS0ow5cqHt7NIcSQXXvhUZv5UhHni/qA8x1rDYNpdc8 kffY6HIZGlMy7xiF52MS5ykIsGHmFp0dXe6Z233Ww4 X-Received: by 2002:a05:6a21:3212:b0:398:a6d4:dbf7 with SMTP id adf61e73a8af0-398ecd98afbmr7285057637.55.1773535805842; Sat, 14 Mar 2026 17:50:05 -0700 (PDT) Received: from udknight.localhost ([120.36.202.188]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-c7402214e1csm66228a12.33.2026.03.14.17.50.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 14 Mar 2026 17:50:05 -0700 (PDT) Received: from udknight.localhost (localhost [127.0.0.1]) by udknight.localhost (8.14.9/8.14.4) with ESMTP id 62F0lmet032279; Sun, 15 Mar 2026 08:47:48 +0800 Received: (from root@localhost) by udknight.localhost (8.14.9/8.14.9/Submit) id 62F0lkjo032278; Sun, 15 Mar 2026 08:47:46 +0800 Date: Sun, 15 Mar 2026 08:47:46 +0800 From: Wang YanQing To: linux@armlinux.org.uk Cc: akpm@linux-foundation.org, willy@infradead.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v2] arm: lpae: fix non-atomic page table entry update issue Message-ID: <20260315004746.GA32062@udknight> Mail-Followup-To: Wang YanQing , linux@armlinux.org.uk, akpm@linux-foundation.org, willy@infradead.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Mutt/1.7.1 (2016-10-04) The ARM Architecture Reference Manual explicitly dictates that writes of 64-bit translation table descriptors must be single-copy atomic: ARM Architecture Reference Manual ARMv7-A and ARMv7-R edition (https://developer.arm.com/documentation/ddi0406/latest) " ... A3.5.3 Atomicity in the ARM architecture ... In an implementation that includes the Large Physical Address Extension, LDRD, and STRD accesses to 64-bit aligned locations are 64-bit single-copy atomic as seen by translation table walks and accesses to translation tables. Note The Large Physical Address Extension adds this requirement to avoid the need for complex measures to avoid atomicity issues when changing translation table entries, without creating a requirement that all locations in the memory system are 64-bit single-copy atomic. This addition means: •The system designer must ensure that all writable memory locations that might be used to hold translations, such as bulk SDRAM, can be accessed with 64-bit single-copy atomicity. •Software must ensure that translation tables are not held in memory locations that cannot meet this atomicity requirement, such as peripherals that are typically accessed using a narrow bus. This requirement places no burden on read-only memory locations for which reads have no side effects, since it is impossible to detect the size of memory accesses to such locations. ... " ARM Architecture Reference Manual for A-profile architecture (https://developer.arm.com/documentation/ddi0487/latest) " ... E2.2.1 Requirements for single-copy atomicity In AArch32 state, the single-copy atomic PE accesses are: •All byte accesses. •All halfword accesses to halfword-aligned locations. •All word accesses to word-aligned locations. •Memory accesses caused by LDREXD and STREXD instructions to doubleword-aligned locations. LDM, LDC, LDRD, STM, STC, STRD, PUSH, POP, RFE, SRS, VLDM, VLDR, VSTM, and VSTR instructions are executed as a sequence of word-aligned word accesses. Each 32-bit word access is guaranteed to be single-copy atomic. The architecture does not require subsequences of two or more word accesses from the sequence to be single-copy atomic. LDRD and STRD accesses to 64-bit aligned locations are 64-bit single-copy atomic as seen by translation table walks and accesses to translation tables. ... " To archieve this 64-bit atomicity on a 32-bit architecture, the linux kernel relies on the STRD (Store Register Dual) instruction, but the copy_pmd() in pgtable-3level.h is C code, then compiler could do very crazy optimization for it and generate code that break the atomicity, for example, we get below copy_pmd() assembly code with gcc 12.4.0 (Using CC_OPTIMIZE_FOR_PERFORMANCE, it is the default compile option): " gdb vmlinux gdb disassemble do_translation_fault gdb ... gdb 0xc020e544 <+136>: ldr.w r4, [r0, r1, lsl #3] @load low 32-bit of pmdps gdb 0xc020e548 <+140>: ldr r0, [r6, #4] @load high 32-bit of pmdps gdb 0xc020e54a <+142>: orrs.w r6, r4, r0 @ pmd_none(pmd_k[index]) gdb 0xc020e54e <+146>: beq.n 0xc020e586 gdb ... gdb 0xc020e562 <+166>: str.w r4, [r5, r1, lsl #3] @store low 32-bit to pmdpd gdb 0xc020e566 <+170>: str r0, [r2, #4] @store hight 32-bit to pmdpd The code breaks the atomicity and valid bit is in the low 32-bit, page table walker could see and cache the partial write entry, this will cause very strange translation-related issues when next page table (level3 PTE table) physical address is larger than 32-bits. So let's use WRITE_ONCE() to protect the page table entry update functions from crazy optimization. Signed-off-by: Wang YanQing --- Changes v1-v2: 1: Add documentation reference in changelog, suggested by Russell King arch/arm/include/asm/pgtable-3level.h | 28 +++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h index 7b71a3d414b7..b077174a4231 100644 --- a/arch/arm/include/asm/pgtable-3level.h +++ b/arch/arm/include/asm/pgtable-3level.h @@ -120,15 +120,15 @@ PMD_TYPE_SECT) #define pmd_leaf(pmd) pmd_sect(pmd) -#define pud_clear(pudp) \ - do { \ - *pudp = __pud(0); \ - clean_pmd_entry(pudp); \ +#define pud_clear(pudp) \ + do { \ + WRITE_ONCE(*pudp, __pud(0)); \ + clean_pmd_entry(pudp); \ } while (0) #define set_pud(pudp, pud) \ do { \ - *pudp = pud; \ + WRITE_ONCE(*pudp, pud); \ flush_pmd_entry(pudp); \ } while (0) @@ -139,16 +139,16 @@ static inline pmd_t *pud_pgtable(pud_t pud) #define pmd_bad(pmd) (!(pmd_val(pmd) & PMD_TABLE_BIT)) -#define copy_pmd(pmdpd,pmdps) \ - do { \ - *pmdpd = *pmdps; \ - flush_pmd_entry(pmdpd); \ +#define copy_pmd(pmdpd, pmdps) \ + do { \ + WRITE_ONCE(*pmdpd, READ_ONCE(*pmdps)); \ + flush_pmd_entry(pmdpd); \ } while (0) -#define pmd_clear(pmdp) \ - do { \ - *pmdp = __pmd(0); \ - clean_pmd_entry(pmdp); \ +#define pmd_clear(pmdp) \ + do { \ + WRITE_ONCE(*pmdp, __pmd(0)); \ + clean_pmd_entry(pmdp); \ } while (0) /* @@ -241,7 +241,7 @@ static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr, else pmd_val(pmd) |= PMD_SECT_AP2; - *pmdp = __pmd(pmd_val(pmd) | PMD_SECT_nG); + WRITE_ONCE(*pmdp, __pmd(pmd_val(pmd) | PMD_SECT_nG)); flush_pmd_entry(pmdp); } -- 2.34.1