[PATCH v2] arm: lpae: fix non-atomic page table entry update issue

[PATCH v2] arm: lpae: fix non-atomic page table entry update issue

[Wang YanQing]

The ARM Architecture Reference Manual explicitly dictates that writes of 64-bit
translation table descriptors must be single-copy atomic:
ARM Architecture Reference Manual ARMv7-A and ARMv7-R edition (https://developer.arm.com/documentation/ddi0406/latest)
"
...
A3.5.3 Atomicity in the ARM architecture
...
In an implementation that includes the Large Physical Address Extension, LDRD, and STRD accesses to 64-bit aligned
locations are 64-bit single-copy atomic as seen by translation table walks and accesses to translation tables.
Note
The Large Physical Address Extension adds this requirement to avoid the need for complex measures to avoid
atomicity issues when changing translation table entries, without creating a requirement that all locations in the
memory system are 64-bit single-copy atomic. This addition means:
•The system designer must ensure that all writable memory locations that might be used to hold translations,
such as bulk SDRAM, can be accessed with 64-bit single-copy atomicity.
•Software must ensure that translation tables are not held in memory locations that cannot meet this atomicity
requirement, such as peripherals that are typically accessed using a narrow bus.
This requirement places no burden on read-only memory locations for which reads have no side effects, since it is
impossible to detect the size of memory accesses to such locations.
...
"

ARM Architecture Reference Manual for A-profile architecture (https://developer.arm.com/documentation/ddi0487/latest)
"
...
E2.2.1 Requirements for single-copy atomicity
In AArch32 state, the single-copy atomic PE accesses are:
•All byte accesses.
•All halfword accesses to halfword-aligned locations.
•All word accesses to word-aligned locations.
•Memory accesses caused by LDREXD and STREXD instructions to doubleword-aligned locations.
LDM, LDC, LDRD, STM, STC, STRD, PUSH, POP, RFE, SRS, VLDM, VLDR, VSTM, and VSTR instructions are executed as a
sequence of word-aligned word accesses. Each 32-bit word access is guaranteed to be single-copy atomic. The
architecture does not require subsequences of two or more word accesses from the sequence to be single-copy atomic.
LDRD and STRD accesses to 64-bit aligned locations are 64-bit single-copy atomic as seen by translation table walks and
accesses to translation tables.
...
"

To archieve this 64-bit atomicity on a 32-bit architecture, the linux kernel relies on the STRD (Store Register Dual)
instruction, but the copy_pmd() in pgtable-3level.h is C code, then compiler could do very crazy optimization for it
and generate code that break the atomicity, for example, we get below copy_pmd() assembly code with gcc 12.4.0
(Using CC_OPTIMIZE_FOR_PERFORMANCE, it is the default compile option):
"
gdb vmlinux
gdb disassemble do_translation_fault
gdb ...
gdb 0xc020e544 <+136>:    ldr.w    r4, [r0, r1, lsl #3] @load low 32-bit of pmdps
gdb 0xc020e548 <+140>:    ldr      r0, [r6, #4]         @load high 32-bit of pmdps
gdb 0xc020e54a <+142>:    orrs.w   r6, r4, r0           @ pmd_none(pmd_k[index])
gdb 0xc020e54e <+146>:    beq.n    0xc020e586 <do_translation_fault+202>
gdb ...
gdb 0xc020e562 <+166>:    str.w    r4, [r5, r1, lsl #3] @store low 32-bit to pmdpd
gdb 0xc020e566 <+170>:    str      r0, [r2, #4]         @store hight 32-bit to pmdpd

The code breaks the atomicity and valid bit is in the low 32-bit, page table walker
could see and cache the partial write entry, this will cause very strange
translation-related issues when next page table (level3 PTE table) physical address
is larger than 32-bits.

So let's use WRITE_ONCE() to protect the page table entry update functions from crazy
optimization.

Signed-off-by: Wang YanQing <udknight@gmail.com>
---
 Changes v1-v2:
 1: Add documentation reference in changelog, suggested by Russell King

 arch/arm/include/asm/pgtable-3level.h | 28 +++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h
index 7b71a3d414b7..b077174a4231 100644
--- a/arch/arm/include/asm/pgtable-3level.h
+++ b/arch/arm/include/asm/pgtable-3level.h
@@ -120,15 +120,15 @@
 						 PMD_TYPE_SECT)
 #define pmd_leaf(pmd)		pmd_sect(pmd)
 
-#define pud_clear(pudp)			\
-	do {				\
-		*pudp = __pud(0);	\
-		clean_pmd_entry(pudp);	\
+#define pud_clear(pudp)				\
+	do {					\
+		WRITE_ONCE(*pudp, __pud(0));	\
+		clean_pmd_entry(pudp);		\
 	} while (0)
 
 #define set_pud(pudp, pud)		\
 	do {				\
-		*pudp = pud;		\
+		WRITE_ONCE(*pudp, pud);	\
 		flush_pmd_entry(pudp);	\
 	} while (0)
 
@@ -139,16 +139,16 @@ static inline pmd_t *pud_pgtable(pud_t pud)
 
 #define pmd_bad(pmd)		(!(pmd_val(pmd) & PMD_TABLE_BIT))
 
-#define copy_pmd(pmdpd,pmdps)		\
-	do {				\
-		*pmdpd = *pmdps;	\
-		flush_pmd_entry(pmdpd);	\
+#define copy_pmd(pmdpd, pmdps)				\
+	do {						\
+		WRITE_ONCE(*pmdpd, READ_ONCE(*pmdps));	\
+		flush_pmd_entry(pmdpd);			\
 	} while (0)
 
-#define pmd_clear(pmdp)			\
-	do {				\
-		*pmdp = __pmd(0);	\
-		clean_pmd_entry(pmdp);	\
+#define pmd_clear(pmdp)				\
+	do {					\
+		WRITE_ONCE(*pmdp, __pmd(0));	\
+		clean_pmd_entry(pmdp);		\
 	} while (0)
 
 /*
@@ -241,7 +241,7 @@ static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr,
 	else
 		pmd_val(pmd) |= PMD_SECT_AP2;
 
-	*pmdp = __pmd(pmd_val(pmd) | PMD_SECT_nG);
+	WRITE_ONCE(*pmdp,  __pmd(pmd_val(pmd) | PMD_SECT_nG));
 	flush_pmd_entry(pmdp);
 }
 
-- 
2.34.1

Re: [PATCH v2] arm: lpae: fix non-atomic page table entry update issue

[Russell King (Oracle)]

On Sun, Mar 15, 2026 at 08:47:46AM +0800, Wang YanQing wrote:
> The ARM Architecture Reference Manual explicitly dictates that writes of 64-bit
> translation table descriptors must be single-copy atomic:
> ARM Architecture Reference Manual ARMv7-A and ARMv7-R edition (https://developer.arm.com/documentation/ddi0406/latest)
> "
> ...
> A3.5.3 Atomicity in the ARM architecture
> ...
> In an implementation that includes the Large Physical Address Extension, LDRD, and STRD accesses to 64-bit aligned
> locations are 64-bit single-copy atomic as seen by translation table walks and accesses to translation tables.
> Note
> The Large Physical Address Extension adds this requirement to avoid the need for complex measures to avoid
> atomicity issues when changing translation table entries, without creating a requirement that all locations in the
> memory system are 64-bit single-copy atomic.

Thanks. Now, please locate where the need for the updates to the page
tables needs to be done atomically, bearing in mind that we program
SCTLR.AFE=1 and SCTLR.HA=0, meaning the hardware won't write-back to
the page tables to e.g. update the access flag.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

Re: [PATCH v2] arm: lpae: fix non-atomic page table entry update issue

[Wang YanQing]

Hi! Russell

On Sun, Mar 15, 2026 at 01:12:28AM +0000, Russell King (Oracle) wrote:
> On Sun, Mar 15, 2026 at 08:47:46AM +0800, Wang YanQing wrote: 
> Thanks. Now, please locate where the need for the updates to the page
> tables needs to be done atomically, bearing in mind that we program
> SCTLR.AFE=1 and SCTLR.HA=0, meaning the hardware won't write-back to
> the page tables to e.g. update the access flag.

When LPAE is enabled and in the 3G/1G userspace & kernel space config, we
use ttbr0 for address space 0-3G, and use ttbr1 for top 1G kernel space,
but wait a moment, the module space is in ttbr0 instead of ttbr1, because
module space is belong to 0-3G.

Then we don't switch ttbr0 to the same value as ttbr1 in task switch, so
when we switch from normal userspace thread to kernel thread, we use the
do_translation_fault() to fault in the module space for the kernel thread
when it accesses the module space. Now please think a situation that
userspace repeats create new short-lived processes (run shell cmds, etc),
we will use do_translation_fault() to fault in the PMD entries repeatly
when switch from new created process to running kernel thread, we need
to update pmd entry automatically here, hw is allowed to do data/instruction
preload and trigger page table walker to see the partial update pmd entry,
page table walker will cache it, and it will cause translation fault,
because it doesn't see the upper 32-bit.

When the userspace process is a multi-threads process, in smp environment,
other cpus could use the same pgd for their according kernel thread, all
the page table walker of the smp cpus have the chance to cache the partial
update entry.

Thanks
> -- 
> RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
> FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!