[PATCH 6.6.y] arm64: mm: clear extra idmap level before use

[PATCH 6.6.y] arm64: mm: clear extra idmap level before use

[Karl Mehltretter]

The 6.6.y adaptation of commit 0e9df1c905d8 ("arm64: mm: Don't remap
pgtables for allocate vs populate") removes the clearing performed by
early_pgtable_alloc().  Its replacement clears allocations made by the
generic page-table walkers, but 6.6's create_idmap() still allocates an
extra root level directly when a sub-48-bit VA kernel is loaded
sufficiently high in physical memory.

memblock_phys_alloc_range() does not zero the returned memory.  The direct
caller therefore publishes an uncleared root page and passes it to
__create_pgd_mapping().  A stale nonzero entry can trip the bad-descriptor
BUG_ON or be followed as a page-table descriptor, preventing the kernel
from booting.

Clear the direct allocation through its linear alias before publishing it.
init_clear_pgtable() also supplies the barrier required before the table
descriptor becomes visible.

Mainline is not affected because commit e6128a8e523c ("arm64: mm: Use
48-bit virtual addressing for the permanent ID map") removed the dynamic
extra level before page-table initialization moved out of the allocator.

Fixes: 54322d95309d ("arm64: mm: Don't remap pgtables for allocate vs populate")
---
A deterministic QEMU A/B used 16 KiB pages, a 36-bit VA, a 48-bit PA, an
Image loaded at 0x1000200000, and test-only instrumentation that filled the
extra root with 0x02.  The affected kernel hit the expected bad-descriptor
BUG_ON.  With this patch, the same poisoned allocation survived
create_idmap() before the test stopped deliberately.

The injected contents make the allocator's permitted nonzero return
deterministic; they do not estimate real-world incidence.

Tested on:
  6.6.y a4a971135a2ff64382ae4235b3ae60503bb1036a (Linux 6.6.155)

 arch/arm64/mm/mmu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index e075792d722575d38488e8352df8947c6aaddd2b..4e9e997e08daf543e5c09799563f991a9f8a7cc0 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -777,9 +777,10 @@ static void __init create_idmap(void)
 	/* check if we need an additional level of translation */
 	if (VA_BITS < 48 && idmap_t0sz < (64 - VA_BITS_MIN)) {
 		pgd_phys = early_pgtable_alloc(PAGE_SHIFT);
+		pgd = __va(pgd_phys);
+		init_clear_pgtable(pgd);
 		set_pgd(&idmap_pg_dir[start >> VA_BITS],
 			__pgd(pgd_phys | P4D_TYPE_TABLE));
-		pgd = __va(pgd_phys);
 	}
 	__create_pgd_mapping(pgd, start, start, size, PAGE_KERNEL_ROX,
 			     early_pgtable_alloc, 0);
-- 
2.39.5 (Apple Git-154)

Re: [PATCH 6.6.y] arm64: mm: clear extra idmap level before use

[Greg Kroah-Hartman]

On Mon, Aug 31, 2026 at 07:42:47AM +0200, Karl Mehltretter wrote:
> The 6.6.y adaptation of commit 0e9df1c905d8 ("arm64: mm: Don't remap
> pgtables for allocate vs populate") removes the clearing performed by
> early_pgtable_alloc().  Its replacement clears allocations made by the
> generic page-table walkers, but 6.6's create_idmap() still allocates an
> extra root level directly when a sub-48-bit VA kernel is loaded
> sufficiently high in physical memory.
> 
> memblock_phys_alloc_range() does not zero the returned memory.  The direct
> caller therefore publishes an uncleared root page and passes it to
> __create_pgd_mapping().  A stale nonzero entry can trip the bad-descriptor
> BUG_ON or be followed as a page-table descriptor, preventing the kernel
> from booting.
> 
> Clear the direct allocation through its linear alias before publishing it.
> init_clear_pgtable() also supplies the barrier required before the table
> descriptor becomes visible.
> 
> Mainline is not affected because commit e6128a8e523c ("arm64: mm: Use
> 48-bit virtual addressing for the permanent ID map") removed the dynamic
> extra level before page-table initialization moved out of the allocator.

Why can't we take that change instead?

> 
> Fixes: 54322d95309d ("arm64: mm: Don't remap pgtables for allocate vs populate")

No signed-off-by?

thanks,

greg k-h

Re: [PATCH 6.6.y] arm64: mm: clear extra idmap level before use

[Ard Biesheuvel]

On Mon, 31 Aug 2026, at 07:54, Greg Kroah-Hartman wrote:
> On Mon, Aug 31, 2026 at 07:42:47AM +0200, Karl Mehltretter wrote:
>> The 6.6.y adaptation of commit 0e9df1c905d8 ("arm64: mm: Don't remap
>> pgtables for allocate vs populate") removes the clearing performed by
>> early_pgtable_alloc().  Its replacement clears allocations made by the
>> generic page-table walkers, but 6.6's create_idmap() still allocates an
>> extra root level directly when a sub-48-bit VA kernel is loaded
>> sufficiently high in physical memory.
>> 
>> memblock_phys_alloc_range() does not zero the returned memory.  The direct
>> caller therefore publishes an uncleared root page and passes it to
>> __create_pgd_mapping().  A stale nonzero entry can trip the bad-descriptor
>> BUG_ON or be followed as a page-table descriptor, preventing the kernel
>> from booting.
>> 
>> Clear the direct allocation through its linear alias before publishing it.
>> init_clear_pgtable() also supplies the barrier required before the table
>> descriptor becomes visible.
>> 
>> Mainline is not affected because commit e6128a8e523c ("arm64: mm: Use
>> 48-bit virtual addressing for the permanent ID map") removed the dynamic
>> extra level before page-table initialization moved out of the allocator.
>
> Why can't we take that change instead?
>

Nope. That relies on a massive refactoring of the arm64 startup code.

Can we just revert 0e9df1c905d8 (aka 54322d95309d) instead?

Re: [PATCH 6.6.y] arm64: mm: clear extra idmap level before use

[Karl Mehltretter]

On Mon, Aug 31, 2026 at 08:15:52AM +0100, Ard Biesheuvel wrote:
> 
> 
> On Mon, 31 Aug 2026, at 07:54, Greg Kroah-Hartman wrote:
> > On Mon, Aug 31, 2026 at 07:42:47AM +0200, Karl Mehltretter wrote:
> >> The 6.6.y adaptation of commit 0e9df1c905d8 ("arm64: mm: Don't remap
> >> pgtables for allocate vs populate") removes the clearing performed by
> >> early_pgtable_alloc().  Its replacement clears allocations made by the
> >> generic page-table walkers, but 6.6's create_idmap() still allocates an
> >> extra root level directly when a sub-48-bit VA kernel is loaded
> >> sufficiently high in physical memory.
> >> 
> >> memblock_phys_alloc_range() does not zero the returned memory.  The direct
> >> caller therefore publishes an uncleared root page and passes it to
> >> __create_pgd_mapping().  A stale nonzero entry can trip the bad-descriptor
> >> BUG_ON or be followed as a page-table descriptor, preventing the kernel
> >> from booting.
> >> 
> >> Clear the direct allocation through its linear alias before publishing it.
> >> init_clear_pgtable() also supplies the barrier required before the table
> >> descriptor becomes visible.
> >> 
> >> Mainline is not affected because commit e6128a8e523c ("arm64: mm: Use
> >> 48-bit virtual addressing for the permanent ID map") removed the dynamic
> >> extra level before page-table initialization moved out of the allocator.
> >
> > Why can't we take that change instead?
> >
> 
> Nope. That relies on a massive refactoring of the arm64 startup code.
> 
> Can we just revert 0e9df1c905d8 (aka 54322d95309d) instead?
> 

Yes, that fixes the issue too.

The commit was patch 3/3 of the following speedup series:

https://lore.kernel.org/r/20240412131908.433043-1-ryan.roberts@arm.com

Patches 1 and 2 accounted for most of the speedup.

I'll wait a bit to give the others a chance to comment.

I can send the revert as v2, or my original patch as v2 with the
missing "Assisted-by: LLM" and "Signed-off-by:" trailers.

Thanks,
Karl

Re: [PATCH 6.6.y] arm64: mm: clear extra idmap level before use

[Greg Kroah-Hartman]

On Mon, Aug 31, 2026 at 01:00:34PM +0200, Karl Mehltretter wrote:
> On Mon, Aug 31, 2026 at 08:15:52AM +0100, Ard Biesheuvel wrote:
> > 
> > 
> > On Mon, 31 Aug 2026, at 07:54, Greg Kroah-Hartman wrote:
> > > On Mon, Aug 31, 2026 at 07:42:47AM +0200, Karl Mehltretter wrote:
> > >> The 6.6.y adaptation of commit 0e9df1c905d8 ("arm64: mm: Don't remap
> > >> pgtables for allocate vs populate") removes the clearing performed by
> > >> early_pgtable_alloc().  Its replacement clears allocations made by the
> > >> generic page-table walkers, but 6.6's create_idmap() still allocates an
> > >> extra root level directly when a sub-48-bit VA kernel is loaded
> > >> sufficiently high in physical memory.
> > >> 
> > >> memblock_phys_alloc_range() does not zero the returned memory.  The direct
> > >> caller therefore publishes an uncleared root page and passes it to
> > >> __create_pgd_mapping().  A stale nonzero entry can trip the bad-descriptor
> > >> BUG_ON or be followed as a page-table descriptor, preventing the kernel
> > >> from booting.
> > >> 
> > >> Clear the direct allocation through its linear alias before publishing it.
> > >> init_clear_pgtable() also supplies the barrier required before the table
> > >> descriptor becomes visible.
> > >> 
> > >> Mainline is not affected because commit e6128a8e523c ("arm64: mm: Use
> > >> 48-bit virtual addressing for the permanent ID map") removed the dynamic
> > >> extra level before page-table initialization moved out of the allocator.
> > >
> > > Why can't we take that change instead?
> > >
> > 
> > Nope. That relies on a massive refactoring of the arm64 startup code.
> > 
> > Can we just revert 0e9df1c905d8 (aka 54322d95309d) instead?
> > 
> 
> Yes, that fixes the issue too.
> 
> The commit was patch 3/3 of the following speedup series:
> 
> https://lore.kernel.org/r/20240412131908.433043-1-ryan.roberts@arm.com
> 
> Patches 1 and 2 accounted for most of the speedup.
> 
> I'll wait a bit to give the others a chance to comment.
> 
> I can send the revert as v2, or my original patch as v2 with the
> missing "Assisted-by: LLM" and "Signed-off-by:" trailers.

REvert would be great, thanks!

greg k-h

[PATCH 6.6.y v2] Revert "arm64: mm: Don't remap pgtables for allocate vs populate"

[Karl Mehltretter]

This reverts commit 54322d95309d9aa4cb77b34ee4b6c8b541f3e21f.

The 6.6.y backport removes the clearing performed by
early_pgtable_alloc(). Its replacement clears allocations made by the
generic page-table walkers, but 6.6's create_idmap() still allocates an
extra root level directly when a sub-48-bit VA kernel is loaded
sufficiently high in physical memory.

memblock_phys_alloc_range() does not zero the returned memory. The direct
caller can therefore publish an uncleared root page. A stale entry can
trip the bad-descriptor BUG_ON or be followed as a page-table descriptor,
preventing the kernel from booting.

Mainline is not affected because commit e6128a8e523c ("arm64: mm: Use
48-bit virtual addressing for the permanent ID map") removed the dynamic
extra level before commit 0e9df1c905d8 ("arm64: mm: Don't remap pgtables
for allocate vs populate") moved page-table initialization out of the
allocator.

Revert the optimization in 6.6.y to restore allocation-time clearing for
all callers.

Fixes: 54322d95309d ("arm64: mm: Don't remap pgtables for allocate vs populate")
Link: https://lore.kernel.org/r/2026083151-mascot-unshaken-5f46@gregkh
Assisted-by: LLM
Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com>
---
Changes in v2:
- Replace the targeted extra-idmap clear with a full revert of
  54322d95309d (Ard).

 arch/arm64/mm/mmu.c | 58 ++++++++++++++++++++++-----------------------
 1 file changed, 29 insertions(+), 29 deletions(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index e075792d72257..c49cf99161881 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -106,12 +106,28 @@ EXPORT_SYMBOL(phys_mem_access_prot);
 static phys_addr_t __init early_pgtable_alloc(int shift)
 {
 	phys_addr_t phys;
+	void *ptr;
 
 	phys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0,
 					 MEMBLOCK_ALLOC_NOLEAKTRACE);
 	if (!phys)
 		panic("Failed to allocate page table page\n");
 
+	/*
+	 * The FIX_{PGD,PUD,PMD} slots may be in active use, but the FIX_PTE
+	 * slot will be free, so we can (ab)use the FIX_PTE slot to initialise
+	 * any level of table.
+	 */
+	ptr = pte_set_fixmap(phys);
+
+	memset(ptr, 0, PAGE_SIZE);
+
+	/*
+	 * Implicit barriers also ensure the zeroed page is visible to the page
+	 * table walker
+	 */
+	pte_clear_fixmap();
+
 	return phys;
 }
 
@@ -153,14 +169,6 @@ bool pgattr_change_is_safe(u64 old, u64 new)
 	return ((old ^ new) & ~mask) == 0;
 }
 
-static void init_clear_pgtable(void *table)
-{
-	clear_page(table);
-
-	/* Ensure the zeroing is observed by page table walks. */
-	dsb(ishst);
-}
-
 static void init_pte(pte_t *ptep, unsigned long addr, unsigned long end,
 		     phys_addr_t phys, pgprot_t prot)
 {
@@ -203,15 +211,12 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr,
 			pmdval |= PMD_TABLE_PXN;
 		BUG_ON(!pgtable_alloc);
 		pte_phys = pgtable_alloc(PAGE_SHIFT);
-		ptep = pte_set_fixmap(pte_phys);
-		init_clear_pgtable(ptep);
-		ptep += pte_index(addr);
 		__pmd_populate(pmdp, pte_phys, pmdval);
-	} else {
-		BUG_ON(pmd_bad(pmd));
-		ptep = pte_set_fixmap_offset(pmdp, addr);
+		pmd = READ_ONCE(*pmdp);
 	}
+	BUG_ON(pmd_bad(pmd));
 
+	ptep = pte_set_fixmap_offset(pmdp, addr);
 	do {
 		pgprot_t __prot = prot;
 
@@ -290,15 +295,12 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr,
 			pudval |= PUD_TABLE_PXN;
 		BUG_ON(!pgtable_alloc);
 		pmd_phys = pgtable_alloc(PMD_SHIFT);
-		pmdp = pmd_set_fixmap(pmd_phys);
-		init_clear_pgtable(pmdp);
-		pmdp += pmd_index(addr);
 		__pud_populate(pudp, pmd_phys, pudval);
-	} else {
-		BUG_ON(pud_bad(pud));
-		pmdp = pmd_set_fixmap_offset(pudp, addr);
+		pud = READ_ONCE(*pudp);
 	}
+	BUG_ON(pud_bad(pud));
 
+	pmdp = pmd_set_fixmap_offset(pudp, addr);
 	do {
 		pgprot_t __prot = prot;
 
@@ -336,15 +338,12 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end,
 			p4dval |= P4D_TABLE_PXN;
 		BUG_ON(!pgtable_alloc);
 		pud_phys = pgtable_alloc(PUD_SHIFT);
-		pudp = pud_set_fixmap(pud_phys);
-		init_clear_pgtable(pudp);
-		pudp += pud_index(addr);
 		__p4d_populate(p4dp, pud_phys, p4dval);
-	} else {
-		BUG_ON(p4d_bad(p4d));
-		pudp = pud_set_fixmap_offset(p4dp, addr);
+		p4d = READ_ONCE(*p4dp);
 	}
+	BUG_ON(p4d_bad(p4d));
 
+	pudp = pud_set_fixmap_offset(p4dp, addr);
 	do {
 		pud_t old_pud = READ_ONCE(*pudp);
 
@@ -426,10 +425,11 @@ void create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long virt,
 
 static phys_addr_t __pgd_pgtable_alloc(int shift)
 {
-	/* Page is zeroed by init_clear_pgtable() so don't duplicate effort. */
-	void *ptr = (void *)__get_free_page(GFP_PGTABLE_KERNEL & ~__GFP_ZERO);
-
+	void *ptr = (void *)__get_free_page(GFP_PGTABLE_KERNEL);
 	BUG_ON(!ptr);
+
+	/* Ensure the zeroed page is visible to the page table walker */
+	dsb(ishst);
 	return __pa(ptr);
 }
 
-- 
2.39.5 (Apple Git-154)

Re: [PATCH 6.6.y v2] Revert "arm64: mm: Don't remap pgtables for allocate vs populate"

[Sasha Levin]

> This reverts commit 54322d95309d9aa4cb77b34ee4b6c8b541f3e21f.
>
> The 6.6.y backport removes the clearing performed by
> early_pgtable_alloc(). Its replacement clears allocations made by the
> generic page-table walkers, but 6.6's create_idmap() still allocates an
> extra root level directly when a sub-48-bit VA kernel is loaded
> sufficiently high in physical memory.

Queued for 6.6, thanks.

-- 
Thanks,
Sasha