* [PATCH V2 01/14] mm: Add read-write accessors for vm_page_prot
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 02/14] arm64/mm: Route all pgtable reads via ptval_get() Anshuman Khandual
` (12 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm,
Lorenzo Stoakes
Currently vma->vm_page_prot is safely read from and written to, without any
locks with READ_ONCE() and WRITE_ONCE(). But with introduction of D128 page
tables on arm64 platform, vm_page_prot grows to 128 bits which can't safely
be handled with READ_ONCE() and WRITE_ONCE().
Add read and write accessors for vm_page_prot like pgprot_[read|write]()
which any platform can override when required, although still defaulting as
READ_ONCE() and WRITE_ONCE(), thus preserving the functionality for others.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
include/linux/pgtable.h | 14 ++++++++++++++
mm/huge_memory.c | 4 ++--
mm/memory.c | 2 +-
mm/migrate.c | 2 +-
mm/mmap.c | 2 +-
5 files changed, 19 insertions(+), 5 deletions(-)
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index e3c8ab96941c..0df57a5c19ef 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -524,6 +524,20 @@ static inline pgd_t pgdp_get(pgd_t *pgdp)
}
#endif
+#ifndef pgprot_read
+static inline pgprot_t pgprot_read(pgprot_t *prot)
+{
+ return READ_ONCE(*prot);
+}
+#endif
+
+#ifndef pgprot_write
+static inline void pgprot_write(pgprot_t *prot, pgprot_t val)
+{
+ WRITE_ONCE(*prot, val);
+}
+#endif
+
#ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG
static inline bool ptep_test_and_clear_young(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index ced400f72d43..33753bffead4 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3391,7 +3391,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
} else {
pte_t entry;
- entry = mk_pte(page, READ_ONCE(vma->vm_page_prot));
+ entry = mk_pte(page, pgprot_read(&vma->vm_page_prot));
if (write)
entry = pte_mkwrite(entry, vma);
if (!young)
@@ -5103,7 +5103,7 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct folio *folio
entry = softleaf_from_pmd(*pvmw->pmd);
folio_get(folio);
- pmde = folio_mk_pmd(folio, READ_ONCE(vma->vm_page_prot));
+ pmde = folio_mk_pmd(folio, pgprot_read(&vma->vm_page_prot));
if (pmd_swp_soft_dirty(*pvmw->pmd))
pmde = pmd_mksoft_dirty(pmde);
diff --git a/mm/memory.c b/mm/memory.c
index aa6b2fc770fc..fcea6098efb2 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -939,7 +939,7 @@ static void restore_exclusive_pte(struct vm_area_struct *vma,
VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
- pte = pte_mkold(mk_pte(page, READ_ONCE(vma->vm_page_prot)));
+ pte = pte_mkold(mk_pte(page, pgprot_read(&vma->vm_page_prot)));
if (pte_swp_soft_dirty(orig_pte))
pte = pte_mksoft_dirty(pte);
diff --git a/mm/migrate.c b/mm/migrate.c
index 15b45832bcfa..579152cce502 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -387,7 +387,7 @@ static bool remove_migration_pte(struct folio *folio,
folio_get(folio);
new = folio_page(folio, idx);
- pte = mk_pte(new, READ_ONCE(vma->vm_page_prot));
+ pte = mk_pte(new, pgprot_read(&vma->vm_page_prot));
if (!softleaf_is_migration_young(entry))
pte = pte_mkold(pte);
if (folio_test_dirty(folio) && softleaf_is_migration_dirty(entry))
diff --git a/mm/mmap.c b/mm/mmap.c
index 4bf26b0f1e6e..729bc244b317 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -89,7 +89,7 @@ void vma_set_page_prot(struct vm_area_struct *vma)
vm_page_prot = vma_pgprot_modify(vm_page_prot, vma_flags);
}
/* remove_protection_ptes reads vma->vm_page_prot without mmap_lock */
- WRITE_ONCE(vma->vm_page_prot, vm_page_prot);
+ pgprot_write(&vma->vm_page_prot, vm_page_prot);
}
/*
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH V2 02/14] arm64/mm: Route all pgtable reads via ptval_get()
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 01/14] mm: Add read-write accessors for vm_page_prot Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 03/14] arm64/mm: Route all pgtable writes via ptval_set() Anshuman Khandual
` (11 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm
Define arm64 platform specific implementations for new pxdp_get() helpers.
These resolve into READ_ONCE(), thus ensuring required single copy atomic
semantics for the page table entry reads.
In future this infrastructure can be used for D128 to maintain single copy
atomicity semantics with inline asm blocks.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
arch/arm64/include/asm/pgtable.h | 38 +++++++++++++++++++++++++++-----
1 file changed, 32 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 83087b70f192..f764132ca34e 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -84,6 +84,32 @@ static inline void arch_leave_lazy_mmu_mode(void)
arch_flush_lazy_mmu_mode();
}
+#define ptval_get(x) READ_ONCE(x)
+
+#define pmdp_get pmdp_get
+static inline pmd_t pmdp_get(pmd_t *pmdp)
+{
+ return ptval_get(*pmdp);
+}
+
+#define pudp_get pudp_get
+static inline pud_t pudp_get(pud_t *pudp)
+{
+ return ptval_get(*pudp);
+}
+
+#define p4dp_get p4dp_get
+static inline p4d_t p4dp_get(p4d_t *p4dp)
+{
+ return ptval_get(*p4dp);
+}
+
+#define pgdp_get pgdp_get
+static inline pgd_t pgdp_get(pgd_t *pgdp)
+{
+ return ptval_get(*pgdp);
+}
+
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE
@@ -381,7 +407,7 @@ static inline void __set_pte(pte_t *ptep, pte_t pte)
static inline pte_t __ptep_get(pte_t *ptep)
{
- return READ_ONCE(*ptep);
+ return ptval_get(*ptep);
}
extern void __sync_icache_dcache(pte_t pteval);
@@ -958,7 +984,7 @@ static inline phys_addr_t pud_offset_phys(p4d_t *p4dp, unsigned long addr)
{
VM_WARN_ON_ONCE(!pgtable_l4_enabled());
- return p4d_page_paddr(READ_ONCE(*p4dp)) + pud_index(addr) * sizeof(pud_t);
+ return p4d_page_paddr(p4dp_get(p4dp)) + pud_index(addr) * sizeof(pud_t);
}
static inline
@@ -972,7 +998,7 @@ pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long addr)
static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long addr)
{
- return pud_offset_lockless(p4dp, READ_ONCE(*p4dp), addr);
+ return pud_offset_lockless(p4dp, p4dp_get(p4dp), addr);
}
#define pud_offset pud_offset
@@ -1078,7 +1104,7 @@ static inline phys_addr_t p4d_offset_phys(pgd_t *pgdp, unsigned long addr)
{
VM_WARN_ON_ONCE(!pgtable_l5_enabled());
- return pgd_page_paddr(READ_ONCE(*pgdp)) + p4d_index(addr) * sizeof(p4d_t);
+ return pgd_page_paddr(pgdp_get(pgdp)) + p4d_index(addr) * sizeof(p4d_t);
}
static inline
@@ -1092,7 +1118,7 @@ p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long addr)
static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long addr)
{
- return p4d_offset_lockless(pgdp, READ_ONCE(*pgdp), addr);
+ return p4d_offset_lockless(pgdp, pgdp_get(pgdp), addr);
}
static inline p4d_t *p4d_set_fixmap(unsigned long addr)
@@ -1287,7 +1313,7 @@ static inline bool pmdp_test_and_clear_young(struct vm_area_struct *vma,
unsigned long address, pmd_t *pmdp)
{
/* Operation applies to PMD table entry only if FEAT_HAFT is enabled */
- VM_WARN_ON(pmd_table(READ_ONCE(*pmdp)) && !system_supports_haft());
+ VM_WARN_ON(pmd_table(pmdp_get(pmdp)) && !system_supports_haft());
return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp);
}
#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH V2 03/14] arm64/mm: Route all pgtable writes via ptval_set()
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 01/14] mm: Add read-write accessors for vm_page_prot Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 02/14] arm64/mm: Route all pgtable reads via ptval_get() Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 04/14] arm64/mm: Route all pgtable atomics to central helpers Anshuman Khandual
` (10 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm
Currently ptval_set() is defined as WRITE_ONCE() but this will change for
D128 pgtable builds, for which WRITE_ONCE() is not sufficient for single
copy atomicity.
In future this infrastructure can be used for D128 to maintain single copy
atomicity semantics with inline asm blocks.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
arch/arm64/include/asm/pgtable.h | 11 ++++++-----
arch/arm64/mm/mmu.c | 4 ++--
2 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index f764132ca34e..692669ea2d29 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -85,6 +85,7 @@ static inline void arch_leave_lazy_mmu_mode(void)
}
#define ptval_get(x) READ_ONCE(x)
+#define ptval_set(x, val) WRITE_ONCE(x, val)
#define pmdp_get pmdp_get
static inline pmd_t pmdp_get(pmd_t *pmdp)
@@ -386,7 +387,7 @@ static inline pte_t pte_clear_uffd(pte_t pte)
static inline void __set_pte_nosync(pte_t *ptep, pte_t pte)
{
- WRITE_ONCE(*ptep, pte);
+ ptval_set(*ptep, pte);
}
static inline void __set_pte_complete(pte_t pte)
@@ -809,7 +810,7 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd)
}
#endif /* __PAGETABLE_PMD_FOLDED */
- WRITE_ONCE(*pmdp, pmd);
+ ptval_set(*pmdp, pmd);
if (pmd_valid(pmd))
queue_pte_barriers();
@@ -871,7 +872,7 @@ static inline void set_pud(pud_t *pudp, pud_t pud)
return;
}
- WRITE_ONCE(*pudp, pud);
+ ptval_set(*pudp, pud);
if (pud_valid(pud))
queue_pte_barriers();
@@ -950,7 +951,7 @@ static inline void set_p4d(p4d_t *p4dp, p4d_t p4d)
return;
}
- WRITE_ONCE(*p4dp, p4d);
+ ptval_set(*p4dp, p4d);
queue_pte_barriers();
}
@@ -1075,7 +1076,7 @@ static inline void set_pgd(pgd_t *pgdp, pgd_t pgd)
return;
}
- WRITE_ONCE(*pgdp, pgd);
+ ptval_set(*pgdp, pgd);
queue_pte_barriers();
}
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 46ff9a92e73a..44af53dc9ba8 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -75,7 +75,7 @@ void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd)
* writable in the kernel mapping.
*/
if (rodata_is_rw) {
- WRITE_ONCE(*pgdp, pgd);
+ ptval_set(*pgdp, pgd);
dsb(ishst);
isb();
return;
@@ -83,7 +83,7 @@ void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd)
spin_lock(&swapper_pgdir_lock);
fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp));
- WRITE_ONCE(*fixmap_pgdp, pgd);
+ ptval_set(*fixmap_pgdp, pgd);
/*
* We need dsb(ishst) here to ensure the page-table-walker sees
* our new entry before set_p?d() returns. The fixmap's
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH V2 04/14] arm64/mm: Route all pgtable atomics to central helpers
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
` (2 preceding siblings ...)
2026-09-07 3:50 ` [PATCH V2 03/14] arm64/mm: Route all pgtable writes via ptval_set() Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 05/14] arm64/mm: Override read-write accessors for vm_page_prot Anshuman Khandual
` (9 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm
Route all cmpxchg() operations performed on various page table entries to a
new ptval_cmpxchg_relaxed() helper. Similarly route all xchg() operations
performed on page table entries to a new ptval_xchg_relaxed() helper.
Currently these helpers just forward to the same APIs that were previously
called direct, but in future we will change the routing for D128 which is
too long to use the standard APIs.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
arch/arm64/include/asm/pgtable.h | 26 +++++++++++++++++++-------
arch/arm64/mm/fault.c | 2 +-
2 files changed, 20 insertions(+), 8 deletions(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 692669ea2d29..66bf5bbe484e 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -87,6 +87,17 @@ static inline void arch_leave_lazy_mmu_mode(void)
#define ptval_get(x) READ_ONCE(x)
#define ptval_set(x, val) WRITE_ONCE(x, val)
+static inline ptval_t ptval_cmpxchg_relaxed(ptval_t *ptep, ptval_t old,
+ ptval_t new)
+{
+ return cmpxchg_relaxed(ptep, old, new);
+}
+
+static inline ptval_t ptval_xchg_relaxed(ptval_t *ptep, ptval_t new)
+{
+ return xchg_relaxed(ptep, new);
+}
+
#define pmdp_get pmdp_get
static inline pmd_t pmdp_get(pmd_t *pmdp)
{
@@ -1281,8 +1292,8 @@ static inline bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
do {
old_pte = pte;
pte = pte_mkold(pte);
- pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep),
- pte_val(old_pte), pte_val(pte));
+ pte_val(pte) = ptval_cmpxchg_relaxed(&pte_val(*ptep),
+ pte_val(old_pte), pte_val(pte));
} while (pte_val(pte) != pte_val(old_pte));
return pte_young(pte);
@@ -1324,7 +1335,7 @@ static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct *mm,
pte_t *ptep,
unsigned long pgsize)
{
- pte_t pte = __pte(xchg_relaxed(&pte_val(*ptep), 0));
+ pte_t pte = __pte(ptval_xchg_relaxed(&pte_val(*ptep), 0));
switch (pgsize) {
case PAGE_SIZE:
@@ -1400,7 +1411,7 @@ static inline void ___ptep_set_wrprotect(struct mm_struct *mm,
do {
old_pte = pte;
pte = pte_wrprotect(pte);
- pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep),
+ pte_val(pte) = ptval_cmpxchg_relaxed(&pte_val(*ptep),
pte_val(old_pte), pte_val(pte));
} while (pte_val(pte) != pte_val(old_pte));
}
@@ -1438,7 +1449,7 @@ static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
if (flags & CYDP_CLEAR_DIRTY)
pte = pte_mkclean(pte);
- pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep),
+ pte_val(pte) = ptval_cmpxchg_relaxed(&pte_val(*ptep),
pte_val(old_pte), pte_val(pte));
} while (pte_val(pte) != pte_val(old_pte));
}
@@ -1477,7 +1488,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
unsigned long address, pmd_t *pmdp, pmd_t pmd)
{
page_table_check_pmd_set(vma->vm_mm, address, pmdp, pmd);
- return __pmd(xchg_relaxed(&pmd_val(*pmdp), pmd_val(pmd)));
+ return __pmd(ptval_xchg_relaxed(&pmd_val(*pmdp), pmd_val(pmd)));
}
#endif
@@ -1804,7 +1815,8 @@ static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte)
{
pteval_t old = 0;
- if (!try_cmpxchg(&pte_val(*ptep), &old, pte_val(new_pte)))
+ old = ptval_cmpxchg_relaxed(&pte_val(*ptep), old, pte_val(new_pte));
+ if (old != 0)
return false;
/*
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 1de76ced0403..6db93371d649 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -244,7 +244,7 @@ int __ptep_set_access_flags_anysz(struct vm_area_struct *vma,
pteval ^= PTE_RDONLY;
pteval |= pte_val(entry);
pteval ^= PTE_RDONLY;
- pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval);
+ pteval = ptval_cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval);
} while (pteval != old_pteval);
/*
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH V2 05/14] arm64/mm: Override read-write accessors for vm_page_prot
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
` (3 preceding siblings ...)
2026-09-07 3:50 ` [PATCH V2 04/14] arm64/mm: Route all pgtable atomics to central helpers Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 06/14] arm64/mm: Enable fixmap with 5 level page table Anshuman Khandual
` (8 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm
Override pgprot_[read|write]() accessors using ptval_[get|set]() providing
required single copy atomic operation for vma->vm_page_prot.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
arch/arm64/include/asm/pgtable.h | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 66bf5bbe484e..2b457db19be0 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -122,6 +122,18 @@ static inline pgd_t pgdp_get(pgd_t *pgdp)
return ptval_get(*pgdp);
}
+#define pgprot_read pgprot_read
+static inline pgprot_t pgprot_read(pgprot_t *prot)
+{
+ return ptval_get(*prot);
+}
+
+#define pgprot_write pgprot_write
+static inline void pgprot_write(pgprot_t *prot, pgprot_t val)
+{
+ ptval_set(*prot, val);
+}
+
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH V2 06/14] arm64/mm: Enable fixmap with 5 level page table
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
` (4 preceding siblings ...)
2026-09-07 3:50 ` [PATCH V2 05/14] arm64/mm: Override read-write accessors for vm_page_prot Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 07/14] arm64/mm: Enable pgtable geometry for FEAT_D128 Anshuman Khandual
` (7 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm
FEAT_D128 halves PTRS_PER_PXX thus shrinking the VA range coverage
for each page table level. Hence in order to preserve all existing
VA range configurations, some geometry now need to become 5-level.
Since fixmap is used to build and manipulate page tables early on
during boot the mapping must also gain that additional level which
was not required earlier.
Enable fixmap with 5 level page table when required. This creates table
entries at the PGD level. Add a fallback stub for pgd_page_paddr() when
(PGTBALE_LEVELS <= 4) which helps in intercepting any unintended usage.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
arch/arm64/include/asm/pgtable.h | 1 +
arch/arm64/mm/fixmap.c | 18 +++++++++++++++---
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 2b457db19be0..7a5d6a14bb52 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1178,6 +1178,7 @@ static inline p4d_t *p4d_offset_kimg(pgd_t *pgdp, u64 addr)
#else
static inline bool pgtable_l5_enabled(void) { return false; }
+#define pgd_page_paddr(pgd) ({ BUILD_BUG(); 0; })
#define p4d_index(addr) (((addr) >> P4D_SHIFT) & (PTRS_PER_P4D - 1))
diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c
index 2a0fe0b0f771..5512521fef8f 100644
--- a/arch/arm64/mm/fixmap.c
+++ b/arch/arm64/mm/fixmap.c
@@ -34,6 +34,7 @@ static_assert(NR_BM_PMD_TABLES == 1);
static pte_t bm_pte[NR_BM_PTE_TABLES][PTRS_PER_PTE] __bss_pgtbl;
static pmd_t bm_pmd[PTRS_PER_PMD] __bss_pgtbl __maybe_unused;
static pud_t bm_pud[PTRS_PER_PUD] __bss_pgtbl __maybe_unused;
+static p4d_t bm_p4d[PTRS_PER_P4D] __bss_pgtbl __maybe_unused;
static inline pte_t *fixmap_pte(unsigned long addr)
{
@@ -95,6 +96,19 @@ static void __init early_fixmap_init_pud(p4d_t *p4dp, unsigned long addr,
early_fixmap_init_pmd(pudp, addr, end);
}
+static void __init early_fixmap_init_p4d(pgd_t *pgdp, unsigned long addr,
+ unsigned long end)
+{
+ pgd_t pgd = pgdp_get(pgdp);
+ p4d_t *p4dp;
+
+ if (pgd_none(pgd))
+ __pgd_populate(pgdp, __pa_symbol(bm_p4d),
+ PGD_TYPE_TABLE | PGD_TABLE_AF);
+ p4dp = p4d_offset_kimg(pgdp, addr);
+ early_fixmap_init_pud(p4dp, addr, end);
+}
+
/*
* The p*d_populate functions call virt_to_phys implicitly so they can't be used
* directly on kernel symbols (bm_p*d). This function is called too early to use
@@ -107,9 +121,7 @@ void __init early_fixmap_init(void)
unsigned long end = FIXADDR_TOP;
pgd_t *pgdp = pgd_offset_k(addr);
- p4d_t *p4dp = p4d_offset_kimg(pgdp, addr);
-
- early_fixmap_init_pud(p4dp, addr, end);
+ early_fixmap_init_p4d(pgdp, addr, end);
}
/*
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH V2 07/14] arm64/mm: Enable pgtable geometry for FEAT_D128
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
` (5 preceding siblings ...)
2026-09-07 3:50 ` [PATCH V2 06/14] arm64/mm: Enable fixmap with 5 level page table Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 08/14] arm64/mm: Enable pgtable descriptor " Anshuman Khandual
` (6 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm
Add build time support for FEAT_D128 page tables with a new Kconfig option
i.e CONFIG_ARM64_D128. When selected, PTE types become 128 bits wide and
PTE bits are mapped to their new locations. Besides the basic page table
geometry is also updated since each table page now holds half the number
of entries (aka PTRS_PER_PXX) as it did previously. Similarly update the
geometry relevant macros ARM64_CONT_[PTE|PMD]_SHIFT, PGTABLE_LEVELS, and
PTDESC_ORDER.
Although for now config ARM64_D128 remains disabled while all the required
dependencies are getting created in subsequent patches.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
arch/arm64/Kconfig | 37 ++++++++++++++++++++++----
arch/arm64/include/asm/pgtable-hwdef.h | 4 +++
arch/arm64/include/asm/pgtable.h | 2 ++
3 files changed, 38 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b5a51b0ef944..306e6c803471 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -284,14 +284,18 @@ config MMU
config ARM64_CONT_PTE_SHIFT
int
- default 5 if PAGE_SIZE_64KB
- default 7 if PAGE_SIZE_16KB
+ default 4 if PAGE_SIZE_64KB && ARM64_D128
+ default 5 if PAGE_SIZE_64KB && !ARM64_D128
+ default 6 if PAGE_SIZE_16KB && ARM64_D128
+ default 7 if PAGE_SIZE_16KB && !ARM64_D128
default 4
config ARM64_CONT_PMD_SHIFT
int
- default 5 if PAGE_SIZE_64KB
- default 5 if PAGE_SIZE_16KB
+ default 6 if PAGE_SIZE_64KB && ARM64_D128
+ default 5 if PAGE_SIZE_64KB && !ARM64_D128
+ default 4 if PAGE_SIZE_16KB && ARM64_D128
+ default 5 if PAGE_SIZE_16KB && !ARM64_D128
default 4
config ARCH_MMAP_RND_BITS_MIN
@@ -362,6 +366,16 @@ config FIX_EARLYCON_MEM
config PGTABLE_LEVELS
int
+ default 4 if ARM64_D128 && ARM64_4K_PAGES && ARM64_VA_BITS_39
+ default 5 if ARM64_D128 && ARM64_4K_PAGES && ARM64_VA_BITS_48
+ default 5 if ARM64_D128 && ARM64_4K_PAGES && ARM64_VA_BITS_52
+ default 3 if ARM64_D128 && ARM64_16K_PAGES && ARM64_VA_BITS_36
+ default 4 if ARM64_D128 && ARM64_16K_PAGES && ARM64_VA_BITS_47
+ default 4 if ARM64_D128 && ARM64_16K_PAGES && ARM64_VA_BITS_48
+ default 4 if ARM64_D128 && ARM64_16K_PAGES && ARM64_VA_BITS_52
+ default 3 if ARM64_D128 && ARM64_64K_PAGES && ARM64_VA_BITS_42
+ default 3 if ARM64_D128 && ARM64_64K_PAGES && ARM64_VA_BITS_48
+ default 3 if ARM64_D128 && ARM64_64K_PAGES && ARM64_VA_BITS_52
default 2 if ARM64_16K_PAGES && ARM64_VA_BITS_36
default 2 if ARM64_64K_PAGES && ARM64_VA_BITS_42
default 3 if ARM64_64K_PAGES && (ARM64_VA_BITS_48 || ARM64_VA_BITS_52)
@@ -1594,7 +1608,7 @@ config ARM64_PA_BITS
config ARM64_LPA2
def_bool y
- depends on ARM64_PA_BITS_52 && !ARM64_64K_PAGES
+ depends on ARM64_PA_BITS_52 && !ARM64_64K_PAGES && !ARM64_D128
choice
prompt "Endianness"
@@ -2289,6 +2303,19 @@ config ARM64_HAFT
endmenu # "ARMv8.9 architectural features"
+menu "ARMv9.3 architectural features"
+
+config ARM64_D128
+ bool
+ default n
+ help
+ ARMv9.3 introduces FEAT_D128, which provides a 128 bit page
+ table format, along with related instructions.
+
+ If unsure, say Y.
+
+endmenu # "ARMv9.3 architectural features"
+
menu "ARMv9.4 architectural features"
config ARM64_GCS
diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index 72f31800c703..1ee1a6d9db6d 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -7,7 +7,11 @@
#include <asm/memory.h>
+#ifdef CONFIG_ARM64_D128
+#define PTDESC_ORDER 4
+#else
#define PTDESC_ORDER 3
+#endif
/* Number of VA bits resolved by a single translation table level */
#define PTDESC_TABLE_SHIFT (PAGE_SHIFT - PTDESC_ORDER)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 7a5d6a14bb52..4b534be3f288 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1075,6 +1075,8 @@ static inline bool pgtable_l4_enabled(void) { return false; }
static __always_inline bool pgtable_l5_enabled(void)
{
+ if (IS_ENABLED(CONFIG_ARM64_D128))
+ return true;
if (!alternative_has_cap_likely(ARM64_ALWAYS_BOOT))
return vabits_actual == VA_BITS;
return alternative_has_cap_unlikely(ARM64_HAS_VA52);
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH V2 08/14] arm64/mm: Enable pgtable descriptor for FEAT_D128
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
` (6 preceding siblings ...)
2026-09-07 3:50 ` [PATCH V2 07/14] arm64/mm: Enable pgtable geometry for FEAT_D128 Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 09/14] arm64/mm: Enable 128 bit translation Anshuman Khandual
` (5 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm
Enable pgtable descriptor fields and related accessors for FEAT_D128. First
expand ptval_t as 128 bits, add all D128 specific pgtable descriptor fields
and update accessors like ptval_get/set() and ptval_xchg_relaxed(). Also do
ensure that ARM64_PA_BITS_52 specific handlings are not applicable for D128
which does not follow PA bits folding mechanisms as in FEAT_[LPA|LPA2].
Although all the above changes gets enabled only with CONFIG_ARM64_D128.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
arch/arm64/include/asm/assembler.h | 4 +-
arch/arm64/include/asm/pgtable-hwdef.h | 133 +++++++++++++++++++++++++
arch/arm64/include/asm/pgtable-prot.h | 20 +++-
arch/arm64/include/asm/pgtable-types.h | 4 +
arch/arm64/include/asm/pgtable.h | 52 +++++++++-
arch/arm64/mm/proc.S | 2 +-
6 files changed, 208 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 0b58b550e8dc..1402f90f0030 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -623,7 +623,7 @@ alternative_else_nop_endif
* ttbr: returns the TTBR value
*/
.macro phys_to_ttbr, ttbr, phys
-#ifdef CONFIG_ARM64_PA_BITS_52
+#if defined(CONFIG_ARM64_PA_BITS_52) && !defined(CONFIG_ARM64_D128)
orr \ttbr, \phys, \phys, lsr #46
and \ttbr, \ttbr, #TTBR_BADDR_MASK_52
#else
@@ -632,7 +632,7 @@ alternative_else_nop_endif
.endm
.macro phys_to_pte, pte, phys
-#ifdef CONFIG_ARM64_PA_BITS_52
+#if defined(CONFIG_ARM64_PA_BITS_52) && !defined(CONFIG_ARM64_D128)
orr \pte, \phys, \phys, lsr #PTE_ADDR_HIGH_SHIFT
and \pte, \pte, #PHYS_TO_PTE_ADDR_MASK
#else
diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index 1ee1a6d9db6d..16fb74c47b74 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -101,6 +101,137 @@
#define CONT_PMD_SIZE (CONT_PMDS * PMD_SIZE)
#define CONT_PMD_MASK (~(CONT_PMD_SIZE - 1))
+#ifdef CONFIG_ARM64_D128
+
+/*
+ * Hardware page table definitions.
+ *
+ * Level -1 descriptor (PGD).
+ */
+#define PGD_SKL_SHIFT 109
+#define PGD_SKL_MASK GENMASK_U128(110, 109)
+#define PGD_SKL_TABLE (_AT(pgdval_t, 0) << PGD_SKL_SHIFT)
+
+#define PGD_TYPE_TABLE _AT(pgdval_t, (PTE_VALID | PGD_SKL_TABLE))
+#define PGD_TYPE_MASK _AT(pgdval_t, (PTE_VALID | PGD_SKL_MASK))
+#define PGD_TABLE_AF (_AT(pgdval_t, 1) << 10) /* Ignored if no FEAT_HAFT */
+#define PGD_TABLE_PXN _AT(pgdval_t, 0) /* Not supported for D128 */
+#define PGD_TABLE_UXN _AT(pgdval_t, 0) /* Not supported for D128 */
+
+/*
+ * Level 0 descriptor (P4D).
+ */
+#define P4D_SKL_SHIFT 109
+#define P4D_SKL_MASK GENMASK_U128(110, 109)
+#define P4D_SKL_TABLE (_AT(p4dval_t, 0) << P4D_SKL_SHIFT)
+#define P4D_SKL_SECT (_AT(p4dval_t, 3) << P4D_SKL_SHIFT)
+
+#define P4D_TYPE_TABLE _AT(p4dval_t, (PTE_VALID | P4D_SKL_TABLE))
+#define P4D_TYPE_MASK _AT(p4dval_t, (PTE_VALID | P4D_SKL_MASK))
+#define P4D_TYPE_SECT _AT(p4dval_t, (PTE_VALID | P4D_SKL_SECT))
+#define P4D_SECT_RDONLY (_AT(p4dval_t, 1) << 7) /* nDirty */
+#define P4D_TABLE_AF (_AT(p4dval_t, 1) << 10) /* Ignored if no FEAT_HAFT */
+#define P4D_TABLE_PXN _AT(p4dval_t, 0) /* Not supported for D128 */
+#define P4D_TABLE_UXN _AT(p4dval_t, 0) /* Not supported for D128 */
+
+/*
+ * Level 1 descriptor (PUD).
+ */
+#define PUD_SKL_SHIFT 109
+#define PUD_SKL_MASK GENMASK_U128(110, 109)
+#define PUD_SKL_TABLE (_AT(pudval_t, 0) << PUD_SKL_SHIFT)
+#define PUD_SKL_SECT (_AT(pudval_t, 2) << PUD_SKL_SHIFT)
+
+#define PUD_TYPE_TABLE _AT(pudval_t, (PTE_VALID | PUD_SKL_TABLE))
+#define PUD_TYPE_MASK _AT(pudval_t, (PTE_VALID | PUD_SKL_MASK))
+#define PUD_TYPE_SECT _AT(pudval_t, (PTE_VALID | PUD_SKL_SECT))
+#define PUD_SECT_RDONLY (_AT(pudval_t, 1) << 7) /* nDirty */
+#define PUD_TABLE_AF (_AT(pudval_t, 1) << 10) /* Ignored if no FEAT_HAFT */
+#define PUD_TABLE_PXN _AT(pudval_t, 0) /* Not supported for D128 */
+#define PUD_TABLE_UXN _AT(pudval_t, 0) /* Not supported for D128 */
+
+/*
+ * Level 2 descriptor (PMD).
+ */
+#define PMD_SKL_SHIFT 109
+#define PMD_SKL_MASK GENMASK_U128(110, 109)
+#define PMD_SKL_TABLE (_AT(pmdval_t, 0) << PMD_SKL_SHIFT)
+#define PMD_SKL_SECT (_AT(pmdval_t, 1) << PMD_SKL_SHIFT)
+
+#define PMD_TYPE_MASK _AT(pmdval_t, (PTE_VALID | PMD_SKL_MASK))
+#define PMD_TYPE_TABLE _AT(pmdval_t, (PTE_VALID | PMD_SKL_TABLE))
+#define PMD_TYPE_SECT _AT(pmdval_t, (PTE_VALID | PMD_SKL_SECT))
+#define PMD_TABLE_AF (_AT(pmdval_t, 1) << 10) /* Ignored if no FEAT_HAFT */
+#define PMD_TABLE_PXN _AT(pmdval_t, 0) /* Not supported for D128 */
+#define PMD_TABLE_UXN _AT(pmdval_t, 0) /* Not supported for D128 */
+
+/*
+ * Section
+ */
+#define PMD_SECT_USER (_AT(pmdval_t, 1) << 115) /* PIIndex[0] */
+#define PMD_SECT_RDONLY (_AT(pmdval_t, 1) << 7) /* nDirty */
+#define PMD_SECT_S (_AT(pmdval_t, 3) << 8)
+#define PMD_SECT_AF (_AT(pmdval_t, 1) << 10)
+#define PMD_SECT_NG (_AT(pmdval_t, 1) << 11)
+#define PMD_SECT_CONT (_AT(pmdval_t, 1) << 111)
+#define PMD_SECT_PXN (_AT(pmdval_t, 1) << 117) /* PIIndex[2] */
+#define PMD_SECT_UXN (_AT(pmdval_t, 1) << 118) /* PIIndex[3] */
+
+/*
+ * AttrIndx[2:0] encoding (mapping attributes defined in the MAIR* registers).
+ */
+#define PMD_ATTRINDX(t) (_AT(pmdval_t, (t)) << 2)
+#define PMD_ATTRINDX_MASK (_AT(pmdval_t, 7) << 2)
+
+/*
+ * Level 3 descriptor (PTE).
+ */
+#define PTE_SKL_SHIFT 109
+#define PTE_SKL_MASK GENMASK_U128(110, 109)
+#define PTE_SKL_SECT (_AT(pteval_t, 0) << PTE_SKL_SHIFT)
+
+#define PTE_VALID (_AT(pteval_t, 1) << 0)
+#define PTE_TYPE_MASK _AT(pteval_t, (PTE_VALID | PTE_SKL_MASK))
+#define PTE_TYPE_PAGE _AT(pteval_t, (PTE_VALID | PTE_SKL_SECT))
+#define PTE_USER (_AT(pteval_t, 1) << 115) /* PIIndex[0] */
+#define PTE_RDONLY (_AT(pteval_t, 1) << 7) /* nDirty */
+#define PTE_SHARED (_AT(pteval_t, 3) << 8) /* SH[1:0], inner shareable */
+#define PTE_AF (_AT(pteval_t, 1) << 10) /* Access Flag */
+#define PTE_NG (_AT(pteval_t, 1) << 11) /* nG */
+#define PTE_GP (_AT(pteval_t, 1) << 113) /* BTI guarded */
+#define PTE_DBM (_AT(pteval_t, 1) << 116) /* PIIndex[1] */
+#define PTE_CONT (_AT(pteval_t, 1) << 111) /* Contiguous range */
+#define PTE_PXN (_AT(pteval_t, 1) << 117) /* PIIndex[2] */
+#define PTE_UXN (_AT(pteval_t, 1) << 118) /* PIIndex[3] */
+#define PTE_SWBITS_MASK _AT(pteval_t, GENMASK_U128(100, 91))
+
+#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (55 - PAGE_SHIFT)) - 1) << PAGE_SHIFT)
+
+/*
+ * AttrIndx[2:0] encoding (mapping attributes defined in the MAIR* registers).
+ */
+#define PTE_ATTRINDX(t) (_AT(pteval_t, (t)) << 2)
+#define PTE_ATTRINDX_MASK (_AT(pteval_t, 7) << 2)
+
+/*
+ * PIIndex[3:0] encoding (Permission Indirection Extension)
+ */
+#define PTE_PI_MASK GENMASK_U128(118, 115)
+#define PTE_PI_SHIFT 115
+
+/*
+ * POIndex[3:0] encoding (Permission Overlay Extension)
+ */
+#define PTE_PO_IDX_0 (_AT(pteval_t, 1) << 121)
+#define PTE_PO_IDX_1 (_AT(pteval_t, 1) << 122)
+#define PTE_PO_IDX_2 (_AT(pteval_t, 1) << 123)
+#define PTE_PO_IDX_3 (_AT(pteval_t, 1) << 124)
+
+#define PTE_PO_IDX_MASK GENMASK_U128(124, 121)
+#define PTE_PO_IDX_SHIFT 121
+
+#else /* !CONFIG_ARM64_D128 */
+
/*
* Hardware page table definitions.
*
@@ -215,7 +346,9 @@
#define PTE_PO_IDX_2 (_AT(pteval_t, 1) << 62)
#define PTE_PO_IDX_MASK GENMASK_ULL(62, 60)
+#define PTE_PO_IDX_SHIFT 60
+#endif /* CONFIG_ARM64_D128 */
/*
* Memory Attribute override for Stage-2 (MemAttr[3:0])
diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h
index 09d7c00cf405..f7df22f222b2 100644
--- a/arch/arm64/include/asm/pgtable-prot.h
+++ b/arch/arm64/include/asm/pgtable-prot.h
@@ -13,10 +13,15 @@
/*
* Software defined PTE bits definition.
*/
-#define PTE_WRITE (PTE_DBM) /* same as DBM (51) */
+#define PTE_WRITE (PTE_DBM) /* same as DBM (51 / 116) */
#define PTE_SWP_EXCLUSIVE (_AT(pteval_t, 1) << 2) /* only for swp ptes */
+#ifdef CONFIG_ARM64_D128
+#define PTE_DIRTY (_AT(pteval_t, 1) << 91)
+#define PTE_SPECIAL (_AT(pteval_t, 1) << 92)
+#else
#define PTE_DIRTY (_AT(pteval_t, 1) << 55)
#define PTE_SPECIAL (_AT(pteval_t, 1) << 56)
+#endif
/*
* PTE_PRESENT_INVALID=1 & PTE_VALID=0 indicates that the pte's fields should be
@@ -28,7 +33,11 @@
#define PTE_PRESENT_VALID_KERNEL (PTE_VALID | PTE_MAYBE_NG)
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_WP
-#define PTE_UFFD (_AT(pteval_t, 1) << 58) /* userfaultfd tracking */
+#ifdef CONFIG_ARM64_D128
+#define PTE_UFFD (_AT(pteval_t, 1) << 94) /* userfaultd tracking */
+#else
+#define PTE_UFFD (_AT(pteval_t, 1) << 58) /* userfaultd tracking */
+#endif /* CONFIG_ARM64_D128 */
#define PTE_SWP_UFFD (_AT(pteval_t, 1) << 3) /* only for swp ptes */
#else
#define PTE_UFFD (_AT(pteval_t, 0))
@@ -131,11 +140,18 @@ static inline bool __pure lpa2_is_enabled(void)
#endif /* __ASSEMBLER__ */
+#ifdef CONFIG_ARM64_D128
+#define pte_pi_index(pte) (((pte) & PTE_PI_MASK) >> PTE_PI_SHIFT)
+#define pte_po_index(pte) ((pte_val(pte) & PTE_PO_IDX_MASK) >> PTE_PO_IDX_SHIFT)
+#else
#define pte_pi_index(pte) ( \
((pte & BIT(PTE_PI_IDX_3)) >> (PTE_PI_IDX_3 - 3)) | \
((pte & BIT(PTE_PI_IDX_2)) >> (PTE_PI_IDX_2 - 2)) | \
((pte & BIT(PTE_PI_IDX_1)) >> (PTE_PI_IDX_1 - 1)) | \
((pte & BIT(PTE_PI_IDX_0)) >> (PTE_PI_IDX_0 - 0)))
+#define pte_po_index(pte) FIELD_GET(PTE_PO_IDX_MASK, pte_val(pte))
+#endif
+
/*
* Page types used via Permission Indirection Extension (PIE). PIE uses
diff --git a/arch/arm64/include/asm/pgtable-types.h b/arch/arm64/include/asm/pgtable-types.h
index 2f2f5527930f..83160f8a19fd 100644
--- a/arch/arm64/include/asm/pgtable-types.h
+++ b/arch/arm64/include/asm/pgtable-types.h
@@ -17,7 +17,11 @@
* Generic page table descriptor format from which
* all level specific descriptors can be derived.
*/
+#ifdef CONFIG_ARM64_D128
+typedef u128 ptval_t;
+#else
typedef u64 ptval_t;
+#endif
typedef ptval_t pteval_t;
typedef ptval_t pmdval_t;
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 4b534be3f288..516681b6db37 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -84,18 +84,62 @@ static inline void arch_leave_lazy_mmu_mode(void)
arch_flush_lazy_mmu_mode();
}
+#ifdef CONFIG_ARM64_D128
+#define ptval_get(x) \
+({ \
+ typeof(&(x)) __x = &(x); \
+ union __u128_halves __v; \
+ \
+ asm volatile ("ldp %[lo], %[hi], %[v]\n" \
+ : [lo] "=r"(__v.low), \
+ [hi] "=r"(__v.high) \
+ : [v] "Q"(*__x) \
+ ); \
+ *(typeof(__x))(&__v.full); \
+})
+
+#define ptval_set(x, val) \
+({ \
+ typeof(&(x)) __x = &(x); \
+ union __u128_halves __v = { .full = *(u128*)(&(val)) }; \
+ \
+ asm volatile ("stp %[lo], %[hi], %[v]\n" \
+ : [v] "=Q"(*__x) \
+ : [lo] "r"(__v.low), \
+ [hi] "r"(__v.high) \
+ ); \
+})
+#else
#define ptval_get(x) READ_ONCE(x)
#define ptval_set(x, val) WRITE_ONCE(x, val)
+#endif
static inline ptval_t ptval_cmpxchg_relaxed(ptval_t *ptep, ptval_t old,
ptval_t new)
{
+#ifdef CONFIG_ARM64_D128
+ return cmpxchg128_relaxed(ptep, old, new);
+#else
return cmpxchg_relaxed(ptep, old, new);
+#endif
}
static inline ptval_t ptval_xchg_relaxed(ptval_t *ptep, ptval_t new)
{
+#ifdef CONFIG_ARM64_D128
+ union __u128_halves r = { .full = new };
+
+ asm volatile(
+ ".arch_extension lse128\n"
+ "swpp %[lo], %[hi], %[v]\n"
+ : [lo] "+r" (r.low),
+ [hi] "+r" (r.high),
+ [v] "+Q" (*ptep)
+ :);
+ return r.full;
+#else
return xchg_relaxed(ptep, new);
+#endif
}
#define pmdp_get pmdp_get
@@ -157,7 +201,7 @@ static inline void pgprot_write(pgprot_t *prot, pgprot_t val)
__flush_tlb_range(vma, address, address + PMD_SIZE, PMD_SIZE, 2, \
TLBF_NOBROADCAST | TLBF_NONOTIFY | TLBF_NOWALKCACHE)
-#ifdef CONFIG_ARM64_PA_BITS_52
+#if defined(CONFIG_ARM64_PA_BITS_52) && !defined(CONFIG_ARM64_D128)
static inline phys_addr_t __pte_to_phys(pte_t pte)
{
pte_val(pte) &= ~PTE_MAYBE_SHARED;
@@ -272,7 +316,7 @@ static inline bool por_el0_allows_pkey(u8 pkey, bool write, bool execute)
(((pte_val(pte) & (PTE_VALID | PTE_USER)) == (PTE_VALID | PTE_USER)) && (!(write) || pte_write(pte)))
#define pte_access_permitted(pte, write) \
(pte_access_permitted_no_overlay(pte, write) && \
- por_el0_allows_pkey(FIELD_GET(PTE_PO_IDX_MASK, pte_val(pte)), write, false))
+ por_el0_allows_pkey(pte_po_index(pte), write, false))
#define pmd_access_permitted(pmd, write) \
(pte_access_permitted(pmd_pte(pmd), (write)))
#define pud_access_permitted(pud, write) \
@@ -1582,11 +1626,15 @@ static inline void update_mmu_cache_range(struct vm_fault *vmf,
update_mmu_cache_range(NULL, vma, addr, ptep, 1)
#define update_mmu_cache_pmd(vma, address, pmd) do { } while (0)
+#ifdef CONFIG_ARM64_D128
+#define phys_to_ttbr(addr) (addr)
+#else
#ifdef CONFIG_ARM64_PA_BITS_52
#define phys_to_ttbr(addr) (((addr) | ((addr) >> 46)) & TTBR_BADDR_MASK_52)
#else
#define phys_to_ttbr(addr) (addr)
#endif
+#endif
/*
* On arm64 without hardware Access Flag, copying from user will fail because
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 12aacc74f764..4e8889961b15 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -222,7 +222,7 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1)
.macro pte_to_phys, phys, pte
and \phys, \pte, #PTE_ADDR_LOW
-#ifdef CONFIG_ARM64_PA_BITS_52
+#if defined(CONFIG_ARM64_PA_BITS_52) && !defined(CONFIG_ARM64_D128)
and \pte, \pte, #PTE_ADDR_HIGH
orr \phys, \phys, \pte, lsl #PTE_ADDR_HIGH_SHIFT
#endif
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH V2 09/14] arm64/mm: Enable 128 bit translation
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
` (7 preceding siblings ...)
2026-09-07 3:50 ` [PATCH V2 08/14] arm64/mm: Enable pgtable descriptor " Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 10/14] arm64/mm: Add an abstraction level for tlbi_op Anshuman Khandual
` (4 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm
All required changes to page table geometry and descriptor are now in place
for D128 translation to be enabled.
Since FEAT_D128 exclusively supports the permission indirection style for
page table entry permission management, given kernel compiled for FEAT_D128
requires both FEAT_S1PIE and FEAT_D128. If these architecture features are
not present during boot, the kernel panics just like it does when there is
a granule size mismatch. So added a new stuck CPU debug identifier for such
scenarios called CPU_STUCK_REASON_NO_D128.
TTBR0/1_EL1 and PAR_EL1 registers become 128 bit wide when D128 is enabled,
thus requiring MSRR/MRRS instructions for their updates. Because PA_BITS is
still capped at 52 bits, MRS/MSR instructions are currently sufficient for
the register accesses that basically operate on the lower 64 bits. Although
entire 128 bits for these registers get cleared during boot via MSRR.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
arch/arm64/Makefile | 4 ++++
arch/arm64/include/asm/el2_setup.h | 9 +++++++++
arch/arm64/include/asm/smp.h | 1 +
arch/arm64/kernel/head.S | 12 ++++++++++++
arch/arm64/mm/proc.S | 23 +++++++++++++++++++++++
5 files changed, 49 insertions(+)
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 6b005c8fef70..12a6e62520e9 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -54,6 +54,10 @@ endif
KBUILD_CFLAGS += $(call cc-option,-mabi=lp64)
KBUILD_AFLAGS += $(call cc-option,-mabi=lp64)
+ifeq ($(CONFIG_ARM64_D128),y)
+KBUILD_AFLAGS += -march=armv9.3-a+d128
+endif
+
# Avoid generating .eh_frame* sections.
ifneq ($(CONFIG_UNWIND_TABLES),y)
KBUILD_CFLAGS += -fno-asynchronous-unwind-tables -fno-unwind-tables
diff --git a/arch/arm64/include/asm/el2_setup.h b/arch/arm64/include/asm/el2_setup.h
index aa8ec9df8024..331e94381dd0 100644
--- a/arch/arm64/include/asm/el2_setup.h
+++ b/arch/arm64/include/asm/el2_setup.h
@@ -78,6 +78,15 @@
cbz x0, .Lskip_hcrx_\@
mov_q x0, (HCRX_EL2_MSCEn | HCRX_EL2_TCR2En | HCRX_EL2_EnFPM)
+#ifdef CONFIG_ARM64_D128
+ mrs_s x1, SYS_ID_AA64MMFR3_EL1
+ ubfx x1, x1, #ID_AA64MMFR3_EL1_D128_SHIFT, #4
+ cbz x1, .Lskip_d128_\@
+
+ orr x0, x0, HCRX_EL2_D128En // Disable MRRS/MSRR traps
+.Lskip_d128_\@:
+#endif
+
/* Enable GCS if supported */
mrs_s x1, SYS_ID_AA64PFR1_EL1
ubfx x1, x1, #ID_AA64PFR1_EL1_GCS_SHIFT, #4
diff --git a/arch/arm64/include/asm/smp.h b/arch/arm64/include/asm/smp.h
index 10ea4f543069..1dd675d2b84d 100644
--- a/arch/arm64/include/asm/smp.h
+++ b/arch/arm64/include/asm/smp.h
@@ -22,6 +22,7 @@
#define CPU_STUCK_REASON_52_BIT_VA (UL(1) << CPU_STUCK_REASON_SHIFT)
#define CPU_STUCK_REASON_NO_GRAN (UL(2) << CPU_STUCK_REASON_SHIFT)
+#define CPU_STUCK_REASON_NO_D128 (UL(3) << CPU_STUCK_REASON_SHIFT)
#ifndef __ASSEMBLER__
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 87a822e5c4ca..4ad8047963ad 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -505,6 +505,18 @@ SYM_FUNC_START_LOCAL(__no_granule_support)
b 1b
SYM_FUNC_END(__no_granule_support)
+#ifdef CONFIG_ARM64_D128
+SYM_FUNC_START(__no_d128_support)
+ /* Indicate that this CPU can't boot and is stuck in the kernel */
+ update_early_cpu_boot_status \
+ CPU_STUCK_IN_KERNEL | CPU_STUCK_REASON_NO_D128, x1, x2
+1:
+ wfe
+ wfi
+ b 1b
+SYM_FUNC_END(__no_d128_support)
+#endif
+
SYM_FUNC_START_LOCAL(__primary_switch)
adrp x1, reserved_pg_dir
adrp x2, __pi_init_idmap_pg_dir
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 4e8889961b15..faf16ffb0528 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -548,7 +548,30 @@ alternative_else_nop_endif
mrs_s x1, SYS_ID_AA64MMFR3_EL1
ubfx x1, x1, #ID_AA64MMFR3_EL1_S1PIE_SHIFT, #4
+#ifdef CONFIG_ARM64_D128
+ cbnz x1, .Lcheck_d128
+ bl __no_d128_support
+.Lcheck_d128:
+ mrs_s x1, SYS_ID_AA64MMFR3_EL1
+ ubfx x1, x1, #ID_AA64MMFR3_EL1_D128_SHIFT, #4
+ cbnz x1, .Linit_d128
+ bl __no_d128_support
+.Linit_d128:
+ /*
+ * Although the lower 64 bits in TTBRx_EL1 registers are now
+ * being used it is prudent to clear out the entire 128 bits
+ * just in case the kernel receives non-zero value in higher
+ * 64 bits from the EL3 which might corrupt the page tables.
+ */
+ mov x4, xzr
+ mov x5, xzr
+
+ msrr ttbr0_el1, x4, x5
+ msrr ttbr1_el1, x4, x5
+ orr tcr2, tcr2, #TCR2_EL1_D128
+#else
cbz x1, .Lskip_indirection
+#endif
mov_q x0, PIE_E0_ASM
msr REG_PIRE0_EL1, x0
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH V2 10/14] arm64/mm: Add an abstraction level for tlbi_op
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
` (8 preceding siblings ...)
2026-09-07 3:50 ` [PATCH V2 09/14] arm64/mm: Enable 128 bit translation Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 11/14] arm64/mm: Enable TLBIP instruction based TLB flush Anshuman Khandual
` (3 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm
From: Linu Cherian <linu.cherian@arm.com>
With FEAT_D128, a new instruction aka TLBIP is being introduced for the TLB
-by-range operations which has an argument size of 128 bit.
Add an abstraction level to void (*tlbi_op)(u64 arg) helpers to support the
D128 variations when applicable.
No functional changes are introduced with this patch.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Linu Cherian <linu.cherian@arm.com>
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
arch/arm64/include/asm/tlbflush.h | 70 ++++++++++++++++---------------
1 file changed, 37 insertions(+), 33 deletions(-)
diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
index 14a78ac0f800..88ae4e443ed6 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -130,49 +130,53 @@ static inline void sme_dvmsync_batch(void)
#define TLBI_TTL_UNKNOWN INT_MAX
-typedef void (*tlbi_op)(u64 arg);
+typedef u64 tlbi_arg_t;
+#define __tlbi_wrapper(op, arg) __tlbi(op, arg)
+#define __tlbi_user_wrapper(op, arg) __tlbi_user(op, arg)
-static __always_inline void vae1is(u64 arg)
+typedef void (*tlbi_op)(tlbi_arg_t arg);
+
+static __always_inline void vae1is(tlbi_arg_t arg)
{
- __tlbi(vae1is, arg);
- __tlbi_user(vae1is, arg);
+ __tlbi_wrapper(vae1is, arg);
+ __tlbi_user_wrapper(vae1is, arg);
}
-static __always_inline void vae2is(u64 arg)
+static __always_inline void vae2is(tlbi_arg_t arg)
{
- __tlbi(vae2is, arg);
+ __tlbi_wrapper(vae2is, arg);
}
-static __always_inline void vale1(u64 arg)
+static __always_inline void vale1(tlbi_arg_t arg)
{
- __tlbi(vale1, arg);
- __tlbi_user(vale1, arg);
+ __tlbi_wrapper(vale1, arg);
+ __tlbi_user_wrapper(vale1, arg);
}
-static __always_inline void vale1is(u64 arg)
+static __always_inline void vale1is(tlbi_arg_t arg)
{
- __tlbi(vale1is, arg);
- __tlbi_user(vale1is, arg);
+ __tlbi_wrapper(vale1is, arg);
+ __tlbi_user_wrapper(vale1is, arg);
}
-static __always_inline void vale2is(u64 arg)
+static __always_inline void vale2is(tlbi_arg_t arg)
{
- __tlbi(vale2is, arg);
+ __tlbi_wrapper(vale2is, arg);
}
-static __always_inline void vaale1is(u64 arg)
+static __always_inline void vaale1is(tlbi_arg_t arg)
{
- __tlbi(vaale1is, arg);
+ __tlbi_wrapper(vaale1is, arg);
}
-static __always_inline void ipas2e1(u64 arg)
+static __always_inline void ipas2e1(tlbi_arg_t arg)
{
- __tlbi(ipas2e1, arg);
+ __tlbi_wrapper(ipas2e1, arg);
}
-static __always_inline void ipas2e1is(u64 arg)
+static __always_inline void ipas2e1is(tlbi_arg_t arg)
{
- __tlbi(ipas2e1is, arg);
+ __tlbi_wrapper(ipas2e1is, arg);
}
static __always_inline void __tlbi_level_asid(tlbi_op op, u64 addr, u32 level,
@@ -443,32 +447,32 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
* operations can only span an even number of pages. We save this for last to
* ensure 64KB start alignment is maintained for the LPA2 case.
*/
-static __always_inline void rvae1is(u64 arg)
+static __always_inline void rvae1is(tlbi_arg_t arg)
{
- __tlbi(rvae1is, arg);
- __tlbi_user(rvae1is, arg);
+ __tlbi_wrapper(rvae1is, arg);
+ __tlbi_user_wrapper(rvae1is, arg);
}
-static __always_inline void rvale1(u64 arg)
+static __always_inline void rvale1(tlbi_arg_t arg)
{
- __tlbi(rvale1, arg);
- __tlbi_user(rvale1, arg);
+ __tlbi_wrapper(rvale1, arg);
+ __tlbi_user_wrapper(rvale1, arg);
}
-static __always_inline void rvale1is(u64 arg)
+static __always_inline void rvale1is(tlbi_arg_t arg)
{
- __tlbi(rvale1is, arg);
- __tlbi_user(rvale1is, arg);
+ __tlbi_wrapper(rvale1is, arg);
+ __tlbi_user_wrapper(rvale1is, arg);
}
-static __always_inline void rvaale1is(u64 arg)
+static __always_inline void rvaale1is(tlbi_arg_t arg)
{
- __tlbi(rvaale1is, arg);
+ __tlbi_wrapper(rvaale1is, arg);
}
-static __always_inline void ripas2e1is(u64 arg)
+static __always_inline void ripas2e1is(tlbi_arg_t arg)
{
- __tlbi(ripas2e1is, arg);
+ __tlbi_wrapper(ripas2e1is, arg);
}
static __always_inline void __tlbi_range(tlbi_op op, u64 addr,
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH V2 11/14] arm64/mm: Enable TLBIP instruction based TLB flush
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
` (9 preceding siblings ...)
2026-09-07 3:50 ` [PATCH V2 10/14] arm64/mm: Add an abstraction level for tlbi_op Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 12/14] arm64/mm: Relax KASAN shadow alignment for FEAT_D128 Anshuman Khandual
` (2 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm
From: Linu Cherian <linu.cherian@arm.com>
Add TLBIP instruction support for TLB flush operation updating all relevant
macros as necessary. This enables back pgtable level hint and address range
support during TLB flush operation for D128 translation.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Linu Cherian <linu.cherian@arm.com>
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
arch/arm64/include/asm/tlbflush.h | 68 +++++++++++++++++++++++++++----
1 file changed, 59 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
index 88ae4e443ed6..f31a555760f0 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -41,6 +41,25 @@
#define __tlbi(op, ...) __TLBI_N(op, ##__VA_ARGS__, 1, 0)
+#ifdef CONFIG_ARM64_D128
+#define __tlbip(op, arg) do { \
+ asm (ARM64_ASM_PREAMBLE \
+ ".arch_extension d128\n\t" \
+ "tlbip " #op ", %0, %H0\n" \
+ : : "r" (arg.full)); \
+} while (0)
+
+#define __tlbip_user(op, arg) do { \
+ if (arm64_kernel_unmapped_at_el0()) { \
+ arg.low |= USER_ASID_FLAG; \
+ __tlbip(op, (arg)); \
+ } \
+} while (0)
+
+#endif
+
+#define TLBI_ASID_MASK GENMASK_ULL(63, 48)
+
#define __tlbi_user(op, arg) do { \
if (arm64_kernel_unmapped_at_el0()) \
__tlbi(op, (arg) | USER_ASID_FLAG); \
@@ -130,9 +149,15 @@ static inline void sme_dvmsync_batch(void)
#define TLBI_TTL_UNKNOWN INT_MAX
+#ifdef CONFIG_ARM64_D128
+typedef union __u128_halves tlbi_arg_t;
+#define __tlbi_wrapper(op, arg) __tlbip(op, arg)
+#define __tlbi_user_wrapper(op, arg) __tlbip_user(op, arg)
+#else
typedef u64 tlbi_arg_t;
#define __tlbi_wrapper(op, arg) __tlbi(op, arg)
#define __tlbi_user_wrapper(op, arg) __tlbi_user(op, arg)
+#endif
typedef void (*tlbi_op)(tlbi_arg_t arg);
@@ -179,17 +204,28 @@ static __always_inline void ipas2e1is(tlbi_arg_t arg)
__tlbi_wrapper(ipas2e1is, arg);
}
-static __always_inline void __tlbi_level_asid(tlbi_op op, u64 addr, u32 level,
- u16 asid)
+static __always_inline void __tlbi_update_level(u32 level, u64 *arg)
{
- u64 arg = __TLBI_VADDR(addr, asid);
-
if (alternative_has_cap_unlikely(ARM64_HAS_ARMv8_4_TTL) && level <= 3) {
u64 ttl = level | (get_trans_granule() << 2);
- FIELD_MODIFY(TLBI_TTL_MASK, &arg, ttl);
+ FIELD_MODIFY(TLBI_TTL_MASK, arg, ttl);
}
+}
+
+static __always_inline void __tlbi_level_asid(tlbi_op op, u64 addr, u32 level, u16 asid)
+{
+#ifdef CONFIG_ARM64_D128
+ union __u128_halves arg;
+
+ arg.low = FIELD_PREP(TLBI_ASID_MASK, asid);
+ __tlbi_update_level(level, &arg.low);
+ arg.high = addr >> 12;
+#else
+ u64 arg = __TLBI_VADDR(addr, asid);
+ __tlbi_update_level(level, &arg);
+#endif
op(arg);
}
@@ -475,19 +511,33 @@ static __always_inline void ripas2e1is(tlbi_arg_t arg)
__tlbi_wrapper(ripas2e1is, arg);
}
-static __always_inline void __tlbi_range(tlbi_op op, u64 addr,
- u16 asid, int scale, int num,
- u32 level, bool lpa2)
+static __always_inline u64 __tlbi_range_args_encode_comm(u16 asid, int scale, int num, u32 level)
{
u64 arg = 0;
- arg |= FIELD_PREP(TLBIR_BADDR_MASK, addr >> (lpa2 ? 16 : PAGE_SHIFT));
arg |= FIELD_PREP(TLBIR_TTL_MASK, level > 3 ? 0 : level);
arg |= FIELD_PREP(TLBIR_NUM_MASK, num);
arg |= FIELD_PREP(TLBIR_SCALE_MASK, scale);
arg |= FIELD_PREP(TLBIR_TG_MASK, get_trans_granule());
arg |= FIELD_PREP(TLBIR_ASID_MASK, asid);
+ return arg;
+}
+
+static __always_inline void __tlbi_range(tlbi_op op, u64 addr,
+ u16 asid, int scale, int num,
+ u32 level, bool lpa2)
+{
+#ifdef CONFIG_ARM64_D128
+ union __u128_halves arg;
+
+ arg.low = __tlbi_range_args_encode_comm(asid, scale, num, level);
+ arg.high = addr >> 12;
+#else
+ u64 arg = __tlbi_range_args_encode_comm(asid, scale, num, level);
+
+ arg |= FIELD_PREP(TLBIR_BADDR_MASK, addr >> (lpa2 ? 16 : PAGE_SHIFT));
+#endif
op(arg);
}
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH V2 12/14] arm64/mm: Relax KASAN shadow alignment for FEAT_D128
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
` (10 preceding siblings ...)
2026-09-07 3:50 ` [PATCH V2 11/14] arm64/mm: Enable TLBIP instruction based TLB flush Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 13/14] arm64/mm: Make ARM64_D128 selectable Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 14/14] arm64/configs: Add d128.config Anshuman Khandual
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm,
Andrey Ryabinin, Alexander Potapenko, Andrey Konovalov, kasan-dev
D128 reduces the number of descriptors in each translation table, causing
some VA size and page granule combinations that use two page table levels
with D64 to now require three levels.
For three-level page tables, the PUD level is folded while the PMD level
remains a real translation level. Hence, PUD_SIZE represents a root-level
span. With D128, some configurations have a KASAN shadow region smaller
than this span, making it impossible for both KASAN shadow boundaries to
satisfy the existing PUD_SIZE alignment requirement.
The KASAN initialization code on arm64 already supports shadow boundaries
which are not root-level aligned, by installing or cloning the next-level
table as required.
Similarly for three level D128 page tables, relax SHADOW_ALIGN to PMD_SIZE,
which corresponds to the first non-folded level below the root level. Keep
the existing alignment requirements unchanged for all other configurations.
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
arch/arm64/mm/kasan_init.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c
index f61f5bc09467..e12abea6ad15 100644
--- a/arch/arm64/mm/kasan_init.c
+++ b/arch/arm64/mm/kasan_init.c
@@ -178,7 +178,9 @@ static void __init kasan_pgd_populate(unsigned long addr, unsigned long end,
} while (pgdp++, addr = next, addr != end);
}
-#if defined(CONFIG_ARM64_64K_PAGES) || CONFIG_PGTABLE_LEVELS > 4
+#if defined(CONFIG_ARM64_D128) && CONFIG_PGTABLE_LEVELS == 3
+#define SHADOW_ALIGN PMD_SIZE
+#elif defined(CONFIG_ARM64_64K_PAGES) || CONFIG_PGTABLE_LEVELS > 4
#define SHADOW_ALIGN P4D_SIZE
#else
#define SHADOW_ALIGN PUD_SIZE
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH V2 13/14] arm64/mm: Make ARM64_D128 selectable
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
` (11 preceding siblings ...)
2026-09-07 3:50 ` [PATCH V2 12/14] arm64/mm: Relax KASAN shadow alignment for FEAT_D128 Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
2026-09-07 3:50 ` [PATCH V2 14/14] arm64/configs: Add d128.config Anshuman Khandual
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm
All required changes are already in place including the TLBIP instruction
support for enabling D128 translation. Just make this new ARM64_D128 user
selectable, after ensuring necessary tools chain support required for the
new instructions being used for D128.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
Changes in V2:
- Dropped !KASAN dependency from ARM64_D128
arch/arm64/Kconfig | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 306e6c803471..483eb6d75c7a 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -276,6 +276,10 @@ config GCC_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS
def_bool CC_IS_GCC
depends on $(cc-option,-fpatchable-function-entry=2)
+config CC_SUPPORTS_LSE128
+ def_bool CC_IS_GCC
+ depends on $(cc-option, -march=armv8.1-a+lse128)
+
config 64BIT
def_bool y
@@ -2305,8 +2309,17 @@ endmenu # "ARMv8.9 architectural features"
menu "ARMv9.3 architectural features"
+config AS_HAS_ARMV9_3
+ def_bool $(cc-option,-Wa$(comma)-march=armv9.3-a)
+
config ARM64_D128
- bool
+ bool "Enable support for 128 bit page table (FEAT_D128)"
+ depends on ARCH_SUPPORTS_INT128
+ depends on CC_SUPPORTS_LSE128
+ depends on AS_HAS_ARMV9_3
+ depends on EXPERT
+ depends on !VIRTUALIZATION
+ depends on !UNMAP_KERNEL_AT_EL0
default n
help
ARMv9.3 introduces FEAT_D128, which provides a 128 bit page
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH V2 14/14] arm64/configs: Add d128.config
2026-09-07 3:50 [PATCH V2 00/14] arm64/mm: Enable 128 bit page table entries Anshuman Khandual
` (12 preceding siblings ...)
2026-09-07 3:50 ` [PATCH V2 13/14] arm64/mm: Make ARM64_D128 selectable Anshuman Khandual
@ 2026-09-07 3:50 ` Anshuman Khandual
13 siblings, 0 replies; 15+ messages in thread
From: Anshuman Khandual @ 2026-09-07 3:50 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Anshuman Khandual, Catalin Marinas, Will Deacon, Ryan Roberts,
Mark Rutland, Lorenzo Stoakes, Andrew Morton, David Hildenbrand,
Mike Rapoport, Linu Cherian, linux-kernel, linux-mm
Provide a slimline configuration intended to be booted with ARM64_D128.
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
arch/arm64/configs/d128.config | 6 ++++++
1 file changed, 6 insertions(+)
create mode 100644 arch/arm64/configs/d128.config
diff --git a/arch/arm64/configs/d128.config b/arch/arm64/configs/d128.config
new file mode 100644
index 000000000000..e0a2d7bd8f20
--- /dev/null
+++ b/arch/arm64/configs/d128.config
@@ -0,0 +1,6 @@
+# ARM64_D128-specific configuration
+
+# CONFIG_UNMAP_KERNEL_AT_EL0 is not set
+# CONFIG_VIRTUALIZATION is not set
+CONFIG_EXPERT=y
+CONFIG_ARM64_D128=y
--
2.43.0
^ permalink raw reply related [flat|nested] 15+ messages in thread