* [PATCH v2 00/15] riscv: add Svnapot PTE folding support
@ 2026-07-16 12:41 Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 01/15] riscv: introduce raw PTE helpers Yunhui Cui
` (14 more replies)
0 siblings, 15 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot can encode a naturally aligned 64KB range as a single folded
PTE block. This series adds the RISC-V MM support needed to use that
encoding for suitable user mappings while preserving the generic MM view
of PAGE_SIZE PTEs.
The series starts by adding raw PTE helpers and switching architecture
page-table users that must see the encoded page-table entry to those raw
helpers. It then introduces Svnapot read-side and update helpers for
folded PTE ranges. Later patches make the generic PTE batch and leaf-size
interfaces work with the raw encoded view where required, avoid redundant
A/D aggregation and TLB flushes, optimize mprotect() for Svnapot ranges,
and request 64KB executable folios when Svnapot is available so text
mappings can benefit from lower iTLB pressure.
Dependency:
- This series depends on the prerequisite RISC-V patch posted at:
https://lore.kernel.org/all/20260618064406.14508-1-cuiyunhui@bytedance.com/
Changes in v2:
- Rework the v1 7-patch series into 15 smaller patches: split raw PTE
helper introduction from the architecture-user conversion, and split
Svnapot contpte handling into read-side and update-side pieces.
- Add the generic MM helper changes needed by folded Svnapot mappings,
including folio-batch comparison flags, raw-entry leaf-size queries and
fast-GUP-specific lockless PTE helpers.
- Make the RISC-V pte_batch_hint() implementation honor folio batch
flags, and preserve Svnapot leaf-size semantics via __ptep_leaf_size().
- Add follow-up fixes and optimizations that were not in v1: avoid
redundant A/D aggregation, avoid Svnapot consistency checks in
ptep_get(), avoid A/D aggregation in fast-GUP, remove a redundant TLB
flush, optimize mprotect() for Svnapot mappings, and request large
executable folios when Svnapot is available.
Yunhui Cui (15):
riscv: introduce raw PTE helpers
riscv: switch arch page-table users to raw PTE helpers
riscv/mm: implement Svnapot contpte read-side helpers
riscv/mm: implement Svnapot contpte update helpers
mm: extend pte batch and leaf-size helpers
riscv: make pte_batch_hint() honor folio batch flags
riscv/mm: preserve Svnapot leaf-size semantics for page-table
consumers
riscv/mm: avoid redundant Svnapot A/D aggregation
riscv/mm: avoid Svnapot consistency checks in ptep_get()
mm/gup: add fast-GUP specific lockless PTE helpers
riscv: mm: avoid Svnapot A/D aggregation in fast-GUP
arm64: mm: avoid contpte A/D aggregation in fast-GUP
riscv/mm: remove redundant TLB flush in napotpte_convert
riscv/mm: optimize mprotect for Svnapot mappings
riscv: mm: Request large exec folios for Svnapot
arch/arm64/include/asm/pgtable.h | 25 +-
arch/riscv/include/asm/kfence.h | 4 +-
arch/riscv/include/asm/pgtable.h | 480 +++++++++++++++++-
arch/riscv/kernel/efi.c | 4 +-
arch/riscv/kernel/hibernate.c | 3 +-
arch/riscv/kvm/gstage.c | 25 +-
arch/riscv/kvm/mmu.c | 4 +-
arch/riscv/mm/Makefile | 1 +
arch/riscv/mm/contpte.c | 804 +++++++++++++++++++++++++++++++
arch/riscv/mm/fault.c | 4 +-
arch/riscv/mm/hugetlbpage.c | 135 ++++--
arch/riscv/mm/init.c | 8 +-
arch/riscv/mm/kasan_init.c | 16 +-
arch/riscv/mm/pageattr.c | 12 +-
arch/riscv/mm/pgtable.c | 52 +-
include/linux/pgtable.h | 71 ++-
kernel/events/core.c | 2 +-
mm/gup.c | 6 +-
mm/internal.h | 39 +-
mm/mincore.c | 2 +-
mm/mremap.c | 2 +-
21 files changed, 1542 insertions(+), 157 deletions(-)
create mode 100644 arch/riscv/mm/contpte.c
base-commit: b8809969e1d7a591e0f49dd464a5d04b3cf02ab1
--
2.39.5
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 01/15] riscv: introduce raw PTE helpers
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 02/15] riscv: switch arch page-table users to " Yunhui Cui
` (13 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot needs RISC-V to distinguish between two PTE views: the raw encoded
entries used by architecture code, and the logical per-page PTE view that
will later be exposed to generic MM paths.
Split the low-level RISC-V PTE operations from the public helpers so the
raw helpers can keep operating on the hardware encoding, while follow-up
changes build the Svnapot-aware public view on top.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 129 +++++++++++++++++++++++++++----
1 file changed, 114 insertions(+), 15 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index a810655ce8f9b..8ca5da9534967 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -9,6 +9,7 @@
#include <linux/mmzone.h>
#include <linux/sizes.h>
+#include <asm/cmpxchg.h>
#include <asm/pgtable-bits.h>
#ifndef CONFIG_MMU
@@ -628,11 +629,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)
* a page table are directly modified. Thus, the following hook is
* made available.
*/
-static inline void set_pte(pte_t *ptep, pte_t pteval)
+static inline void __set_pte(pte_t *ptep, pte_t pteval)
{
WRITE_ONCE(*ptep, pteval);
}
+#define __set_pte __set_pte
+
void flush_icache_pte(struct mm_struct *mm, pte_t pte);
static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval)
@@ -640,13 +643,13 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval)
if (pte_present(pteval) && pte_exec(pteval))
flush_icache_pte(mm, pteval);
- set_pte(ptep, pteval);
+ __set_pte(ptep, pteval);
}
#define PFN_PTE_SHIFT _PAGE_PFN_SHIFT
-static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
- pte_t *ptep, pte_t pteval, unsigned int nr)
+static inline void __set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pteval, unsigned int nr)
{
page_table_check_ptes_set(mm, addr, ptep, pteval, nr);
@@ -658,24 +661,85 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
pte_val(pteval) += 1 << _PAGE_PFN_SHIFT;
}
}
-#define set_ptes set_ptes
-static inline void pte_clear(struct mm_struct *mm,
- unsigned long addr, pte_t *ptep)
+#define __set_ptes __set_ptes
+
+static inline void __pte_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
{
__set_pte_at(mm, ptep, __pte(0));
}
+#define __pte_clear __pte_clear
+
+#define __ptep_get __ptep_get
+static inline pte_t __ptep_get(pte_t *ptep)
+{
+ return READ_ONCE(*ptep);
+}
+
+#define __ptep_get_lockless __ptep_get_lockless
+static inline pte_t __ptep_get_lockless(pte_t *ptep)
+{
+ return __ptep_get(ptep);
+}
+
+static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ pte_t pte, cydp_t flags)
+{
+ pte_t old_pte;
+
+ do {
+ old_pte = pte;
+
+ if (flags & CYDP_CLEAR_YOUNG)
+ pte = pte_mkold(pte);
+ if (flags & CYDP_CLEAR_DIRTY)
+ pte = pte_mkclean(pte);
+
+ pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep),
+ pte_val(old_pte),
+ pte_val(pte));
+ } while (pte_val(pte) != pte_val(old_pte));
+}
+
+static inline void __clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags)
+{
+ pte_t pte;
+
+ for (;;) {
+ pte = __ptep_get(ptep);
+
+ if (flags == (CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY))
+ __set_pte(ptep, pte_mkclean(pte_mkold(pte)));
+ else
+ __clear_young_dirty_pte(vma, addr, ptep, pte, flags);
+
+ if (--nr == 0)
+ break;
+ ptep++;
+ addr += PAGE_SIZE;
+ }
+}
+
#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS /* defined in mm/pgtable.c */
extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address,
pte_t *ptep, pte_t entry, int dirty);
+int __ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty);
#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG /* defined in mm/pgtable.c */
bool ptep_test_and_clear_young(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep);
+ unsigned long address, pte_t *ptep);
+bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep);
#define __HAVE_ARCH_PTEP_GET_AND_CLEAR
-static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
- unsigned long address, pte_t *ptep)
+static inline pte_t __ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep)
{
#ifdef CONFIG_SMP
pte_t pte = __pte(xchg(&ptep->pte, 0));
@@ -690,9 +754,24 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
return pte;
}
+#define __ptep_get_and_clear __ptep_get_and_clear
+
+static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ pte_t pte = __ptep_get_and_clear(vma->vm_mm, address, ptep);
+
+ if (pte_accessible(vma->vm_mm, pte))
+ flush_tlb_page(vma, address);
+
+ return pte;
+}
+
+#define __ptep_clear_flush __ptep_clear_flush
+
#define __HAVE_ARCH_PTEP_SET_WRPROTECT
-static inline void ptep_set_wrprotect(struct mm_struct *mm,
- unsigned long address, pte_t *ptep)
+static inline void __ptep_set_wrprotect(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep)
{
pte_t read_pte = READ_ONCE(*ptep);
/*
@@ -705,9 +784,11 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm,
((pte_val(read_pte) & ~(unsigned long)_PAGE_WRITE) | _PAGE_READ));
}
+#define __ptep_set_wrprotect __ptep_set_wrprotect
+
#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
-static inline bool ptep_clear_flush_young(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep)
+static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
{
/*
* This comment is borrowed from x86, but applies equally to RISC-V:
@@ -724,9 +805,27 @@ static inline bool ptep_clear_flush_young(struct vm_area_struct *vma,
* shouldn't really matter because there's no real memory
* pressure for swapout to react to. ]
*/
- return ptep_test_and_clear_young(vma, address, ptep);
+ return __ptep_test_and_clear_young(vma, address, ptep);
}
+#define __ptep_clear_flush_young __ptep_clear_flush_young
+
+#define ptep_get __ptep_get
+#define ptep_get_lockless __ptep_get_lockless
+#define set_pte __set_pte
+#define set_ptes __set_ptes
+
+static inline void pte_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ __pte_clear(mm, addr, ptep);
+}
+
+#define ptep_get_and_clear __ptep_get_and_clear
+#define clear_young_dirty_ptes __clear_young_dirty_ptes
+#define ptep_set_wrprotect __ptep_set_wrprotect
+#define ptep_clear_flush_young __ptep_clear_flush_young
+
#define pgprot_nx pgprot_nx
static inline pgprot_t pgprot_nx(pgprot_t _prot)
{
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 02/15] riscv: switch arch page-table users to raw PTE helpers
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 01/15] riscv: introduce raw PTE helpers Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 03/15] riscv/mm: implement Svnapot contpte read-side helpers Yunhui Cui
` (12 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Convert RISC-V internal page-table walkers and mutators over to the raw
PTE helper layer so they keep operating on the encoded hardware entries.
This keeps arch-private users independent from the Svnapot-aware public
PTE view that will be added on top for generic mm paths.
Add a raw get-and-clear helper without page-table-check side effects so
NAPOT-aware arch callers can report the effective non-NAPOT leaf view
explicitly when tearing down contiguous mappings.
Also update the hugetlb Svnapot clear/rewrite paths so they keep operating
on the contiguous block with page-table-check behavior consistent with the
raw-helper model.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/kfence.h | 4 +-
arch/riscv/include/asm/pgtable.h | 14 +++-
arch/riscv/kernel/efi.c | 4 +-
arch/riscv/kernel/hibernate.c | 3 +-
arch/riscv/kvm/gstage.c | 25 +++---
arch/riscv/kvm/mmu.c | 4 +-
arch/riscv/mm/fault.c | 4 +-
arch/riscv/mm/hugetlbpage.c | 135 ++++++++++++++++++++-----------
arch/riscv/mm/init.c | 8 +-
arch/riscv/mm/kasan_init.c | 16 ++--
arch/riscv/mm/pageattr.c | 12 +--
arch/riscv/mm/pgtable.c | 34 +++++---
12 files changed, 165 insertions(+), 98 deletions(-)
diff --git a/arch/riscv/include/asm/kfence.h b/arch/riscv/include/asm/kfence.h
index 29cb3a6ee113d..8325962a9d7e8 100644
--- a/arch/riscv/include/asm/kfence.h
+++ b/arch/riscv/include/asm/kfence.h
@@ -19,9 +19,9 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect)
pte_t *pte = virt_to_kpte(addr);
if (protect) {
- set_pte(pte, __pte(pte_val(ptep_get(pte)) & ~_PAGE_PRESENT));
+ __set_pte(pte, __pte(pte_val(__ptep_get(pte)) & ~_PAGE_PRESENT));
} else {
- set_pte(pte, __pte(pte_val(ptep_get(pte)) | _PAGE_PRESENT));
+ __set_pte(pte, __pte(pte_val(__ptep_get(pte)) | _PAGE_PRESENT));
mark_new_valid_map();
}
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 8ca5da9534967..a0f64e980e981 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -9,7 +9,6 @@
#include <linux/mmzone.h>
#include <linux/sizes.h>
-#include <asm/cmpxchg.h>
#include <asm/pgtable-bits.h>
#ifndef CONFIG_MMU
@@ -124,6 +123,7 @@
#ifndef __ASSEMBLER__
+#include <asm/cmpxchg.h>
#include <asm/page.h>
#include <asm/tlbflush.h>
#include <linux/mm_types.h>
@@ -468,6 +468,11 @@ static inline pte_t pte_mkdirty(pte_t pte)
return __pte(pte_val(pte) | _PAGE_DIRTY | _PAGE_SOFT_DIRTY);
}
+static inline pte_t riscv_pte_mkhwdirty(pte_t pte)
+{
+ return __pte(pte_val(pte) | _PAGE_DIRTY);
+}
+
static inline pte_t pte_mkclean(pte_t pte)
{
return __pte(pte_val(pte) & ~(_PAGE_DIRTY));
@@ -756,6 +761,13 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm,
#define __ptep_get_and_clear __ptep_get_and_clear
+static inline pte_t __ptep_get_and_clear_noptc(pte_t *ptep)
+{
+ return __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));
+}
+
+#define __ptep_get_and_clear_noptc __ptep_get_and_clear_noptc
+
static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep)
{
diff --git a/arch/riscv/kernel/efi.c b/arch/riscv/kernel/efi.c
index 2d3cc57b45352..81ea2546d2922 100644
--- a/arch/riscv/kernel/efi.c
+++ b/arch/riscv/kernel/efi.c
@@ -60,7 +60,7 @@ int __init efi_create_mapping(struct mm_struct *mm, efi_memory_desc_t *md)
static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data)
{
efi_memory_desc_t *md = data;
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
unsigned long val;
if (md->attribute & EFI_MEMORY_RO) {
@@ -72,7 +72,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data)
val = pte_val(pte) & ~_PAGE_EXEC;
pte = __pte(val);
}
- set_pte(ptep, pte);
+ __set_pte(ptep, pte);
return 0;
}
diff --git a/arch/riscv/kernel/hibernate.c b/arch/riscv/kernel/hibernate.c
index 982843828adb7..c7d5c7d4bbcde 100644
--- a/arch/riscv/kernel/hibernate.c
+++ b/arch/riscv/kernel/hibernate.c
@@ -186,7 +186,8 @@ static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long
pte_t pte = READ_ONCE(*src_ptep);
if (pte_present(pte))
- set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot)));
+ __set_pte(dst_ptep,
+ __pte(pte_val(pte) | pgprot_val(prot)));
} while (dst_ptep++, src_ptep++, start += PAGE_SIZE, start < end);
return 0;
diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c
index c4c3b79567f10..67b2c75f5cec1 100644
--- a/arch/riscv/kvm/gstage.c
+++ b/arch/riscv/kvm/gstage.c
@@ -88,7 +88,7 @@ bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage, gpa_t addr,
*ptep_level = current_level;
ptep = (pte_t *)gstage->pgd;
ptep = &ptep[gstage_pte_index(gstage, addr, current_level)];
- while (ptep && pte_val(ptep_get(ptep))) {
+ while (ptep && pte_val(__ptep_get(ptep))) {
if (gstage_pte_leaf(ptep)) {
*ptep_level = current_level;
*ptepp = ptep;
@@ -98,7 +98,7 @@ bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage, gpa_t addr,
if (current_level) {
current_level--;
*ptep_level = current_level;
- ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep));
ptep = &ptep[gstage_pte_index(gstage, addr, current_level)];
} else {
ptep = NULL;
@@ -152,18 +152,18 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
if (gstage_pte_leaf(ptep))
return -EEXIST;
- if (!pte_val(ptep_get(ptep))) {
+ if (!pte_val(__ptep_get(ptep))) {
if (!pcache)
return -ENOMEM;
next_ptep = kvm_mmu_memory_cache_alloc(pcache);
if (!next_ptep)
return -ENOMEM;
- set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)),
- __pgprot(_PAGE_TABLE)));
+ __set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)),
+ __pgprot(_PAGE_TABLE)));
} else {
if (gstage_pte_leaf(ptep))
return -EEXIST;
- next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ next_ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep));
}
current_level--;
@@ -171,7 +171,7 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
}
if (pte_val(*ptep) != pte_val(map->pte)) {
- set_pte(ptep, map->pte);
+ __set_pte(ptep, map->pte);
if (gstage_pte_leaf(ptep))
gstage_tlb_flush(gstage, current_level, map->addr);
}
@@ -371,18 +371,18 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
WARN_ON(addr & (page_size - 1));
- if (!pte_val(ptep_get(ptep)))
+ if (!pte_val(__ptep_get(ptep)))
return false;
if (ptep_level && !gstage_pte_leaf(ptep)) {
- next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ next_ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep));
next_ptep_level = ptep_level - 1;
ret = gstage_level_to_page_size(gstage, next_ptep_level, &next_page_size);
if (ret)
return false;
if (op == GSTAGE_OP_CLEAR)
- set_pte(ptep, __pte(0));
+ __set_pte(ptep, __pte(0));
for (i = 0; i < PTRS_PER_PTE; i++)
flush |= kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size,
&next_ptep[i], next_ptep_level, op);
@@ -391,9 +391,10 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
} else {
old_pte = *ptep;
if (op == GSTAGE_OP_CLEAR)
- set_pte(ptep, __pte(0));
+ __set_pte(ptep, __pte(0));
else if (op == GSTAGE_OP_WP)
- set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE));
+ __set_pte(ptep,
+ __pte(pte_val(__ptep_get(ptep)) & ~_PAGE_WRITE));
if (pte_val(*ptep) != pte_val(old_pte))
flush = true;
}
diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c
index 082f9b2617338..48dfc88d8cea6 100644
--- a/arch/riscv/kvm/mmu.c
+++ b/arch/riscv/kvm/mmu.c
@@ -278,7 +278,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
&ptep, &ptep_level))
return false;
- return ptep_test_and_clear_young(NULL, 0, ptep);
+ return __ptep_test_and_clear_young(NULL, 0, ptep);
}
bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
@@ -298,7 +298,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
&ptep, &ptep_level))
return false;
- return pte_young(ptep_get(ptep));
+ return pte_young(__ptep_get(ptep));
}
static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot,
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 04ed6f8acae4f..0b725ed723023 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -69,7 +69,7 @@ static void show_pte(unsigned long addr)
if (!ptep)
goto out;
- pte = ptep_get(ptep);
+ pte = __ptep_get(ptep);
pr_cont(", pte=%016lx", pte_val(pte));
pte_unmap(ptep);
out:
@@ -231,7 +231,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a
* silently loop forever.
*/
pte_k = pte_offset_kernel(pmd_k, addr);
- if (!pte_present(ptep_get(pte_k))) {
+ if (!pte_present(__ptep_get(pte_k))) {
no_context(regs, addr);
return;
}
diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c
index a6d217112cf46..5ededac02161d 100644
--- a/arch/riscv/mm/hugetlbpage.c
+++ b/arch/riscv/mm/hugetlbpage.c
@@ -7,7 +7,7 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
{
unsigned long pte_num;
int i;
- pte_t orig_pte = ptep_get(ptep);
+ pte_t orig_pte = __ptep_get(ptep);
if (!pte_present(orig_pte) || !pte_napot(orig_pte))
return orig_pte;
@@ -15,10 +15,10 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
pte_num = napot_pte_num(napot_cont_order(orig_pte));
for (i = 0; i < pte_num; i++, ptep++) {
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
if (pte_dirty(pte))
- orig_pte = pte_mkdirty(orig_pte);
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
if (pte_young(pte))
orig_pte = pte_mkyoung(orig_pte);
@@ -74,7 +74,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm,
out:
if (pte) {
- pte_t pteval = ptep_get_lockless(pte);
+ pte_t pteval = __ptep_get_lockless(pte);
WARN_ON_ONCE(pte_present(pteval) && !pte_huge(pteval));
}
@@ -145,28 +145,52 @@ unsigned long hugetlb_mask_last_page(struct hstate *h)
return 0UL;
}
+static unsigned long napot_hugetlb_block_addr(pte_t pte, unsigned long addr)
+{
+ unsigned long order;
+
+ if (!pte_napot(pte))
+ return addr;
+
+ order = napot_cont_order(pte);
+
+ return addr & napot_cont_mask(order);
+}
+
+static pte_t napot_hugetlb_get_and_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ pte_t pte;
+
+ pte = __ptep_get_and_clear_noptc(ptep);
+ page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
+
+ return pte;
+}
+
static pte_t get_clear_contig(struct mm_struct *mm,
unsigned long addr,
pte_t *ptep,
unsigned long ncontig)
{
- pte_t pte, tmp_pte;
- bool present;
-
- pte = ptep_get_and_clear(mm, addr, ptep);
- present = pte_present(pte);
- while (--ncontig) {
- ptep++;
- addr += PAGE_SIZE;
- tmp_pte = ptep_get_and_clear(mm, addr, ptep);
- if (present) {
- if (pte_dirty(tmp_pte))
- pte = pte_mkdirty(pte);
- if (pte_young(tmp_pte))
- pte = pte_mkyoung(pte);
- }
+ pte_t orig_pte = __ptep_get(ptep);
+ unsigned long i;
+
+ addr = napot_hugetlb_block_addr(orig_pte, addr);
+ if (pte_napot(orig_pte))
+ ptep = huge_pte_offset(mm, addr,
+ napot_cont_size(napot_cont_order(orig_pte)));
+
+ for (i = 0; i < ncontig; i++, addr += PAGE_SIZE, ptep++) {
+ pte_t pte = napot_hugetlb_get_and_clear(mm, addr, ptep);
+
+ if (pte_dirty(pte))
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
+ if (pte_young(pte))
+ orig_pte = pte_mkyoung(orig_pte);
}
- return pte;
+
+ return orig_pte;
}
static pte_t get_clear_contig_flush(struct mm_struct *mm,
@@ -174,10 +198,13 @@ static pte_t get_clear_contig_flush(struct mm_struct *mm,
pte_t *ptep,
unsigned long pte_num)
{
+ pte_t pte = __ptep_get(ptep);
pte_t orig_pte = get_clear_contig(mm, addr, ptep, pte_num);
struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0);
bool valid = !pte_none(orig_pte);
+ addr = napot_hugetlb_block_addr(pte, addr);
+
if (valid)
flush_tlb_range(&vma, addr, addr + (PAGE_SIZE * pte_num));
@@ -207,14 +234,31 @@ static void clear_flush(struct mm_struct *mm,
unsigned long ncontig)
{
struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0);
+ pte_t pte = __ptep_get(ptep);
unsigned long i, saddr = addr;
+ addr = napot_hugetlb_block_addr(pte, addr);
+ if (pte_napot(pte))
+ ptep = huge_pte_offset(mm, addr,
+ napot_cont_size(napot_cont_order(pte)));
+ saddr = addr;
+
for (i = 0; i < ncontig; i++, addr += pgsize, ptep++)
- ptep_get_and_clear(mm, addr, ptep);
+ napot_hugetlb_get_and_clear(mm, addr, ptep);
flush_tlb_range(&vma, saddr, addr);
}
+static void set_huge_napot_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte, unsigned long pte_num)
+{
+ unsigned long i;
+
+ page_table_check_ptes_set(mm, addr, ptep, pte, pte_num);
+ for (i = 0; i < pte_num; i++)
+ __set_pte_at(mm, ptep + i, pte);
+}
+
static int num_contig_ptes_from_size(unsigned long sz, size_t *pgsize)
{
unsigned long hugepage_shift;
@@ -256,19 +300,18 @@ void set_huge_pte_at(struct mm_struct *mm,
if (!pte_present(pte)) {
for (i = 0; i < pte_num; i++, ptep++, addr += pgsize)
- set_ptes(mm, addr, ptep, pte, 1);
+ __set_ptes(mm, addr, ptep, pte, 1);
return;
}
if (!pte_napot(pte)) {
- set_ptes(mm, addr, ptep, pte, 1);
+ __set_ptes(mm, addr, ptep, pte, 1);
return;
}
clear_flush(mm, addr, ptep, pgsize, pte_num);
- for (i = 0; i < pte_num; i++, ptep++, addr += pgsize)
- set_pte_at(mm, addr, ptep, pte);
+ set_huge_napot_ptes(mm, addr, ptep, pte, pte_num);
}
int huge_ptep_set_access_flags(struct vm_area_struct *vma,
@@ -280,10 +323,10 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,
struct mm_struct *mm = vma->vm_mm;
unsigned long order;
pte_t orig_pte;
- int i, pte_num;
+ int pte_num;
if (!pte_napot(pte))
- return ptep_set_access_flags(vma, addr, ptep, pte, dirty);
+ return __ptep_set_access_flags(vma, addr, ptep, pte, dirty);
order = napot_cont_order(pte);
pte_num = napot_pte_num(order);
@@ -291,13 +334,12 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,
orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num);
if (pte_dirty(orig_pte))
- pte = pte_mkdirty(pte);
+ pte = riscv_pte_mkhwdirty(pte);
if (pte_young(orig_pte))
pte = pte_mkyoung(pte);
- for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++)
- set_pte_at(mm, addr, ptep, pte);
+ set_huge_napot_ptes(mm, addr, ptep, pte, pte_num);
return true;
}
@@ -306,14 +348,13 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm,
unsigned long addr,
pte_t *ptep, unsigned long sz)
{
- size_t pgsize;
- pte_t orig_pte = ptep_get(ptep);
+ pte_t orig_pte = __ptep_get(ptep);
int pte_num;
if (!pte_napot(orig_pte))
- return ptep_get_and_clear(mm, addr, ptep);
+ return __ptep_get_and_clear(mm, addr, ptep);
- pte_num = num_contig_ptes_from_size(sz, &pgsize);
+ pte_num = napot_pte_num(napot_cont_order(orig_pte));
return get_clear_contig(mm, addr, ptep, pte_num);
}
@@ -322,13 +363,13 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm,
unsigned long addr,
pte_t *ptep)
{
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
unsigned long order;
pte_t orig_pte;
- int i, pte_num;
+ int pte_num;
if (!pte_napot(pte)) {
- ptep_set_wrprotect(mm, addr, ptep);
+ __ptep_set_wrprotect(mm, addr, ptep);
return;
}
@@ -339,19 +380,18 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm,
orig_pte = pte_wrprotect(orig_pte);
- for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++)
- set_pte_at(mm, addr, ptep, orig_pte);
+ set_huge_napot_ptes(mm, addr, ptep, orig_pte, pte_num);
}
pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,
unsigned long addr,
pte_t *ptep)
{
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
int pte_num;
if (!pte_napot(pte))
- return ptep_clear_flush(vma, addr, ptep);
+ return __ptep_clear_flush(vma, addr, ptep);
pte_num = napot_pte_num(napot_cont_order(pte));
@@ -363,19 +403,16 @@ void huge_pte_clear(struct mm_struct *mm,
pte_t *ptep,
unsigned long sz)
{
- size_t pgsize;
- pte_t pte = ptep_get(ptep);
- int i, pte_num;
+ pte_t pte = __ptep_get(ptep);
+ int pte_num;
if (!pte_napot(pte)) {
- pte_clear(mm, addr, ptep);
+ __pte_clear(mm, addr, ptep);
return;
}
- pte_num = num_contig_ptes_from_size(sz, &pgsize);
-
- for (i = 0; i < pte_num; i++, addr += pgsize, ptep++)
- pte_clear(mm, addr, ptep);
+ pte_num = napot_pte_num(napot_cont_order(pte));
+ get_clear_contig(mm, addr, ptep, pte_num);
}
static bool is_napot_size(unsigned long size)
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index e2be64bacf160..d44bb7db22b43 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -378,9 +378,9 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot)
ptep = &fixmap_pte[pte_index(addr)];
if (pgprot_val(prot))
- set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot));
+ __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot));
else
- pte_clear(&init_mm, addr, ptep);
+ __pte_clear(&init_mm, addr, ptep);
local_flush_tlb_page(addr);
}
@@ -1571,11 +1571,11 @@ static void __meminit remove_pte_mapping(pte_t *pte_base, unsigned long addr, un
next = end;
ptep = pte_base + pte_index(addr);
- pte = ptep_get(ptep);
+ pte = __ptep_get(ptep);
if (!pte_present(*ptep))
continue;
- pte_clear(&init_mm, addr, ptep);
+ __pte_clear(&init_mm, addr, ptep);
if (is_vmemmap)
free_vmemmap_storage(pte_page(pte), PAGE_SIZE, altmap);
}
diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c
index 1f3aa9611187f..c5d5513b206ca 100644
--- a/arch/riscv/mm/kasan_init.c
+++ b/arch/riscv/mm/kasan_init.c
@@ -39,9 +39,9 @@ static void __init kasan_populate_pte(pmd_t *pmd, unsigned long vaddr, unsigned
ptep = pte_offset_kernel(pmd, vaddr);
do {
- if (pte_none(ptep_get(ptep))) {
+ if (pte_none(__ptep_get(ptep))) {
phys_addr = memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE);
- set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL));
+ __set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL));
memset(__va(phys_addr), KASAN_SHADOW_INIT, PAGE_SIZE);
}
} while (ptep++, vaddr += PAGE_SIZE, vaddr != end);
@@ -327,8 +327,8 @@ asmlinkage void __init kasan_early_init(void)
KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT)));
for (i = 0; i < PTRS_PER_PTE; ++i)
- set_pte(kasan_early_shadow_pte + i,
- pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL));
+ __set_pte(kasan_early_shadow_pte + i,
+ pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL));
for (i = 0; i < PTRS_PER_PMD; ++i)
set_pmd(kasan_early_shadow_pmd + i,
@@ -520,10 +520,10 @@ void __init kasan_init(void)
kasan_mem_to_shadow((const void *)MODULES_VADDR + SZ_2G));
for (i = 0; i < PTRS_PER_PTE; i++)
- set_pte(&kasan_early_shadow_pte[i],
- mk_pte(virt_to_page(kasan_early_shadow_page),
- __pgprot(_PAGE_PRESENT | _PAGE_READ |
- _PAGE_ACCESSED)));
+ __set_pte(&kasan_early_shadow_pte[i],
+ mk_pte(virt_to_page(kasan_early_shadow_page),
+ __pgprot(_PAGE_PRESENT | _PAGE_READ |
+ _PAGE_ACCESSED)));
memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE);
init_task.kasan_depth = 0;
diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c
index 3f76db3d27699..e0271e2a0b295 100644
--- a/arch/riscv/mm/pageattr.c
+++ b/arch/riscv/mm/pageattr.c
@@ -68,10 +68,10 @@ static int pageattr_pmd_entry(pmd_t *pmd, unsigned long addr,
static int pageattr_pte_entry(pte_t *pte, unsigned long addr,
unsigned long next, struct mm_walk *walk)
{
- pte_t val = ptep_get(pte);
+ pte_t val = __ptep_get(pte);
val = __pte(set_pageattr_masks(pte_val(val), walk));
- set_pte(pte, val);
+ __set_pte(pte, val);
return 0;
}
@@ -121,7 +121,7 @@ static int __split_linear_mapping_pmd(pud_t *pudp,
ptep_new = (pte_t *)page_address(pte_page);
for (i = 0; i < PTRS_PER_PTE; ++i, ++ptep_new)
- set_pte(ptep_new, pfn_pte(pfn + i, prot));
+ __set_pte(ptep_new, pfn_pte(pfn + i, prot));
smp_wmb();
@@ -406,14 +406,14 @@ static int debug_pagealloc_set_page(pte_t *pte, unsigned long addr, void *data)
{
int enable = *(int *)data;
- unsigned long val = pte_val(ptep_get(pte));
+ unsigned long val = pte_val(__ptep_get(pte));
if (enable)
val |= _PAGE_PRESENT;
else
val &= ~_PAGE_PRESENT;
- set_pte(pte, __pte(val));
+ __set_pte(pte, __pte(val));
return 0;
}
@@ -466,5 +466,5 @@ bool kernel_page_present(struct page *page)
return true;
pte = pte_offset_kernel(pmd, addr);
- return pte_present(ptep_get(pte));
+ return pte_present(__ptep_get(pte));
}
diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c
index 9c4427d0b1874..43bd542c6ff02 100644
--- a/arch/riscv/mm/pgtable.c
+++ b/arch/riscv/mm/pgtable.c
@@ -5,14 +5,13 @@
#include <linux/kernel.h>
#include <linux/pgtable.h>
-int ptep_set_access_flags(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep,
- pte_t entry, int dirty)
+int __ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty)
{
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC)) {
- if (!pte_same(ptep_get(ptep), entry)) {
+ if (!pte_same(__ptep_get(ptep), entry)) {
__set_pte_at(vma->vm_mm, ptep, entry);
- /* Here only not svadu is impacted */
flush_tlb_page(vma, address);
return true;
}
@@ -20,7 +19,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
return false;
}
- if (!pte_same(ptep_get(ptep), entry))
+ if (!pte_same(__ptep_get(ptep), entry))
__set_pte_at(vma->vm_mm, ptep, entry);
/*
* update_mmu_cache will unconditionally execute, handling both
@@ -29,13 +28,30 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
return true;
}
-bool ptep_test_and_clear_young(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep)
+int ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty)
{
- if (!pte_young(ptep_get(ptep)))
+ return __ptep_set_access_flags(vma, address, ptep, entry, dirty);
+}
+
+bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address,
+ pte_t *ptep)
+{
+ if (!pte_young(__ptep_get(ptep)))
return false;
+
return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep));
}
+EXPORT_SYMBOL_GPL(__ptep_test_and_clear_young);
+
+bool ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address,
+ pte_t *ptep)
+{
+ return __ptep_test_and_clear_young(vma, address, ptep);
+}
EXPORT_SYMBOL_GPL(ptep_test_and_clear_young);
#ifdef CONFIG_64BIT
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 03/15] riscv/mm: implement Svnapot contpte read-side helpers
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 01/15] riscv: introduce raw PTE helpers Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 02/15] riscv: switch arch page-table users to " Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 04/15] riscv/mm: implement Svnapot contpte update helpers Yunhui Cui
` (11 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot keeps a contiguous mapping encoded in a block of page-table
entries, while generic MM code expects the public PTE getters to return
a normal per-page view for the address being queried. Without a dedicated
read path, exposing the raw NAPOT encoding through the public helpers
would leak arch-private representation details into generic MM users and
make folded mappings harder to integrate with existing PTE walkers.
Add Svnapot-aware contpte read-side helpers and route the public PTE
getters through them, so generic MM sees the logical sub-PTE for the
current page while RISC-V raw helpers continue to operate on the encoded
entries.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 50 +++++-
arch/riscv/mm/Makefile | 1 +
arch/riscv/mm/contpte.c | 273 +++++++++++++++++++++++++++++++
3 files changed, 322 insertions(+), 2 deletions(-)
create mode 100644 arch/riscv/mm/contpte.c
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index a0f64e980e981..fce6d04ea65db 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -312,6 +312,11 @@ static inline unsigned long pte_napot(pte_t pte)
return 0;
}
+static inline pte_t pte_mknapot(pte_t pte, unsigned int order)
+{
+ return pte;
+}
+
#endif /* CONFIG_RISCV_ISA_SVNAPOT */
/* Yields the page frame number (PFN) of a page table entry */
@@ -350,6 +355,11 @@ static inline int pte_present(pte_t pte)
return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE));
}
+static inline bool pte_present_napot(pte_t pte)
+{
+ return pte_present(pte) && pte_napot(pte);
+}
+
#define pte_accessible pte_accessible
static inline unsigned long pte_accessible(struct mm_struct *mm, pte_t a)
{
@@ -538,6 +548,13 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
#define pte_leaf_size(pte) (pte_napot(pte) ? \
napot_cont_size(napot_cont_order(pte)) :\
PAGE_SIZE)
+
+void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte);
+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte);
+pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte);
+pte_t napotpte_ptep_get_lockless(pte_t *ptep);
#endif
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
@@ -689,6 +706,37 @@ static inline pte_t __ptep_get_lockless(pte_t *ptep)
return __ptep_get(ptep);
}
+#ifdef CONFIG_RISCV_ISA_SVNAPOT
+
+#define ptep_get ptep_get
+static inline pte_t ptep_get(pte_t *ptep)
+{
+ pte_t pte = __ptep_get(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get(ptep, pte);
+}
+
+#define ptep_get_lockless ptep_get_lockless
+static inline pte_t ptep_get_lockless(pte_t *ptep)
+{
+ pte_t pte = __ptep_get_lockless(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get_lockless(ptep);
+}
+
+#else
+
+#define ptep_get __ptep_get
+#define ptep_get_lockless __ptep_get_lockless
+
+#endif
+
static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep,
pte_t pte, cydp_t flags)
@@ -822,8 +870,6 @@ static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
#define __ptep_clear_flush_young __ptep_clear_flush_young
-#define ptep_get __ptep_get
-#define ptep_get_lockless __ptep_get_lockless
#define set_pte __set_pte
#define set_ptes __set_ptes
diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile
index b916a68d324ad..fd260903cd340 100644
--- a/arch/riscv/mm/Makefile
+++ b/arch/riscv/mm/Makefile
@@ -19,6 +19,7 @@ obj-y += context.o
obj-y += pmem.o
obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o
+obj-$(CONFIG_RISCV_ISA_SVNAPOT) += contpte.o
obj-$(CONFIG_PTDUMP) += ptdump.o
obj-$(CONFIG_KASAN) += kasan_init.o
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
new file mode 100644
index 0000000000000..43ea76e424492
--- /dev/null
+++ b/arch/riscv/mm/contpte.c
@@ -0,0 +1,273 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/align.h>
+#include <linux/cpufeature.h>
+#include <linux/efi.h>
+#include <linux/export.h>
+#include <linux/mm.h>
+#include <linux/mm_types.h>
+#include <linux/page_table_check.h>
+#include <linux/pgtable.h>
+
+#include <asm/tlbflush.h>
+
+static inline bool napot_hw_supported(void)
+{
+ return riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT);
+}
+
+static inline bool mm_is_user(struct mm_struct *mm)
+{
+ if (unlikely(mm_is_efi(mm)))
+ return false;
+
+ return mm != &init_mm;
+}
+
+static inline unsigned int napotpte_order(void)
+{
+ return NAPOT_CONT64KB_ORDER;
+}
+
+static inline unsigned long napotpte_size(void)
+{
+ return napot_cont_size(napotpte_order());
+}
+
+static inline unsigned int napotpte_pte_num(void)
+{
+ return napot_pte_num(napotpte_order());
+}
+
+static inline unsigned long napot_align_addr(unsigned long addr)
+{
+ return ALIGN_DOWN(addr, napotpte_size());
+}
+
+static inline pte_t *napot_align_ptep(pte_t *ptep)
+{
+ return PTR_ALIGN_DOWN(ptep, napotpte_pte_num() * sizeof(*ptep));
+}
+
+static inline pte_t pte_mask_ad(pte_t pte)
+{
+ return pte_mkold(pte_mkclean(pte));
+}
+
+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte);
+
+static inline unsigned long pte_protval_no_pfn_no_napot(pte_t pte)
+{
+ return (pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT;
+}
+
+static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
+{
+ unsigned long pfn;
+ pgprot_t prot;
+
+ if (!pte_present_napot(pte))
+ return pte;
+
+ pfn = pte_pfn(pte) + (ptep - napot_align_ptep(ptep));
+ prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
+
+ return pfn_pte(pfn, prot);
+}
+
+static inline pte_t __napot_ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ pte_t pte;
+
+ pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));
+ page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
+
+ return pte;
+}
+
+static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t target)
+{
+ unsigned long start_addr, end, ptent_addr;
+ pte_t *start_ptep;
+ pte_t ptent, pte;
+ unsigned int i, nr;
+
+ start_addr = napot_align_addr(addr);
+ start_ptep = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+ end = start_addr + napotpte_size();
+
+ for (i = 0; i < nr; i++) {
+ ptent_addr = start_addr + i * PAGE_SIZE;
+ ptent = __napot_ptep_get_and_clear(mm, ptent_addr,
+ start_ptep + i);
+ if (pte_dirty(ptent))
+ target = riscv_pte_mkhwdirty(target);
+ if (pte_young(ptent))
+ target = pte_mkyoung(target);
+ }
+
+ flush_tlb_mm_range(mm, start_addr, end, PAGE_SIZE);
+
+ page_table_check_ptes_set(mm, start_addr, start_ptep, target, nr);
+ if (pte_napot(target)) {
+ for (i = 0; i < nr; i++)
+ __set_pte_at(mm, start_ptep + i, target);
+ return;
+ }
+
+ for (i = 0; i < nr; i++) {
+ pte = pfn_pte(pte_pfn(target) + i,
+ __pgprot(pte_protval_no_pfn_no_napot(target)));
+ if (pte_dirty(target))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(target))
+ pte = pte_mkyoung(pte);
+ __set_pte_at(mm, start_ptep + i, pte);
+ }
+}
+
+static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte)
+{
+ return pte_present_napot(pte) &&
+ pte_val(pte_mask_ad(pte)) == pte_val(pte_mask_ad(orig_pte));
+}
+
+void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte)
+{
+ struct page *page;
+ struct folio *folio;
+ unsigned long folio_start, folio_end;
+ unsigned long cont_start, cont_end;
+ unsigned long pfn;
+ pgprot_t prot;
+ pte_t expected, cur;
+ pte_t *start;
+ unsigned int i, nr;
+
+ if (!napot_hw_supported() || !mm_is_user(mm))
+ return;
+
+ if (!pte_present(pte) || pte_napot(pte) || pte_special(pte))
+ return;
+
+ /*
+ * Driver __GFP_COMP pages inserted by vm_insert_page() have valid
+ * compound metadata. Fold them only after verifying the whole supported
+ * Svnapot range is identical and contiguous.
+ */
+ page = pte_page(pte);
+ folio = page_folio(page);
+ folio_start = addr - (page - &folio->page) * PAGE_SIZE;
+ folio_end = folio_start + folio_nr_pages(folio) * PAGE_SIZE;
+ cont_start = napot_align_addr(addr);
+ cont_end = cont_start + napotpte_size();
+ if (folio_start > cont_start || folio_end < cont_end)
+ return;
+
+ nr = napotpte_pte_num();
+ start = napot_align_ptep(ptep);
+
+ pfn = ALIGN_DOWN(pte_pfn(pte), nr);
+ prot = pte_pgprot(pte_mask_ad(pte));
+ expected = pfn_pte(pfn, prot);
+
+ /*
+ * The caller must hold the PTL across validation and conversion.
+ * Software PTE updates are serialized by the PTL; hardware A/D updates
+ * may race and are re-read and folded into target by napotpte_convert().
+ */
+ for (i = 0; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (pte_val(pte_mask_ad(cur)) != pte_val(expected))
+ return;
+ pte_val(expected) += 1UL << _PAGE_PFN_SHIFT;
+ }
+
+ expected = pte_mknapot(pfn_pte(pfn, prot), napotpte_order());
+ napotpte_convert(mm, addr, ptep, expected);
+}
+EXPORT_SYMBOL(__napotpte_try_fold);
+
+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte)
+{
+ pte_t target;
+ pgprot_t prot;
+
+ if (!napot_hw_supported() || !mm_is_user(mm) ||
+ !pte_present_napot(pte))
+ return;
+
+ prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
+ target = pfn_pte(pte_pfn(pte), prot);
+
+ napotpte_convert(mm, addr, ptep, target);
+}
+EXPORT_SYMBOL(__napotpte_try_unfold);
+
+pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
+{
+ pte_t pte, cur;
+ pte_t *start;
+ unsigned int i, nr;
+
+ if (!napot_hw_supported() || !pte_present_napot(orig_pte))
+ return orig_pte;
+
+ pte = orig_pte;
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+
+ for (i = 0; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (!napotpte_is_consistent(cur, orig_pte))
+ return napotpte_subpte(ptep, orig_pte);
+ if (pte_dirty(cur))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(cur))
+ pte = pte_mkyoung(pte);
+ }
+
+ return napotpte_subpte(ptep, pte);
+}
+EXPORT_SYMBOL(napotpte_ptep_get);
+
+pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
+{
+ pte_t orig_pte, pte;
+ pte_t *ptep;
+ unsigned int i, nr;
+
+ if (!napot_hw_supported())
+ return READ_ONCE(*orig_ptep);
+
+ nr = napotpte_pte_num();
+
+retry:
+ orig_pte = READ_ONCE(*orig_ptep);
+ if (!pte_present_napot(orig_pte))
+ return orig_pte;
+
+ ptep = napot_align_ptep(orig_ptep);
+
+ for (i = 0; i < nr; i++, ptep++) {
+ pte = READ_ONCE(*ptep);
+
+ if (!napotpte_is_consistent(pte, orig_pte))
+ goto retry;
+
+ if (pte_dirty(pte))
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
+
+ if (pte_young(pte))
+ orig_pte = pte_mkyoung(orig_pte);
+ }
+
+ return napotpte_subpte(orig_ptep, orig_pte);
+}
+EXPORT_SYMBOL(napotpte_ptep_get_lockless);
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 04/15] riscv/mm: implement Svnapot contpte update helpers
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
` (2 preceding siblings ...)
2026-07-16 12:41 ` [PATCH v2 03/15] riscv/mm: implement Svnapot contpte read-side helpers Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 05/15] mm: extend pte batch and leaf-size helpers Yunhui Cui
` (10 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot support needs the public RISC-V PTE update helpers to preserve
the same logical per-page view that generic MM users see on the read
side. Without a dedicated update path, operations such as set_ptes(),
ptep_set_access_flags(), and related clear or write-protect flows would
end up touching the raw NAPOT encoding directly, which leaks arch-private
representation details into generic MM code and makes folded mappings
harder to update safely.
Add Svnapot-aware contpte update helpers and route the public update-side
PTE operations through them, while keeping the raw helpers available for
RISC-V code that still needs direct access to the encoded entries.
This also wires the update path into the related mechanics needed when a
Svnapot block is unfolded and rewritten, so page-table-check, access-flag
updates, and non-Svnapot builds continue to observe the expected public
PTE semantics.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 281 ++++++++++++++++++++++----
arch/riscv/mm/contpte.c | 329 ++++++++++++++++++++++++++++++-
arch/riscv/mm/pgtable.c | 18 ++
3 files changed, 588 insertions(+), 40 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index fce6d04ea65db..771faabb2b00b 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -413,6 +413,30 @@ static inline int pte_special(pte_t pte)
return pte_val(pte) & _PAGE_SPECIAL;
}
+#ifdef CONFIG_RISCV_ISA_SVNAPOT
+static inline pte_t pte_mknonnapot(pte_t pte, unsigned long addr)
+{
+ unsigned long pfn;
+ unsigned long offset;
+ pgprot_t prot;
+
+ if (!pte_present_napot(pte))
+ return pte;
+
+ offset = (addr & (napot_cont_size(napot_cont_order(pte)) - 1)) >>
+ PAGE_SHIFT;
+ pfn = pte_pfn(pte) + offset;
+ prot = __pgprot((pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT);
+
+ return pfn_pte(pfn, prot);
+}
+#else
+static inline pte_t pte_mknonnapot(pte_t pte, unsigned long addr)
+{
+ return pte;
+}
+#endif
+
/* static inline pte_t pte_rdprotect(pte_t pte) */
static inline pte_t pte_wrprotect(pte_t pte)
@@ -555,6 +579,25 @@ void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte);
pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte);
pte_t napotpte_ptep_get_lockless(pte_t *ptep);
+void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte, unsigned int nr);
+void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full);
+pte_t napotpte_get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full);
+void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags);
+void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr);
+int napotpte_ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty);
+int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep);
+int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep);
#endif
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
@@ -706,37 +749,6 @@ static inline pte_t __ptep_get_lockless(pte_t *ptep)
return __ptep_get(ptep);
}
-#ifdef CONFIG_RISCV_ISA_SVNAPOT
-
-#define ptep_get ptep_get
-static inline pte_t ptep_get(pte_t *ptep)
-{
- pte_t pte = __ptep_get(ptep);
-
- if (likely(!pte_present_napot(pte)))
- return pte;
-
- return napotpte_ptep_get(ptep, pte);
-}
-
-#define ptep_get_lockless ptep_get_lockless
-static inline pte_t ptep_get_lockless(pte_t *ptep)
-{
- pte_t pte = __ptep_get_lockless(ptep);
-
- if (likely(!pte_present_napot(pte)))
- return pte;
-
- return napotpte_ptep_get_lockless(ptep);
-}
-
-#else
-
-#define ptep_get __ptep_get
-#define ptep_get_lockless __ptep_get_lockless
-
-#endif
-
static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep,
pte_t pte, cydp_t flags)
@@ -829,7 +841,6 @@ static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,
#define __ptep_clear_flush __ptep_clear_flush
-#define __HAVE_ARCH_PTEP_SET_WRPROTECT
static inline void __ptep_set_wrprotect(struct mm_struct *mm,
unsigned long address, pte_t *ptep)
{
@@ -870,7 +881,169 @@ static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
#define __ptep_clear_flush_young __ptep_clear_flush_young
-#define set_pte __set_pte
+#ifdef CONFIG_RISCV_ISA_SVNAPOT
+
+static __always_inline void napotpte_try_fold(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ pte_t pte)
+{
+ const unsigned long contmask = napot_pte_num(NAPOT_CONT64KB_ORDER) - 1;
+ bool valign = ((addr >> PAGE_SHIFT) & contmask) == contmask;
+
+ if (unlikely(valign)) {
+ bool palign = (pte_pfn(pte) & contmask) == contmask;
+
+ if (unlikely(palign && pte_present(pte) && !pte_napot(pte) &&
+ !pte_special(pte)))
+ __napotpte_try_fold(mm, addr, ptep, pte);
+ }
+}
+
+static __always_inline void napotpte_try_unfold(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ pte_t pte)
+{
+ if (unlikely(pte_present_napot(pte)))
+ __napotpte_try_unfold(mm, addr, ptep, pte);
+}
+
+/*
+ * Public PTE helpers may transparently handle Svnapot-encoded mappings.
+ * NAPOT-aware arch users should stick to the private/raw __pte* helpers.
+ */
+#define ptep_get ptep_get
+static inline pte_t ptep_get(pte_t *ptep)
+{
+ pte_t pte = __ptep_get(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get(ptep, pte);
+}
+
+#define ptep_get_lockless ptep_get_lockless
+static inline pte_t ptep_get_lockless(pte_t *ptep)
+{
+ pte_t pte = __ptep_get_lockless(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get_lockless(ptep);
+}
+
+#define set_ptes set_ptes
+static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pteval, unsigned int nr)
+{
+ pteval = pte_mknonnapot(pteval, addr);
+
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ __set_ptes(mm, addr, ptep, pteval, 1);
+ napotpte_try_fold(mm, addr, ptep, pteval);
+ return;
+ }
+
+ napotpte_set_ptes(mm, addr, ptep, pteval, nr);
+}
+
+static inline void pte_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ __pte_clear(mm, addr, ptep);
+}
+
+static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ return __ptep_get_and_clear(mm, addr, ptep);
+}
+
+#define clear_young_dirty_ptes clear_young_dirty_ptes
+static inline void clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags)
+{
+ if (likely(nr == 1 && !pte_present_napot(__ptep_get(ptep))))
+ __clear_young_dirty_ptes(vma, addr, ptep, nr, flags);
+ else
+ napotpte_clear_young_dirty_ptes(vma, addr, ptep, nr, flags);
+}
+
+#define clear_full_ptes clear_full_ptes
+static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full)
+{
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ __ptep_get_and_clear(mm, addr, ptep);
+ return;
+ }
+
+ napotpte_clear_full_ptes(mm, addr, ptep, nr, full);
+}
+
+#define get_and_clear_full_ptes get_and_clear_full_ptes
+static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ return __ptep_get_and_clear(mm, addr, ptep);
+ }
+
+ return napotpte_get_and_clear_full_ptes(mm, addr, ptep, nr, full);
+}
+
+#define wrprotect_ptes wrprotect_ptes
+static inline void wrprotect_ptes(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep,
+ unsigned int nr)
+{
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, address, ptep, __ptep_get(ptep));
+ __ptep_set_wrprotect(mm, address, ptep);
+ return;
+ }
+
+ napotpte_wrprotect_ptes(mm, address, ptep, nr);
+}
+
+#define __HAVE_ARCH_PTEP_SET_WRPROTECT
+static inline void ptep_set_wrprotect(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep)
+{
+ wrprotect_ptes(mm, address, ptep, 1);
+}
+
+#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
+static inline int ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ pte_t orig_pte = __ptep_get(ptep);
+
+ if (likely(!pte_present_napot(orig_pte)))
+ return __ptep_clear_flush_young(vma, address, ptep);
+
+ return napotpte_ptep_clear_flush_young(vma, address, ptep);
+}
+
+#else /* CONFIG_RISCV_ISA_SVNAPOT */
+
+#define napotpte_ptep_set_access_flags(vma, address, ptep, entry, dirty) \
+ ({ (void)(vma); (void)(address); (void)(ptep); (void)(entry); \
+ (void)(dirty); 0; })
+
+#define napotpte_ptep_test_and_clear_young(vma, address, ptep) \
+ ({ (void)(vma); (void)(address); (void)(ptep); 0; })
+
+#define ptep_get __ptep_get
+#define ptep_get_lockless __ptep_get_lockless
#define set_ptes __set_ptes
static inline void pte_clear(struct mm_struct *mm,
@@ -881,9 +1054,51 @@ static inline void pte_clear(struct mm_struct *mm,
#define ptep_get_and_clear __ptep_get_and_clear
#define clear_young_dirty_ptes __clear_young_dirty_ptes
+
+static inline void __clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ for (;;) {
+ __ptep_get_and_clear(mm, addr, ptep);
+ if (--nr == 0)
+ break;
+ ptep++;
+ addr += PAGE_SIZE;
+ }
+}
+
+static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ pte_t pte, tmp_pte;
+
+ pte = __ptep_get_and_clear(mm, addr, ptep);
+ while (--nr) {
+ ptep++;
+ addr += PAGE_SIZE;
+ tmp_pte = __ptep_get_and_clear(mm, addr, ptep);
+ if (pte_dirty(tmp_pte))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(tmp_pte))
+ pte = pte_mkyoung(pte);
+ }
+
+ return pte;
+}
+
+#define clear_full_ptes __clear_full_ptes
+#define get_and_clear_full_ptes __get_and_clear_full_ptes
+#define __HAVE_ARCH_PTEP_SET_WRPROTECT
#define ptep_set_wrprotect __ptep_set_wrprotect
+#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
#define ptep_clear_flush_young __ptep_clear_flush_young
+#endif /* CONFIG_RISCV_ISA_SVNAPOT */
+
+#define set_pte __set_pte
+
#define pgprot_nx pgprot_nx
static inline pgprot_t pgprot_nx(pgprot_t _prot)
{
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 43ea76e424492..3f8a0a2970afb 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -62,6 +62,24 @@ static inline unsigned long pte_protval_no_pfn_no_napot(pte_t pte)
return (pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT;
}
+static inline void napotpte_clear_young_dirty_pte(pte_t *ptep, cydp_t flags)
+{
+ pte_t old_pte, new_pte;
+ unsigned long old_val, new_val;
+
+ do {
+ old_pte = READ_ONCE(*ptep);
+ new_pte = old_pte;
+ if (flags & CYDP_CLEAR_YOUNG)
+ new_pte = pte_mkold(new_pte);
+ if (flags & CYDP_CLEAR_DIRTY)
+ new_pte = pte_mkclean(new_pte);
+
+ old_val = pte_val(old_pte);
+ new_val = pte_val(new_pte);
+ } while (cmpxchg_relaxed(&pte_val(*ptep), old_val, new_val) != old_val);
+}
+
static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
{
unsigned long pfn;
@@ -76,13 +94,34 @@ static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
return pfn_pte(pfn, prot);
}
-static inline pte_t __napot_ptep_get_and_clear(struct mm_struct *mm,
- unsigned long addr, pte_t *ptep)
+static void __clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full)
{
- pte_t pte;
+ for (;;) {
+ __ptep_get_and_clear(mm, addr, ptep);
+ if (--nr == 0)
+ break;
+ ptep++;
+ addr += PAGE_SIZE;
+ }
+}
- pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));
- page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
+static pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ pte_t pte, tmp_pte;
+
+ pte = __ptep_get_and_clear(mm, addr, ptep);
+ while (--nr) {
+ ptep++;
+ addr += PAGE_SIZE;
+ tmp_pte = __ptep_get_and_clear(mm, addr, ptep);
+ if (pte_dirty(tmp_pte))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(tmp_pte))
+ pte = pte_mkyoung(pte);
+ }
return pte;
}
@@ -102,8 +141,9 @@ static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
for (i = 0; i < nr; i++) {
ptent_addr = start_addr + i * PAGE_SIZE;
- ptent = __napot_ptep_get_and_clear(mm, ptent_addr,
- start_ptep + i);
+ ptent = __ptep_get_and_clear_noptc(start_ptep + i);
+ page_table_check_pte_clear(mm, ptent_addr,
+ pte_mknonnapot(ptent, ptent_addr));
if (pte_dirty(ptent))
target = riscv_pte_mkhwdirty(target);
if (pte_young(ptent))
@@ -136,6 +176,22 @@ static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte)
pte_val(pte_mask_ad(pte)) == pte_val(pte_mask_ad(orig_pte));
}
+static bool napotpte_all_subptes_same(pte_t *ptep, pte_t expected_pte)
+{
+ pte_t *start;
+ unsigned int i, nr;
+
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+
+ for (i = 0; i < nr; i++) {
+ if (!pte_same(READ_ONCE(start[i]), expected_pte))
+ return false;
+ }
+
+ return true;
+}
+
void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte)
{
@@ -271,3 +327,262 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
return napotpte_subpte(orig_ptep, orig_pte);
}
EXPORT_SYMBOL(napotpte_ptep_get_lockless);
+
+static void napotpte_try_unfold_range(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr)
+{
+ unsigned long next;
+ pte_t pte;
+ unsigned int chunk;
+
+ while (nr) {
+ pte = READ_ONCE(*ptep);
+ if (pte_present_napot(pte)) {
+ __napotpte_try_unfold(mm, addr, ptep, pte);
+ next = napot_align_addr(addr) + napotpte_size();
+ chunk = (next - addr) >> PAGE_SHIFT;
+ } else {
+ chunk = 1;
+ }
+
+ if (chunk > nr)
+ chunk = nr;
+
+ ptep += chunk;
+ addr += chunk * PAGE_SIZE;
+ nr -= chunk;
+ }
+}
+
+static void napotpte_try_unfold_partial(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr)
+{
+ pte_t pte;
+
+ if (ptep != napot_align_ptep(ptep) || nr < napotpte_pte_num()) {
+ pte = READ_ONCE(*ptep);
+ if (pte_present_napot(pte))
+ __napotpte_try_unfold(mm, addr, ptep, pte);
+ }
+
+ if (ptep + nr != napot_align_ptep(ptep + nr)) {
+ unsigned long last_addr;
+ pte_t *last_ptep;
+
+ last_addr = addr + PAGE_SIZE * (nr - 1);
+ last_ptep = ptep + nr - 1;
+ pte = READ_ONCE(*last_ptep);
+ if (pte_present_napot(pte))
+ __napotpte_try_unfold(mm, last_addr, last_ptep, pte);
+ }
+}
+
+void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte, unsigned int nr)
+{
+ unsigned long next, end;
+ unsigned long pfn, size, boundary;
+ pgprot_t prot;
+ unsigned int chunk, i;
+ pte_t cur;
+
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ __set_ptes(mm, addr, ptep, pte, nr);
+ return;
+ }
+
+ size = napotpte_size();
+ end = addr + ((unsigned long)nr << PAGE_SHIFT);
+ pfn = pte_pfn(pte);
+ prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
+
+ do {
+ boundary = (addr + size) & ~(size - 1);
+ next = (boundary - 1 < end - 1) ? boundary : end;
+ chunk = (next - addr) >> PAGE_SHIFT;
+
+ cur = pfn_pte(pfn, prot);
+ if (((addr | next | (pfn << PAGE_SHIFT)) & (size - 1)) == 0) {
+ cur = pte_mknapot(cur, napotpte_order());
+ page_table_check_ptes_set(mm, addr, ptep, cur, chunk);
+ for (i = 0; i < chunk; i++)
+ __set_pte_at(mm, ptep + i, cur);
+ } else {
+ __set_ptes(mm, addr, ptep, cur, chunk);
+ }
+
+ addr = next;
+ ptep += chunk;
+ pfn += chunk;
+ } while (addr != end);
+}
+EXPORT_SYMBOL(napotpte_set_ptes);
+
+void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full)
+{
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ __clear_full_ptes(mm, addr, ptep, nr, full);
+ return;
+ }
+
+ /*
+ * Svnapot stores identical napot-encoded entries across the whole block
+ * rather than per-page PFNs, so batch zap paths must unfold the covered
+ * range before the generic MM consumes ordinary per-page PTEs.
+ */
+ napotpte_try_unfold_range(mm, addr, ptep, nr);
+ __clear_full_ptes(mm, addr, ptep, nr, full);
+}
+EXPORT_SYMBOL(napotpte_clear_full_ptes);
+
+pte_t napotpte_get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ if (!napot_hw_supported() || !mm_is_user(mm))
+ return __get_and_clear_full_ptes(mm, addr, ptep, nr, full);
+
+ napotpte_try_unfold_range(mm, addr, ptep, nr);
+
+ return __get_and_clear_full_ptes(mm, addr, ptep, nr, full);
+}
+EXPORT_SYMBOL(napotpte_get_and_clear_full_ptes);
+
+void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags)
+{
+ struct mm_struct *mm;
+ unsigned long start, end;
+ unsigned int total;
+
+ mm = vma->vm_mm;
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ __clear_young_dirty_ptes(vma, addr, ptep, nr, flags);
+ return;
+ }
+
+ start = addr;
+ end = start + nr * PAGE_SIZE;
+
+ if (pte_present_napot(READ_ONCE(*(ptep + nr - 1))))
+ end = ALIGN(end, napotpte_size());
+
+ if (pte_present_napot(READ_ONCE(*ptep))) {
+ start = napot_align_addr(start);
+ ptep = napot_align_ptep(ptep);
+ }
+
+ total = (end - start) >> PAGE_SHIFT;
+ for (; total; total--, ptep++, start += PAGE_SIZE)
+ napotpte_clear_young_dirty_pte(ptep, flags);
+}
+EXPORT_SYMBOL(napotpte_clear_young_dirty_ptes);
+
+void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr)
+{
+ unsigned int i;
+
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ for (i = 0; i < nr; i++, ptep++, addr += PAGE_SIZE)
+ __ptep_set_wrprotect(mm, addr, ptep);
+ return;
+ }
+
+ napotpte_try_unfold_partial(mm, addr, ptep, nr);
+
+ for (i = 0; i < nr; i++, ptep++, addr += PAGE_SIZE)
+ __ptep_set_wrprotect(mm, addr, ptep);
+}
+EXPORT_SYMBOL(napotpte_wrprotect_ptes);
+
+int napotpte_ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty)
+{
+ pte_t raw_pte, napot_pte;
+ pte_t *start;
+ pgprot_t prot;
+ unsigned long start_addr;
+ unsigned int i, nr;
+ bool changed;
+
+ raw_pte = READ_ONCE(*ptep);
+ if (!napot_hw_supported() || !pte_present_napot(raw_pte))
+ return 0;
+
+ prot = pte_pgprot(entry);
+ napot_pte = pfn_pte(pte_pfn(raw_pte), prot);
+ napot_pte = pte_mknapot(napot_pte, napotpte_order());
+
+ if (napotpte_all_subptes_same(ptep, napot_pte))
+ return !riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC);
+
+ if (pte_write(raw_pte) != pte_write(napot_pte)) {
+ __napotpte_try_unfold(vma->vm_mm, address, ptep, raw_pte);
+ entry = pte_mknonnapot(entry, address);
+
+ return __ptep_set_access_flags(vma, address, ptep, entry,
+ dirty);
+ }
+
+ start = napot_align_ptep(ptep);
+ address = napot_align_addr(address);
+ start_addr = address;
+ nr = napotpte_pte_num();
+ changed = false;
+
+ for (i = 0; i < nr; i++, start++, address += PAGE_SIZE) {
+ if (__ptep_set_access_flags(vma, address, start, napot_pte, 0))
+ changed = true;
+ }
+
+ if (changed)
+ flush_tlb_range(vma, start_addr, start_addr + napotpte_size());
+
+ return changed;
+}
+EXPORT_SYMBOL(napotpte_ptep_set_access_flags);
+
+int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ pte_t *start;
+ unsigned int i, nr;
+ int young;
+
+ if (!napot_hw_supported() || !pte_present_napot(READ_ONCE(*ptep)))
+ return 0;
+
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+ young = 0;
+
+ for (i = 0; i < nr; i++)
+ young |= test_and_clear_bit(_PAGE_ACCESSED_OFFSET,
+ &pte_val(start[i]));
+
+ return young;
+}
+EXPORT_SYMBOL(napotpte_ptep_test_and_clear_young);
+
+int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ unsigned long start_addr;
+ int young;
+
+ young = napotpte_ptep_test_and_clear_young(vma, address, ptep);
+ if (!young)
+ return 0;
+
+ start_addr = napot_align_addr(address);
+ flush_tlb_range(vma, start_addr, start_addr + napotpte_size());
+
+ return young;
+}
+EXPORT_SYMBOL(napotpte_ptep_clear_flush_young);
diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c
index 43bd542c6ff02..50e4c4cc10691 100644
--- a/arch/riscv/mm/pgtable.c
+++ b/arch/riscv/mm/pgtable.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0
#include <asm/pgalloc.h>
+#include <linux/cpufeature.h>
#include <linux/gfp.h>
#include <linux/kernel.h>
#include <linux/pgtable.h>
@@ -32,6 +33,16 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep,
pte_t entry, int dirty)
{
+ pte_t raw_pte;
+
+ entry = pte_mknonnapot(entry, address);
+
+ raw_pte = READ_ONCE(*ptep);
+ if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT) &&
+ pte_present_napot(raw_pte))
+ return napotpte_ptep_set_access_flags(vma, address, ptep, entry,
+ dirty);
+
return __ptep_set_access_flags(vma, address, ptep, entry, dirty);
}
@@ -50,6 +61,13 @@ bool ptep_test_and_clear_young(struct vm_area_struct *vma,
unsigned long address,
pte_t *ptep)
{
+ pte_t raw_pte;
+
+ raw_pte = READ_ONCE(*ptep);
+ if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT) &&
+ pte_present_napot(raw_pte))
+ return napotpte_ptep_test_and_clear_young(vma, address, ptep);
+
return __ptep_test_and_clear_young(vma, address, ptep);
}
EXPORT_SYMBOL_GPL(ptep_test_and_clear_young);
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 05/15] mm: extend pte batch and leaf-size helpers
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
` (3 preceding siblings ...)
2026-07-16 12:41 ` [PATCH v2 04/15] riscv/mm: implement Svnapot contpte update helpers Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 06/15] riscv: make pte_batch_hint() honor folio batch flags Yunhui Cui
` (9 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Folded mappings such as RISC-V Svnapot need two extra pieces of generic
MM support.
First, architecture pte_batch_hint() implementations need access to the
same batch flags and normalization rules used by folio_pte_batch(), so
folded ranges can participate in batching with semantics consistent with
generic folio comparisons.
Second, leaf-size users may need the original PTE pointer instead of the
public ptep_get() view. For folded mappings, the public helper can return
a logical per-page sub-PTE, while the effective leaf size still depends
on the raw encoded entry in the page table.
Move the batch flag helpers into pgtable.h, extend pte_batch_hint() to
take the batch flags, add __ptep_leaf_size(), and update the generic
callers to use the new interfaces.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/arm64/include/asm/pgtable.h | 9 +++++-
include/linux/pgtable.h | 53 +++++++++++++++++++++++++++++++-
kernel/events/core.c | 2 +-
mm/internal.h | 39 ++---------------------
mm/mincore.c | 2 +-
mm/mremap.c | 2 +-
6 files changed, 65 insertions(+), 42 deletions(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index a2681d7553584..018781a1fcbee 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -5,6 +5,13 @@
#ifndef __ASM_PGTABLE_H
#define __ASM_PGTABLE_H
+#ifndef __ASSEMBLY__
+#ifndef __LINUX_FPB_T
+#define __LINUX_FPB_T
+typedef int __bitwise fpb_t;
+#endif
+#endif
+
#include <asm/bug.h>
#include <asm/proc-fns.h>
@@ -1705,7 +1712,7 @@ static __always_inline void contpte_try_unfold(struct mm_struct *mm,
}
#define pte_batch_hint pte_batch_hint
-static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte)
+static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
{
if (!pte_valid_cont(pte))
return 1;
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 8c093c119e5a8..da14328093a86 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -3,6 +3,35 @@
#define _LINUX_PGTABLE_H
#include <linux/pfn.h>
+
+#ifndef __ASSEMBLY__
+#ifndef __LINUX_FPB_T
+#define __LINUX_FPB_T
+typedef int __bitwise fpb_t;
+#endif
+
+/* Compare PTEs respecting the dirty bit. */
+#define FPB_RESPECT_DIRTY ((__force fpb_t)BIT(0))
+
+/* Compare PTEs respecting the soft-dirty bit. */
+#define FPB_RESPECT_SOFT_DIRTY ((__force fpb_t)BIT(1))
+
+/* Compare PTEs respecting the writable bit. */
+#define FPB_RESPECT_WRITE ((__force fpb_t)BIT(2))
+
+/*
+ * Merge PTE write bits: if any PTE in the batch is writable, modify the
+ * PTE at @ptentp to be writable.
+ */
+#define FPB_MERGE_WRITE ((__force fpb_t)BIT(3))
+
+/*
+ * Merge PTE young and dirty bits: if any PTE in the batch is young or dirty,
+ * modify the PTE at @ptentp to be young or dirty, respectively.
+ */
+#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(4))
+#endif /* !__ASSEMBLY__ */
+
#include <asm/pgtable.h>
#define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT)
@@ -397,6 +426,7 @@ static inline void lazy_mmu_mode_resume(void) {}
* pte_batch_hint - Number of pages that can be added to batch without scanning.
* @ptep: Page table pointer for the entry.
* @pte: Page table entry.
+ * @flags: Flags that control PTE normalization for batch comparisons.
*
* Some architectures know that a set of contiguous ptes all map the same
* contiguous memory with the same permissions. In this case, it can provide a
@@ -407,7 +437,7 @@ static inline void lazy_mmu_mode_resume(void) {}
*
* May be overridden by the architecture, else pte_batch_hint is always 1.
*/
-static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte)
+static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
{
return 1;
}
@@ -1855,6 +1885,7 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
return pmd;
}
#endif
+
#else /* !CONFIG_HAVE_ARCH_SOFT_DIRTY */
static inline int pte_soft_dirty(pte_t pte)
{
@@ -1917,6 +1948,17 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
}
#endif
+static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
+{
+ if (!(flags & FPB_RESPECT_DIRTY))
+ pte = pte_mkclean(pte);
+ if (likely(!(flags & FPB_RESPECT_SOFT_DIRTY)))
+ pte = pte_clear_soft_dirty(pte);
+ if (likely(!(flags & FPB_RESPECT_WRITE)))
+ pte = pte_wrprotect(pte);
+ return pte_mkold(pte);
+}
+
#ifndef __HAVE_PFNMAP_TRACKING
/*
* Interfaces that can be used by architecture code to keep track of
@@ -2414,6 +2456,15 @@ static inline const char *pgtable_level_to_str(enum pgtable_level level)
#define __pte_leaf_size(x,y) pte_leaf_size(y)
#endif
+#ifndef __ptep_leaf_size
+#ifndef __ASSEMBLY__
+static inline unsigned long __ptep_leaf_size(pmd_t pmd, pte_t *ptep, pte_t pte)
+{
+ return __pte_leaf_size(pmd, pte);
+}
+#endif /* !__ASSEMBLY__ */
+#endif
+
/*
* We always define pmd_pfn for all archs as it's used in lots of generic
* code. Now it happens too for pud_pfn (and can happen for larger
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 51c1200ea3fdd..37b860abe9e61 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8551,7 +8551,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr)
pte = ptep_get_lockless(ptep);
if (pte_present(pte))
- size = __pte_leaf_size(pmd, pte);
+ size = __ptep_leaf_size(pmd, ptep, pte);
pte_unmap(ptep);
#endif /* CONFIG_HAVE_GUP_FAST */
diff --git a/mm/internal.h b/mm/internal.h
index 874be94cf2570..aff00a3fdff50 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -343,41 +343,6 @@ static inline int anon_vma_prepare(struct vm_area_struct *vma)
return __anon_vma_prepare(vma);
}
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs respecting the dirty bit. */
-#define FPB_RESPECT_DIRTY ((__force fpb_t)BIT(0))
-
-/* Compare PTEs respecting the soft-dirty bit. */
-#define FPB_RESPECT_SOFT_DIRTY ((__force fpb_t)BIT(1))
-
-/* Compare PTEs respecting the writable bit. */
-#define FPB_RESPECT_WRITE ((__force fpb_t)BIT(2))
-
-/*
- * Merge PTE write bits: if any PTE in the batch is writable, modify the
- * PTE at @ptentp to be writable.
- */
-#define FPB_MERGE_WRITE ((__force fpb_t)BIT(3))
-
-/*
- * Merge PTE young and dirty bits: if any PTE in the batch is young or dirty,
- * modify the PTE at @ptentp to be young or dirty, respectively.
- */
-#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(4))
-
-static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
-{
- if (!(flags & FPB_RESPECT_DIRTY))
- pte = pte_mkclean(pte);
- if (likely(!(flags & FPB_RESPECT_SOFT_DIRTY)))
- pte = pte_clear_soft_dirty(pte);
- if (likely(!(flags & FPB_RESPECT_WRITE)))
- pte = pte_wrprotect(pte);
- return pte_mkold(pte);
-}
-
/**
* folio_pte_batch_flags - detect a PTE batch for a large folio
* @folio: The large folio to detect a PTE batch for.
@@ -430,7 +395,7 @@ static inline unsigned int folio_pte_batch_flags(struct folio *folio,
max_nr = min_t(unsigned long, max_nr,
folio_pfn(folio) + folio_nr_pages(folio) - pte_pfn(pte));
- nr = pte_batch_hint(ptep, pte);
+ nr = pte_batch_hint(ptep, pte, flags);
expected_pte = __pte_batch_clear_ignored(pte_advance_pfn(pte, nr), flags);
ptep = ptep + nr;
@@ -447,7 +412,7 @@ static inline unsigned int folio_pte_batch_flags(struct folio *folio,
any_dirty |= pte_dirty(pte);
}
- cur_nr = pte_batch_hint(ptep, pte);
+ cur_nr = pte_batch_hint(ptep, pte, flags);
expected_pte = pte_advance_pfn(expected_pte, cur_nr);
ptep += cur_nr;
nr += cur_nr;
diff --git a/mm/mincore.c b/mm/mincore.c
index 53b9828037713..e23ef3f270db6 100644
--- a/mm/mincore.c
+++ b/mm/mincore.c
@@ -191,7 +191,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
__mincore_unmapped_range(addr, addr + PAGE_SIZE,
vma, vec);
else if (pte_present(pte)) {
- unsigned int batch = pte_batch_hint(ptep, pte);
+ unsigned int batch = pte_batch_hint(ptep, pte, 0);
if (batch > 1) {
unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
diff --git a/mm/mremap.c b/mm/mremap.c
index 384ef4cc2195b..f8b585e59771d 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -184,7 +184,7 @@ static int mremap_folio_pte_batch(struct vm_area_struct *vma, unsigned long addr
return 1;
/* Avoid expensive folio lookup if we stand no chance of benefit. */
- if (pte_batch_hint(ptep, pte) == 1)
+ if (pte_batch_hint(ptep, pte, 0) == 1)
return 1;
folio = vm_normal_folio(vma, addr, pte);
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 06/15] riscv: make pte_batch_hint() honor folio batch flags
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
` (4 preceding siblings ...)
2026-07-16 12:41 ` [PATCH v2 05/15] mm: extend pte batch and leaf-size helpers Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 07/15] riscv/mm: preserve Svnapot leaf-size semantics for page-table consumers Yunhui Cui
` (8 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot folded mappings may be exposed through a logical per-page PTE
view instead of the raw encoded block entry. Generic folio batching code
already defines which PTE bits are ignored when comparing entries, and
folded mappings need to use the same comparison rules. Otherwise RISC-V
can reject a batch that generic MM would consider equivalent simply
because the hint path compared a different view of the same mapping.
Install a Svnapot-aware pte_batch_hint() implementation. If the caller
passes a logical sub-PTE instead of a raw NAPOT entry, rebuild the
corresponding block PTE first, apply the same folio batch flags as the
generic batching code, and only return a multi-entry hint if the
remaining raw PTEs in the folded block still match.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 26 +++++++++++++
arch/riscv/mm/contpte.c | 66 ++++++++++++++++++++++++++++++++
2 files changed, 92 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 771faabb2b00b..da7ffea0dcd98 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -6,6 +6,13 @@
#ifndef _ASM_RISCV_PGTABLE_H
#define _ASM_RISCV_PGTABLE_H
+#ifndef __ASSEMBLY__
+#ifndef __LINUX_FPB_T
+#define __LINUX_FPB_T
+typedef int __bitwise fpb_t;
+#endif
+#endif
+
#include <linux/mmzone.h>
#include <linux/sizes.h>
@@ -591,6 +598,8 @@ void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,
unsigned int nr, cydp_t flags);
void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, unsigned int nr);
+unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte,
+ fpb_t flags);
int napotpte_ptep_set_access_flags(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep,
pte_t entry, int dirty);
@@ -1033,6 +1042,16 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma,
return napotpte_ptep_clear_flush_young(vma, address, ptep);
}
+#define pte_batch_hint pte_batch_hint
+static inline unsigned int
+pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
+{
+ if (!pte_present(pte))
+ return 1;
+
+ return napotpte_pte_batch_hint_from_pte(ptep, pte, flags);
+}
+
#else /* CONFIG_RISCV_ISA_SVNAPOT */
#define napotpte_ptep_set_access_flags(vma, address, ptep, entry, dirty) \
@@ -1095,6 +1114,13 @@ static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
#define ptep_clear_flush_young __ptep_clear_flush_young
+#define pte_batch_hint pte_batch_hint
+static inline unsigned int
+pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
+{
+ return 1;
+}
+
#endif /* CONFIG_RISCV_ISA_SVNAPOT */
#define set_pte __set_pte
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 3f8a0a2970afb..9ffdffdc2b0c0 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -176,6 +176,32 @@ static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte)
pte_val(pte_mask_ad(pte)) == pte_val(pte_mask_ad(orig_pte));
}
+static inline bool
+napotpte_is_batch_consistent(pte_t pte, pte_t batch_pte, fpb_t flags)
+{
+ return pte_present_napot(pte) &&
+ pte_val(__pte_batch_clear_ignored(pte, flags)) ==
+ pte_val(batch_pte);
+}
+
+static inline pte_t
+napotpte_normalize_batch_pte(pte_t *ptep, pte_t orig_pte, fpb_t flags)
+{
+ unsigned long pfn;
+ pgprot_t prot;
+ unsigned int off;
+
+ if (pte_present_napot(orig_pte))
+ return __pte_batch_clear_ignored(orig_pte, flags);
+
+ off = ptep - napot_align_ptep(ptep);
+ pfn = pte_pfn(orig_pte) - off;
+ prot = __pgprot(pte_protval_no_pfn_no_napot(orig_pte));
+
+ return __pte_batch_clear_ignored(pte_mknapot(pfn_pte(pfn, prot),
+ napotpte_order()), flags);
+}
+
static bool napotpte_all_subptes_same(pte_t *ptep, pte_t expected_pte)
{
pte_t *start;
@@ -328,6 +354,46 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
}
EXPORT_SYMBOL(napotpte_ptep_get_lockless);
+unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte,
+ fpb_t flags)
+{
+ pte_t batch_pte, pte;
+ pte_t *start;
+ unsigned int i, nr, off;
+
+ if (!napot_hw_supported())
+ return 1;
+
+ if (!pte_present_napot(orig_pte) && !pte_present_napot(READ_ONCE(*ptep)))
+ return 1;
+
+ /*
+ * @orig_pte may be either the raw NAPOT entry read from the page
+ * table or the logical sub-PTE returned by ptep_get(). In the latter
+ * case, the public view has the NAPOT bit stripped and the PFN adjusted
+ * by the slot offset within the folded block.
+ *
+ * If the caller passes a logical sub-PTE, rebuild the corresponding
+ * block PTE first. Then apply the same folio batch flags as the generic
+ * batching code, and only return a multi-entry hint if every remaining
+ * raw PTE in the folded block still matches.
+ */
+ batch_pte = napotpte_normalize_batch_pte(ptep, orig_pte, flags);
+
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+ off = ptep - start;
+
+ for (i = off; i < nr; i++) {
+ pte = READ_ONCE(start[i]);
+ if (!napotpte_is_batch_consistent(pte, batch_pte, flags))
+ return 1;
+ }
+
+ return nr - off;
+}
+EXPORT_SYMBOL(napotpte_pte_batch_hint_from_pte);
+
static void napotpte_try_unfold_range(struct mm_struct *mm,
unsigned long addr, pte_t *ptep,
unsigned int nr)
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 07/15] riscv/mm: preserve Svnapot leaf-size semantics for page-table consumers
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
` (5 preceding siblings ...)
2026-07-16 12:41 ` [PATCH v2 06/15] riscv: make pte_batch_hint() honor folio batch flags Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 08/15] riscv/mm: avoid redundant Svnapot A/D aggregation Yunhui Cui
` (7 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot keeps a folded 64KB mapping encoded in the page table, while the
public PTE helpers may expose a logical per-page sub-PTE view to generic
MM users. That logical view is suitable for leaf-style PTE consumers, but
it must not change the effective leaf size reported for the mapping.
Use the original page-table entry for leaf-size queries, so page-table
consumers continue to observe the Svnapot leaf size instead of incorrectly
treating a folded mapping as PAGE_SIZE entries.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index da7ffea0dcd98..e4b9c05014c00 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -580,6 +580,17 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
napot_cont_size(napot_cont_order(pte)) :\
PAGE_SIZE)
+#define __ptep_leaf_size __ptep_leaf_size
+static inline unsigned long __ptep_leaf_size(pmd_t pmd, pte_t *ptep, pte_t pte)
+{
+ pte_t raw_pte = READ_ONCE(*ptep);
+
+ if (pte_present_napot(raw_pte))
+ return pte_leaf_size(raw_pte);
+
+ return PAGE_SIZE;
+}
+
void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte);
void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 08/15] riscv/mm: avoid redundant Svnapot A/D aggregation
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
` (6 preceding siblings ...)
2026-07-16 12:41 ` [PATCH v2 07/15] riscv/mm: preserve Svnapot leaf-size semantics for page-table consumers Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 09/15] riscv/mm: avoid Svnapot consistency checks in ptep_get() Yunhui Cui
` (6 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot mappings encode a 64KB range as a NAPOT PTE block, while the
public getter returns a logical base-page PTE. To preserve that PTE view,
the getter folds accessed and dirty information from the whole NAPOT block
into the returned PTE.
This aggregation is an OR-style operation. Once the returned PTE has both
accessed and dirty state set, later entries in the block cannot change the
result. Avoid reading the rest of the block in that case.
The lockless getter still validates each entry before consuming its A/D
state, so it preserves the self-consistent range requirement while also
avoiding reads that cannot affect the returned PTE.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/mm/contpte.c | 60 ++++++++++++++++++++++++++++++++++++++---
1 file changed, 56 insertions(+), 4 deletions(-)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 9ffdffdc2b0c0..70f9a0668b50c 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -309,10 +309,34 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
cur = READ_ONCE(start[i]);
if (!napotpte_is_consistent(cur, orig_pte))
return napotpte_subpte(ptep, orig_pte);
- if (pte_dirty(cur))
+ if (pte_dirty(cur)) {
pte = riscv_pte_mkhwdirty(pte);
- if (pte_young(cur))
+ for (; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (!napotpte_is_consistent(cur, orig_pte))
+ return napotpte_subpte(ptep, orig_pte);
+ if (pte_young(cur)) {
+ pte = pte_mkyoung(pte);
+ break;
+ }
+ }
+ break;
+ }
+
+ if (pte_young(cur)) {
pte = pte_mkyoung(pte);
+ i++;
+ for (; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (!napotpte_is_consistent(cur, orig_pte))
+ return napotpte_subpte(ptep, orig_pte);
+ if (pte_dirty(cur)) {
+ pte = riscv_pte_mkhwdirty(pte);
+ break;
+ }
+ }
+ break;
+ }
}
return napotpte_subpte(ptep, pte);
@@ -343,11 +367,39 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
if (!napotpte_is_consistent(pte, orig_pte))
goto retry;
- if (pte_dirty(pte))
+ if (pte_dirty(pte)) {
orig_pte = riscv_pte_mkhwdirty(orig_pte);
+ for (; i < nr; i++, ptep++) {
+ pte = READ_ONCE(*ptep);
+
+ if (!napotpte_is_consistent(pte, orig_pte))
+ goto retry;
- if (pte_young(pte))
+ if (pte_young(pte)) {
+ orig_pte = pte_mkyoung(orig_pte);
+ break;
+ }
+ }
+ break;
+ }
+
+ if (pte_young(pte)) {
orig_pte = pte_mkyoung(orig_pte);
+ i++;
+ ptep++;
+ for (; i < nr; i++, ptep++) {
+ pte = READ_ONCE(*ptep);
+
+ if (!napotpte_is_consistent(pte, orig_pte))
+ goto retry;
+
+ if (pte_dirty(pte)) {
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
+ break;
+ }
+ }
+ break;
+ }
}
return napotpte_subpte(orig_ptep, orig_pte);
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 09/15] riscv/mm: avoid Svnapot consistency checks in ptep_get()
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
` (7 preceding siblings ...)
2026-07-16 12:41 ` [PATCH v2 08/15] riscv/mm: avoid redundant Svnapot A/D aggregation Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 10/15] mm/gup: add fast-GUP specific lockless PTE helpers Yunhui Cui
` (5 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
napotpte_ptep_get() builds the returned logical sub-PTE from the
initial PTE observed at the target address. Neighbouring entries in the
NAPOT block are only scanned to gather accessed and dirty state, so they
cannot contribute PFN, permission, present or writable state to the
returned PTE.
Drop the range consistency checks from this ptep_get() path. Callers
that rely on the block-wide A/D view are expected to hold the PTL. There
are generic lockless ptep_get() users, such as userfaultfd_must_wait()
and HMM walks after pte_offset_map(), but those paths use the present,
marker, write and PFN state and do not consume the gathered A/D bits.
Keep the consistency checks in napotpte_ptep_get_lockless(). That API
must return a self-consistent PTE snapshot to lockless walkers and fault
retry paths. napotpte_is_consistent() verifies that every scanned entry
still belongs to the same NAPOT mapping, ignoring A/D bits, and retries
if a concurrent unfold, rewrite or refold tears the range while A/D
state is being gathered.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 3 +++
arch/riscv/mm/contpte.c | 19 +++++++++++++------
2 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index e4b9c05014c00..dd56c20a817bb 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -930,6 +930,9 @@ static __always_inline void napotpte_try_unfold(struct mm_struct *mm,
/*
* Public PTE helpers may transparently handle Svnapot-encoded mappings.
* NAPOT-aware arch users should stick to the private/raw __pte* helpers.
+ * Callers relying on the block-wide A/D view from ptep_get() are
+ * expected to hold the PTL. Lockless callers that require a
+ * self-consistent Svnapot range must use ptep_get_lockless().
*/
#define ptep_get ptep_get
static inline pte_t ptep_get(pte_t *ptep)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 70f9a0668b50c..0f66426b0ecfd 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -294,6 +294,13 @@ EXPORT_SYMBOL(__napotpte_try_unfold);
pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
{
+ /*
+ * Gather access/dirty bits from the whole NAPOT range for the
+ * ptep_get() view. The returned sub-PTE is built from orig_pte;
+ * neighbouring entries only contribute A/D state. Lockless callers
+ * requiring a self-consistent range must use ptep_get_lockless().
+ */
+
pte_t pte, cur;
pte_t *start;
unsigned int i, nr;
@@ -307,14 +314,10 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
for (i = 0; i < nr; i++) {
cur = READ_ONCE(start[i]);
- if (!napotpte_is_consistent(cur, orig_pte))
- return napotpte_subpte(ptep, orig_pte);
if (pte_dirty(cur)) {
pte = riscv_pte_mkhwdirty(pte);
for (; i < nr; i++) {
cur = READ_ONCE(start[i]);
- if (!napotpte_is_consistent(cur, orig_pte))
- return napotpte_subpte(ptep, orig_pte);
if (pte_young(cur)) {
pte = pte_mkyoung(pte);
break;
@@ -328,8 +331,6 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
i++;
for (; i < nr; i++) {
cur = READ_ONCE(start[i]);
- if (!napotpte_is_consistent(cur, orig_pte))
- return napotpte_subpte(ptep, orig_pte);
if (pte_dirty(cur)) {
pte = riscv_pte_mkhwdirty(pte);
break;
@@ -345,6 +346,12 @@ EXPORT_SYMBOL(napotpte_ptep_get);
pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
{
+ /*
+ * ptep_get_lockless() must return a self-consistent PTE without the
+ * PTL. Recheck that the whole NAPOT range still describes the same
+ * mapping, ignoring A/D bits, and retry if a concurrent update tears
+ * the range while A/D state is being gathered.
+ */
pte_t orig_pte, pte;
pte_t *ptep;
unsigned int i, nr;
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 10/15] mm/gup: add fast-GUP specific lockless PTE helpers
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
` (8 preceding siblings ...)
2026-07-16 12:41 ` [PATCH v2 09/15] riscv/mm: avoid Svnapot consistency checks in ptep_get() Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 11/15] riscv: mm: avoid Svnapot A/D aggregation in fast-GUP Yunhui Cui
` (4 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
fast-GUP snapshots a PTE without holding the PTL, pins the page, and
then revalidates that the PTE did not change. Keep both reads under the
same lockless PTE semantics by defaulting the fast-GUP revalidation to
ptep_get_lockless().
Introduce fast-GUP specific PTE snapshot and revalidation helpers. The
default implementation preserves the existing ptep_get_lockless()
semantics, while allowing architectures to override the helpers when the
public lockless getter provides extra semantics that fast-GUP does not
consume.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
include/linux/pgtable.h | 18 ++++++++++++++++++
mm/gup.c | 6 ++++--
2 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index da14328093a86..62943fcbf7046 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -813,6 +813,24 @@ static inline pte_t ptep_get_lockless(pte_t *ptep)
}
#endif
+#ifndef gup_ptep_get_lockless
+static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp)
+{
+ pte_t pte = ptep_get_lockless(ptep);
+
+ *rawp = pte;
+
+ return pte;
+}
+#endif
+
+#ifndef gup_ptep_revalidate
+static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte)
+{
+ return pte_val(raw_pte) == pte_val(ptep_get_lockless(ptep));
+}
+#endif
+
#ifndef pmdp_get_lockless
static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
{
diff --git a/mm/gup.c b/mm/gup.c
index 99902c15703b0..72fb147193e55 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2842,10 +2842,12 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,
if (!ptep)
return 0;
do {
- pte_t pte = ptep_get_lockless(ptep);
+ pte_t raw_pte, pte;
struct page *page;
struct folio *folio;
+ pte = gup_ptep_get_lockless(ptep, &raw_pte);
+
/*
* Always fallback to ordinary GUP on PROT_NONE-mapped pages:
* pte_access_permitted() better should reject these pages
@@ -2871,7 +2873,7 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,
goto pte_unmap;
if (unlikely(pmd_val(pmd) != pmd_val(pmdp_get_lockless(pmdp))) ||
- unlikely(pte_val(pte) != pte_val(ptep_get_lockless(ptep)))) {
+ unlikely(!gup_ptep_revalidate(ptep, raw_pte))) {
gup_put_folio(folio, 1, flags);
goto pte_unmap;
}
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 11/15] riscv: mm: avoid Svnapot A/D aggregation in fast-GUP
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
` (9 preceding siblings ...)
2026-07-16 12:41 ` [PATCH v2 10/15] mm/gup: add fast-GUP specific lockless PTE helpers Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 12/15] arm64: mm: avoid contpte " Yunhui Cui
` (3 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
fast-GUP only needs the PTE state for the current virtual address and
revalidates the raw PTE after pinning the page. It does not consume the
block-wide accessed/dirty aggregation provided by the public Svnapot
lockless getter.
Override the fast-GUP PTE helpers to translate a NAPOT PTE into the
current sub-PTE without scanning the whole NAPOT block. Keep the public
ptep_get_lockless() semantics unchanged for other lockless users.
With 64K THP enabled in always mode, fio 4K random direct read
throughput improves by about 10% over the previous implementation on the
same system, from 2.615M to 2.874M IOPS and from 9.97GiB/s to 11.0GiB/s,
while average clat drops from 95.54 usec to 86.97 usec.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 20 ++++++++++++++++++++
arch/riscv/mm/contpte.c | 9 +++++++++
2 files changed, 29 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index dd56c20a817bb..3727c148fa0fc 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -597,6 +597,7 @@ void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte);
pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte);
pte_t napotpte_ptep_get_lockless(pte_t *ptep);
+pte_t napotpte_ptep_get_gup_fast(pte_t *ptep, pte_t orig_pte);
void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte, unsigned int nr);
void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr,
@@ -956,6 +957,25 @@ static inline pte_t ptep_get_lockless(pte_t *ptep)
return napotpte_ptep_get_lockless(ptep);
}
+#define gup_ptep_get_lockless gup_ptep_get_lockless
+static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp)
+{
+ pte_t pte = __ptep_get_lockless(ptep);
+
+ *rawp = pte;
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get_gup_fast(ptep, pte);
+}
+
+#define gup_ptep_revalidate gup_ptep_revalidate
+static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte)
+{
+ return pte_val(raw_pte) == pte_val(__ptep_get_lockless(ptep));
+}
+
#define set_ptes set_ptes
static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pteval, unsigned int nr)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 0f66426b0ecfd..b0b8ff0aade3a 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -413,6 +413,15 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
}
EXPORT_SYMBOL(napotpte_ptep_get_lockless);
+pte_t napotpte_ptep_get_gup_fast(pte_t *orig_ptep, pte_t orig_pte)
+{
+ if (!napot_hw_supported() || !pte_present_napot(orig_pte))
+ return orig_pte;
+
+ return napotpte_subpte(orig_ptep, orig_pte);
+}
+EXPORT_SYMBOL(napotpte_ptep_get_gup_fast);
+
unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte,
fpb_t flags)
{
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 12/15] arm64: mm: avoid contpte A/D aggregation in fast-GUP
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
` (10 preceding siblings ...)
2026-07-16 12:41 ` [PATCH v2 11/15] riscv: mm: avoid Svnapot A/D aggregation in fast-GUP Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 13/15] riscv/mm: remove redundant TLB flush in napotpte_convert Yunhui Cui
` (2 subsequent siblings)
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
fast-GUP only needs the PTE state for the current virtual address and
revalidates the raw PTE after pinning the page. It does not consume the
contpte block-wide accessed/dirty aggregation provided by the public
lockless getter.
Override the fast-GUP PTE helpers to use the raw PTE for the current
entry and raw revalidation. Keep the public ptep_get_lockless() semantics
unchanged for other lockless users.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/arm64/include/asm/pgtable.h | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 018781a1fcbee..d1e08dce9f7c5 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1760,6 +1760,22 @@ static inline pte_t ptep_get_lockless(pte_t *ptep)
return contpte_ptep_get_lockless(ptep);
}
+#define gup_ptep_get_lockless gup_ptep_get_lockless
+static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp)
+{
+ pte_t pte = __ptep_get(ptep);
+
+ *rawp = pte;
+
+ return pte;
+}
+
+#define gup_ptep_revalidate gup_ptep_revalidate
+static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte)
+{
+ return pte_val(raw_pte) == pte_val(__ptep_get(ptep));
+}
+
static inline void set_pte(pte_t *ptep, pte_t pte)
{
/*
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 13/15] riscv/mm: remove redundant TLB flush in napotpte_convert
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
` (11 preceding siblings ...)
2026-07-16 12:41 ` [PATCH v2 12/15] arm64: mm: avoid contpte " Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 14/15] riscv/mm: optimize mprotect for Svnapot mappings Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 15/15] riscv: mm: Request large exec folios for Svnapot Yunhui Cui
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
napotpte_convert() switches a NAPOT block between the Svnapot encoding
and the equivalent set of base PTEs. The conversion itself preserves the
effective translations: PFNs and permissions are kept the same while the
page-table representation changes.
Do not issue an SFENCE.VMA in the middle of this representation change.
For partial updates, such as unfolding a writable NAPOT block before
write-protecting one subrange, a flush at conversion time would not be the
operation that enforces the final permission downgrade. Writable base-PTE
translations could still be refilled after the conversion and before the
subrange is write-protected. The required TLB maintenance belongs to the
operation that actually changes the effective mapping.
The RISC-V privileged architecture discussion clarifies that Svnapot is
an encoding of a naturally aligned range. When napotpte_convert() only
switches between the NAPOT encoding and the corresponding set of base PTEs
for the same effective translations, the conversion itself does not
require an extra TLB invalidation beyond the usual page-table update
rules.
Link: https://lists.riscv.org/g/tech-privileged/message/2950
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/mm/contpte.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index b0b8ff0aade3a..8289f39d6e188 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -129,7 +129,7 @@ static pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t target)
{
- unsigned long start_addr, end, ptent_addr;
+ unsigned long start_addr, ptent_addr;
pte_t *start_ptep;
pte_t ptent, pte;
unsigned int i, nr;
@@ -137,7 +137,6 @@ static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
start_addr = napot_align_addr(addr);
start_ptep = napot_align_ptep(ptep);
nr = napotpte_pte_num();
- end = start_addr + napotpte_size();
for (i = 0; i < nr; i++) {
ptent_addr = start_addr + i * PAGE_SIZE;
@@ -150,8 +149,15 @@ static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
target = pte_mkyoung(target);
}
- flush_tlb_mm_range(mm, start_addr, end, PAGE_SIZE);
-
+ /*
+ * This conversion only changes the representation of the same
+ * effective translations: NAPOT encoding <-> equivalent base PTEs.
+ * A flush here would not replace the TLB maintenance required by a
+ * later permission downgrade, because writable base-PTE translations
+ * could be refilled after this conversion and before that downgrade.
+ * Leave TLB invalidation to the operation that changes the effective
+ * mapping.
+ */
page_table_check_ptes_set(mm, start_addr, start_ptep, target, nr);
if (pte_napot(target)) {
for (i = 0; i < nr; i++)
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 14/15] riscv/mm: optimize mprotect for Svnapot mappings
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
` (12 preceding siblings ...)
2026-07-16 12:41 ` [PATCH v2 13/15] riscv/mm: remove redundant TLB flush in napotpte_convert Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 15/15] riscv: mm: Request large exec folios for Svnapot Yunhui Cui
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot represents a naturally aligned range as a block of base PTEs, but
generic mprotect() still processes a full block one base PTE at a time.
For a full-block permission change, that repeatedly clears and reinstalls
the same logical mapping and then has to reconstruct the folded Svnapot
block afterwards, making the operation unnecessarily expensive.
Use the batched protection transaction hooks to handle a full Svnapot
block as one operation. The fast path is restricted to user mappings
where hardware Svnapot is supported, the range is exactly one aligned
Svnapot block, and every sub-PTE is consistent with the block PTE. The
generic per-PTE fallback remains in place for partial, unaligned, special,
non-user, or otherwise inconsistent mappings.
libMicro mprotect median latency improves on the full-block cases:
mprot_tw4m (-l 4m -w -t -f /dev/zero): 40.4 us -> 4.3 us
mprot_tw128k (-l 128k -w -t -f /dev/zero): about 15-16% faster
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 8 ++++
arch/riscv/mm/contpte.c | 76 ++++++++++++++++++++++++++++++++
2 files changed, 84 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 3727c148fa0fc..b546e6614c558 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -619,6 +619,14 @@ int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep);
int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep);
+#define modify_prot_start_ptes modify_prot_start_ptes
+pte_t modify_prot_start_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr);
+#define modify_prot_commit_ptes modify_prot_commit_ptes
+void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pte_t *ptep, pte_t old_pte, pte_t pte,
+ unsigned int nr);
#endif
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 8289f39d6e188..6a1156629d115 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -190,6 +190,82 @@ napotpte_is_batch_consistent(pte_t pte, pte_t batch_pte, fpb_t flags)
pte_val(batch_pte);
}
+static bool napotpte_can_modify_prot_block(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, pte_t *raw_pte)
+{
+ pte_t raw, pte;
+ unsigned int i;
+
+ if (!napot_hw_supported() || !mm_is_user(mm))
+ return false;
+
+ if (nr != napotpte_pte_num())
+ return false;
+
+ if (addr != napot_align_addr(addr) || ptep != napot_align_ptep(ptep))
+ return false;
+
+ raw = READ_ONCE(*ptep);
+ if (!pte_present_napot(raw) || pte_special(raw))
+ return false;
+
+ for (i = 0; i < nr; i++) {
+ pte = READ_ONCE(ptep[i]);
+ if (!napotpte_is_consistent(pte, raw))
+ return false;
+ }
+
+ *raw_pte = raw;
+ return true;
+}
+
+pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pte_t *ptep, unsigned int nr)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ unsigned long ptent_addr;
+ pte_t raw, pte, ptent;
+ unsigned int i;
+
+ if (napotpte_can_modify_prot_block(mm, addr, ptep, nr, &raw)) {
+ pte = pte_mknonnapot(raw, addr);
+
+ for (i = 0; i < nr; i++) {
+ ptent_addr = addr + i * PAGE_SIZE;
+ ptent = __ptep_get_and_clear_noptc(ptep + i);
+ page_table_check_pte_clear(mm, ptent_addr,
+ pte_mknonnapot(ptent, ptent_addr));
+ if (pte_dirty(ptent))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(ptent))
+ pte = pte_mkyoung(pte);
+ }
+
+ return pte;
+ }
+
+ pte = __ptep_modify_prot_start(vma, addr, ptep);
+ while (--nr) {
+ ptep++;
+ addr += PAGE_SIZE;
+ ptent = __ptep_modify_prot_start(vma, addr, ptep);
+ if (pte_dirty(ptent))
+ pte = pte_mkdirty(pte);
+ if (pte_young(ptent))
+ pte = pte_mkyoung(pte);
+ }
+
+ return pte;
+}
+
+void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pte_t *ptep, pte_t old_pte, pte_t pte,
+ unsigned int nr)
+{
+ set_ptes(vma->vm_mm, addr, ptep, pte, nr);
+}
+
static inline pte_t
napotpte_normalize_batch_pte(pte_t *ptep, pte_t orig_pte, fpb_t flags)
{
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 15/15] riscv: mm: Request large exec folios for Svnapot
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
` (13 preceding siblings ...)
2026-07-16 12:41 ` [PATCH v2 14/15] riscv/mm: optimize mprotect for Svnapot mappings Yunhui Cui
@ 2026-07-16 12:41 ` Yunhui Cui
14 siblings, 0 replies; 16+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
The Svnapot MM support in this series allows generic MM paths to operate
on folded 64KB executable mappings while preserving a per-page public PTE
view. To benefit from that more often, executable file-backed mappings
should be faulted in with a folio size that can be folded into a Svnapot
range.
Request 64KB executable folios when Svnapot is available, so executable
text mappings are more likely to be installed as folded Svnapot blocks
and benefit from lower iTLB pressure. Keep the default order-0 behavior
when Svnapot is unavailable.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index b546e6614c558..6801ba135c483 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -296,6 +296,14 @@ static __always_inline bool has_svnapot(void)
return riscv_has_extension_likely(RISCV_ISA_EXT_SVNAPOT);
}
+/*
+ * Request executable file-backed mappings to be read into 64KB folios when
+ * Svnapot is available. A naturally aligned 64KB folio can be folded into a
+ * Svnapot PTE range, reducing iTLB pressure for executable text.
+ */
+#define exec_folio_order() \
+ (has_svnapot() ? NAPOT_CONT64KB_ORDER : 0)
+
static inline unsigned long pte_napot(pte_t pte)
{
return pte_val(pte) & _PAGE_NAPOT;
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
end of thread, other threads:[~2026-07-16 12:44 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 01/15] riscv: introduce raw PTE helpers Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 02/15] riscv: switch arch page-table users to " Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 03/15] riscv/mm: implement Svnapot contpte read-side helpers Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 04/15] riscv/mm: implement Svnapot contpte update helpers Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 05/15] mm: extend pte batch and leaf-size helpers Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 06/15] riscv: make pte_batch_hint() honor folio batch flags Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 07/15] riscv/mm: preserve Svnapot leaf-size semantics for page-table consumers Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 08/15] riscv/mm: avoid redundant Svnapot A/D aggregation Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 09/15] riscv/mm: avoid Svnapot consistency checks in ptep_get() Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 10/15] mm/gup: add fast-GUP specific lockless PTE helpers Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 11/15] riscv: mm: avoid Svnapot A/D aggregation in fast-GUP Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 12/15] arm64: mm: avoid contpte " Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 13/15] riscv/mm: remove redundant TLB flush in napotpte_convert Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 14/15] riscv/mm: optimize mprotect for Svnapot mappings Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 15/15] riscv: mm: Request large exec folios for Svnapot Yunhui Cui
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox