* [PATCH v2 00/15] riscv: add Svnapot PTE folding support
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot can encode a naturally aligned 64KB range as a single folded
PTE block. This series adds the RISC-V MM support needed to use that
encoding for suitable user mappings while preserving the generic MM view
of PAGE_SIZE PTEs.
The series starts by adding raw PTE helpers and switching architecture
page-table users that must see the encoded page-table entry to those raw
helpers. It then introduces Svnapot read-side and update helpers for
folded PTE ranges. Later patches make the generic PTE batch and leaf-size
interfaces work with the raw encoded view where required, avoid redundant
A/D aggregation and TLB flushes, optimize mprotect() for Svnapot ranges,
and request 64KB executable folios when Svnapot is available so text
mappings can benefit from lower iTLB pressure.
Dependency:
- This series depends on the prerequisite RISC-V patch posted at:
https://lore.kernel.org/all/20260618064406.14508-1-cuiyunhui@bytedance.com/
Changes in v2:
- Rework the v1 7-patch series into 15 smaller patches: split raw PTE
helper introduction from the architecture-user conversion, and split
Svnapot contpte handling into read-side and update-side pieces.
- Add the generic MM helper changes needed by folded Svnapot mappings,
including folio-batch comparison flags, raw-entry leaf-size queries and
fast-GUP-specific lockless PTE helpers.
- Make the RISC-V pte_batch_hint() implementation honor folio batch
flags, and preserve Svnapot leaf-size semantics via __ptep_leaf_size().
- Add follow-up fixes and optimizations that were not in v1: avoid
redundant A/D aggregation, avoid Svnapot consistency checks in
ptep_get(), avoid A/D aggregation in fast-GUP, remove a redundant TLB
flush, optimize mprotect() for Svnapot mappings, and request large
executable folios when Svnapot is available.
Yunhui Cui (15):
riscv: introduce raw PTE helpers
riscv: switch arch page-table users to raw PTE helpers
riscv/mm: implement Svnapot contpte read-side helpers
riscv/mm: implement Svnapot contpte update helpers
mm: extend pte batch and leaf-size helpers
riscv: make pte_batch_hint() honor folio batch flags
riscv/mm: preserve Svnapot leaf-size semantics for page-table
consumers
riscv/mm: avoid redundant Svnapot A/D aggregation
riscv/mm: avoid Svnapot consistency checks in ptep_get()
mm/gup: add fast-GUP specific lockless PTE helpers
riscv: mm: avoid Svnapot A/D aggregation in fast-GUP
arm64: mm: avoid contpte A/D aggregation in fast-GUP
riscv/mm: remove redundant TLB flush in napotpte_convert
riscv/mm: optimize mprotect for Svnapot mappings
riscv: mm: Request large exec folios for Svnapot
arch/arm64/include/asm/pgtable.h | 25 +-
arch/riscv/include/asm/kfence.h | 4 +-
arch/riscv/include/asm/pgtable.h | 480 +++++++++++++++++-
arch/riscv/kernel/efi.c | 4 +-
arch/riscv/kernel/hibernate.c | 3 +-
arch/riscv/kvm/gstage.c | 25 +-
arch/riscv/kvm/mmu.c | 4 +-
arch/riscv/mm/Makefile | 1 +
arch/riscv/mm/contpte.c | 804 +++++++++++++++++++++++++++++++
arch/riscv/mm/fault.c | 4 +-
arch/riscv/mm/hugetlbpage.c | 135 ++++--
arch/riscv/mm/init.c | 8 +-
arch/riscv/mm/kasan_init.c | 16 +-
arch/riscv/mm/pageattr.c | 12 +-
arch/riscv/mm/pgtable.c | 52 +-
include/linux/pgtable.h | 71 ++-
kernel/events/core.c | 2 +-
mm/gup.c | 6 +-
mm/internal.h | 39 +-
mm/mincore.c | 2 +-
mm/mremap.c | 2 +-
21 files changed, 1542 insertions(+), 157 deletions(-)
create mode 100644 arch/riscv/mm/contpte.c
base-commit: b8809969e1d7a591e0f49dd464a5d04b3cf02ab1
--
2.39.5
^ permalink raw reply [flat|nested] 57+ messages in thread
* [PATCH v2 00/15] riscv: add Svnapot PTE folding support
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot can encode a naturally aligned 64KB range as a single folded
PTE block. This series adds the RISC-V MM support needed to use that
encoding for suitable user mappings while preserving the generic MM view
of PAGE_SIZE PTEs.
The series starts by adding raw PTE helpers and switching architecture
page-table users that must see the encoded page-table entry to those raw
helpers. It then introduces Svnapot read-side and update helpers for
folded PTE ranges. Later patches make the generic PTE batch and leaf-size
interfaces work with the raw encoded view where required, avoid redundant
A/D aggregation and TLB flushes, optimize mprotect() for Svnapot ranges,
and request 64KB executable folios when Svnapot is available so text
mappings can benefit from lower iTLB pressure.
Dependency:
- This series depends on the prerequisite RISC-V patch posted at:
https://lore.kernel.org/all/20260618064406.14508-1-cuiyunhui@bytedance.com/
Changes in v2:
- Rework the v1 7-patch series into 15 smaller patches: split raw PTE
helper introduction from the architecture-user conversion, and split
Svnapot contpte handling into read-side and update-side pieces.
- Add the generic MM helper changes needed by folded Svnapot mappings,
including folio-batch comparison flags, raw-entry leaf-size queries and
fast-GUP-specific lockless PTE helpers.
- Make the RISC-V pte_batch_hint() implementation honor folio batch
flags, and preserve Svnapot leaf-size semantics via __ptep_leaf_size().
- Add follow-up fixes and optimizations that were not in v1: avoid
redundant A/D aggregation, avoid Svnapot consistency checks in
ptep_get(), avoid A/D aggregation in fast-GUP, remove a redundant TLB
flush, optimize mprotect() for Svnapot mappings, and request large
executable folios when Svnapot is available.
Yunhui Cui (15):
riscv: introduce raw PTE helpers
riscv: switch arch page-table users to raw PTE helpers
riscv/mm: implement Svnapot contpte read-side helpers
riscv/mm: implement Svnapot contpte update helpers
mm: extend pte batch and leaf-size helpers
riscv: make pte_batch_hint() honor folio batch flags
riscv/mm: preserve Svnapot leaf-size semantics for page-table
consumers
riscv/mm: avoid redundant Svnapot A/D aggregation
riscv/mm: avoid Svnapot consistency checks in ptep_get()
mm/gup: add fast-GUP specific lockless PTE helpers
riscv: mm: avoid Svnapot A/D aggregation in fast-GUP
arm64: mm: avoid contpte A/D aggregation in fast-GUP
riscv/mm: remove redundant TLB flush in napotpte_convert
riscv/mm: optimize mprotect for Svnapot mappings
riscv: mm: Request large exec folios for Svnapot
arch/arm64/include/asm/pgtable.h | 25 +-
arch/riscv/include/asm/kfence.h | 4 +-
arch/riscv/include/asm/pgtable.h | 480 +++++++++++++++++-
arch/riscv/kernel/efi.c | 4 +-
arch/riscv/kernel/hibernate.c | 3 +-
arch/riscv/kvm/gstage.c | 25 +-
arch/riscv/kvm/mmu.c | 4 +-
arch/riscv/mm/Makefile | 1 +
arch/riscv/mm/contpte.c | 804 +++++++++++++++++++++++++++++++
arch/riscv/mm/fault.c | 4 +-
arch/riscv/mm/hugetlbpage.c | 135 ++++--
arch/riscv/mm/init.c | 8 +-
arch/riscv/mm/kasan_init.c | 16 +-
arch/riscv/mm/pageattr.c | 12 +-
arch/riscv/mm/pgtable.c | 52 +-
include/linux/pgtable.h | 71 ++-
kernel/events/core.c | 2 +-
mm/gup.c | 6 +-
mm/internal.h | 39 +-
mm/mincore.c | 2 +-
mm/mremap.c | 2 +-
21 files changed, 1542 insertions(+), 157 deletions(-)
create mode 100644 arch/riscv/mm/contpte.c
base-commit: b8809969e1d7a591e0f49dd464a5d04b3cf02ab1
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply [flat|nested] 57+ messages in thread
* [PATCH v2 00/15] riscv: add Svnapot PTE folding support
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot can encode a naturally aligned 64KB range as a single folded
PTE block. This series adds the RISC-V MM support needed to use that
encoding for suitable user mappings while preserving the generic MM view
of PAGE_SIZE PTEs.
The series starts by adding raw PTE helpers and switching architecture
page-table users that must see the encoded page-table entry to those raw
helpers. It then introduces Svnapot read-side and update helpers for
folded PTE ranges. Later patches make the generic PTE batch and leaf-size
interfaces work with the raw encoded view where required, avoid redundant
A/D aggregation and TLB flushes, optimize mprotect() for Svnapot ranges,
and request 64KB executable folios when Svnapot is available so text
mappings can benefit from lower iTLB pressure.
Dependency:
- This series depends on the prerequisite RISC-V patch posted at:
https://lore.kernel.org/all/20260618064406.14508-1-cuiyunhui@bytedance.com/
Changes in v2:
- Rework the v1 7-patch series into 15 smaller patches: split raw PTE
helper introduction from the architecture-user conversion, and split
Svnapot contpte handling into read-side and update-side pieces.
- Add the generic MM helper changes needed by folded Svnapot mappings,
including folio-batch comparison flags, raw-entry leaf-size queries and
fast-GUP-specific lockless PTE helpers.
- Make the RISC-V pte_batch_hint() implementation honor folio batch
flags, and preserve Svnapot leaf-size semantics via __ptep_leaf_size().
- Add follow-up fixes and optimizations that were not in v1: avoid
redundant A/D aggregation, avoid Svnapot consistency checks in
ptep_get(), avoid A/D aggregation in fast-GUP, remove a redundant TLB
flush, optimize mprotect() for Svnapot mappings, and request large
executable folios when Svnapot is available.
Yunhui Cui (15):
riscv: introduce raw PTE helpers
riscv: switch arch page-table users to raw PTE helpers
riscv/mm: implement Svnapot contpte read-side helpers
riscv/mm: implement Svnapot contpte update helpers
mm: extend pte batch and leaf-size helpers
riscv: make pte_batch_hint() honor folio batch flags
riscv/mm: preserve Svnapot leaf-size semantics for page-table
consumers
riscv/mm: avoid redundant Svnapot A/D aggregation
riscv/mm: avoid Svnapot consistency checks in ptep_get()
mm/gup: add fast-GUP specific lockless PTE helpers
riscv: mm: avoid Svnapot A/D aggregation in fast-GUP
arm64: mm: avoid contpte A/D aggregation in fast-GUP
riscv/mm: remove redundant TLB flush in napotpte_convert
riscv/mm: optimize mprotect for Svnapot mappings
riscv: mm: Request large exec folios for Svnapot
arch/arm64/include/asm/pgtable.h | 25 +-
arch/riscv/include/asm/kfence.h | 4 +-
arch/riscv/include/asm/pgtable.h | 480 +++++++++++++++++-
arch/riscv/kernel/efi.c | 4 +-
arch/riscv/kernel/hibernate.c | 3 +-
arch/riscv/kvm/gstage.c | 25 +-
arch/riscv/kvm/mmu.c | 4 +-
arch/riscv/mm/Makefile | 1 +
arch/riscv/mm/contpte.c | 804 +++++++++++++++++++++++++++++++
arch/riscv/mm/fault.c | 4 +-
arch/riscv/mm/hugetlbpage.c | 135 ++++--
arch/riscv/mm/init.c | 8 +-
arch/riscv/mm/kasan_init.c | 16 +-
arch/riscv/mm/pageattr.c | 12 +-
arch/riscv/mm/pgtable.c | 52 +-
include/linux/pgtable.h | 71 ++-
kernel/events/core.c | 2 +-
mm/gup.c | 6 +-
mm/internal.h | 39 +-
mm/mincore.c | 2 +-
mm/mremap.c | 2 +-
21 files changed, 1542 insertions(+), 157 deletions(-)
create mode 100644 arch/riscv/mm/contpte.c
base-commit: b8809969e1d7a591e0f49dd464a5d04b3cf02ab1
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 57+ messages in thread
* [PATCH v2 01/15] riscv: introduce raw PTE helpers
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot needs RISC-V to distinguish between two PTE views: the raw encoded
entries used by architecture code, and the logical per-page PTE view that
will later be exposed to generic MM paths.
Split the low-level RISC-V PTE operations from the public helpers so the
raw helpers can keep operating on the hardware encoding, while follow-up
changes build the Svnapot-aware public view on top.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 129 +++++++++++++++++++++++++++----
1 file changed, 114 insertions(+), 15 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index a810655ce8f9b..8ca5da9534967 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -9,6 +9,7 @@
#include <linux/mmzone.h>
#include <linux/sizes.h>
+#include <asm/cmpxchg.h>
#include <asm/pgtable-bits.h>
#ifndef CONFIG_MMU
@@ -628,11 +629,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)
* a page table are directly modified. Thus, the following hook is
* made available.
*/
-static inline void set_pte(pte_t *ptep, pte_t pteval)
+static inline void __set_pte(pte_t *ptep, pte_t pteval)
{
WRITE_ONCE(*ptep, pteval);
}
+#define __set_pte __set_pte
+
void flush_icache_pte(struct mm_struct *mm, pte_t pte);
static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval)
@@ -640,13 +643,13 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval)
if (pte_present(pteval) && pte_exec(pteval))
flush_icache_pte(mm, pteval);
- set_pte(ptep, pteval);
+ __set_pte(ptep, pteval);
}
#define PFN_PTE_SHIFT _PAGE_PFN_SHIFT
-static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
- pte_t *ptep, pte_t pteval, unsigned int nr)
+static inline void __set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pteval, unsigned int nr)
{
page_table_check_ptes_set(mm, addr, ptep, pteval, nr);
@@ -658,24 +661,85 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
pte_val(pteval) += 1 << _PAGE_PFN_SHIFT;
}
}
-#define set_ptes set_ptes
-static inline void pte_clear(struct mm_struct *mm,
- unsigned long addr, pte_t *ptep)
+#define __set_ptes __set_ptes
+
+static inline void __pte_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
{
__set_pte_at(mm, ptep, __pte(0));
}
+#define __pte_clear __pte_clear
+
+#define __ptep_get __ptep_get
+static inline pte_t __ptep_get(pte_t *ptep)
+{
+ return READ_ONCE(*ptep);
+}
+
+#define __ptep_get_lockless __ptep_get_lockless
+static inline pte_t __ptep_get_lockless(pte_t *ptep)
+{
+ return __ptep_get(ptep);
+}
+
+static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ pte_t pte, cydp_t flags)
+{
+ pte_t old_pte;
+
+ do {
+ old_pte = pte;
+
+ if (flags & CYDP_CLEAR_YOUNG)
+ pte = pte_mkold(pte);
+ if (flags & CYDP_CLEAR_DIRTY)
+ pte = pte_mkclean(pte);
+
+ pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep),
+ pte_val(old_pte),
+ pte_val(pte));
+ } while (pte_val(pte) != pte_val(old_pte));
+}
+
+static inline void __clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags)
+{
+ pte_t pte;
+
+ for (;;) {
+ pte = __ptep_get(ptep);
+
+ if (flags == (CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY))
+ __set_pte(ptep, pte_mkclean(pte_mkold(pte)));
+ else
+ __clear_young_dirty_pte(vma, addr, ptep, pte, flags);
+
+ if (--nr == 0)
+ break;
+ ptep++;
+ addr += PAGE_SIZE;
+ }
+}
+
#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS /* defined in mm/pgtable.c */
extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address,
pte_t *ptep, pte_t entry, int dirty);
+int __ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty);
#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG /* defined in mm/pgtable.c */
bool ptep_test_and_clear_young(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep);
+ unsigned long address, pte_t *ptep);
+bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep);
#define __HAVE_ARCH_PTEP_GET_AND_CLEAR
-static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
- unsigned long address, pte_t *ptep)
+static inline pte_t __ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep)
{
#ifdef CONFIG_SMP
pte_t pte = __pte(xchg(&ptep->pte, 0));
@@ -690,9 +754,24 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
return pte;
}
+#define __ptep_get_and_clear __ptep_get_and_clear
+
+static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ pte_t pte = __ptep_get_and_clear(vma->vm_mm, address, ptep);
+
+ if (pte_accessible(vma->vm_mm, pte))
+ flush_tlb_page(vma, address);
+
+ return pte;
+}
+
+#define __ptep_clear_flush __ptep_clear_flush
+
#define __HAVE_ARCH_PTEP_SET_WRPROTECT
-static inline void ptep_set_wrprotect(struct mm_struct *mm,
- unsigned long address, pte_t *ptep)
+static inline void __ptep_set_wrprotect(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep)
{
pte_t read_pte = READ_ONCE(*ptep);
/*
@@ -705,9 +784,11 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm,
((pte_val(read_pte) & ~(unsigned long)_PAGE_WRITE) | _PAGE_READ));
}
+#define __ptep_set_wrprotect __ptep_set_wrprotect
+
#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
-static inline bool ptep_clear_flush_young(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep)
+static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
{
/*
* This comment is borrowed from x86, but applies equally to RISC-V:
@@ -724,9 +805,27 @@ static inline bool ptep_clear_flush_young(struct vm_area_struct *vma,
* shouldn't really matter because there's no real memory
* pressure for swapout to react to. ]
*/
- return ptep_test_and_clear_young(vma, address, ptep);
+ return __ptep_test_and_clear_young(vma, address, ptep);
}
+#define __ptep_clear_flush_young __ptep_clear_flush_young
+
+#define ptep_get __ptep_get
+#define ptep_get_lockless __ptep_get_lockless
+#define set_pte __set_pte
+#define set_ptes __set_ptes
+
+static inline void pte_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ __pte_clear(mm, addr, ptep);
+}
+
+#define ptep_get_and_clear __ptep_get_and_clear
+#define clear_young_dirty_ptes __clear_young_dirty_ptes
+#define ptep_set_wrprotect __ptep_set_wrprotect
+#define ptep_clear_flush_young __ptep_clear_flush_young
+
#define pgprot_nx pgprot_nx
static inline pgprot_t pgprot_nx(pgprot_t _prot)
{
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 01/15] riscv: introduce raw PTE helpers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot needs RISC-V to distinguish between two PTE views: the raw encoded
entries used by architecture code, and the logical per-page PTE view that
will later be exposed to generic MM paths.
Split the low-level RISC-V PTE operations from the public helpers so the
raw helpers can keep operating on the hardware encoding, while follow-up
changes build the Svnapot-aware public view on top.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 129 +++++++++++++++++++++++++++----
1 file changed, 114 insertions(+), 15 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index a810655ce8f9b..8ca5da9534967 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -9,6 +9,7 @@
#include <linux/mmzone.h>
#include <linux/sizes.h>
+#include <asm/cmpxchg.h>
#include <asm/pgtable-bits.h>
#ifndef CONFIG_MMU
@@ -628,11 +629,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)
* a page table are directly modified. Thus, the following hook is
* made available.
*/
-static inline void set_pte(pte_t *ptep, pte_t pteval)
+static inline void __set_pte(pte_t *ptep, pte_t pteval)
{
WRITE_ONCE(*ptep, pteval);
}
+#define __set_pte __set_pte
+
void flush_icache_pte(struct mm_struct *mm, pte_t pte);
static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval)
@@ -640,13 +643,13 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval)
if (pte_present(pteval) && pte_exec(pteval))
flush_icache_pte(mm, pteval);
- set_pte(ptep, pteval);
+ __set_pte(ptep, pteval);
}
#define PFN_PTE_SHIFT _PAGE_PFN_SHIFT
-static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
- pte_t *ptep, pte_t pteval, unsigned int nr)
+static inline void __set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pteval, unsigned int nr)
{
page_table_check_ptes_set(mm, addr, ptep, pteval, nr);
@@ -658,24 +661,85 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
pte_val(pteval) += 1 << _PAGE_PFN_SHIFT;
}
}
-#define set_ptes set_ptes
-static inline void pte_clear(struct mm_struct *mm,
- unsigned long addr, pte_t *ptep)
+#define __set_ptes __set_ptes
+
+static inline void __pte_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
{
__set_pte_at(mm, ptep, __pte(0));
}
+#define __pte_clear __pte_clear
+
+#define __ptep_get __ptep_get
+static inline pte_t __ptep_get(pte_t *ptep)
+{
+ return READ_ONCE(*ptep);
+}
+
+#define __ptep_get_lockless __ptep_get_lockless
+static inline pte_t __ptep_get_lockless(pte_t *ptep)
+{
+ return __ptep_get(ptep);
+}
+
+static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ pte_t pte, cydp_t flags)
+{
+ pte_t old_pte;
+
+ do {
+ old_pte = pte;
+
+ if (flags & CYDP_CLEAR_YOUNG)
+ pte = pte_mkold(pte);
+ if (flags & CYDP_CLEAR_DIRTY)
+ pte = pte_mkclean(pte);
+
+ pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep),
+ pte_val(old_pte),
+ pte_val(pte));
+ } while (pte_val(pte) != pte_val(old_pte));
+}
+
+static inline void __clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags)
+{
+ pte_t pte;
+
+ for (;;) {
+ pte = __ptep_get(ptep);
+
+ if (flags == (CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY))
+ __set_pte(ptep, pte_mkclean(pte_mkold(pte)));
+ else
+ __clear_young_dirty_pte(vma, addr, ptep, pte, flags);
+
+ if (--nr == 0)
+ break;
+ ptep++;
+ addr += PAGE_SIZE;
+ }
+}
+
#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS /* defined in mm/pgtable.c */
extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address,
pte_t *ptep, pte_t entry, int dirty);
+int __ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty);
#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG /* defined in mm/pgtable.c */
bool ptep_test_and_clear_young(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep);
+ unsigned long address, pte_t *ptep);
+bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep);
#define __HAVE_ARCH_PTEP_GET_AND_CLEAR
-static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
- unsigned long address, pte_t *ptep)
+static inline pte_t __ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep)
{
#ifdef CONFIG_SMP
pte_t pte = __pte(xchg(&ptep->pte, 0));
@@ -690,9 +754,24 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
return pte;
}
+#define __ptep_get_and_clear __ptep_get_and_clear
+
+static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ pte_t pte = __ptep_get_and_clear(vma->vm_mm, address, ptep);
+
+ if (pte_accessible(vma->vm_mm, pte))
+ flush_tlb_page(vma, address);
+
+ return pte;
+}
+
+#define __ptep_clear_flush __ptep_clear_flush
+
#define __HAVE_ARCH_PTEP_SET_WRPROTECT
-static inline void ptep_set_wrprotect(struct mm_struct *mm,
- unsigned long address, pte_t *ptep)
+static inline void __ptep_set_wrprotect(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep)
{
pte_t read_pte = READ_ONCE(*ptep);
/*
@@ -705,9 +784,11 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm,
((pte_val(read_pte) & ~(unsigned long)_PAGE_WRITE) | _PAGE_READ));
}
+#define __ptep_set_wrprotect __ptep_set_wrprotect
+
#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
-static inline bool ptep_clear_flush_young(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep)
+static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
{
/*
* This comment is borrowed from x86, but applies equally to RISC-V:
@@ -724,9 +805,27 @@ static inline bool ptep_clear_flush_young(struct vm_area_struct *vma,
* shouldn't really matter because there's no real memory
* pressure for swapout to react to. ]
*/
- return ptep_test_and_clear_young(vma, address, ptep);
+ return __ptep_test_and_clear_young(vma, address, ptep);
}
+#define __ptep_clear_flush_young __ptep_clear_flush_young
+
+#define ptep_get __ptep_get
+#define ptep_get_lockless __ptep_get_lockless
+#define set_pte __set_pte
+#define set_ptes __set_ptes
+
+static inline void pte_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ __pte_clear(mm, addr, ptep);
+}
+
+#define ptep_get_and_clear __ptep_get_and_clear
+#define clear_young_dirty_ptes __clear_young_dirty_ptes
+#define ptep_set_wrprotect __ptep_set_wrprotect
+#define ptep_clear_flush_young __ptep_clear_flush_young
+
#define pgprot_nx pgprot_nx
static inline pgprot_t pgprot_nx(pgprot_t _prot)
{
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 01/15] riscv: introduce raw PTE helpers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot needs RISC-V to distinguish between two PTE views: the raw encoded
entries used by architecture code, and the logical per-page PTE view that
will later be exposed to generic MM paths.
Split the low-level RISC-V PTE operations from the public helpers so the
raw helpers can keep operating on the hardware encoding, while follow-up
changes build the Svnapot-aware public view on top.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 129 +++++++++++++++++++++++++++----
1 file changed, 114 insertions(+), 15 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index a810655ce8f9b..8ca5da9534967 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -9,6 +9,7 @@
#include <linux/mmzone.h>
#include <linux/sizes.h>
+#include <asm/cmpxchg.h>
#include <asm/pgtable-bits.h>
#ifndef CONFIG_MMU
@@ -628,11 +629,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)
* a page table are directly modified. Thus, the following hook is
* made available.
*/
-static inline void set_pte(pte_t *ptep, pte_t pteval)
+static inline void __set_pte(pte_t *ptep, pte_t pteval)
{
WRITE_ONCE(*ptep, pteval);
}
+#define __set_pte __set_pte
+
void flush_icache_pte(struct mm_struct *mm, pte_t pte);
static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval)
@@ -640,13 +643,13 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval)
if (pte_present(pteval) && pte_exec(pteval))
flush_icache_pte(mm, pteval);
- set_pte(ptep, pteval);
+ __set_pte(ptep, pteval);
}
#define PFN_PTE_SHIFT _PAGE_PFN_SHIFT
-static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
- pte_t *ptep, pte_t pteval, unsigned int nr)
+static inline void __set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pteval, unsigned int nr)
{
page_table_check_ptes_set(mm, addr, ptep, pteval, nr);
@@ -658,24 +661,85 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
pte_val(pteval) += 1 << _PAGE_PFN_SHIFT;
}
}
-#define set_ptes set_ptes
-static inline void pte_clear(struct mm_struct *mm,
- unsigned long addr, pte_t *ptep)
+#define __set_ptes __set_ptes
+
+static inline void __pte_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
{
__set_pte_at(mm, ptep, __pte(0));
}
+#define __pte_clear __pte_clear
+
+#define __ptep_get __ptep_get
+static inline pte_t __ptep_get(pte_t *ptep)
+{
+ return READ_ONCE(*ptep);
+}
+
+#define __ptep_get_lockless __ptep_get_lockless
+static inline pte_t __ptep_get_lockless(pte_t *ptep)
+{
+ return __ptep_get(ptep);
+}
+
+static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ pte_t pte, cydp_t flags)
+{
+ pte_t old_pte;
+
+ do {
+ old_pte = pte;
+
+ if (flags & CYDP_CLEAR_YOUNG)
+ pte = pte_mkold(pte);
+ if (flags & CYDP_CLEAR_DIRTY)
+ pte = pte_mkclean(pte);
+
+ pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep),
+ pte_val(old_pte),
+ pte_val(pte));
+ } while (pte_val(pte) != pte_val(old_pte));
+}
+
+static inline void __clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags)
+{
+ pte_t pte;
+
+ for (;;) {
+ pte = __ptep_get(ptep);
+
+ if (flags == (CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY))
+ __set_pte(ptep, pte_mkclean(pte_mkold(pte)));
+ else
+ __clear_young_dirty_pte(vma, addr, ptep, pte, flags);
+
+ if (--nr == 0)
+ break;
+ ptep++;
+ addr += PAGE_SIZE;
+ }
+}
+
#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS /* defined in mm/pgtable.c */
extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address,
pte_t *ptep, pte_t entry, int dirty);
+int __ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty);
#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG /* defined in mm/pgtable.c */
bool ptep_test_and_clear_young(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep);
+ unsigned long address, pte_t *ptep);
+bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep);
#define __HAVE_ARCH_PTEP_GET_AND_CLEAR
-static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
- unsigned long address, pte_t *ptep)
+static inline pte_t __ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep)
{
#ifdef CONFIG_SMP
pte_t pte = __pte(xchg(&ptep->pte, 0));
@@ -690,9 +754,24 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
return pte;
}
+#define __ptep_get_and_clear __ptep_get_and_clear
+
+static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ pte_t pte = __ptep_get_and_clear(vma->vm_mm, address, ptep);
+
+ if (pte_accessible(vma->vm_mm, pte))
+ flush_tlb_page(vma, address);
+
+ return pte;
+}
+
+#define __ptep_clear_flush __ptep_clear_flush
+
#define __HAVE_ARCH_PTEP_SET_WRPROTECT
-static inline void ptep_set_wrprotect(struct mm_struct *mm,
- unsigned long address, pte_t *ptep)
+static inline void __ptep_set_wrprotect(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep)
{
pte_t read_pte = READ_ONCE(*ptep);
/*
@@ -705,9 +784,11 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm,
((pte_val(read_pte) & ~(unsigned long)_PAGE_WRITE) | _PAGE_READ));
}
+#define __ptep_set_wrprotect __ptep_set_wrprotect
+
#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
-static inline bool ptep_clear_flush_young(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep)
+static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
{
/*
* This comment is borrowed from x86, but applies equally to RISC-V:
@@ -724,9 +805,27 @@ static inline bool ptep_clear_flush_young(struct vm_area_struct *vma,
* shouldn't really matter because there's no real memory
* pressure for swapout to react to. ]
*/
- return ptep_test_and_clear_young(vma, address, ptep);
+ return __ptep_test_and_clear_young(vma, address, ptep);
}
+#define __ptep_clear_flush_young __ptep_clear_flush_young
+
+#define ptep_get __ptep_get
+#define ptep_get_lockless __ptep_get_lockless
+#define set_pte __set_pte
+#define set_ptes __set_ptes
+
+static inline void pte_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ __pte_clear(mm, addr, ptep);
+}
+
+#define ptep_get_and_clear __ptep_get_and_clear
+#define clear_young_dirty_ptes __clear_young_dirty_ptes
+#define ptep_set_wrprotect __ptep_set_wrprotect
+#define ptep_clear_flush_young __ptep_clear_flush_young
+
#define pgprot_nx pgprot_nx
static inline pgprot_t pgprot_nx(pgprot_t _prot)
{
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 02/15] riscv: switch arch page-table users to raw PTE helpers
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Convert RISC-V internal page-table walkers and mutators over to the raw
PTE helper layer so they keep operating on the encoded hardware entries.
This keeps arch-private users independent from the Svnapot-aware public
PTE view that will be added on top for generic mm paths.
Add a raw get-and-clear helper without page-table-check side effects so
NAPOT-aware arch callers can report the effective non-NAPOT leaf view
explicitly when tearing down contiguous mappings.
Also update the hugetlb Svnapot clear/rewrite paths so they keep operating
on the contiguous block with page-table-check behavior consistent with the
raw-helper model.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/kfence.h | 4 +-
arch/riscv/include/asm/pgtable.h | 14 +++-
arch/riscv/kernel/efi.c | 4 +-
arch/riscv/kernel/hibernate.c | 3 +-
arch/riscv/kvm/gstage.c | 25 +++---
arch/riscv/kvm/mmu.c | 4 +-
arch/riscv/mm/fault.c | 4 +-
arch/riscv/mm/hugetlbpage.c | 135 ++++++++++++++++++++-----------
arch/riscv/mm/init.c | 8 +-
arch/riscv/mm/kasan_init.c | 16 ++--
arch/riscv/mm/pageattr.c | 12 +--
arch/riscv/mm/pgtable.c | 34 +++++---
12 files changed, 165 insertions(+), 98 deletions(-)
diff --git a/arch/riscv/include/asm/kfence.h b/arch/riscv/include/asm/kfence.h
index 29cb3a6ee113d..8325962a9d7e8 100644
--- a/arch/riscv/include/asm/kfence.h
+++ b/arch/riscv/include/asm/kfence.h
@@ -19,9 +19,9 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect)
pte_t *pte = virt_to_kpte(addr);
if (protect) {
- set_pte(pte, __pte(pte_val(ptep_get(pte)) & ~_PAGE_PRESENT));
+ __set_pte(pte, __pte(pte_val(__ptep_get(pte)) & ~_PAGE_PRESENT));
} else {
- set_pte(pte, __pte(pte_val(ptep_get(pte)) | _PAGE_PRESENT));
+ __set_pte(pte, __pte(pte_val(__ptep_get(pte)) | _PAGE_PRESENT));
mark_new_valid_map();
}
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 8ca5da9534967..a0f64e980e981 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -9,7 +9,6 @@
#include <linux/mmzone.h>
#include <linux/sizes.h>
-#include <asm/cmpxchg.h>
#include <asm/pgtable-bits.h>
#ifndef CONFIG_MMU
@@ -124,6 +123,7 @@
#ifndef __ASSEMBLER__
+#include <asm/cmpxchg.h>
#include <asm/page.h>
#include <asm/tlbflush.h>
#include <linux/mm_types.h>
@@ -468,6 +468,11 @@ static inline pte_t pte_mkdirty(pte_t pte)
return __pte(pte_val(pte) | _PAGE_DIRTY | _PAGE_SOFT_DIRTY);
}
+static inline pte_t riscv_pte_mkhwdirty(pte_t pte)
+{
+ return __pte(pte_val(pte) | _PAGE_DIRTY);
+}
+
static inline pte_t pte_mkclean(pte_t pte)
{
return __pte(pte_val(pte) & ~(_PAGE_DIRTY));
@@ -756,6 +761,13 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm,
#define __ptep_get_and_clear __ptep_get_and_clear
+static inline pte_t __ptep_get_and_clear_noptc(pte_t *ptep)
+{
+ return __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));
+}
+
+#define __ptep_get_and_clear_noptc __ptep_get_and_clear_noptc
+
static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep)
{
diff --git a/arch/riscv/kernel/efi.c b/arch/riscv/kernel/efi.c
index 2d3cc57b45352..81ea2546d2922 100644
--- a/arch/riscv/kernel/efi.c
+++ b/arch/riscv/kernel/efi.c
@@ -60,7 +60,7 @@ int __init efi_create_mapping(struct mm_struct *mm, efi_memory_desc_t *md)
static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data)
{
efi_memory_desc_t *md = data;
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
unsigned long val;
if (md->attribute & EFI_MEMORY_RO) {
@@ -72,7 +72,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data)
val = pte_val(pte) & ~_PAGE_EXEC;
pte = __pte(val);
}
- set_pte(ptep, pte);
+ __set_pte(ptep, pte);
return 0;
}
diff --git a/arch/riscv/kernel/hibernate.c b/arch/riscv/kernel/hibernate.c
index 982843828adb7..c7d5c7d4bbcde 100644
--- a/arch/riscv/kernel/hibernate.c
+++ b/arch/riscv/kernel/hibernate.c
@@ -186,7 +186,8 @@ static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long
pte_t pte = READ_ONCE(*src_ptep);
if (pte_present(pte))
- set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot)));
+ __set_pte(dst_ptep,
+ __pte(pte_val(pte) | pgprot_val(prot)));
} while (dst_ptep++, src_ptep++, start += PAGE_SIZE, start < end);
return 0;
diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c
index c4c3b79567f10..67b2c75f5cec1 100644
--- a/arch/riscv/kvm/gstage.c
+++ b/arch/riscv/kvm/gstage.c
@@ -88,7 +88,7 @@ bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage, gpa_t addr,
*ptep_level = current_level;
ptep = (pte_t *)gstage->pgd;
ptep = &ptep[gstage_pte_index(gstage, addr, current_level)];
- while (ptep && pte_val(ptep_get(ptep))) {
+ while (ptep && pte_val(__ptep_get(ptep))) {
if (gstage_pte_leaf(ptep)) {
*ptep_level = current_level;
*ptepp = ptep;
@@ -98,7 +98,7 @@ bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage, gpa_t addr,
if (current_level) {
current_level--;
*ptep_level = current_level;
- ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep));
ptep = &ptep[gstage_pte_index(gstage, addr, current_level)];
} else {
ptep = NULL;
@@ -152,18 +152,18 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
if (gstage_pte_leaf(ptep))
return -EEXIST;
- if (!pte_val(ptep_get(ptep))) {
+ if (!pte_val(__ptep_get(ptep))) {
if (!pcache)
return -ENOMEM;
next_ptep = kvm_mmu_memory_cache_alloc(pcache);
if (!next_ptep)
return -ENOMEM;
- set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)),
- __pgprot(_PAGE_TABLE)));
+ __set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)),
+ __pgprot(_PAGE_TABLE)));
} else {
if (gstage_pte_leaf(ptep))
return -EEXIST;
- next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ next_ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep));
}
current_level--;
@@ -171,7 +171,7 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
}
if (pte_val(*ptep) != pte_val(map->pte)) {
- set_pte(ptep, map->pte);
+ __set_pte(ptep, map->pte);
if (gstage_pte_leaf(ptep))
gstage_tlb_flush(gstage, current_level, map->addr);
}
@@ -371,18 +371,18 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
WARN_ON(addr & (page_size - 1));
- if (!pte_val(ptep_get(ptep)))
+ if (!pte_val(__ptep_get(ptep)))
return false;
if (ptep_level && !gstage_pte_leaf(ptep)) {
- next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ next_ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep));
next_ptep_level = ptep_level - 1;
ret = gstage_level_to_page_size(gstage, next_ptep_level, &next_page_size);
if (ret)
return false;
if (op == GSTAGE_OP_CLEAR)
- set_pte(ptep, __pte(0));
+ __set_pte(ptep, __pte(0));
for (i = 0; i < PTRS_PER_PTE; i++)
flush |= kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size,
&next_ptep[i], next_ptep_level, op);
@@ -391,9 +391,10 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
} else {
old_pte = *ptep;
if (op == GSTAGE_OP_CLEAR)
- set_pte(ptep, __pte(0));
+ __set_pte(ptep, __pte(0));
else if (op == GSTAGE_OP_WP)
- set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE));
+ __set_pte(ptep,
+ __pte(pte_val(__ptep_get(ptep)) & ~_PAGE_WRITE));
if (pte_val(*ptep) != pte_val(old_pte))
flush = true;
}
diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c
index 082f9b2617338..48dfc88d8cea6 100644
--- a/arch/riscv/kvm/mmu.c
+++ b/arch/riscv/kvm/mmu.c
@@ -278,7 +278,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
&ptep, &ptep_level))
return false;
- return ptep_test_and_clear_young(NULL, 0, ptep);
+ return __ptep_test_and_clear_young(NULL, 0, ptep);
}
bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
@@ -298,7 +298,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
&ptep, &ptep_level))
return false;
- return pte_young(ptep_get(ptep));
+ return pte_young(__ptep_get(ptep));
}
static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot,
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 04ed6f8acae4f..0b725ed723023 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -69,7 +69,7 @@ static void show_pte(unsigned long addr)
if (!ptep)
goto out;
- pte = ptep_get(ptep);
+ pte = __ptep_get(ptep);
pr_cont(", pte=%016lx", pte_val(pte));
pte_unmap(ptep);
out:
@@ -231,7 +231,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a
* silently loop forever.
*/
pte_k = pte_offset_kernel(pmd_k, addr);
- if (!pte_present(ptep_get(pte_k))) {
+ if (!pte_present(__ptep_get(pte_k))) {
no_context(regs, addr);
return;
}
diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c
index a6d217112cf46..5ededac02161d 100644
--- a/arch/riscv/mm/hugetlbpage.c
+++ b/arch/riscv/mm/hugetlbpage.c
@@ -7,7 +7,7 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
{
unsigned long pte_num;
int i;
- pte_t orig_pte = ptep_get(ptep);
+ pte_t orig_pte = __ptep_get(ptep);
if (!pte_present(orig_pte) || !pte_napot(orig_pte))
return orig_pte;
@@ -15,10 +15,10 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
pte_num = napot_pte_num(napot_cont_order(orig_pte));
for (i = 0; i < pte_num; i++, ptep++) {
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
if (pte_dirty(pte))
- orig_pte = pte_mkdirty(orig_pte);
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
if (pte_young(pte))
orig_pte = pte_mkyoung(orig_pte);
@@ -74,7 +74,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm,
out:
if (pte) {
- pte_t pteval = ptep_get_lockless(pte);
+ pte_t pteval = __ptep_get_lockless(pte);
WARN_ON_ONCE(pte_present(pteval) && !pte_huge(pteval));
}
@@ -145,28 +145,52 @@ unsigned long hugetlb_mask_last_page(struct hstate *h)
return 0UL;
}
+static unsigned long napot_hugetlb_block_addr(pte_t pte, unsigned long addr)
+{
+ unsigned long order;
+
+ if (!pte_napot(pte))
+ return addr;
+
+ order = napot_cont_order(pte);
+
+ return addr & napot_cont_mask(order);
+}
+
+static pte_t napot_hugetlb_get_and_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ pte_t pte;
+
+ pte = __ptep_get_and_clear_noptc(ptep);
+ page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
+
+ return pte;
+}
+
static pte_t get_clear_contig(struct mm_struct *mm,
unsigned long addr,
pte_t *ptep,
unsigned long ncontig)
{
- pte_t pte, tmp_pte;
- bool present;
-
- pte = ptep_get_and_clear(mm, addr, ptep);
- present = pte_present(pte);
- while (--ncontig) {
- ptep++;
- addr += PAGE_SIZE;
- tmp_pte = ptep_get_and_clear(mm, addr, ptep);
- if (present) {
- if (pte_dirty(tmp_pte))
- pte = pte_mkdirty(pte);
- if (pte_young(tmp_pte))
- pte = pte_mkyoung(pte);
- }
+ pte_t orig_pte = __ptep_get(ptep);
+ unsigned long i;
+
+ addr = napot_hugetlb_block_addr(orig_pte, addr);
+ if (pte_napot(orig_pte))
+ ptep = huge_pte_offset(mm, addr,
+ napot_cont_size(napot_cont_order(orig_pte)));
+
+ for (i = 0; i < ncontig; i++, addr += PAGE_SIZE, ptep++) {
+ pte_t pte = napot_hugetlb_get_and_clear(mm, addr, ptep);
+
+ if (pte_dirty(pte))
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
+ if (pte_young(pte))
+ orig_pte = pte_mkyoung(orig_pte);
}
- return pte;
+
+ return orig_pte;
}
static pte_t get_clear_contig_flush(struct mm_struct *mm,
@@ -174,10 +198,13 @@ static pte_t get_clear_contig_flush(struct mm_struct *mm,
pte_t *ptep,
unsigned long pte_num)
{
+ pte_t pte = __ptep_get(ptep);
pte_t orig_pte = get_clear_contig(mm, addr, ptep, pte_num);
struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0);
bool valid = !pte_none(orig_pte);
+ addr = napot_hugetlb_block_addr(pte, addr);
+
if (valid)
flush_tlb_range(&vma, addr, addr + (PAGE_SIZE * pte_num));
@@ -207,14 +234,31 @@ static void clear_flush(struct mm_struct *mm,
unsigned long ncontig)
{
struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0);
+ pte_t pte = __ptep_get(ptep);
unsigned long i, saddr = addr;
+ addr = napot_hugetlb_block_addr(pte, addr);
+ if (pte_napot(pte))
+ ptep = huge_pte_offset(mm, addr,
+ napot_cont_size(napot_cont_order(pte)));
+ saddr = addr;
+
for (i = 0; i < ncontig; i++, addr += pgsize, ptep++)
- ptep_get_and_clear(mm, addr, ptep);
+ napot_hugetlb_get_and_clear(mm, addr, ptep);
flush_tlb_range(&vma, saddr, addr);
}
+static void set_huge_napot_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte, unsigned long pte_num)
+{
+ unsigned long i;
+
+ page_table_check_ptes_set(mm, addr, ptep, pte, pte_num);
+ for (i = 0; i < pte_num; i++)
+ __set_pte_at(mm, ptep + i, pte);
+}
+
static int num_contig_ptes_from_size(unsigned long sz, size_t *pgsize)
{
unsigned long hugepage_shift;
@@ -256,19 +300,18 @@ void set_huge_pte_at(struct mm_struct *mm,
if (!pte_present(pte)) {
for (i = 0; i < pte_num; i++, ptep++, addr += pgsize)
- set_ptes(mm, addr, ptep, pte, 1);
+ __set_ptes(mm, addr, ptep, pte, 1);
return;
}
if (!pte_napot(pte)) {
- set_ptes(mm, addr, ptep, pte, 1);
+ __set_ptes(mm, addr, ptep, pte, 1);
return;
}
clear_flush(mm, addr, ptep, pgsize, pte_num);
- for (i = 0; i < pte_num; i++, ptep++, addr += pgsize)
- set_pte_at(mm, addr, ptep, pte);
+ set_huge_napot_ptes(mm, addr, ptep, pte, pte_num);
}
int huge_ptep_set_access_flags(struct vm_area_struct *vma,
@@ -280,10 +323,10 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,
struct mm_struct *mm = vma->vm_mm;
unsigned long order;
pte_t orig_pte;
- int i, pte_num;
+ int pte_num;
if (!pte_napot(pte))
- return ptep_set_access_flags(vma, addr, ptep, pte, dirty);
+ return __ptep_set_access_flags(vma, addr, ptep, pte, dirty);
order = napot_cont_order(pte);
pte_num = napot_pte_num(order);
@@ -291,13 +334,12 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,
orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num);
if (pte_dirty(orig_pte))
- pte = pte_mkdirty(pte);
+ pte = riscv_pte_mkhwdirty(pte);
if (pte_young(orig_pte))
pte = pte_mkyoung(pte);
- for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++)
- set_pte_at(mm, addr, ptep, pte);
+ set_huge_napot_ptes(mm, addr, ptep, pte, pte_num);
return true;
}
@@ -306,14 +348,13 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm,
unsigned long addr,
pte_t *ptep, unsigned long sz)
{
- size_t pgsize;
- pte_t orig_pte = ptep_get(ptep);
+ pte_t orig_pte = __ptep_get(ptep);
int pte_num;
if (!pte_napot(orig_pte))
- return ptep_get_and_clear(mm, addr, ptep);
+ return __ptep_get_and_clear(mm, addr, ptep);
- pte_num = num_contig_ptes_from_size(sz, &pgsize);
+ pte_num = napot_pte_num(napot_cont_order(orig_pte));
return get_clear_contig(mm, addr, ptep, pte_num);
}
@@ -322,13 +363,13 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm,
unsigned long addr,
pte_t *ptep)
{
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
unsigned long order;
pte_t orig_pte;
- int i, pte_num;
+ int pte_num;
if (!pte_napot(pte)) {
- ptep_set_wrprotect(mm, addr, ptep);
+ __ptep_set_wrprotect(mm, addr, ptep);
return;
}
@@ -339,19 +380,18 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm,
orig_pte = pte_wrprotect(orig_pte);
- for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++)
- set_pte_at(mm, addr, ptep, orig_pte);
+ set_huge_napot_ptes(mm, addr, ptep, orig_pte, pte_num);
}
pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,
unsigned long addr,
pte_t *ptep)
{
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
int pte_num;
if (!pte_napot(pte))
- return ptep_clear_flush(vma, addr, ptep);
+ return __ptep_clear_flush(vma, addr, ptep);
pte_num = napot_pte_num(napot_cont_order(pte));
@@ -363,19 +403,16 @@ void huge_pte_clear(struct mm_struct *mm,
pte_t *ptep,
unsigned long sz)
{
- size_t pgsize;
- pte_t pte = ptep_get(ptep);
- int i, pte_num;
+ pte_t pte = __ptep_get(ptep);
+ int pte_num;
if (!pte_napot(pte)) {
- pte_clear(mm, addr, ptep);
+ __pte_clear(mm, addr, ptep);
return;
}
- pte_num = num_contig_ptes_from_size(sz, &pgsize);
-
- for (i = 0; i < pte_num; i++, addr += pgsize, ptep++)
- pte_clear(mm, addr, ptep);
+ pte_num = napot_pte_num(napot_cont_order(pte));
+ get_clear_contig(mm, addr, ptep, pte_num);
}
static bool is_napot_size(unsigned long size)
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index e2be64bacf160..d44bb7db22b43 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -378,9 +378,9 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot)
ptep = &fixmap_pte[pte_index(addr)];
if (pgprot_val(prot))
- set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot));
+ __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot));
else
- pte_clear(&init_mm, addr, ptep);
+ __pte_clear(&init_mm, addr, ptep);
local_flush_tlb_page(addr);
}
@@ -1571,11 +1571,11 @@ static void __meminit remove_pte_mapping(pte_t *pte_base, unsigned long addr, un
next = end;
ptep = pte_base + pte_index(addr);
- pte = ptep_get(ptep);
+ pte = __ptep_get(ptep);
if (!pte_present(*ptep))
continue;
- pte_clear(&init_mm, addr, ptep);
+ __pte_clear(&init_mm, addr, ptep);
if (is_vmemmap)
free_vmemmap_storage(pte_page(pte), PAGE_SIZE, altmap);
}
diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c
index 1f3aa9611187f..c5d5513b206ca 100644
--- a/arch/riscv/mm/kasan_init.c
+++ b/arch/riscv/mm/kasan_init.c
@@ -39,9 +39,9 @@ static void __init kasan_populate_pte(pmd_t *pmd, unsigned long vaddr, unsigned
ptep = pte_offset_kernel(pmd, vaddr);
do {
- if (pte_none(ptep_get(ptep))) {
+ if (pte_none(__ptep_get(ptep))) {
phys_addr = memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE);
- set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL));
+ __set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL));
memset(__va(phys_addr), KASAN_SHADOW_INIT, PAGE_SIZE);
}
} while (ptep++, vaddr += PAGE_SIZE, vaddr != end);
@@ -327,8 +327,8 @@ asmlinkage void __init kasan_early_init(void)
KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT)));
for (i = 0; i < PTRS_PER_PTE; ++i)
- set_pte(kasan_early_shadow_pte + i,
- pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL));
+ __set_pte(kasan_early_shadow_pte + i,
+ pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL));
for (i = 0; i < PTRS_PER_PMD; ++i)
set_pmd(kasan_early_shadow_pmd + i,
@@ -520,10 +520,10 @@ void __init kasan_init(void)
kasan_mem_to_shadow((const void *)MODULES_VADDR + SZ_2G));
for (i = 0; i < PTRS_PER_PTE; i++)
- set_pte(&kasan_early_shadow_pte[i],
- mk_pte(virt_to_page(kasan_early_shadow_page),
- __pgprot(_PAGE_PRESENT | _PAGE_READ |
- _PAGE_ACCESSED)));
+ __set_pte(&kasan_early_shadow_pte[i],
+ mk_pte(virt_to_page(kasan_early_shadow_page),
+ __pgprot(_PAGE_PRESENT | _PAGE_READ |
+ _PAGE_ACCESSED)));
memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE);
init_task.kasan_depth = 0;
diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c
index 3f76db3d27699..e0271e2a0b295 100644
--- a/arch/riscv/mm/pageattr.c
+++ b/arch/riscv/mm/pageattr.c
@@ -68,10 +68,10 @@ static int pageattr_pmd_entry(pmd_t *pmd, unsigned long addr,
static int pageattr_pte_entry(pte_t *pte, unsigned long addr,
unsigned long next, struct mm_walk *walk)
{
- pte_t val = ptep_get(pte);
+ pte_t val = __ptep_get(pte);
val = __pte(set_pageattr_masks(pte_val(val), walk));
- set_pte(pte, val);
+ __set_pte(pte, val);
return 0;
}
@@ -121,7 +121,7 @@ static int __split_linear_mapping_pmd(pud_t *pudp,
ptep_new = (pte_t *)page_address(pte_page);
for (i = 0; i < PTRS_PER_PTE; ++i, ++ptep_new)
- set_pte(ptep_new, pfn_pte(pfn + i, prot));
+ __set_pte(ptep_new, pfn_pte(pfn + i, prot));
smp_wmb();
@@ -406,14 +406,14 @@ static int debug_pagealloc_set_page(pte_t *pte, unsigned long addr, void *data)
{
int enable = *(int *)data;
- unsigned long val = pte_val(ptep_get(pte));
+ unsigned long val = pte_val(__ptep_get(pte));
if (enable)
val |= _PAGE_PRESENT;
else
val &= ~_PAGE_PRESENT;
- set_pte(pte, __pte(val));
+ __set_pte(pte, __pte(val));
return 0;
}
@@ -466,5 +466,5 @@ bool kernel_page_present(struct page *page)
return true;
pte = pte_offset_kernel(pmd, addr);
- return pte_present(ptep_get(pte));
+ return pte_present(__ptep_get(pte));
}
diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c
index 9c4427d0b1874..43bd542c6ff02 100644
--- a/arch/riscv/mm/pgtable.c
+++ b/arch/riscv/mm/pgtable.c
@@ -5,14 +5,13 @@
#include <linux/kernel.h>
#include <linux/pgtable.h>
-int ptep_set_access_flags(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep,
- pte_t entry, int dirty)
+int __ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty)
{
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC)) {
- if (!pte_same(ptep_get(ptep), entry)) {
+ if (!pte_same(__ptep_get(ptep), entry)) {
__set_pte_at(vma->vm_mm, ptep, entry);
- /* Here only not svadu is impacted */
flush_tlb_page(vma, address);
return true;
}
@@ -20,7 +19,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
return false;
}
- if (!pte_same(ptep_get(ptep), entry))
+ if (!pte_same(__ptep_get(ptep), entry))
__set_pte_at(vma->vm_mm, ptep, entry);
/*
* update_mmu_cache will unconditionally execute, handling both
@@ -29,13 +28,30 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
return true;
}
-bool ptep_test_and_clear_young(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep)
+int ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty)
{
- if (!pte_young(ptep_get(ptep)))
+ return __ptep_set_access_flags(vma, address, ptep, entry, dirty);
+}
+
+bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address,
+ pte_t *ptep)
+{
+ if (!pte_young(__ptep_get(ptep)))
return false;
+
return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep));
}
+EXPORT_SYMBOL_GPL(__ptep_test_and_clear_young);
+
+bool ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address,
+ pte_t *ptep)
+{
+ return __ptep_test_and_clear_young(vma, address, ptep);
+}
EXPORT_SYMBOL_GPL(ptep_test_and_clear_young);
#ifdef CONFIG_64BIT
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 02/15] riscv: switch arch page-table users to raw PTE helpers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Convert RISC-V internal page-table walkers and mutators over to the raw
PTE helper layer so they keep operating on the encoded hardware entries.
This keeps arch-private users independent from the Svnapot-aware public
PTE view that will be added on top for generic mm paths.
Add a raw get-and-clear helper without page-table-check side effects so
NAPOT-aware arch callers can report the effective non-NAPOT leaf view
explicitly when tearing down contiguous mappings.
Also update the hugetlb Svnapot clear/rewrite paths so they keep operating
on the contiguous block with page-table-check behavior consistent with the
raw-helper model.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/kfence.h | 4 +-
arch/riscv/include/asm/pgtable.h | 14 +++-
arch/riscv/kernel/efi.c | 4 +-
arch/riscv/kernel/hibernate.c | 3 +-
arch/riscv/kvm/gstage.c | 25 +++---
arch/riscv/kvm/mmu.c | 4 +-
arch/riscv/mm/fault.c | 4 +-
arch/riscv/mm/hugetlbpage.c | 135 ++++++++++++++++++++-----------
arch/riscv/mm/init.c | 8 +-
arch/riscv/mm/kasan_init.c | 16 ++--
arch/riscv/mm/pageattr.c | 12 +--
arch/riscv/mm/pgtable.c | 34 +++++---
12 files changed, 165 insertions(+), 98 deletions(-)
diff --git a/arch/riscv/include/asm/kfence.h b/arch/riscv/include/asm/kfence.h
index 29cb3a6ee113d..8325962a9d7e8 100644
--- a/arch/riscv/include/asm/kfence.h
+++ b/arch/riscv/include/asm/kfence.h
@@ -19,9 +19,9 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect)
pte_t *pte = virt_to_kpte(addr);
if (protect) {
- set_pte(pte, __pte(pte_val(ptep_get(pte)) & ~_PAGE_PRESENT));
+ __set_pte(pte, __pte(pte_val(__ptep_get(pte)) & ~_PAGE_PRESENT));
} else {
- set_pte(pte, __pte(pte_val(ptep_get(pte)) | _PAGE_PRESENT));
+ __set_pte(pte, __pte(pte_val(__ptep_get(pte)) | _PAGE_PRESENT));
mark_new_valid_map();
}
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 8ca5da9534967..a0f64e980e981 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -9,7 +9,6 @@
#include <linux/mmzone.h>
#include <linux/sizes.h>
-#include <asm/cmpxchg.h>
#include <asm/pgtable-bits.h>
#ifndef CONFIG_MMU
@@ -124,6 +123,7 @@
#ifndef __ASSEMBLER__
+#include <asm/cmpxchg.h>
#include <asm/page.h>
#include <asm/tlbflush.h>
#include <linux/mm_types.h>
@@ -468,6 +468,11 @@ static inline pte_t pte_mkdirty(pte_t pte)
return __pte(pte_val(pte) | _PAGE_DIRTY | _PAGE_SOFT_DIRTY);
}
+static inline pte_t riscv_pte_mkhwdirty(pte_t pte)
+{
+ return __pte(pte_val(pte) | _PAGE_DIRTY);
+}
+
static inline pte_t pte_mkclean(pte_t pte)
{
return __pte(pte_val(pte) & ~(_PAGE_DIRTY));
@@ -756,6 +761,13 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm,
#define __ptep_get_and_clear __ptep_get_and_clear
+static inline pte_t __ptep_get_and_clear_noptc(pte_t *ptep)
+{
+ return __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));
+}
+
+#define __ptep_get_and_clear_noptc __ptep_get_and_clear_noptc
+
static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep)
{
diff --git a/arch/riscv/kernel/efi.c b/arch/riscv/kernel/efi.c
index 2d3cc57b45352..81ea2546d2922 100644
--- a/arch/riscv/kernel/efi.c
+++ b/arch/riscv/kernel/efi.c
@@ -60,7 +60,7 @@ int __init efi_create_mapping(struct mm_struct *mm, efi_memory_desc_t *md)
static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data)
{
efi_memory_desc_t *md = data;
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
unsigned long val;
if (md->attribute & EFI_MEMORY_RO) {
@@ -72,7 +72,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data)
val = pte_val(pte) & ~_PAGE_EXEC;
pte = __pte(val);
}
- set_pte(ptep, pte);
+ __set_pte(ptep, pte);
return 0;
}
diff --git a/arch/riscv/kernel/hibernate.c b/arch/riscv/kernel/hibernate.c
index 982843828adb7..c7d5c7d4bbcde 100644
--- a/arch/riscv/kernel/hibernate.c
+++ b/arch/riscv/kernel/hibernate.c
@@ -186,7 +186,8 @@ static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long
pte_t pte = READ_ONCE(*src_ptep);
if (pte_present(pte))
- set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot)));
+ __set_pte(dst_ptep,
+ __pte(pte_val(pte) | pgprot_val(prot)));
} while (dst_ptep++, src_ptep++, start += PAGE_SIZE, start < end);
return 0;
diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c
index c4c3b79567f10..67b2c75f5cec1 100644
--- a/arch/riscv/kvm/gstage.c
+++ b/arch/riscv/kvm/gstage.c
@@ -88,7 +88,7 @@ bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage, gpa_t addr,
*ptep_level = current_level;
ptep = (pte_t *)gstage->pgd;
ptep = &ptep[gstage_pte_index(gstage, addr, current_level)];
- while (ptep && pte_val(ptep_get(ptep))) {
+ while (ptep && pte_val(__ptep_get(ptep))) {
if (gstage_pte_leaf(ptep)) {
*ptep_level = current_level;
*ptepp = ptep;
@@ -98,7 +98,7 @@ bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage, gpa_t addr,
if (current_level) {
current_level--;
*ptep_level = current_level;
- ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep));
ptep = &ptep[gstage_pte_index(gstage, addr, current_level)];
} else {
ptep = NULL;
@@ -152,18 +152,18 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
if (gstage_pte_leaf(ptep))
return -EEXIST;
- if (!pte_val(ptep_get(ptep))) {
+ if (!pte_val(__ptep_get(ptep))) {
if (!pcache)
return -ENOMEM;
next_ptep = kvm_mmu_memory_cache_alloc(pcache);
if (!next_ptep)
return -ENOMEM;
- set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)),
- __pgprot(_PAGE_TABLE)));
+ __set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)),
+ __pgprot(_PAGE_TABLE)));
} else {
if (gstage_pte_leaf(ptep))
return -EEXIST;
- next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ next_ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep));
}
current_level--;
@@ -171,7 +171,7 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
}
if (pte_val(*ptep) != pte_val(map->pte)) {
- set_pte(ptep, map->pte);
+ __set_pte(ptep, map->pte);
if (gstage_pte_leaf(ptep))
gstage_tlb_flush(gstage, current_level, map->addr);
}
@@ -371,18 +371,18 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
WARN_ON(addr & (page_size - 1));
- if (!pte_val(ptep_get(ptep)))
+ if (!pte_val(__ptep_get(ptep)))
return false;
if (ptep_level && !gstage_pte_leaf(ptep)) {
- next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ next_ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep));
next_ptep_level = ptep_level - 1;
ret = gstage_level_to_page_size(gstage, next_ptep_level, &next_page_size);
if (ret)
return false;
if (op == GSTAGE_OP_CLEAR)
- set_pte(ptep, __pte(0));
+ __set_pte(ptep, __pte(0));
for (i = 0; i < PTRS_PER_PTE; i++)
flush |= kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size,
&next_ptep[i], next_ptep_level, op);
@@ -391,9 +391,10 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
} else {
old_pte = *ptep;
if (op == GSTAGE_OP_CLEAR)
- set_pte(ptep, __pte(0));
+ __set_pte(ptep, __pte(0));
else if (op == GSTAGE_OP_WP)
- set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE));
+ __set_pte(ptep,
+ __pte(pte_val(__ptep_get(ptep)) & ~_PAGE_WRITE));
if (pte_val(*ptep) != pte_val(old_pte))
flush = true;
}
diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c
index 082f9b2617338..48dfc88d8cea6 100644
--- a/arch/riscv/kvm/mmu.c
+++ b/arch/riscv/kvm/mmu.c
@@ -278,7 +278,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
&ptep, &ptep_level))
return false;
- return ptep_test_and_clear_young(NULL, 0, ptep);
+ return __ptep_test_and_clear_young(NULL, 0, ptep);
}
bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
@@ -298,7 +298,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
&ptep, &ptep_level))
return false;
- return pte_young(ptep_get(ptep));
+ return pte_young(__ptep_get(ptep));
}
static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot,
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 04ed6f8acae4f..0b725ed723023 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -69,7 +69,7 @@ static void show_pte(unsigned long addr)
if (!ptep)
goto out;
- pte = ptep_get(ptep);
+ pte = __ptep_get(ptep);
pr_cont(", pte=%016lx", pte_val(pte));
pte_unmap(ptep);
out:
@@ -231,7 +231,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a
* silently loop forever.
*/
pte_k = pte_offset_kernel(pmd_k, addr);
- if (!pte_present(ptep_get(pte_k))) {
+ if (!pte_present(__ptep_get(pte_k))) {
no_context(regs, addr);
return;
}
diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c
index a6d217112cf46..5ededac02161d 100644
--- a/arch/riscv/mm/hugetlbpage.c
+++ b/arch/riscv/mm/hugetlbpage.c
@@ -7,7 +7,7 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
{
unsigned long pte_num;
int i;
- pte_t orig_pte = ptep_get(ptep);
+ pte_t orig_pte = __ptep_get(ptep);
if (!pte_present(orig_pte) || !pte_napot(orig_pte))
return orig_pte;
@@ -15,10 +15,10 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
pte_num = napot_pte_num(napot_cont_order(orig_pte));
for (i = 0; i < pte_num; i++, ptep++) {
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
if (pte_dirty(pte))
- orig_pte = pte_mkdirty(orig_pte);
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
if (pte_young(pte))
orig_pte = pte_mkyoung(orig_pte);
@@ -74,7 +74,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm,
out:
if (pte) {
- pte_t pteval = ptep_get_lockless(pte);
+ pte_t pteval = __ptep_get_lockless(pte);
WARN_ON_ONCE(pte_present(pteval) && !pte_huge(pteval));
}
@@ -145,28 +145,52 @@ unsigned long hugetlb_mask_last_page(struct hstate *h)
return 0UL;
}
+static unsigned long napot_hugetlb_block_addr(pte_t pte, unsigned long addr)
+{
+ unsigned long order;
+
+ if (!pte_napot(pte))
+ return addr;
+
+ order = napot_cont_order(pte);
+
+ return addr & napot_cont_mask(order);
+}
+
+static pte_t napot_hugetlb_get_and_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ pte_t pte;
+
+ pte = __ptep_get_and_clear_noptc(ptep);
+ page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
+
+ return pte;
+}
+
static pte_t get_clear_contig(struct mm_struct *mm,
unsigned long addr,
pte_t *ptep,
unsigned long ncontig)
{
- pte_t pte, tmp_pte;
- bool present;
-
- pte = ptep_get_and_clear(mm, addr, ptep);
- present = pte_present(pte);
- while (--ncontig) {
- ptep++;
- addr += PAGE_SIZE;
- tmp_pte = ptep_get_and_clear(mm, addr, ptep);
- if (present) {
- if (pte_dirty(tmp_pte))
- pte = pte_mkdirty(pte);
- if (pte_young(tmp_pte))
- pte = pte_mkyoung(pte);
- }
+ pte_t orig_pte = __ptep_get(ptep);
+ unsigned long i;
+
+ addr = napot_hugetlb_block_addr(orig_pte, addr);
+ if (pte_napot(orig_pte))
+ ptep = huge_pte_offset(mm, addr,
+ napot_cont_size(napot_cont_order(orig_pte)));
+
+ for (i = 0; i < ncontig; i++, addr += PAGE_SIZE, ptep++) {
+ pte_t pte = napot_hugetlb_get_and_clear(mm, addr, ptep);
+
+ if (pte_dirty(pte))
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
+ if (pte_young(pte))
+ orig_pte = pte_mkyoung(orig_pte);
}
- return pte;
+
+ return orig_pte;
}
static pte_t get_clear_contig_flush(struct mm_struct *mm,
@@ -174,10 +198,13 @@ static pte_t get_clear_contig_flush(struct mm_struct *mm,
pte_t *ptep,
unsigned long pte_num)
{
+ pte_t pte = __ptep_get(ptep);
pte_t orig_pte = get_clear_contig(mm, addr, ptep, pte_num);
struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0);
bool valid = !pte_none(orig_pte);
+ addr = napot_hugetlb_block_addr(pte, addr);
+
if (valid)
flush_tlb_range(&vma, addr, addr + (PAGE_SIZE * pte_num));
@@ -207,14 +234,31 @@ static void clear_flush(struct mm_struct *mm,
unsigned long ncontig)
{
struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0);
+ pte_t pte = __ptep_get(ptep);
unsigned long i, saddr = addr;
+ addr = napot_hugetlb_block_addr(pte, addr);
+ if (pte_napot(pte))
+ ptep = huge_pte_offset(mm, addr,
+ napot_cont_size(napot_cont_order(pte)));
+ saddr = addr;
+
for (i = 0; i < ncontig; i++, addr += pgsize, ptep++)
- ptep_get_and_clear(mm, addr, ptep);
+ napot_hugetlb_get_and_clear(mm, addr, ptep);
flush_tlb_range(&vma, saddr, addr);
}
+static void set_huge_napot_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte, unsigned long pte_num)
+{
+ unsigned long i;
+
+ page_table_check_ptes_set(mm, addr, ptep, pte, pte_num);
+ for (i = 0; i < pte_num; i++)
+ __set_pte_at(mm, ptep + i, pte);
+}
+
static int num_contig_ptes_from_size(unsigned long sz, size_t *pgsize)
{
unsigned long hugepage_shift;
@@ -256,19 +300,18 @@ void set_huge_pte_at(struct mm_struct *mm,
if (!pte_present(pte)) {
for (i = 0; i < pte_num; i++, ptep++, addr += pgsize)
- set_ptes(mm, addr, ptep, pte, 1);
+ __set_ptes(mm, addr, ptep, pte, 1);
return;
}
if (!pte_napot(pte)) {
- set_ptes(mm, addr, ptep, pte, 1);
+ __set_ptes(mm, addr, ptep, pte, 1);
return;
}
clear_flush(mm, addr, ptep, pgsize, pte_num);
- for (i = 0; i < pte_num; i++, ptep++, addr += pgsize)
- set_pte_at(mm, addr, ptep, pte);
+ set_huge_napot_ptes(mm, addr, ptep, pte, pte_num);
}
int huge_ptep_set_access_flags(struct vm_area_struct *vma,
@@ -280,10 +323,10 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,
struct mm_struct *mm = vma->vm_mm;
unsigned long order;
pte_t orig_pte;
- int i, pte_num;
+ int pte_num;
if (!pte_napot(pte))
- return ptep_set_access_flags(vma, addr, ptep, pte, dirty);
+ return __ptep_set_access_flags(vma, addr, ptep, pte, dirty);
order = napot_cont_order(pte);
pte_num = napot_pte_num(order);
@@ -291,13 +334,12 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,
orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num);
if (pte_dirty(orig_pte))
- pte = pte_mkdirty(pte);
+ pte = riscv_pte_mkhwdirty(pte);
if (pte_young(orig_pte))
pte = pte_mkyoung(pte);
- for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++)
- set_pte_at(mm, addr, ptep, pte);
+ set_huge_napot_ptes(mm, addr, ptep, pte, pte_num);
return true;
}
@@ -306,14 +348,13 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm,
unsigned long addr,
pte_t *ptep, unsigned long sz)
{
- size_t pgsize;
- pte_t orig_pte = ptep_get(ptep);
+ pte_t orig_pte = __ptep_get(ptep);
int pte_num;
if (!pte_napot(orig_pte))
- return ptep_get_and_clear(mm, addr, ptep);
+ return __ptep_get_and_clear(mm, addr, ptep);
- pte_num = num_contig_ptes_from_size(sz, &pgsize);
+ pte_num = napot_pte_num(napot_cont_order(orig_pte));
return get_clear_contig(mm, addr, ptep, pte_num);
}
@@ -322,13 +363,13 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm,
unsigned long addr,
pte_t *ptep)
{
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
unsigned long order;
pte_t orig_pte;
- int i, pte_num;
+ int pte_num;
if (!pte_napot(pte)) {
- ptep_set_wrprotect(mm, addr, ptep);
+ __ptep_set_wrprotect(mm, addr, ptep);
return;
}
@@ -339,19 +380,18 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm,
orig_pte = pte_wrprotect(orig_pte);
- for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++)
- set_pte_at(mm, addr, ptep, orig_pte);
+ set_huge_napot_ptes(mm, addr, ptep, orig_pte, pte_num);
}
pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,
unsigned long addr,
pte_t *ptep)
{
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
int pte_num;
if (!pte_napot(pte))
- return ptep_clear_flush(vma, addr, ptep);
+ return __ptep_clear_flush(vma, addr, ptep);
pte_num = napot_pte_num(napot_cont_order(pte));
@@ -363,19 +403,16 @@ void huge_pte_clear(struct mm_struct *mm,
pte_t *ptep,
unsigned long sz)
{
- size_t pgsize;
- pte_t pte = ptep_get(ptep);
- int i, pte_num;
+ pte_t pte = __ptep_get(ptep);
+ int pte_num;
if (!pte_napot(pte)) {
- pte_clear(mm, addr, ptep);
+ __pte_clear(mm, addr, ptep);
return;
}
- pte_num = num_contig_ptes_from_size(sz, &pgsize);
-
- for (i = 0; i < pte_num; i++, addr += pgsize, ptep++)
- pte_clear(mm, addr, ptep);
+ pte_num = napot_pte_num(napot_cont_order(pte));
+ get_clear_contig(mm, addr, ptep, pte_num);
}
static bool is_napot_size(unsigned long size)
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index e2be64bacf160..d44bb7db22b43 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -378,9 +378,9 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot)
ptep = &fixmap_pte[pte_index(addr)];
if (pgprot_val(prot))
- set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot));
+ __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot));
else
- pte_clear(&init_mm, addr, ptep);
+ __pte_clear(&init_mm, addr, ptep);
local_flush_tlb_page(addr);
}
@@ -1571,11 +1571,11 @@ static void __meminit remove_pte_mapping(pte_t *pte_base, unsigned long addr, un
next = end;
ptep = pte_base + pte_index(addr);
- pte = ptep_get(ptep);
+ pte = __ptep_get(ptep);
if (!pte_present(*ptep))
continue;
- pte_clear(&init_mm, addr, ptep);
+ __pte_clear(&init_mm, addr, ptep);
if (is_vmemmap)
free_vmemmap_storage(pte_page(pte), PAGE_SIZE, altmap);
}
diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c
index 1f3aa9611187f..c5d5513b206ca 100644
--- a/arch/riscv/mm/kasan_init.c
+++ b/arch/riscv/mm/kasan_init.c
@@ -39,9 +39,9 @@ static void __init kasan_populate_pte(pmd_t *pmd, unsigned long vaddr, unsigned
ptep = pte_offset_kernel(pmd, vaddr);
do {
- if (pte_none(ptep_get(ptep))) {
+ if (pte_none(__ptep_get(ptep))) {
phys_addr = memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE);
- set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL));
+ __set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL));
memset(__va(phys_addr), KASAN_SHADOW_INIT, PAGE_SIZE);
}
} while (ptep++, vaddr += PAGE_SIZE, vaddr != end);
@@ -327,8 +327,8 @@ asmlinkage void __init kasan_early_init(void)
KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT)));
for (i = 0; i < PTRS_PER_PTE; ++i)
- set_pte(kasan_early_shadow_pte + i,
- pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL));
+ __set_pte(kasan_early_shadow_pte + i,
+ pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL));
for (i = 0; i < PTRS_PER_PMD; ++i)
set_pmd(kasan_early_shadow_pmd + i,
@@ -520,10 +520,10 @@ void __init kasan_init(void)
kasan_mem_to_shadow((const void *)MODULES_VADDR + SZ_2G));
for (i = 0; i < PTRS_PER_PTE; i++)
- set_pte(&kasan_early_shadow_pte[i],
- mk_pte(virt_to_page(kasan_early_shadow_page),
- __pgprot(_PAGE_PRESENT | _PAGE_READ |
- _PAGE_ACCESSED)));
+ __set_pte(&kasan_early_shadow_pte[i],
+ mk_pte(virt_to_page(kasan_early_shadow_page),
+ __pgprot(_PAGE_PRESENT | _PAGE_READ |
+ _PAGE_ACCESSED)));
memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE);
init_task.kasan_depth = 0;
diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c
index 3f76db3d27699..e0271e2a0b295 100644
--- a/arch/riscv/mm/pageattr.c
+++ b/arch/riscv/mm/pageattr.c
@@ -68,10 +68,10 @@ static int pageattr_pmd_entry(pmd_t *pmd, unsigned long addr,
static int pageattr_pte_entry(pte_t *pte, unsigned long addr,
unsigned long next, struct mm_walk *walk)
{
- pte_t val = ptep_get(pte);
+ pte_t val = __ptep_get(pte);
val = __pte(set_pageattr_masks(pte_val(val), walk));
- set_pte(pte, val);
+ __set_pte(pte, val);
return 0;
}
@@ -121,7 +121,7 @@ static int __split_linear_mapping_pmd(pud_t *pudp,
ptep_new = (pte_t *)page_address(pte_page);
for (i = 0; i < PTRS_PER_PTE; ++i, ++ptep_new)
- set_pte(ptep_new, pfn_pte(pfn + i, prot));
+ __set_pte(ptep_new, pfn_pte(pfn + i, prot));
smp_wmb();
@@ -406,14 +406,14 @@ static int debug_pagealloc_set_page(pte_t *pte, unsigned long addr, void *data)
{
int enable = *(int *)data;
- unsigned long val = pte_val(ptep_get(pte));
+ unsigned long val = pte_val(__ptep_get(pte));
if (enable)
val |= _PAGE_PRESENT;
else
val &= ~_PAGE_PRESENT;
- set_pte(pte, __pte(val));
+ __set_pte(pte, __pte(val));
return 0;
}
@@ -466,5 +466,5 @@ bool kernel_page_present(struct page *page)
return true;
pte = pte_offset_kernel(pmd, addr);
- return pte_present(ptep_get(pte));
+ return pte_present(__ptep_get(pte));
}
diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c
index 9c4427d0b1874..43bd542c6ff02 100644
--- a/arch/riscv/mm/pgtable.c
+++ b/arch/riscv/mm/pgtable.c
@@ -5,14 +5,13 @@
#include <linux/kernel.h>
#include <linux/pgtable.h>
-int ptep_set_access_flags(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep,
- pte_t entry, int dirty)
+int __ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty)
{
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC)) {
- if (!pte_same(ptep_get(ptep), entry)) {
+ if (!pte_same(__ptep_get(ptep), entry)) {
__set_pte_at(vma->vm_mm, ptep, entry);
- /* Here only not svadu is impacted */
flush_tlb_page(vma, address);
return true;
}
@@ -20,7 +19,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
return false;
}
- if (!pte_same(ptep_get(ptep), entry))
+ if (!pte_same(__ptep_get(ptep), entry))
__set_pte_at(vma->vm_mm, ptep, entry);
/*
* update_mmu_cache will unconditionally execute, handling both
@@ -29,13 +28,30 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
return true;
}
-bool ptep_test_and_clear_young(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep)
+int ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty)
{
- if (!pte_young(ptep_get(ptep)))
+ return __ptep_set_access_flags(vma, address, ptep, entry, dirty);
+}
+
+bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address,
+ pte_t *ptep)
+{
+ if (!pte_young(__ptep_get(ptep)))
return false;
+
return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep));
}
+EXPORT_SYMBOL_GPL(__ptep_test_and_clear_young);
+
+bool ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address,
+ pte_t *ptep)
+{
+ return __ptep_test_and_clear_young(vma, address, ptep);
+}
EXPORT_SYMBOL_GPL(ptep_test_and_clear_young);
#ifdef CONFIG_64BIT
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 02/15] riscv: switch arch page-table users to raw PTE helpers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Convert RISC-V internal page-table walkers and mutators over to the raw
PTE helper layer so they keep operating on the encoded hardware entries.
This keeps arch-private users independent from the Svnapot-aware public
PTE view that will be added on top for generic mm paths.
Add a raw get-and-clear helper without page-table-check side effects so
NAPOT-aware arch callers can report the effective non-NAPOT leaf view
explicitly when tearing down contiguous mappings.
Also update the hugetlb Svnapot clear/rewrite paths so they keep operating
on the contiguous block with page-table-check behavior consistent with the
raw-helper model.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/kfence.h | 4 +-
arch/riscv/include/asm/pgtable.h | 14 +++-
arch/riscv/kernel/efi.c | 4 +-
arch/riscv/kernel/hibernate.c | 3 +-
arch/riscv/kvm/gstage.c | 25 +++---
arch/riscv/kvm/mmu.c | 4 +-
arch/riscv/mm/fault.c | 4 +-
arch/riscv/mm/hugetlbpage.c | 135 ++++++++++++++++++++-----------
arch/riscv/mm/init.c | 8 +-
arch/riscv/mm/kasan_init.c | 16 ++--
arch/riscv/mm/pageattr.c | 12 +--
arch/riscv/mm/pgtable.c | 34 +++++---
12 files changed, 165 insertions(+), 98 deletions(-)
diff --git a/arch/riscv/include/asm/kfence.h b/arch/riscv/include/asm/kfence.h
index 29cb3a6ee113d..8325962a9d7e8 100644
--- a/arch/riscv/include/asm/kfence.h
+++ b/arch/riscv/include/asm/kfence.h
@@ -19,9 +19,9 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect)
pte_t *pte = virt_to_kpte(addr);
if (protect) {
- set_pte(pte, __pte(pte_val(ptep_get(pte)) & ~_PAGE_PRESENT));
+ __set_pte(pte, __pte(pte_val(__ptep_get(pte)) & ~_PAGE_PRESENT));
} else {
- set_pte(pte, __pte(pte_val(ptep_get(pte)) | _PAGE_PRESENT));
+ __set_pte(pte, __pte(pte_val(__ptep_get(pte)) | _PAGE_PRESENT));
mark_new_valid_map();
}
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 8ca5da9534967..a0f64e980e981 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -9,7 +9,6 @@
#include <linux/mmzone.h>
#include <linux/sizes.h>
-#include <asm/cmpxchg.h>
#include <asm/pgtable-bits.h>
#ifndef CONFIG_MMU
@@ -124,6 +123,7 @@
#ifndef __ASSEMBLER__
+#include <asm/cmpxchg.h>
#include <asm/page.h>
#include <asm/tlbflush.h>
#include <linux/mm_types.h>
@@ -468,6 +468,11 @@ static inline pte_t pte_mkdirty(pte_t pte)
return __pte(pte_val(pte) | _PAGE_DIRTY | _PAGE_SOFT_DIRTY);
}
+static inline pte_t riscv_pte_mkhwdirty(pte_t pte)
+{
+ return __pte(pte_val(pte) | _PAGE_DIRTY);
+}
+
static inline pte_t pte_mkclean(pte_t pte)
{
return __pte(pte_val(pte) & ~(_PAGE_DIRTY));
@@ -756,6 +761,13 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm,
#define __ptep_get_and_clear __ptep_get_and_clear
+static inline pte_t __ptep_get_and_clear_noptc(pte_t *ptep)
+{
+ return __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));
+}
+
+#define __ptep_get_and_clear_noptc __ptep_get_and_clear_noptc
+
static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep)
{
diff --git a/arch/riscv/kernel/efi.c b/arch/riscv/kernel/efi.c
index 2d3cc57b45352..81ea2546d2922 100644
--- a/arch/riscv/kernel/efi.c
+++ b/arch/riscv/kernel/efi.c
@@ -60,7 +60,7 @@ int __init efi_create_mapping(struct mm_struct *mm, efi_memory_desc_t *md)
static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data)
{
efi_memory_desc_t *md = data;
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
unsigned long val;
if (md->attribute & EFI_MEMORY_RO) {
@@ -72,7 +72,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data)
val = pte_val(pte) & ~_PAGE_EXEC;
pte = __pte(val);
}
- set_pte(ptep, pte);
+ __set_pte(ptep, pte);
return 0;
}
diff --git a/arch/riscv/kernel/hibernate.c b/arch/riscv/kernel/hibernate.c
index 982843828adb7..c7d5c7d4bbcde 100644
--- a/arch/riscv/kernel/hibernate.c
+++ b/arch/riscv/kernel/hibernate.c
@@ -186,7 +186,8 @@ static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long
pte_t pte = READ_ONCE(*src_ptep);
if (pte_present(pte))
- set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot)));
+ __set_pte(dst_ptep,
+ __pte(pte_val(pte) | pgprot_val(prot)));
} while (dst_ptep++, src_ptep++, start += PAGE_SIZE, start < end);
return 0;
diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c
index c4c3b79567f10..67b2c75f5cec1 100644
--- a/arch/riscv/kvm/gstage.c
+++ b/arch/riscv/kvm/gstage.c
@@ -88,7 +88,7 @@ bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage, gpa_t addr,
*ptep_level = current_level;
ptep = (pte_t *)gstage->pgd;
ptep = &ptep[gstage_pte_index(gstage, addr, current_level)];
- while (ptep && pte_val(ptep_get(ptep))) {
+ while (ptep && pte_val(__ptep_get(ptep))) {
if (gstage_pte_leaf(ptep)) {
*ptep_level = current_level;
*ptepp = ptep;
@@ -98,7 +98,7 @@ bool kvm_riscv_gstage_get_leaf(struct kvm_gstage *gstage, gpa_t addr,
if (current_level) {
current_level--;
*ptep_level = current_level;
- ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep));
ptep = &ptep[gstage_pte_index(gstage, addr, current_level)];
} else {
ptep = NULL;
@@ -152,18 +152,18 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
if (gstage_pte_leaf(ptep))
return -EEXIST;
- if (!pte_val(ptep_get(ptep))) {
+ if (!pte_val(__ptep_get(ptep))) {
if (!pcache)
return -ENOMEM;
next_ptep = kvm_mmu_memory_cache_alloc(pcache);
if (!next_ptep)
return -ENOMEM;
- set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)),
- __pgprot(_PAGE_TABLE)));
+ __set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)),
+ __pgprot(_PAGE_TABLE)));
} else {
if (gstage_pte_leaf(ptep))
return -EEXIST;
- next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ next_ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep));
}
current_level--;
@@ -171,7 +171,7 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
}
if (pte_val(*ptep) != pte_val(map->pte)) {
- set_pte(ptep, map->pte);
+ __set_pte(ptep, map->pte);
if (gstage_pte_leaf(ptep))
gstage_tlb_flush(gstage, current_level, map->addr);
}
@@ -371,18 +371,18 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
WARN_ON(addr & (page_size - 1));
- if (!pte_val(ptep_get(ptep)))
+ if (!pte_val(__ptep_get(ptep)))
return false;
if (ptep_level && !gstage_pte_leaf(ptep)) {
- next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
+ next_ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep));
next_ptep_level = ptep_level - 1;
ret = gstage_level_to_page_size(gstage, next_ptep_level, &next_page_size);
if (ret)
return false;
if (op == GSTAGE_OP_CLEAR)
- set_pte(ptep, __pte(0));
+ __set_pte(ptep, __pte(0));
for (i = 0; i < PTRS_PER_PTE; i++)
flush |= kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size,
&next_ptep[i], next_ptep_level, op);
@@ -391,9 +391,10 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
} else {
old_pte = *ptep;
if (op == GSTAGE_OP_CLEAR)
- set_pte(ptep, __pte(0));
+ __set_pte(ptep, __pte(0));
else if (op == GSTAGE_OP_WP)
- set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE));
+ __set_pte(ptep,
+ __pte(pte_val(__ptep_get(ptep)) & ~_PAGE_WRITE));
if (pte_val(*ptep) != pte_val(old_pte))
flush = true;
}
diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c
index 082f9b2617338..48dfc88d8cea6 100644
--- a/arch/riscv/kvm/mmu.c
+++ b/arch/riscv/kvm/mmu.c
@@ -278,7 +278,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
&ptep, &ptep_level))
return false;
- return ptep_test_and_clear_young(NULL, 0, ptep);
+ return __ptep_test_and_clear_young(NULL, 0, ptep);
}
bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
@@ -298,7 +298,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
&ptep, &ptep_level))
return false;
- return pte_young(ptep_get(ptep));
+ return pte_young(__ptep_get(ptep));
}
static bool fault_supports_gstage_huge_mapping(struct kvm_memory_slot *memslot,
diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c
index 04ed6f8acae4f..0b725ed723023 100644
--- a/arch/riscv/mm/fault.c
+++ b/arch/riscv/mm/fault.c
@@ -69,7 +69,7 @@ static void show_pte(unsigned long addr)
if (!ptep)
goto out;
- pte = ptep_get(ptep);
+ pte = __ptep_get(ptep);
pr_cont(", pte=%016lx", pte_val(pte));
pte_unmap(ptep);
out:
@@ -231,7 +231,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a
* silently loop forever.
*/
pte_k = pte_offset_kernel(pmd_k, addr);
- if (!pte_present(ptep_get(pte_k))) {
+ if (!pte_present(__ptep_get(pte_k))) {
no_context(regs, addr);
return;
}
diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c
index a6d217112cf46..5ededac02161d 100644
--- a/arch/riscv/mm/hugetlbpage.c
+++ b/arch/riscv/mm/hugetlbpage.c
@@ -7,7 +7,7 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
{
unsigned long pte_num;
int i;
- pte_t orig_pte = ptep_get(ptep);
+ pte_t orig_pte = __ptep_get(ptep);
if (!pte_present(orig_pte) || !pte_napot(orig_pte))
return orig_pte;
@@ -15,10 +15,10 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
pte_num = napot_pte_num(napot_cont_order(orig_pte));
for (i = 0; i < pte_num; i++, ptep++) {
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
if (pte_dirty(pte))
- orig_pte = pte_mkdirty(orig_pte);
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
if (pte_young(pte))
orig_pte = pte_mkyoung(orig_pte);
@@ -74,7 +74,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm,
out:
if (pte) {
- pte_t pteval = ptep_get_lockless(pte);
+ pte_t pteval = __ptep_get_lockless(pte);
WARN_ON_ONCE(pte_present(pteval) && !pte_huge(pteval));
}
@@ -145,28 +145,52 @@ unsigned long hugetlb_mask_last_page(struct hstate *h)
return 0UL;
}
+static unsigned long napot_hugetlb_block_addr(pte_t pte, unsigned long addr)
+{
+ unsigned long order;
+
+ if (!pte_napot(pte))
+ return addr;
+
+ order = napot_cont_order(pte);
+
+ return addr & napot_cont_mask(order);
+}
+
+static pte_t napot_hugetlb_get_and_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ pte_t pte;
+
+ pte = __ptep_get_and_clear_noptc(ptep);
+ page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
+
+ return pte;
+}
+
static pte_t get_clear_contig(struct mm_struct *mm,
unsigned long addr,
pte_t *ptep,
unsigned long ncontig)
{
- pte_t pte, tmp_pte;
- bool present;
-
- pte = ptep_get_and_clear(mm, addr, ptep);
- present = pte_present(pte);
- while (--ncontig) {
- ptep++;
- addr += PAGE_SIZE;
- tmp_pte = ptep_get_and_clear(mm, addr, ptep);
- if (present) {
- if (pte_dirty(tmp_pte))
- pte = pte_mkdirty(pte);
- if (pte_young(tmp_pte))
- pte = pte_mkyoung(pte);
- }
+ pte_t orig_pte = __ptep_get(ptep);
+ unsigned long i;
+
+ addr = napot_hugetlb_block_addr(orig_pte, addr);
+ if (pte_napot(orig_pte))
+ ptep = huge_pte_offset(mm, addr,
+ napot_cont_size(napot_cont_order(orig_pte)));
+
+ for (i = 0; i < ncontig; i++, addr += PAGE_SIZE, ptep++) {
+ pte_t pte = napot_hugetlb_get_and_clear(mm, addr, ptep);
+
+ if (pte_dirty(pte))
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
+ if (pte_young(pte))
+ orig_pte = pte_mkyoung(orig_pte);
}
- return pte;
+
+ return orig_pte;
}
static pte_t get_clear_contig_flush(struct mm_struct *mm,
@@ -174,10 +198,13 @@ static pte_t get_clear_contig_flush(struct mm_struct *mm,
pte_t *ptep,
unsigned long pte_num)
{
+ pte_t pte = __ptep_get(ptep);
pte_t orig_pte = get_clear_contig(mm, addr, ptep, pte_num);
struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0);
bool valid = !pte_none(orig_pte);
+ addr = napot_hugetlb_block_addr(pte, addr);
+
if (valid)
flush_tlb_range(&vma, addr, addr + (PAGE_SIZE * pte_num));
@@ -207,14 +234,31 @@ static void clear_flush(struct mm_struct *mm,
unsigned long ncontig)
{
struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0);
+ pte_t pte = __ptep_get(ptep);
unsigned long i, saddr = addr;
+ addr = napot_hugetlb_block_addr(pte, addr);
+ if (pte_napot(pte))
+ ptep = huge_pte_offset(mm, addr,
+ napot_cont_size(napot_cont_order(pte)));
+ saddr = addr;
+
for (i = 0; i < ncontig; i++, addr += pgsize, ptep++)
- ptep_get_and_clear(mm, addr, ptep);
+ napot_hugetlb_get_and_clear(mm, addr, ptep);
flush_tlb_range(&vma, saddr, addr);
}
+static void set_huge_napot_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte, unsigned long pte_num)
+{
+ unsigned long i;
+
+ page_table_check_ptes_set(mm, addr, ptep, pte, pte_num);
+ for (i = 0; i < pte_num; i++)
+ __set_pte_at(mm, ptep + i, pte);
+}
+
static int num_contig_ptes_from_size(unsigned long sz, size_t *pgsize)
{
unsigned long hugepage_shift;
@@ -256,19 +300,18 @@ void set_huge_pte_at(struct mm_struct *mm,
if (!pte_present(pte)) {
for (i = 0; i < pte_num; i++, ptep++, addr += pgsize)
- set_ptes(mm, addr, ptep, pte, 1);
+ __set_ptes(mm, addr, ptep, pte, 1);
return;
}
if (!pte_napot(pte)) {
- set_ptes(mm, addr, ptep, pte, 1);
+ __set_ptes(mm, addr, ptep, pte, 1);
return;
}
clear_flush(mm, addr, ptep, pgsize, pte_num);
- for (i = 0; i < pte_num; i++, ptep++, addr += pgsize)
- set_pte_at(mm, addr, ptep, pte);
+ set_huge_napot_ptes(mm, addr, ptep, pte, pte_num);
}
int huge_ptep_set_access_flags(struct vm_area_struct *vma,
@@ -280,10 +323,10 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,
struct mm_struct *mm = vma->vm_mm;
unsigned long order;
pte_t orig_pte;
- int i, pte_num;
+ int pte_num;
if (!pte_napot(pte))
- return ptep_set_access_flags(vma, addr, ptep, pte, dirty);
+ return __ptep_set_access_flags(vma, addr, ptep, pte, dirty);
order = napot_cont_order(pte);
pte_num = napot_pte_num(order);
@@ -291,13 +334,12 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma,
orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num);
if (pte_dirty(orig_pte))
- pte = pte_mkdirty(pte);
+ pte = riscv_pte_mkhwdirty(pte);
if (pte_young(orig_pte))
pte = pte_mkyoung(pte);
- for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++)
- set_pte_at(mm, addr, ptep, pte);
+ set_huge_napot_ptes(mm, addr, ptep, pte, pte_num);
return true;
}
@@ -306,14 +348,13 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm,
unsigned long addr,
pte_t *ptep, unsigned long sz)
{
- size_t pgsize;
- pte_t orig_pte = ptep_get(ptep);
+ pte_t orig_pte = __ptep_get(ptep);
int pte_num;
if (!pte_napot(orig_pte))
- return ptep_get_and_clear(mm, addr, ptep);
+ return __ptep_get_and_clear(mm, addr, ptep);
- pte_num = num_contig_ptes_from_size(sz, &pgsize);
+ pte_num = napot_pte_num(napot_cont_order(orig_pte));
return get_clear_contig(mm, addr, ptep, pte_num);
}
@@ -322,13 +363,13 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm,
unsigned long addr,
pte_t *ptep)
{
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
unsigned long order;
pte_t orig_pte;
- int i, pte_num;
+ int pte_num;
if (!pte_napot(pte)) {
- ptep_set_wrprotect(mm, addr, ptep);
+ __ptep_set_wrprotect(mm, addr, ptep);
return;
}
@@ -339,19 +380,18 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm,
orig_pte = pte_wrprotect(orig_pte);
- for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++)
- set_pte_at(mm, addr, ptep, orig_pte);
+ set_huge_napot_ptes(mm, addr, ptep, orig_pte, pte_num);
}
pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,
unsigned long addr,
pte_t *ptep)
{
- pte_t pte = ptep_get(ptep);
+ pte_t pte = __ptep_get(ptep);
int pte_num;
if (!pte_napot(pte))
- return ptep_clear_flush(vma, addr, ptep);
+ return __ptep_clear_flush(vma, addr, ptep);
pte_num = napot_pte_num(napot_cont_order(pte));
@@ -363,19 +403,16 @@ void huge_pte_clear(struct mm_struct *mm,
pte_t *ptep,
unsigned long sz)
{
- size_t pgsize;
- pte_t pte = ptep_get(ptep);
- int i, pte_num;
+ pte_t pte = __ptep_get(ptep);
+ int pte_num;
if (!pte_napot(pte)) {
- pte_clear(mm, addr, ptep);
+ __pte_clear(mm, addr, ptep);
return;
}
- pte_num = num_contig_ptes_from_size(sz, &pgsize);
-
- for (i = 0; i < pte_num; i++, addr += pgsize, ptep++)
- pte_clear(mm, addr, ptep);
+ pte_num = napot_pte_num(napot_cont_order(pte));
+ get_clear_contig(mm, addr, ptep, pte_num);
}
static bool is_napot_size(unsigned long size)
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index e2be64bacf160..d44bb7db22b43 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -378,9 +378,9 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot)
ptep = &fixmap_pte[pte_index(addr)];
if (pgprot_val(prot))
- set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot));
+ __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot));
else
- pte_clear(&init_mm, addr, ptep);
+ __pte_clear(&init_mm, addr, ptep);
local_flush_tlb_page(addr);
}
@@ -1571,11 +1571,11 @@ static void __meminit remove_pte_mapping(pte_t *pte_base, unsigned long addr, un
next = end;
ptep = pte_base + pte_index(addr);
- pte = ptep_get(ptep);
+ pte = __ptep_get(ptep);
if (!pte_present(*ptep))
continue;
- pte_clear(&init_mm, addr, ptep);
+ __pte_clear(&init_mm, addr, ptep);
if (is_vmemmap)
free_vmemmap_storage(pte_page(pte), PAGE_SIZE, altmap);
}
diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c
index 1f3aa9611187f..c5d5513b206ca 100644
--- a/arch/riscv/mm/kasan_init.c
+++ b/arch/riscv/mm/kasan_init.c
@@ -39,9 +39,9 @@ static void __init kasan_populate_pte(pmd_t *pmd, unsigned long vaddr, unsigned
ptep = pte_offset_kernel(pmd, vaddr);
do {
- if (pte_none(ptep_get(ptep))) {
+ if (pte_none(__ptep_get(ptep))) {
phys_addr = memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE);
- set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL));
+ __set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL));
memset(__va(phys_addr), KASAN_SHADOW_INIT, PAGE_SIZE);
}
} while (ptep++, vaddr += PAGE_SIZE, vaddr != end);
@@ -327,8 +327,8 @@ asmlinkage void __init kasan_early_init(void)
KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT)));
for (i = 0; i < PTRS_PER_PTE; ++i)
- set_pte(kasan_early_shadow_pte + i,
- pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL));
+ __set_pte(kasan_early_shadow_pte + i,
+ pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL));
for (i = 0; i < PTRS_PER_PMD; ++i)
set_pmd(kasan_early_shadow_pmd + i,
@@ -520,10 +520,10 @@ void __init kasan_init(void)
kasan_mem_to_shadow((const void *)MODULES_VADDR + SZ_2G));
for (i = 0; i < PTRS_PER_PTE; i++)
- set_pte(&kasan_early_shadow_pte[i],
- mk_pte(virt_to_page(kasan_early_shadow_page),
- __pgprot(_PAGE_PRESENT | _PAGE_READ |
- _PAGE_ACCESSED)));
+ __set_pte(&kasan_early_shadow_pte[i],
+ mk_pte(virt_to_page(kasan_early_shadow_page),
+ __pgprot(_PAGE_PRESENT | _PAGE_READ |
+ _PAGE_ACCESSED)));
memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE);
init_task.kasan_depth = 0;
diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c
index 3f76db3d27699..e0271e2a0b295 100644
--- a/arch/riscv/mm/pageattr.c
+++ b/arch/riscv/mm/pageattr.c
@@ -68,10 +68,10 @@ static int pageattr_pmd_entry(pmd_t *pmd, unsigned long addr,
static int pageattr_pte_entry(pte_t *pte, unsigned long addr,
unsigned long next, struct mm_walk *walk)
{
- pte_t val = ptep_get(pte);
+ pte_t val = __ptep_get(pte);
val = __pte(set_pageattr_masks(pte_val(val), walk));
- set_pte(pte, val);
+ __set_pte(pte, val);
return 0;
}
@@ -121,7 +121,7 @@ static int __split_linear_mapping_pmd(pud_t *pudp,
ptep_new = (pte_t *)page_address(pte_page);
for (i = 0; i < PTRS_PER_PTE; ++i, ++ptep_new)
- set_pte(ptep_new, pfn_pte(pfn + i, prot));
+ __set_pte(ptep_new, pfn_pte(pfn + i, prot));
smp_wmb();
@@ -406,14 +406,14 @@ static int debug_pagealloc_set_page(pte_t *pte, unsigned long addr, void *data)
{
int enable = *(int *)data;
- unsigned long val = pte_val(ptep_get(pte));
+ unsigned long val = pte_val(__ptep_get(pte));
if (enable)
val |= _PAGE_PRESENT;
else
val &= ~_PAGE_PRESENT;
- set_pte(pte, __pte(val));
+ __set_pte(pte, __pte(val));
return 0;
}
@@ -466,5 +466,5 @@ bool kernel_page_present(struct page *page)
return true;
pte = pte_offset_kernel(pmd, addr);
- return pte_present(ptep_get(pte));
+ return pte_present(__ptep_get(pte));
}
diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c
index 9c4427d0b1874..43bd542c6ff02 100644
--- a/arch/riscv/mm/pgtable.c
+++ b/arch/riscv/mm/pgtable.c
@@ -5,14 +5,13 @@
#include <linux/kernel.h>
#include <linux/pgtable.h>
-int ptep_set_access_flags(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep,
- pte_t entry, int dirty)
+int __ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty)
{
if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC)) {
- if (!pte_same(ptep_get(ptep), entry)) {
+ if (!pte_same(__ptep_get(ptep), entry)) {
__set_pte_at(vma->vm_mm, ptep, entry);
- /* Here only not svadu is impacted */
flush_tlb_page(vma, address);
return true;
}
@@ -20,7 +19,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
return false;
}
- if (!pte_same(ptep_get(ptep), entry))
+ if (!pte_same(__ptep_get(ptep), entry))
__set_pte_at(vma->vm_mm, ptep, entry);
/*
* update_mmu_cache will unconditionally execute, handling both
@@ -29,13 +28,30 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
return true;
}
-bool ptep_test_and_clear_young(struct vm_area_struct *vma,
- unsigned long address, pte_t *ptep)
+int ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty)
{
- if (!pte_young(ptep_get(ptep)))
+ return __ptep_set_access_flags(vma, address, ptep, entry, dirty);
+}
+
+bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address,
+ pte_t *ptep)
+{
+ if (!pte_young(__ptep_get(ptep)))
return false;
+
return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep));
}
+EXPORT_SYMBOL_GPL(__ptep_test_and_clear_young);
+
+bool ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address,
+ pte_t *ptep)
+{
+ return __ptep_test_and_clear_young(vma, address, ptep);
+}
EXPORT_SYMBOL_GPL(ptep_test_and_clear_young);
#ifdef CONFIG_64BIT
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 03/15] riscv/mm: implement Svnapot contpte read-side helpers
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot keeps a contiguous mapping encoded in a block of page-table
entries, while generic MM code expects the public PTE getters to return
a normal per-page view for the address being queried. Without a dedicated
read path, exposing the raw NAPOT encoding through the public helpers
would leak arch-private representation details into generic MM users and
make folded mappings harder to integrate with existing PTE walkers.
Add Svnapot-aware contpte read-side helpers and route the public PTE
getters through them, so generic MM sees the logical sub-PTE for the
current page while RISC-V raw helpers continue to operate on the encoded
entries.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 50 +++++-
arch/riscv/mm/Makefile | 1 +
arch/riscv/mm/contpte.c | 273 +++++++++++++++++++++++++++++++
3 files changed, 322 insertions(+), 2 deletions(-)
create mode 100644 arch/riscv/mm/contpte.c
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index a0f64e980e981..fce6d04ea65db 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -312,6 +312,11 @@ static inline unsigned long pte_napot(pte_t pte)
return 0;
}
+static inline pte_t pte_mknapot(pte_t pte, unsigned int order)
+{
+ return pte;
+}
+
#endif /* CONFIG_RISCV_ISA_SVNAPOT */
/* Yields the page frame number (PFN) of a page table entry */
@@ -350,6 +355,11 @@ static inline int pte_present(pte_t pte)
return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE));
}
+static inline bool pte_present_napot(pte_t pte)
+{
+ return pte_present(pte) && pte_napot(pte);
+}
+
#define pte_accessible pte_accessible
static inline unsigned long pte_accessible(struct mm_struct *mm, pte_t a)
{
@@ -538,6 +548,13 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
#define pte_leaf_size(pte) (pte_napot(pte) ? \
napot_cont_size(napot_cont_order(pte)) :\
PAGE_SIZE)
+
+void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte);
+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte);
+pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte);
+pte_t napotpte_ptep_get_lockless(pte_t *ptep);
#endif
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
@@ -689,6 +706,37 @@ static inline pte_t __ptep_get_lockless(pte_t *ptep)
return __ptep_get(ptep);
}
+#ifdef CONFIG_RISCV_ISA_SVNAPOT
+
+#define ptep_get ptep_get
+static inline pte_t ptep_get(pte_t *ptep)
+{
+ pte_t pte = __ptep_get(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get(ptep, pte);
+}
+
+#define ptep_get_lockless ptep_get_lockless
+static inline pte_t ptep_get_lockless(pte_t *ptep)
+{
+ pte_t pte = __ptep_get_lockless(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get_lockless(ptep);
+}
+
+#else
+
+#define ptep_get __ptep_get
+#define ptep_get_lockless __ptep_get_lockless
+
+#endif
+
static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep,
pte_t pte, cydp_t flags)
@@ -822,8 +870,6 @@ static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
#define __ptep_clear_flush_young __ptep_clear_flush_young
-#define ptep_get __ptep_get
-#define ptep_get_lockless __ptep_get_lockless
#define set_pte __set_pte
#define set_ptes __set_ptes
diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile
index b916a68d324ad..fd260903cd340 100644
--- a/arch/riscv/mm/Makefile
+++ b/arch/riscv/mm/Makefile
@@ -19,6 +19,7 @@ obj-y += context.o
obj-y += pmem.o
obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o
+obj-$(CONFIG_RISCV_ISA_SVNAPOT) += contpte.o
obj-$(CONFIG_PTDUMP) += ptdump.o
obj-$(CONFIG_KASAN) += kasan_init.o
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
new file mode 100644
index 0000000000000..43ea76e424492
--- /dev/null
+++ b/arch/riscv/mm/contpte.c
@@ -0,0 +1,273 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/align.h>
+#include <linux/cpufeature.h>
+#include <linux/efi.h>
+#include <linux/export.h>
+#include <linux/mm.h>
+#include <linux/mm_types.h>
+#include <linux/page_table_check.h>
+#include <linux/pgtable.h>
+
+#include <asm/tlbflush.h>
+
+static inline bool napot_hw_supported(void)
+{
+ return riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT);
+}
+
+static inline bool mm_is_user(struct mm_struct *mm)
+{
+ if (unlikely(mm_is_efi(mm)))
+ return false;
+
+ return mm != &init_mm;
+}
+
+static inline unsigned int napotpte_order(void)
+{
+ return NAPOT_CONT64KB_ORDER;
+}
+
+static inline unsigned long napotpte_size(void)
+{
+ return napot_cont_size(napotpte_order());
+}
+
+static inline unsigned int napotpte_pte_num(void)
+{
+ return napot_pte_num(napotpte_order());
+}
+
+static inline unsigned long napot_align_addr(unsigned long addr)
+{
+ return ALIGN_DOWN(addr, napotpte_size());
+}
+
+static inline pte_t *napot_align_ptep(pte_t *ptep)
+{
+ return PTR_ALIGN_DOWN(ptep, napotpte_pte_num() * sizeof(*ptep));
+}
+
+static inline pte_t pte_mask_ad(pte_t pte)
+{
+ return pte_mkold(pte_mkclean(pte));
+}
+
+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte);
+
+static inline unsigned long pte_protval_no_pfn_no_napot(pte_t pte)
+{
+ return (pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT;
+}
+
+static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
+{
+ unsigned long pfn;
+ pgprot_t prot;
+
+ if (!pte_present_napot(pte))
+ return pte;
+
+ pfn = pte_pfn(pte) + (ptep - napot_align_ptep(ptep));
+ prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
+
+ return pfn_pte(pfn, prot);
+}
+
+static inline pte_t __napot_ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ pte_t pte;
+
+ pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));
+ page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
+
+ return pte;
+}
+
+static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t target)
+{
+ unsigned long start_addr, end, ptent_addr;
+ pte_t *start_ptep;
+ pte_t ptent, pte;
+ unsigned int i, nr;
+
+ start_addr = napot_align_addr(addr);
+ start_ptep = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+ end = start_addr + napotpte_size();
+
+ for (i = 0; i < nr; i++) {
+ ptent_addr = start_addr + i * PAGE_SIZE;
+ ptent = __napot_ptep_get_and_clear(mm, ptent_addr,
+ start_ptep + i);
+ if (pte_dirty(ptent))
+ target = riscv_pte_mkhwdirty(target);
+ if (pte_young(ptent))
+ target = pte_mkyoung(target);
+ }
+
+ flush_tlb_mm_range(mm, start_addr, end, PAGE_SIZE);
+
+ page_table_check_ptes_set(mm, start_addr, start_ptep, target, nr);
+ if (pte_napot(target)) {
+ for (i = 0; i < nr; i++)
+ __set_pte_at(mm, start_ptep + i, target);
+ return;
+ }
+
+ for (i = 0; i < nr; i++) {
+ pte = pfn_pte(pte_pfn(target) + i,
+ __pgprot(pte_protval_no_pfn_no_napot(target)));
+ if (pte_dirty(target))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(target))
+ pte = pte_mkyoung(pte);
+ __set_pte_at(mm, start_ptep + i, pte);
+ }
+}
+
+static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte)
+{
+ return pte_present_napot(pte) &&
+ pte_val(pte_mask_ad(pte)) == pte_val(pte_mask_ad(orig_pte));
+}
+
+void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte)
+{
+ struct page *page;
+ struct folio *folio;
+ unsigned long folio_start, folio_end;
+ unsigned long cont_start, cont_end;
+ unsigned long pfn;
+ pgprot_t prot;
+ pte_t expected, cur;
+ pte_t *start;
+ unsigned int i, nr;
+
+ if (!napot_hw_supported() || !mm_is_user(mm))
+ return;
+
+ if (!pte_present(pte) || pte_napot(pte) || pte_special(pte))
+ return;
+
+ /*
+ * Driver __GFP_COMP pages inserted by vm_insert_page() have valid
+ * compound metadata. Fold them only after verifying the whole supported
+ * Svnapot range is identical and contiguous.
+ */
+ page = pte_page(pte);
+ folio = page_folio(page);
+ folio_start = addr - (page - &folio->page) * PAGE_SIZE;
+ folio_end = folio_start + folio_nr_pages(folio) * PAGE_SIZE;
+ cont_start = napot_align_addr(addr);
+ cont_end = cont_start + napotpte_size();
+ if (folio_start > cont_start || folio_end < cont_end)
+ return;
+
+ nr = napotpte_pte_num();
+ start = napot_align_ptep(ptep);
+
+ pfn = ALIGN_DOWN(pte_pfn(pte), nr);
+ prot = pte_pgprot(pte_mask_ad(pte));
+ expected = pfn_pte(pfn, prot);
+
+ /*
+ * The caller must hold the PTL across validation and conversion.
+ * Software PTE updates are serialized by the PTL; hardware A/D updates
+ * may race and are re-read and folded into target by napotpte_convert().
+ */
+ for (i = 0; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (pte_val(pte_mask_ad(cur)) != pte_val(expected))
+ return;
+ pte_val(expected) += 1UL << _PAGE_PFN_SHIFT;
+ }
+
+ expected = pte_mknapot(pfn_pte(pfn, prot), napotpte_order());
+ napotpte_convert(mm, addr, ptep, expected);
+}
+EXPORT_SYMBOL(__napotpte_try_fold);
+
+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte)
+{
+ pte_t target;
+ pgprot_t prot;
+
+ if (!napot_hw_supported() || !mm_is_user(mm) ||
+ !pte_present_napot(pte))
+ return;
+
+ prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
+ target = pfn_pte(pte_pfn(pte), prot);
+
+ napotpte_convert(mm, addr, ptep, target);
+}
+EXPORT_SYMBOL(__napotpte_try_unfold);
+
+pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
+{
+ pte_t pte, cur;
+ pte_t *start;
+ unsigned int i, nr;
+
+ if (!napot_hw_supported() || !pte_present_napot(orig_pte))
+ return orig_pte;
+
+ pte = orig_pte;
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+
+ for (i = 0; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (!napotpte_is_consistent(cur, orig_pte))
+ return napotpte_subpte(ptep, orig_pte);
+ if (pte_dirty(cur))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(cur))
+ pte = pte_mkyoung(pte);
+ }
+
+ return napotpte_subpte(ptep, pte);
+}
+EXPORT_SYMBOL(napotpte_ptep_get);
+
+pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
+{
+ pte_t orig_pte, pte;
+ pte_t *ptep;
+ unsigned int i, nr;
+
+ if (!napot_hw_supported())
+ return READ_ONCE(*orig_ptep);
+
+ nr = napotpte_pte_num();
+
+retry:
+ orig_pte = READ_ONCE(*orig_ptep);
+ if (!pte_present_napot(orig_pte))
+ return orig_pte;
+
+ ptep = napot_align_ptep(orig_ptep);
+
+ for (i = 0; i < nr; i++, ptep++) {
+ pte = READ_ONCE(*ptep);
+
+ if (!napotpte_is_consistent(pte, orig_pte))
+ goto retry;
+
+ if (pte_dirty(pte))
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
+
+ if (pte_young(pte))
+ orig_pte = pte_mkyoung(orig_pte);
+ }
+
+ return napotpte_subpte(orig_ptep, orig_pte);
+}
+EXPORT_SYMBOL(napotpte_ptep_get_lockless);
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 03/15] riscv/mm: implement Svnapot contpte read-side helpers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot keeps a contiguous mapping encoded in a block of page-table
entries, while generic MM code expects the public PTE getters to return
a normal per-page view for the address being queried. Without a dedicated
read path, exposing the raw NAPOT encoding through the public helpers
would leak arch-private representation details into generic MM users and
make folded mappings harder to integrate with existing PTE walkers.
Add Svnapot-aware contpte read-side helpers and route the public PTE
getters through them, so generic MM sees the logical sub-PTE for the
current page while RISC-V raw helpers continue to operate on the encoded
entries.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 50 +++++-
arch/riscv/mm/Makefile | 1 +
arch/riscv/mm/contpte.c | 273 +++++++++++++++++++++++++++++++
3 files changed, 322 insertions(+), 2 deletions(-)
create mode 100644 arch/riscv/mm/contpte.c
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index a0f64e980e981..fce6d04ea65db 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -312,6 +312,11 @@ static inline unsigned long pte_napot(pte_t pte)
return 0;
}
+static inline pte_t pte_mknapot(pte_t pte, unsigned int order)
+{
+ return pte;
+}
+
#endif /* CONFIG_RISCV_ISA_SVNAPOT */
/* Yields the page frame number (PFN) of a page table entry */
@@ -350,6 +355,11 @@ static inline int pte_present(pte_t pte)
return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE));
}
+static inline bool pte_present_napot(pte_t pte)
+{
+ return pte_present(pte) && pte_napot(pte);
+}
+
#define pte_accessible pte_accessible
static inline unsigned long pte_accessible(struct mm_struct *mm, pte_t a)
{
@@ -538,6 +548,13 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
#define pte_leaf_size(pte) (pte_napot(pte) ? \
napot_cont_size(napot_cont_order(pte)) :\
PAGE_SIZE)
+
+void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte);
+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte);
+pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte);
+pte_t napotpte_ptep_get_lockless(pte_t *ptep);
#endif
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
@@ -689,6 +706,37 @@ static inline pte_t __ptep_get_lockless(pte_t *ptep)
return __ptep_get(ptep);
}
+#ifdef CONFIG_RISCV_ISA_SVNAPOT
+
+#define ptep_get ptep_get
+static inline pte_t ptep_get(pte_t *ptep)
+{
+ pte_t pte = __ptep_get(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get(ptep, pte);
+}
+
+#define ptep_get_lockless ptep_get_lockless
+static inline pte_t ptep_get_lockless(pte_t *ptep)
+{
+ pte_t pte = __ptep_get_lockless(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get_lockless(ptep);
+}
+
+#else
+
+#define ptep_get __ptep_get
+#define ptep_get_lockless __ptep_get_lockless
+
+#endif
+
static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep,
pte_t pte, cydp_t flags)
@@ -822,8 +870,6 @@ static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
#define __ptep_clear_flush_young __ptep_clear_flush_young
-#define ptep_get __ptep_get
-#define ptep_get_lockless __ptep_get_lockless
#define set_pte __set_pte
#define set_ptes __set_ptes
diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile
index b916a68d324ad..fd260903cd340 100644
--- a/arch/riscv/mm/Makefile
+++ b/arch/riscv/mm/Makefile
@@ -19,6 +19,7 @@ obj-y += context.o
obj-y += pmem.o
obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o
+obj-$(CONFIG_RISCV_ISA_SVNAPOT) += contpte.o
obj-$(CONFIG_PTDUMP) += ptdump.o
obj-$(CONFIG_KASAN) += kasan_init.o
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
new file mode 100644
index 0000000000000..43ea76e424492
--- /dev/null
+++ b/arch/riscv/mm/contpte.c
@@ -0,0 +1,273 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/align.h>
+#include <linux/cpufeature.h>
+#include <linux/efi.h>
+#include <linux/export.h>
+#include <linux/mm.h>
+#include <linux/mm_types.h>
+#include <linux/page_table_check.h>
+#include <linux/pgtable.h>
+
+#include <asm/tlbflush.h>
+
+static inline bool napot_hw_supported(void)
+{
+ return riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT);
+}
+
+static inline bool mm_is_user(struct mm_struct *mm)
+{
+ if (unlikely(mm_is_efi(mm)))
+ return false;
+
+ return mm != &init_mm;
+}
+
+static inline unsigned int napotpte_order(void)
+{
+ return NAPOT_CONT64KB_ORDER;
+}
+
+static inline unsigned long napotpte_size(void)
+{
+ return napot_cont_size(napotpte_order());
+}
+
+static inline unsigned int napotpte_pte_num(void)
+{
+ return napot_pte_num(napotpte_order());
+}
+
+static inline unsigned long napot_align_addr(unsigned long addr)
+{
+ return ALIGN_DOWN(addr, napotpte_size());
+}
+
+static inline pte_t *napot_align_ptep(pte_t *ptep)
+{
+ return PTR_ALIGN_DOWN(ptep, napotpte_pte_num() * sizeof(*ptep));
+}
+
+static inline pte_t pte_mask_ad(pte_t pte)
+{
+ return pte_mkold(pte_mkclean(pte));
+}
+
+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte);
+
+static inline unsigned long pte_protval_no_pfn_no_napot(pte_t pte)
+{
+ return (pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT;
+}
+
+static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
+{
+ unsigned long pfn;
+ pgprot_t prot;
+
+ if (!pte_present_napot(pte))
+ return pte;
+
+ pfn = pte_pfn(pte) + (ptep - napot_align_ptep(ptep));
+ prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
+
+ return pfn_pte(pfn, prot);
+}
+
+static inline pte_t __napot_ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ pte_t pte;
+
+ pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));
+ page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
+
+ return pte;
+}
+
+static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t target)
+{
+ unsigned long start_addr, end, ptent_addr;
+ pte_t *start_ptep;
+ pte_t ptent, pte;
+ unsigned int i, nr;
+
+ start_addr = napot_align_addr(addr);
+ start_ptep = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+ end = start_addr + napotpte_size();
+
+ for (i = 0; i < nr; i++) {
+ ptent_addr = start_addr + i * PAGE_SIZE;
+ ptent = __napot_ptep_get_and_clear(mm, ptent_addr,
+ start_ptep + i);
+ if (pte_dirty(ptent))
+ target = riscv_pte_mkhwdirty(target);
+ if (pte_young(ptent))
+ target = pte_mkyoung(target);
+ }
+
+ flush_tlb_mm_range(mm, start_addr, end, PAGE_SIZE);
+
+ page_table_check_ptes_set(mm, start_addr, start_ptep, target, nr);
+ if (pte_napot(target)) {
+ for (i = 0; i < nr; i++)
+ __set_pte_at(mm, start_ptep + i, target);
+ return;
+ }
+
+ for (i = 0; i < nr; i++) {
+ pte = pfn_pte(pte_pfn(target) + i,
+ __pgprot(pte_protval_no_pfn_no_napot(target)));
+ if (pte_dirty(target))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(target))
+ pte = pte_mkyoung(pte);
+ __set_pte_at(mm, start_ptep + i, pte);
+ }
+}
+
+static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte)
+{
+ return pte_present_napot(pte) &&
+ pte_val(pte_mask_ad(pte)) == pte_val(pte_mask_ad(orig_pte));
+}
+
+void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte)
+{
+ struct page *page;
+ struct folio *folio;
+ unsigned long folio_start, folio_end;
+ unsigned long cont_start, cont_end;
+ unsigned long pfn;
+ pgprot_t prot;
+ pte_t expected, cur;
+ pte_t *start;
+ unsigned int i, nr;
+
+ if (!napot_hw_supported() || !mm_is_user(mm))
+ return;
+
+ if (!pte_present(pte) || pte_napot(pte) || pte_special(pte))
+ return;
+
+ /*
+ * Driver __GFP_COMP pages inserted by vm_insert_page() have valid
+ * compound metadata. Fold them only after verifying the whole supported
+ * Svnapot range is identical and contiguous.
+ */
+ page = pte_page(pte);
+ folio = page_folio(page);
+ folio_start = addr - (page - &folio->page) * PAGE_SIZE;
+ folio_end = folio_start + folio_nr_pages(folio) * PAGE_SIZE;
+ cont_start = napot_align_addr(addr);
+ cont_end = cont_start + napotpte_size();
+ if (folio_start > cont_start || folio_end < cont_end)
+ return;
+
+ nr = napotpte_pte_num();
+ start = napot_align_ptep(ptep);
+
+ pfn = ALIGN_DOWN(pte_pfn(pte), nr);
+ prot = pte_pgprot(pte_mask_ad(pte));
+ expected = pfn_pte(pfn, prot);
+
+ /*
+ * The caller must hold the PTL across validation and conversion.
+ * Software PTE updates are serialized by the PTL; hardware A/D updates
+ * may race and are re-read and folded into target by napotpte_convert().
+ */
+ for (i = 0; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (pte_val(pte_mask_ad(cur)) != pte_val(expected))
+ return;
+ pte_val(expected) += 1UL << _PAGE_PFN_SHIFT;
+ }
+
+ expected = pte_mknapot(pfn_pte(pfn, prot), napotpte_order());
+ napotpte_convert(mm, addr, ptep, expected);
+}
+EXPORT_SYMBOL(__napotpte_try_fold);
+
+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte)
+{
+ pte_t target;
+ pgprot_t prot;
+
+ if (!napot_hw_supported() || !mm_is_user(mm) ||
+ !pte_present_napot(pte))
+ return;
+
+ prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
+ target = pfn_pte(pte_pfn(pte), prot);
+
+ napotpte_convert(mm, addr, ptep, target);
+}
+EXPORT_SYMBOL(__napotpte_try_unfold);
+
+pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
+{
+ pte_t pte, cur;
+ pte_t *start;
+ unsigned int i, nr;
+
+ if (!napot_hw_supported() || !pte_present_napot(orig_pte))
+ return orig_pte;
+
+ pte = orig_pte;
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+
+ for (i = 0; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (!napotpte_is_consistent(cur, orig_pte))
+ return napotpte_subpte(ptep, orig_pte);
+ if (pte_dirty(cur))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(cur))
+ pte = pte_mkyoung(pte);
+ }
+
+ return napotpte_subpte(ptep, pte);
+}
+EXPORT_SYMBOL(napotpte_ptep_get);
+
+pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
+{
+ pte_t orig_pte, pte;
+ pte_t *ptep;
+ unsigned int i, nr;
+
+ if (!napot_hw_supported())
+ return READ_ONCE(*orig_ptep);
+
+ nr = napotpte_pte_num();
+
+retry:
+ orig_pte = READ_ONCE(*orig_ptep);
+ if (!pte_present_napot(orig_pte))
+ return orig_pte;
+
+ ptep = napot_align_ptep(orig_ptep);
+
+ for (i = 0; i < nr; i++, ptep++) {
+ pte = READ_ONCE(*ptep);
+
+ if (!napotpte_is_consistent(pte, orig_pte))
+ goto retry;
+
+ if (pte_dirty(pte))
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
+
+ if (pte_young(pte))
+ orig_pte = pte_mkyoung(orig_pte);
+ }
+
+ return napotpte_subpte(orig_ptep, orig_pte);
+}
+EXPORT_SYMBOL(napotpte_ptep_get_lockless);
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 03/15] riscv/mm: implement Svnapot contpte read-side helpers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot keeps a contiguous mapping encoded in a block of page-table
entries, while generic MM code expects the public PTE getters to return
a normal per-page view for the address being queried. Without a dedicated
read path, exposing the raw NAPOT encoding through the public helpers
would leak arch-private representation details into generic MM users and
make folded mappings harder to integrate with existing PTE walkers.
Add Svnapot-aware contpte read-side helpers and route the public PTE
getters through them, so generic MM sees the logical sub-PTE for the
current page while RISC-V raw helpers continue to operate on the encoded
entries.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 50 +++++-
arch/riscv/mm/Makefile | 1 +
arch/riscv/mm/contpte.c | 273 +++++++++++++++++++++++++++++++
3 files changed, 322 insertions(+), 2 deletions(-)
create mode 100644 arch/riscv/mm/contpte.c
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index a0f64e980e981..fce6d04ea65db 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -312,6 +312,11 @@ static inline unsigned long pte_napot(pte_t pte)
return 0;
}
+static inline pte_t pte_mknapot(pte_t pte, unsigned int order)
+{
+ return pte;
+}
+
#endif /* CONFIG_RISCV_ISA_SVNAPOT */
/* Yields the page frame number (PFN) of a page table entry */
@@ -350,6 +355,11 @@ static inline int pte_present(pte_t pte)
return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE));
}
+static inline bool pte_present_napot(pte_t pte)
+{
+ return pte_present(pte) && pte_napot(pte);
+}
+
#define pte_accessible pte_accessible
static inline unsigned long pte_accessible(struct mm_struct *mm, pte_t a)
{
@@ -538,6 +548,13 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
#define pte_leaf_size(pte) (pte_napot(pte) ? \
napot_cont_size(napot_cont_order(pte)) :\
PAGE_SIZE)
+
+void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte);
+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte);
+pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte);
+pte_t napotpte_ptep_get_lockless(pte_t *ptep);
#endif
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
@@ -689,6 +706,37 @@ static inline pte_t __ptep_get_lockless(pte_t *ptep)
return __ptep_get(ptep);
}
+#ifdef CONFIG_RISCV_ISA_SVNAPOT
+
+#define ptep_get ptep_get
+static inline pte_t ptep_get(pte_t *ptep)
+{
+ pte_t pte = __ptep_get(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get(ptep, pte);
+}
+
+#define ptep_get_lockless ptep_get_lockless
+static inline pte_t ptep_get_lockless(pte_t *ptep)
+{
+ pte_t pte = __ptep_get_lockless(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get_lockless(ptep);
+}
+
+#else
+
+#define ptep_get __ptep_get
+#define ptep_get_lockless __ptep_get_lockless
+
+#endif
+
static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep,
pte_t pte, cydp_t flags)
@@ -822,8 +870,6 @@ static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
#define __ptep_clear_flush_young __ptep_clear_flush_young
-#define ptep_get __ptep_get
-#define ptep_get_lockless __ptep_get_lockless
#define set_pte __set_pte
#define set_ptes __set_ptes
diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile
index b916a68d324ad..fd260903cd340 100644
--- a/arch/riscv/mm/Makefile
+++ b/arch/riscv/mm/Makefile
@@ -19,6 +19,7 @@ obj-y += context.o
obj-y += pmem.o
obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o
+obj-$(CONFIG_RISCV_ISA_SVNAPOT) += contpte.o
obj-$(CONFIG_PTDUMP) += ptdump.o
obj-$(CONFIG_KASAN) += kasan_init.o
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
new file mode 100644
index 0000000000000..43ea76e424492
--- /dev/null
+++ b/arch/riscv/mm/contpte.c
@@ -0,0 +1,273 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/align.h>
+#include <linux/cpufeature.h>
+#include <linux/efi.h>
+#include <linux/export.h>
+#include <linux/mm.h>
+#include <linux/mm_types.h>
+#include <linux/page_table_check.h>
+#include <linux/pgtable.h>
+
+#include <asm/tlbflush.h>
+
+static inline bool napot_hw_supported(void)
+{
+ return riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT);
+}
+
+static inline bool mm_is_user(struct mm_struct *mm)
+{
+ if (unlikely(mm_is_efi(mm)))
+ return false;
+
+ return mm != &init_mm;
+}
+
+static inline unsigned int napotpte_order(void)
+{
+ return NAPOT_CONT64KB_ORDER;
+}
+
+static inline unsigned long napotpte_size(void)
+{
+ return napot_cont_size(napotpte_order());
+}
+
+static inline unsigned int napotpte_pte_num(void)
+{
+ return napot_pte_num(napotpte_order());
+}
+
+static inline unsigned long napot_align_addr(unsigned long addr)
+{
+ return ALIGN_DOWN(addr, napotpte_size());
+}
+
+static inline pte_t *napot_align_ptep(pte_t *ptep)
+{
+ return PTR_ALIGN_DOWN(ptep, napotpte_pte_num() * sizeof(*ptep));
+}
+
+static inline pte_t pte_mask_ad(pte_t pte)
+{
+ return pte_mkold(pte_mkclean(pte));
+}
+
+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte);
+
+static inline unsigned long pte_protval_no_pfn_no_napot(pte_t pte)
+{
+ return (pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT;
+}
+
+static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
+{
+ unsigned long pfn;
+ pgprot_t prot;
+
+ if (!pte_present_napot(pte))
+ return pte;
+
+ pfn = pte_pfn(pte) + (ptep - napot_align_ptep(ptep));
+ prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
+
+ return pfn_pte(pfn, prot);
+}
+
+static inline pte_t __napot_ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ pte_t pte;
+
+ pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));
+ page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
+
+ return pte;
+}
+
+static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t target)
+{
+ unsigned long start_addr, end, ptent_addr;
+ pte_t *start_ptep;
+ pte_t ptent, pte;
+ unsigned int i, nr;
+
+ start_addr = napot_align_addr(addr);
+ start_ptep = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+ end = start_addr + napotpte_size();
+
+ for (i = 0; i < nr; i++) {
+ ptent_addr = start_addr + i * PAGE_SIZE;
+ ptent = __napot_ptep_get_and_clear(mm, ptent_addr,
+ start_ptep + i);
+ if (pte_dirty(ptent))
+ target = riscv_pte_mkhwdirty(target);
+ if (pte_young(ptent))
+ target = pte_mkyoung(target);
+ }
+
+ flush_tlb_mm_range(mm, start_addr, end, PAGE_SIZE);
+
+ page_table_check_ptes_set(mm, start_addr, start_ptep, target, nr);
+ if (pte_napot(target)) {
+ for (i = 0; i < nr; i++)
+ __set_pte_at(mm, start_ptep + i, target);
+ return;
+ }
+
+ for (i = 0; i < nr; i++) {
+ pte = pfn_pte(pte_pfn(target) + i,
+ __pgprot(pte_protval_no_pfn_no_napot(target)));
+ if (pte_dirty(target))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(target))
+ pte = pte_mkyoung(pte);
+ __set_pte_at(mm, start_ptep + i, pte);
+ }
+}
+
+static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte)
+{
+ return pte_present_napot(pte) &&
+ pte_val(pte_mask_ad(pte)) == pte_val(pte_mask_ad(orig_pte));
+}
+
+void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte)
+{
+ struct page *page;
+ struct folio *folio;
+ unsigned long folio_start, folio_end;
+ unsigned long cont_start, cont_end;
+ unsigned long pfn;
+ pgprot_t prot;
+ pte_t expected, cur;
+ pte_t *start;
+ unsigned int i, nr;
+
+ if (!napot_hw_supported() || !mm_is_user(mm))
+ return;
+
+ if (!pte_present(pte) || pte_napot(pte) || pte_special(pte))
+ return;
+
+ /*
+ * Driver __GFP_COMP pages inserted by vm_insert_page() have valid
+ * compound metadata. Fold them only after verifying the whole supported
+ * Svnapot range is identical and contiguous.
+ */
+ page = pte_page(pte);
+ folio = page_folio(page);
+ folio_start = addr - (page - &folio->page) * PAGE_SIZE;
+ folio_end = folio_start + folio_nr_pages(folio) * PAGE_SIZE;
+ cont_start = napot_align_addr(addr);
+ cont_end = cont_start + napotpte_size();
+ if (folio_start > cont_start || folio_end < cont_end)
+ return;
+
+ nr = napotpte_pte_num();
+ start = napot_align_ptep(ptep);
+
+ pfn = ALIGN_DOWN(pte_pfn(pte), nr);
+ prot = pte_pgprot(pte_mask_ad(pte));
+ expected = pfn_pte(pfn, prot);
+
+ /*
+ * The caller must hold the PTL across validation and conversion.
+ * Software PTE updates are serialized by the PTL; hardware A/D updates
+ * may race and are re-read and folded into target by napotpte_convert().
+ */
+ for (i = 0; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (pte_val(pte_mask_ad(cur)) != pte_val(expected))
+ return;
+ pte_val(expected) += 1UL << _PAGE_PFN_SHIFT;
+ }
+
+ expected = pte_mknapot(pfn_pte(pfn, prot), napotpte_order());
+ napotpte_convert(mm, addr, ptep, expected);
+}
+EXPORT_SYMBOL(__napotpte_try_fold);
+
+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte)
+{
+ pte_t target;
+ pgprot_t prot;
+
+ if (!napot_hw_supported() || !mm_is_user(mm) ||
+ !pte_present_napot(pte))
+ return;
+
+ prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
+ target = pfn_pte(pte_pfn(pte), prot);
+
+ napotpte_convert(mm, addr, ptep, target);
+}
+EXPORT_SYMBOL(__napotpte_try_unfold);
+
+pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
+{
+ pte_t pte, cur;
+ pte_t *start;
+ unsigned int i, nr;
+
+ if (!napot_hw_supported() || !pte_present_napot(orig_pte))
+ return orig_pte;
+
+ pte = orig_pte;
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+
+ for (i = 0; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (!napotpte_is_consistent(cur, orig_pte))
+ return napotpte_subpte(ptep, orig_pte);
+ if (pte_dirty(cur))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(cur))
+ pte = pte_mkyoung(pte);
+ }
+
+ return napotpte_subpte(ptep, pte);
+}
+EXPORT_SYMBOL(napotpte_ptep_get);
+
+pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
+{
+ pte_t orig_pte, pte;
+ pte_t *ptep;
+ unsigned int i, nr;
+
+ if (!napot_hw_supported())
+ return READ_ONCE(*orig_ptep);
+
+ nr = napotpte_pte_num();
+
+retry:
+ orig_pte = READ_ONCE(*orig_ptep);
+ if (!pte_present_napot(orig_pte))
+ return orig_pte;
+
+ ptep = napot_align_ptep(orig_ptep);
+
+ for (i = 0; i < nr; i++, ptep++) {
+ pte = READ_ONCE(*ptep);
+
+ if (!napotpte_is_consistent(pte, orig_pte))
+ goto retry;
+
+ if (pte_dirty(pte))
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
+
+ if (pte_young(pte))
+ orig_pte = pte_mkyoung(orig_pte);
+ }
+
+ return napotpte_subpte(orig_ptep, orig_pte);
+}
+EXPORT_SYMBOL(napotpte_ptep_get_lockless);
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 04/15] riscv/mm: implement Svnapot contpte update helpers
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot support needs the public RISC-V PTE update helpers to preserve
the same logical per-page view that generic MM users see on the read
side. Without a dedicated update path, operations such as set_ptes(),
ptep_set_access_flags(), and related clear or write-protect flows would
end up touching the raw NAPOT encoding directly, which leaks arch-private
representation details into generic MM code and makes folded mappings
harder to update safely.
Add Svnapot-aware contpte update helpers and route the public update-side
PTE operations through them, while keeping the raw helpers available for
RISC-V code that still needs direct access to the encoded entries.
This also wires the update path into the related mechanics needed when a
Svnapot block is unfolded and rewritten, so page-table-check, access-flag
updates, and non-Svnapot builds continue to observe the expected public
PTE semantics.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 281 ++++++++++++++++++++++----
arch/riscv/mm/contpte.c | 329 ++++++++++++++++++++++++++++++-
arch/riscv/mm/pgtable.c | 18 ++
3 files changed, 588 insertions(+), 40 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index fce6d04ea65db..771faabb2b00b 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -413,6 +413,30 @@ static inline int pte_special(pte_t pte)
return pte_val(pte) & _PAGE_SPECIAL;
}
+#ifdef CONFIG_RISCV_ISA_SVNAPOT
+static inline pte_t pte_mknonnapot(pte_t pte, unsigned long addr)
+{
+ unsigned long pfn;
+ unsigned long offset;
+ pgprot_t prot;
+
+ if (!pte_present_napot(pte))
+ return pte;
+
+ offset = (addr & (napot_cont_size(napot_cont_order(pte)) - 1)) >>
+ PAGE_SHIFT;
+ pfn = pte_pfn(pte) + offset;
+ prot = __pgprot((pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT);
+
+ return pfn_pte(pfn, prot);
+}
+#else
+static inline pte_t pte_mknonnapot(pte_t pte, unsigned long addr)
+{
+ return pte;
+}
+#endif
+
/* static inline pte_t pte_rdprotect(pte_t pte) */
static inline pte_t pte_wrprotect(pte_t pte)
@@ -555,6 +579,25 @@ void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte);
pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte);
pte_t napotpte_ptep_get_lockless(pte_t *ptep);
+void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte, unsigned int nr);
+void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full);
+pte_t napotpte_get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full);
+void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags);
+void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr);
+int napotpte_ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty);
+int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep);
+int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep);
#endif
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
@@ -706,37 +749,6 @@ static inline pte_t __ptep_get_lockless(pte_t *ptep)
return __ptep_get(ptep);
}
-#ifdef CONFIG_RISCV_ISA_SVNAPOT
-
-#define ptep_get ptep_get
-static inline pte_t ptep_get(pte_t *ptep)
-{
- pte_t pte = __ptep_get(ptep);
-
- if (likely(!pte_present_napot(pte)))
- return pte;
-
- return napotpte_ptep_get(ptep, pte);
-}
-
-#define ptep_get_lockless ptep_get_lockless
-static inline pte_t ptep_get_lockless(pte_t *ptep)
-{
- pte_t pte = __ptep_get_lockless(ptep);
-
- if (likely(!pte_present_napot(pte)))
- return pte;
-
- return napotpte_ptep_get_lockless(ptep);
-}
-
-#else
-
-#define ptep_get __ptep_get
-#define ptep_get_lockless __ptep_get_lockless
-
-#endif
-
static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep,
pte_t pte, cydp_t flags)
@@ -829,7 +841,6 @@ static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,
#define __ptep_clear_flush __ptep_clear_flush
-#define __HAVE_ARCH_PTEP_SET_WRPROTECT
static inline void __ptep_set_wrprotect(struct mm_struct *mm,
unsigned long address, pte_t *ptep)
{
@@ -870,7 +881,169 @@ static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
#define __ptep_clear_flush_young __ptep_clear_flush_young
-#define set_pte __set_pte
+#ifdef CONFIG_RISCV_ISA_SVNAPOT
+
+static __always_inline void napotpte_try_fold(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ pte_t pte)
+{
+ const unsigned long contmask = napot_pte_num(NAPOT_CONT64KB_ORDER) - 1;
+ bool valign = ((addr >> PAGE_SHIFT) & contmask) == contmask;
+
+ if (unlikely(valign)) {
+ bool palign = (pte_pfn(pte) & contmask) == contmask;
+
+ if (unlikely(palign && pte_present(pte) && !pte_napot(pte) &&
+ !pte_special(pte)))
+ __napotpte_try_fold(mm, addr, ptep, pte);
+ }
+}
+
+static __always_inline void napotpte_try_unfold(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ pte_t pte)
+{
+ if (unlikely(pte_present_napot(pte)))
+ __napotpte_try_unfold(mm, addr, ptep, pte);
+}
+
+/*
+ * Public PTE helpers may transparently handle Svnapot-encoded mappings.
+ * NAPOT-aware arch users should stick to the private/raw __pte* helpers.
+ */
+#define ptep_get ptep_get
+static inline pte_t ptep_get(pte_t *ptep)
+{
+ pte_t pte = __ptep_get(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get(ptep, pte);
+}
+
+#define ptep_get_lockless ptep_get_lockless
+static inline pte_t ptep_get_lockless(pte_t *ptep)
+{
+ pte_t pte = __ptep_get_lockless(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get_lockless(ptep);
+}
+
+#define set_ptes set_ptes
+static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pteval, unsigned int nr)
+{
+ pteval = pte_mknonnapot(pteval, addr);
+
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ __set_ptes(mm, addr, ptep, pteval, 1);
+ napotpte_try_fold(mm, addr, ptep, pteval);
+ return;
+ }
+
+ napotpte_set_ptes(mm, addr, ptep, pteval, nr);
+}
+
+static inline void pte_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ __pte_clear(mm, addr, ptep);
+}
+
+static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ return __ptep_get_and_clear(mm, addr, ptep);
+}
+
+#define clear_young_dirty_ptes clear_young_dirty_ptes
+static inline void clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags)
+{
+ if (likely(nr == 1 && !pte_present_napot(__ptep_get(ptep))))
+ __clear_young_dirty_ptes(vma, addr, ptep, nr, flags);
+ else
+ napotpte_clear_young_dirty_ptes(vma, addr, ptep, nr, flags);
+}
+
+#define clear_full_ptes clear_full_ptes
+static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full)
+{
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ __ptep_get_and_clear(mm, addr, ptep);
+ return;
+ }
+
+ napotpte_clear_full_ptes(mm, addr, ptep, nr, full);
+}
+
+#define get_and_clear_full_ptes get_and_clear_full_ptes
+static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ return __ptep_get_and_clear(mm, addr, ptep);
+ }
+
+ return napotpte_get_and_clear_full_ptes(mm, addr, ptep, nr, full);
+}
+
+#define wrprotect_ptes wrprotect_ptes
+static inline void wrprotect_ptes(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep,
+ unsigned int nr)
+{
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, address, ptep, __ptep_get(ptep));
+ __ptep_set_wrprotect(mm, address, ptep);
+ return;
+ }
+
+ napotpte_wrprotect_ptes(mm, address, ptep, nr);
+}
+
+#define __HAVE_ARCH_PTEP_SET_WRPROTECT
+static inline void ptep_set_wrprotect(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep)
+{
+ wrprotect_ptes(mm, address, ptep, 1);
+}
+
+#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
+static inline int ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ pte_t orig_pte = __ptep_get(ptep);
+
+ if (likely(!pte_present_napot(orig_pte)))
+ return __ptep_clear_flush_young(vma, address, ptep);
+
+ return napotpte_ptep_clear_flush_young(vma, address, ptep);
+}
+
+#else /* CONFIG_RISCV_ISA_SVNAPOT */
+
+#define napotpte_ptep_set_access_flags(vma, address, ptep, entry, dirty) \
+ ({ (void)(vma); (void)(address); (void)(ptep); (void)(entry); \
+ (void)(dirty); 0; })
+
+#define napotpte_ptep_test_and_clear_young(vma, address, ptep) \
+ ({ (void)(vma); (void)(address); (void)(ptep); 0; })
+
+#define ptep_get __ptep_get
+#define ptep_get_lockless __ptep_get_lockless
#define set_ptes __set_ptes
static inline void pte_clear(struct mm_struct *mm,
@@ -881,9 +1054,51 @@ static inline void pte_clear(struct mm_struct *mm,
#define ptep_get_and_clear __ptep_get_and_clear
#define clear_young_dirty_ptes __clear_young_dirty_ptes
+
+static inline void __clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ for (;;) {
+ __ptep_get_and_clear(mm, addr, ptep);
+ if (--nr == 0)
+ break;
+ ptep++;
+ addr += PAGE_SIZE;
+ }
+}
+
+static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ pte_t pte, tmp_pte;
+
+ pte = __ptep_get_and_clear(mm, addr, ptep);
+ while (--nr) {
+ ptep++;
+ addr += PAGE_SIZE;
+ tmp_pte = __ptep_get_and_clear(mm, addr, ptep);
+ if (pte_dirty(tmp_pte))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(tmp_pte))
+ pte = pte_mkyoung(pte);
+ }
+
+ return pte;
+}
+
+#define clear_full_ptes __clear_full_ptes
+#define get_and_clear_full_ptes __get_and_clear_full_ptes
+#define __HAVE_ARCH_PTEP_SET_WRPROTECT
#define ptep_set_wrprotect __ptep_set_wrprotect
+#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
#define ptep_clear_flush_young __ptep_clear_flush_young
+#endif /* CONFIG_RISCV_ISA_SVNAPOT */
+
+#define set_pte __set_pte
+
#define pgprot_nx pgprot_nx
static inline pgprot_t pgprot_nx(pgprot_t _prot)
{
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 43ea76e424492..3f8a0a2970afb 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -62,6 +62,24 @@ static inline unsigned long pte_protval_no_pfn_no_napot(pte_t pte)
return (pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT;
}
+static inline void napotpte_clear_young_dirty_pte(pte_t *ptep, cydp_t flags)
+{
+ pte_t old_pte, new_pte;
+ unsigned long old_val, new_val;
+
+ do {
+ old_pte = READ_ONCE(*ptep);
+ new_pte = old_pte;
+ if (flags & CYDP_CLEAR_YOUNG)
+ new_pte = pte_mkold(new_pte);
+ if (flags & CYDP_CLEAR_DIRTY)
+ new_pte = pte_mkclean(new_pte);
+
+ old_val = pte_val(old_pte);
+ new_val = pte_val(new_pte);
+ } while (cmpxchg_relaxed(&pte_val(*ptep), old_val, new_val) != old_val);
+}
+
static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
{
unsigned long pfn;
@@ -76,13 +94,34 @@ static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
return pfn_pte(pfn, prot);
}
-static inline pte_t __napot_ptep_get_and_clear(struct mm_struct *mm,
- unsigned long addr, pte_t *ptep)
+static void __clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full)
{
- pte_t pte;
+ for (;;) {
+ __ptep_get_and_clear(mm, addr, ptep);
+ if (--nr == 0)
+ break;
+ ptep++;
+ addr += PAGE_SIZE;
+ }
+}
- pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));
- page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
+static pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ pte_t pte, tmp_pte;
+
+ pte = __ptep_get_and_clear(mm, addr, ptep);
+ while (--nr) {
+ ptep++;
+ addr += PAGE_SIZE;
+ tmp_pte = __ptep_get_and_clear(mm, addr, ptep);
+ if (pte_dirty(tmp_pte))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(tmp_pte))
+ pte = pte_mkyoung(pte);
+ }
return pte;
}
@@ -102,8 +141,9 @@ static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
for (i = 0; i < nr; i++) {
ptent_addr = start_addr + i * PAGE_SIZE;
- ptent = __napot_ptep_get_and_clear(mm, ptent_addr,
- start_ptep + i);
+ ptent = __ptep_get_and_clear_noptc(start_ptep + i);
+ page_table_check_pte_clear(mm, ptent_addr,
+ pte_mknonnapot(ptent, ptent_addr));
if (pte_dirty(ptent))
target = riscv_pte_mkhwdirty(target);
if (pte_young(ptent))
@@ -136,6 +176,22 @@ static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte)
pte_val(pte_mask_ad(pte)) == pte_val(pte_mask_ad(orig_pte));
}
+static bool napotpte_all_subptes_same(pte_t *ptep, pte_t expected_pte)
+{
+ pte_t *start;
+ unsigned int i, nr;
+
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+
+ for (i = 0; i < nr; i++) {
+ if (!pte_same(READ_ONCE(start[i]), expected_pte))
+ return false;
+ }
+
+ return true;
+}
+
void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte)
{
@@ -271,3 +327,262 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
return napotpte_subpte(orig_ptep, orig_pte);
}
EXPORT_SYMBOL(napotpte_ptep_get_lockless);
+
+static void napotpte_try_unfold_range(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr)
+{
+ unsigned long next;
+ pte_t pte;
+ unsigned int chunk;
+
+ while (nr) {
+ pte = READ_ONCE(*ptep);
+ if (pte_present_napot(pte)) {
+ __napotpte_try_unfold(mm, addr, ptep, pte);
+ next = napot_align_addr(addr) + napotpte_size();
+ chunk = (next - addr) >> PAGE_SHIFT;
+ } else {
+ chunk = 1;
+ }
+
+ if (chunk > nr)
+ chunk = nr;
+
+ ptep += chunk;
+ addr += chunk * PAGE_SIZE;
+ nr -= chunk;
+ }
+}
+
+static void napotpte_try_unfold_partial(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr)
+{
+ pte_t pte;
+
+ if (ptep != napot_align_ptep(ptep) || nr < napotpte_pte_num()) {
+ pte = READ_ONCE(*ptep);
+ if (pte_present_napot(pte))
+ __napotpte_try_unfold(mm, addr, ptep, pte);
+ }
+
+ if (ptep + nr != napot_align_ptep(ptep + nr)) {
+ unsigned long last_addr;
+ pte_t *last_ptep;
+
+ last_addr = addr + PAGE_SIZE * (nr - 1);
+ last_ptep = ptep + nr - 1;
+ pte = READ_ONCE(*last_ptep);
+ if (pte_present_napot(pte))
+ __napotpte_try_unfold(mm, last_addr, last_ptep, pte);
+ }
+}
+
+void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte, unsigned int nr)
+{
+ unsigned long next, end;
+ unsigned long pfn, size, boundary;
+ pgprot_t prot;
+ unsigned int chunk, i;
+ pte_t cur;
+
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ __set_ptes(mm, addr, ptep, pte, nr);
+ return;
+ }
+
+ size = napotpte_size();
+ end = addr + ((unsigned long)nr << PAGE_SHIFT);
+ pfn = pte_pfn(pte);
+ prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
+
+ do {
+ boundary = (addr + size) & ~(size - 1);
+ next = (boundary - 1 < end - 1) ? boundary : end;
+ chunk = (next - addr) >> PAGE_SHIFT;
+
+ cur = pfn_pte(pfn, prot);
+ if (((addr | next | (pfn << PAGE_SHIFT)) & (size - 1)) == 0) {
+ cur = pte_mknapot(cur, napotpte_order());
+ page_table_check_ptes_set(mm, addr, ptep, cur, chunk);
+ for (i = 0; i < chunk; i++)
+ __set_pte_at(mm, ptep + i, cur);
+ } else {
+ __set_ptes(mm, addr, ptep, cur, chunk);
+ }
+
+ addr = next;
+ ptep += chunk;
+ pfn += chunk;
+ } while (addr != end);
+}
+EXPORT_SYMBOL(napotpte_set_ptes);
+
+void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full)
+{
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ __clear_full_ptes(mm, addr, ptep, nr, full);
+ return;
+ }
+
+ /*
+ * Svnapot stores identical napot-encoded entries across the whole block
+ * rather than per-page PFNs, so batch zap paths must unfold the covered
+ * range before the generic MM consumes ordinary per-page PTEs.
+ */
+ napotpte_try_unfold_range(mm, addr, ptep, nr);
+ __clear_full_ptes(mm, addr, ptep, nr, full);
+}
+EXPORT_SYMBOL(napotpte_clear_full_ptes);
+
+pte_t napotpte_get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ if (!napot_hw_supported() || !mm_is_user(mm))
+ return __get_and_clear_full_ptes(mm, addr, ptep, nr, full);
+
+ napotpte_try_unfold_range(mm, addr, ptep, nr);
+
+ return __get_and_clear_full_ptes(mm, addr, ptep, nr, full);
+}
+EXPORT_SYMBOL(napotpte_get_and_clear_full_ptes);
+
+void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags)
+{
+ struct mm_struct *mm;
+ unsigned long start, end;
+ unsigned int total;
+
+ mm = vma->vm_mm;
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ __clear_young_dirty_ptes(vma, addr, ptep, nr, flags);
+ return;
+ }
+
+ start = addr;
+ end = start + nr * PAGE_SIZE;
+
+ if (pte_present_napot(READ_ONCE(*(ptep + nr - 1))))
+ end = ALIGN(end, napotpte_size());
+
+ if (pte_present_napot(READ_ONCE(*ptep))) {
+ start = napot_align_addr(start);
+ ptep = napot_align_ptep(ptep);
+ }
+
+ total = (end - start) >> PAGE_SHIFT;
+ for (; total; total--, ptep++, start += PAGE_SIZE)
+ napotpte_clear_young_dirty_pte(ptep, flags);
+}
+EXPORT_SYMBOL(napotpte_clear_young_dirty_ptes);
+
+void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr)
+{
+ unsigned int i;
+
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ for (i = 0; i < nr; i++, ptep++, addr += PAGE_SIZE)
+ __ptep_set_wrprotect(mm, addr, ptep);
+ return;
+ }
+
+ napotpte_try_unfold_partial(mm, addr, ptep, nr);
+
+ for (i = 0; i < nr; i++, ptep++, addr += PAGE_SIZE)
+ __ptep_set_wrprotect(mm, addr, ptep);
+}
+EXPORT_SYMBOL(napotpte_wrprotect_ptes);
+
+int napotpte_ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty)
+{
+ pte_t raw_pte, napot_pte;
+ pte_t *start;
+ pgprot_t prot;
+ unsigned long start_addr;
+ unsigned int i, nr;
+ bool changed;
+
+ raw_pte = READ_ONCE(*ptep);
+ if (!napot_hw_supported() || !pte_present_napot(raw_pte))
+ return 0;
+
+ prot = pte_pgprot(entry);
+ napot_pte = pfn_pte(pte_pfn(raw_pte), prot);
+ napot_pte = pte_mknapot(napot_pte, napotpte_order());
+
+ if (napotpte_all_subptes_same(ptep, napot_pte))
+ return !riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC);
+
+ if (pte_write(raw_pte) != pte_write(napot_pte)) {
+ __napotpte_try_unfold(vma->vm_mm, address, ptep, raw_pte);
+ entry = pte_mknonnapot(entry, address);
+
+ return __ptep_set_access_flags(vma, address, ptep, entry,
+ dirty);
+ }
+
+ start = napot_align_ptep(ptep);
+ address = napot_align_addr(address);
+ start_addr = address;
+ nr = napotpte_pte_num();
+ changed = false;
+
+ for (i = 0; i < nr; i++, start++, address += PAGE_SIZE) {
+ if (__ptep_set_access_flags(vma, address, start, napot_pte, 0))
+ changed = true;
+ }
+
+ if (changed)
+ flush_tlb_range(vma, start_addr, start_addr + napotpte_size());
+
+ return changed;
+}
+EXPORT_SYMBOL(napotpte_ptep_set_access_flags);
+
+int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ pte_t *start;
+ unsigned int i, nr;
+ int young;
+
+ if (!napot_hw_supported() || !pte_present_napot(READ_ONCE(*ptep)))
+ return 0;
+
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+ young = 0;
+
+ for (i = 0; i < nr; i++)
+ young |= test_and_clear_bit(_PAGE_ACCESSED_OFFSET,
+ &pte_val(start[i]));
+
+ return young;
+}
+EXPORT_SYMBOL(napotpte_ptep_test_and_clear_young);
+
+int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ unsigned long start_addr;
+ int young;
+
+ young = napotpte_ptep_test_and_clear_young(vma, address, ptep);
+ if (!young)
+ return 0;
+
+ start_addr = napot_align_addr(address);
+ flush_tlb_range(vma, start_addr, start_addr + napotpte_size());
+
+ return young;
+}
+EXPORT_SYMBOL(napotpte_ptep_clear_flush_young);
diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c
index 43bd542c6ff02..50e4c4cc10691 100644
--- a/arch/riscv/mm/pgtable.c
+++ b/arch/riscv/mm/pgtable.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0
#include <asm/pgalloc.h>
+#include <linux/cpufeature.h>
#include <linux/gfp.h>
#include <linux/kernel.h>
#include <linux/pgtable.h>
@@ -32,6 +33,16 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep,
pte_t entry, int dirty)
{
+ pte_t raw_pte;
+
+ entry = pte_mknonnapot(entry, address);
+
+ raw_pte = READ_ONCE(*ptep);
+ if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT) &&
+ pte_present_napot(raw_pte))
+ return napotpte_ptep_set_access_flags(vma, address, ptep, entry,
+ dirty);
+
return __ptep_set_access_flags(vma, address, ptep, entry, dirty);
}
@@ -50,6 +61,13 @@ bool ptep_test_and_clear_young(struct vm_area_struct *vma,
unsigned long address,
pte_t *ptep)
{
+ pte_t raw_pte;
+
+ raw_pte = READ_ONCE(*ptep);
+ if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT) &&
+ pte_present_napot(raw_pte))
+ return napotpte_ptep_test_and_clear_young(vma, address, ptep);
+
return __ptep_test_and_clear_young(vma, address, ptep);
}
EXPORT_SYMBOL_GPL(ptep_test_and_clear_young);
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 04/15] riscv/mm: implement Svnapot contpte update helpers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot support needs the public RISC-V PTE update helpers to preserve
the same logical per-page view that generic MM users see on the read
side. Without a dedicated update path, operations such as set_ptes(),
ptep_set_access_flags(), and related clear or write-protect flows would
end up touching the raw NAPOT encoding directly, which leaks arch-private
representation details into generic MM code and makes folded mappings
harder to update safely.
Add Svnapot-aware contpte update helpers and route the public update-side
PTE operations through them, while keeping the raw helpers available for
RISC-V code that still needs direct access to the encoded entries.
This also wires the update path into the related mechanics needed when a
Svnapot block is unfolded and rewritten, so page-table-check, access-flag
updates, and non-Svnapot builds continue to observe the expected public
PTE semantics.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 281 ++++++++++++++++++++++----
arch/riscv/mm/contpte.c | 329 ++++++++++++++++++++++++++++++-
arch/riscv/mm/pgtable.c | 18 ++
3 files changed, 588 insertions(+), 40 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index fce6d04ea65db..771faabb2b00b 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -413,6 +413,30 @@ static inline int pte_special(pte_t pte)
return pte_val(pte) & _PAGE_SPECIAL;
}
+#ifdef CONFIG_RISCV_ISA_SVNAPOT
+static inline pte_t pte_mknonnapot(pte_t pte, unsigned long addr)
+{
+ unsigned long pfn;
+ unsigned long offset;
+ pgprot_t prot;
+
+ if (!pte_present_napot(pte))
+ return pte;
+
+ offset = (addr & (napot_cont_size(napot_cont_order(pte)) - 1)) >>
+ PAGE_SHIFT;
+ pfn = pte_pfn(pte) + offset;
+ prot = __pgprot((pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT);
+
+ return pfn_pte(pfn, prot);
+}
+#else
+static inline pte_t pte_mknonnapot(pte_t pte, unsigned long addr)
+{
+ return pte;
+}
+#endif
+
/* static inline pte_t pte_rdprotect(pte_t pte) */
static inline pte_t pte_wrprotect(pte_t pte)
@@ -555,6 +579,25 @@ void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte);
pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte);
pte_t napotpte_ptep_get_lockless(pte_t *ptep);
+void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte, unsigned int nr);
+void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full);
+pte_t napotpte_get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full);
+void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags);
+void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr);
+int napotpte_ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty);
+int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep);
+int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep);
#endif
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
@@ -706,37 +749,6 @@ static inline pte_t __ptep_get_lockless(pte_t *ptep)
return __ptep_get(ptep);
}
-#ifdef CONFIG_RISCV_ISA_SVNAPOT
-
-#define ptep_get ptep_get
-static inline pte_t ptep_get(pte_t *ptep)
-{
- pte_t pte = __ptep_get(ptep);
-
- if (likely(!pte_present_napot(pte)))
- return pte;
-
- return napotpte_ptep_get(ptep, pte);
-}
-
-#define ptep_get_lockless ptep_get_lockless
-static inline pte_t ptep_get_lockless(pte_t *ptep)
-{
- pte_t pte = __ptep_get_lockless(ptep);
-
- if (likely(!pte_present_napot(pte)))
- return pte;
-
- return napotpte_ptep_get_lockless(ptep);
-}
-
-#else
-
-#define ptep_get __ptep_get
-#define ptep_get_lockless __ptep_get_lockless
-
-#endif
-
static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep,
pte_t pte, cydp_t flags)
@@ -829,7 +841,6 @@ static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,
#define __ptep_clear_flush __ptep_clear_flush
-#define __HAVE_ARCH_PTEP_SET_WRPROTECT
static inline void __ptep_set_wrprotect(struct mm_struct *mm,
unsigned long address, pte_t *ptep)
{
@@ -870,7 +881,169 @@ static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
#define __ptep_clear_flush_young __ptep_clear_flush_young
-#define set_pte __set_pte
+#ifdef CONFIG_RISCV_ISA_SVNAPOT
+
+static __always_inline void napotpte_try_fold(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ pte_t pte)
+{
+ const unsigned long contmask = napot_pte_num(NAPOT_CONT64KB_ORDER) - 1;
+ bool valign = ((addr >> PAGE_SHIFT) & contmask) == contmask;
+
+ if (unlikely(valign)) {
+ bool palign = (pte_pfn(pte) & contmask) == contmask;
+
+ if (unlikely(palign && pte_present(pte) && !pte_napot(pte) &&
+ !pte_special(pte)))
+ __napotpte_try_fold(mm, addr, ptep, pte);
+ }
+}
+
+static __always_inline void napotpte_try_unfold(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ pte_t pte)
+{
+ if (unlikely(pte_present_napot(pte)))
+ __napotpte_try_unfold(mm, addr, ptep, pte);
+}
+
+/*
+ * Public PTE helpers may transparently handle Svnapot-encoded mappings.
+ * NAPOT-aware arch users should stick to the private/raw __pte* helpers.
+ */
+#define ptep_get ptep_get
+static inline pte_t ptep_get(pte_t *ptep)
+{
+ pte_t pte = __ptep_get(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get(ptep, pte);
+}
+
+#define ptep_get_lockless ptep_get_lockless
+static inline pte_t ptep_get_lockless(pte_t *ptep)
+{
+ pte_t pte = __ptep_get_lockless(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get_lockless(ptep);
+}
+
+#define set_ptes set_ptes
+static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pteval, unsigned int nr)
+{
+ pteval = pte_mknonnapot(pteval, addr);
+
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ __set_ptes(mm, addr, ptep, pteval, 1);
+ napotpte_try_fold(mm, addr, ptep, pteval);
+ return;
+ }
+
+ napotpte_set_ptes(mm, addr, ptep, pteval, nr);
+}
+
+static inline void pte_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ __pte_clear(mm, addr, ptep);
+}
+
+static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ return __ptep_get_and_clear(mm, addr, ptep);
+}
+
+#define clear_young_dirty_ptes clear_young_dirty_ptes
+static inline void clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags)
+{
+ if (likely(nr == 1 && !pte_present_napot(__ptep_get(ptep))))
+ __clear_young_dirty_ptes(vma, addr, ptep, nr, flags);
+ else
+ napotpte_clear_young_dirty_ptes(vma, addr, ptep, nr, flags);
+}
+
+#define clear_full_ptes clear_full_ptes
+static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full)
+{
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ __ptep_get_and_clear(mm, addr, ptep);
+ return;
+ }
+
+ napotpte_clear_full_ptes(mm, addr, ptep, nr, full);
+}
+
+#define get_and_clear_full_ptes get_and_clear_full_ptes
+static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ return __ptep_get_and_clear(mm, addr, ptep);
+ }
+
+ return napotpte_get_and_clear_full_ptes(mm, addr, ptep, nr, full);
+}
+
+#define wrprotect_ptes wrprotect_ptes
+static inline void wrprotect_ptes(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep,
+ unsigned int nr)
+{
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, address, ptep, __ptep_get(ptep));
+ __ptep_set_wrprotect(mm, address, ptep);
+ return;
+ }
+
+ napotpte_wrprotect_ptes(mm, address, ptep, nr);
+}
+
+#define __HAVE_ARCH_PTEP_SET_WRPROTECT
+static inline void ptep_set_wrprotect(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep)
+{
+ wrprotect_ptes(mm, address, ptep, 1);
+}
+
+#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
+static inline int ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ pte_t orig_pte = __ptep_get(ptep);
+
+ if (likely(!pte_present_napot(orig_pte)))
+ return __ptep_clear_flush_young(vma, address, ptep);
+
+ return napotpte_ptep_clear_flush_young(vma, address, ptep);
+}
+
+#else /* CONFIG_RISCV_ISA_SVNAPOT */
+
+#define napotpte_ptep_set_access_flags(vma, address, ptep, entry, dirty) \
+ ({ (void)(vma); (void)(address); (void)(ptep); (void)(entry); \
+ (void)(dirty); 0; })
+
+#define napotpte_ptep_test_and_clear_young(vma, address, ptep) \
+ ({ (void)(vma); (void)(address); (void)(ptep); 0; })
+
+#define ptep_get __ptep_get
+#define ptep_get_lockless __ptep_get_lockless
#define set_ptes __set_ptes
static inline void pte_clear(struct mm_struct *mm,
@@ -881,9 +1054,51 @@ static inline void pte_clear(struct mm_struct *mm,
#define ptep_get_and_clear __ptep_get_and_clear
#define clear_young_dirty_ptes __clear_young_dirty_ptes
+
+static inline void __clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ for (;;) {
+ __ptep_get_and_clear(mm, addr, ptep);
+ if (--nr == 0)
+ break;
+ ptep++;
+ addr += PAGE_SIZE;
+ }
+}
+
+static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ pte_t pte, tmp_pte;
+
+ pte = __ptep_get_and_clear(mm, addr, ptep);
+ while (--nr) {
+ ptep++;
+ addr += PAGE_SIZE;
+ tmp_pte = __ptep_get_and_clear(mm, addr, ptep);
+ if (pte_dirty(tmp_pte))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(tmp_pte))
+ pte = pte_mkyoung(pte);
+ }
+
+ return pte;
+}
+
+#define clear_full_ptes __clear_full_ptes
+#define get_and_clear_full_ptes __get_and_clear_full_ptes
+#define __HAVE_ARCH_PTEP_SET_WRPROTECT
#define ptep_set_wrprotect __ptep_set_wrprotect
+#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
#define ptep_clear_flush_young __ptep_clear_flush_young
+#endif /* CONFIG_RISCV_ISA_SVNAPOT */
+
+#define set_pte __set_pte
+
#define pgprot_nx pgprot_nx
static inline pgprot_t pgprot_nx(pgprot_t _prot)
{
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 43ea76e424492..3f8a0a2970afb 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -62,6 +62,24 @@ static inline unsigned long pte_protval_no_pfn_no_napot(pte_t pte)
return (pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT;
}
+static inline void napotpte_clear_young_dirty_pte(pte_t *ptep, cydp_t flags)
+{
+ pte_t old_pte, new_pte;
+ unsigned long old_val, new_val;
+
+ do {
+ old_pte = READ_ONCE(*ptep);
+ new_pte = old_pte;
+ if (flags & CYDP_CLEAR_YOUNG)
+ new_pte = pte_mkold(new_pte);
+ if (flags & CYDP_CLEAR_DIRTY)
+ new_pte = pte_mkclean(new_pte);
+
+ old_val = pte_val(old_pte);
+ new_val = pte_val(new_pte);
+ } while (cmpxchg_relaxed(&pte_val(*ptep), old_val, new_val) != old_val);
+}
+
static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
{
unsigned long pfn;
@@ -76,13 +94,34 @@ static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
return pfn_pte(pfn, prot);
}
-static inline pte_t __napot_ptep_get_and_clear(struct mm_struct *mm,
- unsigned long addr, pte_t *ptep)
+static void __clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full)
{
- pte_t pte;
+ for (;;) {
+ __ptep_get_and_clear(mm, addr, ptep);
+ if (--nr == 0)
+ break;
+ ptep++;
+ addr += PAGE_SIZE;
+ }
+}
- pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));
- page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
+static pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ pte_t pte, tmp_pte;
+
+ pte = __ptep_get_and_clear(mm, addr, ptep);
+ while (--nr) {
+ ptep++;
+ addr += PAGE_SIZE;
+ tmp_pte = __ptep_get_and_clear(mm, addr, ptep);
+ if (pte_dirty(tmp_pte))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(tmp_pte))
+ pte = pte_mkyoung(pte);
+ }
return pte;
}
@@ -102,8 +141,9 @@ static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
for (i = 0; i < nr; i++) {
ptent_addr = start_addr + i * PAGE_SIZE;
- ptent = __napot_ptep_get_and_clear(mm, ptent_addr,
- start_ptep + i);
+ ptent = __ptep_get_and_clear_noptc(start_ptep + i);
+ page_table_check_pte_clear(mm, ptent_addr,
+ pte_mknonnapot(ptent, ptent_addr));
if (pte_dirty(ptent))
target = riscv_pte_mkhwdirty(target);
if (pte_young(ptent))
@@ -136,6 +176,22 @@ static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte)
pte_val(pte_mask_ad(pte)) == pte_val(pte_mask_ad(orig_pte));
}
+static bool napotpte_all_subptes_same(pte_t *ptep, pte_t expected_pte)
+{
+ pte_t *start;
+ unsigned int i, nr;
+
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+
+ for (i = 0; i < nr; i++) {
+ if (!pte_same(READ_ONCE(start[i]), expected_pte))
+ return false;
+ }
+
+ return true;
+}
+
void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte)
{
@@ -271,3 +327,262 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
return napotpte_subpte(orig_ptep, orig_pte);
}
EXPORT_SYMBOL(napotpte_ptep_get_lockless);
+
+static void napotpte_try_unfold_range(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr)
+{
+ unsigned long next;
+ pte_t pte;
+ unsigned int chunk;
+
+ while (nr) {
+ pte = READ_ONCE(*ptep);
+ if (pte_present_napot(pte)) {
+ __napotpte_try_unfold(mm, addr, ptep, pte);
+ next = napot_align_addr(addr) + napotpte_size();
+ chunk = (next - addr) >> PAGE_SHIFT;
+ } else {
+ chunk = 1;
+ }
+
+ if (chunk > nr)
+ chunk = nr;
+
+ ptep += chunk;
+ addr += chunk * PAGE_SIZE;
+ nr -= chunk;
+ }
+}
+
+static void napotpte_try_unfold_partial(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr)
+{
+ pte_t pte;
+
+ if (ptep != napot_align_ptep(ptep) || nr < napotpte_pte_num()) {
+ pte = READ_ONCE(*ptep);
+ if (pte_present_napot(pte))
+ __napotpte_try_unfold(mm, addr, ptep, pte);
+ }
+
+ if (ptep + nr != napot_align_ptep(ptep + nr)) {
+ unsigned long last_addr;
+ pte_t *last_ptep;
+
+ last_addr = addr + PAGE_SIZE * (nr - 1);
+ last_ptep = ptep + nr - 1;
+ pte = READ_ONCE(*last_ptep);
+ if (pte_present_napot(pte))
+ __napotpte_try_unfold(mm, last_addr, last_ptep, pte);
+ }
+}
+
+void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte, unsigned int nr)
+{
+ unsigned long next, end;
+ unsigned long pfn, size, boundary;
+ pgprot_t prot;
+ unsigned int chunk, i;
+ pte_t cur;
+
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ __set_ptes(mm, addr, ptep, pte, nr);
+ return;
+ }
+
+ size = napotpte_size();
+ end = addr + ((unsigned long)nr << PAGE_SHIFT);
+ pfn = pte_pfn(pte);
+ prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
+
+ do {
+ boundary = (addr + size) & ~(size - 1);
+ next = (boundary - 1 < end - 1) ? boundary : end;
+ chunk = (next - addr) >> PAGE_SHIFT;
+
+ cur = pfn_pte(pfn, prot);
+ if (((addr | next | (pfn << PAGE_SHIFT)) & (size - 1)) == 0) {
+ cur = pte_mknapot(cur, napotpte_order());
+ page_table_check_ptes_set(mm, addr, ptep, cur, chunk);
+ for (i = 0; i < chunk; i++)
+ __set_pte_at(mm, ptep + i, cur);
+ } else {
+ __set_ptes(mm, addr, ptep, cur, chunk);
+ }
+
+ addr = next;
+ ptep += chunk;
+ pfn += chunk;
+ } while (addr != end);
+}
+EXPORT_SYMBOL(napotpte_set_ptes);
+
+void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full)
+{
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ __clear_full_ptes(mm, addr, ptep, nr, full);
+ return;
+ }
+
+ /*
+ * Svnapot stores identical napot-encoded entries across the whole block
+ * rather than per-page PFNs, so batch zap paths must unfold the covered
+ * range before the generic MM consumes ordinary per-page PTEs.
+ */
+ napotpte_try_unfold_range(mm, addr, ptep, nr);
+ __clear_full_ptes(mm, addr, ptep, nr, full);
+}
+EXPORT_SYMBOL(napotpte_clear_full_ptes);
+
+pte_t napotpte_get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ if (!napot_hw_supported() || !mm_is_user(mm))
+ return __get_and_clear_full_ptes(mm, addr, ptep, nr, full);
+
+ napotpte_try_unfold_range(mm, addr, ptep, nr);
+
+ return __get_and_clear_full_ptes(mm, addr, ptep, nr, full);
+}
+EXPORT_SYMBOL(napotpte_get_and_clear_full_ptes);
+
+void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags)
+{
+ struct mm_struct *mm;
+ unsigned long start, end;
+ unsigned int total;
+
+ mm = vma->vm_mm;
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ __clear_young_dirty_ptes(vma, addr, ptep, nr, flags);
+ return;
+ }
+
+ start = addr;
+ end = start + nr * PAGE_SIZE;
+
+ if (pte_present_napot(READ_ONCE(*(ptep + nr - 1))))
+ end = ALIGN(end, napotpte_size());
+
+ if (pte_present_napot(READ_ONCE(*ptep))) {
+ start = napot_align_addr(start);
+ ptep = napot_align_ptep(ptep);
+ }
+
+ total = (end - start) >> PAGE_SHIFT;
+ for (; total; total--, ptep++, start += PAGE_SIZE)
+ napotpte_clear_young_dirty_pte(ptep, flags);
+}
+EXPORT_SYMBOL(napotpte_clear_young_dirty_ptes);
+
+void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr)
+{
+ unsigned int i;
+
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ for (i = 0; i < nr; i++, ptep++, addr += PAGE_SIZE)
+ __ptep_set_wrprotect(mm, addr, ptep);
+ return;
+ }
+
+ napotpte_try_unfold_partial(mm, addr, ptep, nr);
+
+ for (i = 0; i < nr; i++, ptep++, addr += PAGE_SIZE)
+ __ptep_set_wrprotect(mm, addr, ptep);
+}
+EXPORT_SYMBOL(napotpte_wrprotect_ptes);
+
+int napotpte_ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty)
+{
+ pte_t raw_pte, napot_pte;
+ pte_t *start;
+ pgprot_t prot;
+ unsigned long start_addr;
+ unsigned int i, nr;
+ bool changed;
+
+ raw_pte = READ_ONCE(*ptep);
+ if (!napot_hw_supported() || !pte_present_napot(raw_pte))
+ return 0;
+
+ prot = pte_pgprot(entry);
+ napot_pte = pfn_pte(pte_pfn(raw_pte), prot);
+ napot_pte = pte_mknapot(napot_pte, napotpte_order());
+
+ if (napotpte_all_subptes_same(ptep, napot_pte))
+ return !riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC);
+
+ if (pte_write(raw_pte) != pte_write(napot_pte)) {
+ __napotpte_try_unfold(vma->vm_mm, address, ptep, raw_pte);
+ entry = pte_mknonnapot(entry, address);
+
+ return __ptep_set_access_flags(vma, address, ptep, entry,
+ dirty);
+ }
+
+ start = napot_align_ptep(ptep);
+ address = napot_align_addr(address);
+ start_addr = address;
+ nr = napotpte_pte_num();
+ changed = false;
+
+ for (i = 0; i < nr; i++, start++, address += PAGE_SIZE) {
+ if (__ptep_set_access_flags(vma, address, start, napot_pte, 0))
+ changed = true;
+ }
+
+ if (changed)
+ flush_tlb_range(vma, start_addr, start_addr + napotpte_size());
+
+ return changed;
+}
+EXPORT_SYMBOL(napotpte_ptep_set_access_flags);
+
+int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ pte_t *start;
+ unsigned int i, nr;
+ int young;
+
+ if (!napot_hw_supported() || !pte_present_napot(READ_ONCE(*ptep)))
+ return 0;
+
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+ young = 0;
+
+ for (i = 0; i < nr; i++)
+ young |= test_and_clear_bit(_PAGE_ACCESSED_OFFSET,
+ &pte_val(start[i]));
+
+ return young;
+}
+EXPORT_SYMBOL(napotpte_ptep_test_and_clear_young);
+
+int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ unsigned long start_addr;
+ int young;
+
+ young = napotpte_ptep_test_and_clear_young(vma, address, ptep);
+ if (!young)
+ return 0;
+
+ start_addr = napot_align_addr(address);
+ flush_tlb_range(vma, start_addr, start_addr + napotpte_size());
+
+ return young;
+}
+EXPORT_SYMBOL(napotpte_ptep_clear_flush_young);
diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c
index 43bd542c6ff02..50e4c4cc10691 100644
--- a/arch/riscv/mm/pgtable.c
+++ b/arch/riscv/mm/pgtable.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0
#include <asm/pgalloc.h>
+#include <linux/cpufeature.h>
#include <linux/gfp.h>
#include <linux/kernel.h>
#include <linux/pgtable.h>
@@ -32,6 +33,16 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep,
pte_t entry, int dirty)
{
+ pte_t raw_pte;
+
+ entry = pte_mknonnapot(entry, address);
+
+ raw_pte = READ_ONCE(*ptep);
+ if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT) &&
+ pte_present_napot(raw_pte))
+ return napotpte_ptep_set_access_flags(vma, address, ptep, entry,
+ dirty);
+
return __ptep_set_access_flags(vma, address, ptep, entry, dirty);
}
@@ -50,6 +61,13 @@ bool ptep_test_and_clear_young(struct vm_area_struct *vma,
unsigned long address,
pte_t *ptep)
{
+ pte_t raw_pte;
+
+ raw_pte = READ_ONCE(*ptep);
+ if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT) &&
+ pte_present_napot(raw_pte))
+ return napotpte_ptep_test_and_clear_young(vma, address, ptep);
+
return __ptep_test_and_clear_young(vma, address, ptep);
}
EXPORT_SYMBOL_GPL(ptep_test_and_clear_young);
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 04/15] riscv/mm: implement Svnapot contpte update helpers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot support needs the public RISC-V PTE update helpers to preserve
the same logical per-page view that generic MM users see on the read
side. Without a dedicated update path, operations such as set_ptes(),
ptep_set_access_flags(), and related clear or write-protect flows would
end up touching the raw NAPOT encoding directly, which leaks arch-private
representation details into generic MM code and makes folded mappings
harder to update safely.
Add Svnapot-aware contpte update helpers and route the public update-side
PTE operations through them, while keeping the raw helpers available for
RISC-V code that still needs direct access to the encoded entries.
This also wires the update path into the related mechanics needed when a
Svnapot block is unfolded and rewritten, so page-table-check, access-flag
updates, and non-Svnapot builds continue to observe the expected public
PTE semantics.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 281 ++++++++++++++++++++++----
arch/riscv/mm/contpte.c | 329 ++++++++++++++++++++++++++++++-
arch/riscv/mm/pgtable.c | 18 ++
3 files changed, 588 insertions(+), 40 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index fce6d04ea65db..771faabb2b00b 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -413,6 +413,30 @@ static inline int pte_special(pte_t pte)
return pte_val(pte) & _PAGE_SPECIAL;
}
+#ifdef CONFIG_RISCV_ISA_SVNAPOT
+static inline pte_t pte_mknonnapot(pte_t pte, unsigned long addr)
+{
+ unsigned long pfn;
+ unsigned long offset;
+ pgprot_t prot;
+
+ if (!pte_present_napot(pte))
+ return pte;
+
+ offset = (addr & (napot_cont_size(napot_cont_order(pte)) - 1)) >>
+ PAGE_SHIFT;
+ pfn = pte_pfn(pte) + offset;
+ prot = __pgprot((pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT);
+
+ return pfn_pte(pfn, prot);
+}
+#else
+static inline pte_t pte_mknonnapot(pte_t pte, unsigned long addr)
+{
+ return pte;
+}
+#endif
+
/* static inline pte_t pte_rdprotect(pte_t pte) */
static inline pte_t pte_wrprotect(pte_t pte)
@@ -555,6 +579,25 @@ void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte);
pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte);
pte_t napotpte_ptep_get_lockless(pte_t *ptep);
+void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte, unsigned int nr);
+void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full);
+pte_t napotpte_get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full);
+void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags);
+void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr);
+int napotpte_ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty);
+int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep);
+int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep);
#endif
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
@@ -706,37 +749,6 @@ static inline pte_t __ptep_get_lockless(pte_t *ptep)
return __ptep_get(ptep);
}
-#ifdef CONFIG_RISCV_ISA_SVNAPOT
-
-#define ptep_get ptep_get
-static inline pte_t ptep_get(pte_t *ptep)
-{
- pte_t pte = __ptep_get(ptep);
-
- if (likely(!pte_present_napot(pte)))
- return pte;
-
- return napotpte_ptep_get(ptep, pte);
-}
-
-#define ptep_get_lockless ptep_get_lockless
-static inline pte_t ptep_get_lockless(pte_t *ptep)
-{
- pte_t pte = __ptep_get_lockless(ptep);
-
- if (likely(!pte_present_napot(pte)))
- return pte;
-
- return napotpte_ptep_get_lockless(ptep);
-}
-
-#else
-
-#define ptep_get __ptep_get
-#define ptep_get_lockless __ptep_get_lockless
-
-#endif
-
static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,
unsigned long addr, pte_t *ptep,
pte_t pte, cydp_t flags)
@@ -829,7 +841,6 @@ static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,
#define __ptep_clear_flush __ptep_clear_flush
-#define __HAVE_ARCH_PTEP_SET_WRPROTECT
static inline void __ptep_set_wrprotect(struct mm_struct *mm,
unsigned long address, pte_t *ptep)
{
@@ -870,7 +881,169 @@ static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
#define __ptep_clear_flush_young __ptep_clear_flush_young
-#define set_pte __set_pte
+#ifdef CONFIG_RISCV_ISA_SVNAPOT
+
+static __always_inline void napotpte_try_fold(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ pte_t pte)
+{
+ const unsigned long contmask = napot_pte_num(NAPOT_CONT64KB_ORDER) - 1;
+ bool valign = ((addr >> PAGE_SHIFT) & contmask) == contmask;
+
+ if (unlikely(valign)) {
+ bool palign = (pte_pfn(pte) & contmask) == contmask;
+
+ if (unlikely(palign && pte_present(pte) && !pte_napot(pte) &&
+ !pte_special(pte)))
+ __napotpte_try_fold(mm, addr, ptep, pte);
+ }
+}
+
+static __always_inline void napotpte_try_unfold(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ pte_t pte)
+{
+ if (unlikely(pte_present_napot(pte)))
+ __napotpte_try_unfold(mm, addr, ptep, pte);
+}
+
+/*
+ * Public PTE helpers may transparently handle Svnapot-encoded mappings.
+ * NAPOT-aware arch users should stick to the private/raw __pte* helpers.
+ */
+#define ptep_get ptep_get
+static inline pte_t ptep_get(pte_t *ptep)
+{
+ pte_t pte = __ptep_get(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get(ptep, pte);
+}
+
+#define ptep_get_lockless ptep_get_lockless
+static inline pte_t ptep_get_lockless(pte_t *ptep)
+{
+ pte_t pte = __ptep_get_lockless(ptep);
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get_lockless(ptep);
+}
+
+#define set_ptes set_ptes
+static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pteval, unsigned int nr)
+{
+ pteval = pte_mknonnapot(pteval, addr);
+
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ __set_ptes(mm, addr, ptep, pteval, 1);
+ napotpte_try_fold(mm, addr, ptep, pteval);
+ return;
+ }
+
+ napotpte_set_ptes(mm, addr, ptep, pteval, nr);
+}
+
+static inline void pte_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ __pte_clear(mm, addr, ptep);
+}
+
+static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep)
+{
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ return __ptep_get_and_clear(mm, addr, ptep);
+}
+
+#define clear_young_dirty_ptes clear_young_dirty_ptes
+static inline void clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags)
+{
+ if (likely(nr == 1 && !pte_present_napot(__ptep_get(ptep))))
+ __clear_young_dirty_ptes(vma, addr, ptep, nr, flags);
+ else
+ napotpte_clear_young_dirty_ptes(vma, addr, ptep, nr, flags);
+}
+
+#define clear_full_ptes clear_full_ptes
+static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full)
+{
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ __ptep_get_and_clear(mm, addr, ptep);
+ return;
+ }
+
+ napotpte_clear_full_ptes(mm, addr, ptep, nr, full);
+}
+
+#define get_and_clear_full_ptes get_and_clear_full_ptes
+static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
+ return __ptep_get_and_clear(mm, addr, ptep);
+ }
+
+ return napotpte_get_and_clear_full_ptes(mm, addr, ptep, nr, full);
+}
+
+#define wrprotect_ptes wrprotect_ptes
+static inline void wrprotect_ptes(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep,
+ unsigned int nr)
+{
+ if (likely(nr == 1)) {
+ napotpte_try_unfold(mm, address, ptep, __ptep_get(ptep));
+ __ptep_set_wrprotect(mm, address, ptep);
+ return;
+ }
+
+ napotpte_wrprotect_ptes(mm, address, ptep, nr);
+}
+
+#define __HAVE_ARCH_PTEP_SET_WRPROTECT
+static inline void ptep_set_wrprotect(struct mm_struct *mm,
+ unsigned long address, pte_t *ptep)
+{
+ wrprotect_ptes(mm, address, ptep, 1);
+}
+
+#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
+static inline int ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ pte_t orig_pte = __ptep_get(ptep);
+
+ if (likely(!pte_present_napot(orig_pte)))
+ return __ptep_clear_flush_young(vma, address, ptep);
+
+ return napotpte_ptep_clear_flush_young(vma, address, ptep);
+}
+
+#else /* CONFIG_RISCV_ISA_SVNAPOT */
+
+#define napotpte_ptep_set_access_flags(vma, address, ptep, entry, dirty) \
+ ({ (void)(vma); (void)(address); (void)(ptep); (void)(entry); \
+ (void)(dirty); 0; })
+
+#define napotpte_ptep_test_and_clear_young(vma, address, ptep) \
+ ({ (void)(vma); (void)(address); (void)(ptep); 0; })
+
+#define ptep_get __ptep_get
+#define ptep_get_lockless __ptep_get_lockless
#define set_ptes __set_ptes
static inline void pte_clear(struct mm_struct *mm,
@@ -881,9 +1054,51 @@ static inline void pte_clear(struct mm_struct *mm,
#define ptep_get_and_clear __ptep_get_and_clear
#define clear_young_dirty_ptes __clear_young_dirty_ptes
+
+static inline void __clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ for (;;) {
+ __ptep_get_and_clear(mm, addr, ptep);
+ if (--nr == 0)
+ break;
+ ptep++;
+ addr += PAGE_SIZE;
+ }
+}
+
+static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ pte_t pte, tmp_pte;
+
+ pte = __ptep_get_and_clear(mm, addr, ptep);
+ while (--nr) {
+ ptep++;
+ addr += PAGE_SIZE;
+ tmp_pte = __ptep_get_and_clear(mm, addr, ptep);
+ if (pte_dirty(tmp_pte))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(tmp_pte))
+ pte = pte_mkyoung(pte);
+ }
+
+ return pte;
+}
+
+#define clear_full_ptes __clear_full_ptes
+#define get_and_clear_full_ptes __get_and_clear_full_ptes
+#define __HAVE_ARCH_PTEP_SET_WRPROTECT
#define ptep_set_wrprotect __ptep_set_wrprotect
+#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
#define ptep_clear_flush_young __ptep_clear_flush_young
+#endif /* CONFIG_RISCV_ISA_SVNAPOT */
+
+#define set_pte __set_pte
+
#define pgprot_nx pgprot_nx
static inline pgprot_t pgprot_nx(pgprot_t _prot)
{
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 43ea76e424492..3f8a0a2970afb 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -62,6 +62,24 @@ static inline unsigned long pte_protval_no_pfn_no_napot(pte_t pte)
return (pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT;
}
+static inline void napotpte_clear_young_dirty_pte(pte_t *ptep, cydp_t flags)
+{
+ pte_t old_pte, new_pte;
+ unsigned long old_val, new_val;
+
+ do {
+ old_pte = READ_ONCE(*ptep);
+ new_pte = old_pte;
+ if (flags & CYDP_CLEAR_YOUNG)
+ new_pte = pte_mkold(new_pte);
+ if (flags & CYDP_CLEAR_DIRTY)
+ new_pte = pte_mkclean(new_pte);
+
+ old_val = pte_val(old_pte);
+ new_val = pte_val(new_pte);
+ } while (cmpxchg_relaxed(&pte_val(*ptep), old_val, new_val) != old_val);
+}
+
static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
{
unsigned long pfn;
@@ -76,13 +94,34 @@ static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
return pfn_pte(pfn, prot);
}
-static inline pte_t __napot_ptep_get_and_clear(struct mm_struct *mm,
- unsigned long addr, pte_t *ptep)
+static void __clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full)
{
- pte_t pte;
+ for (;;) {
+ __ptep_get_and_clear(mm, addr, ptep);
+ if (--nr == 0)
+ break;
+ ptep++;
+ addr += PAGE_SIZE;
+ }
+}
- pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));
- page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
+static pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ pte_t pte, tmp_pte;
+
+ pte = __ptep_get_and_clear(mm, addr, ptep);
+ while (--nr) {
+ ptep++;
+ addr += PAGE_SIZE;
+ tmp_pte = __ptep_get_and_clear(mm, addr, ptep);
+ if (pte_dirty(tmp_pte))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(tmp_pte))
+ pte = pte_mkyoung(pte);
+ }
return pte;
}
@@ -102,8 +141,9 @@ static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
for (i = 0; i < nr; i++) {
ptent_addr = start_addr + i * PAGE_SIZE;
- ptent = __napot_ptep_get_and_clear(mm, ptent_addr,
- start_ptep + i);
+ ptent = __ptep_get_and_clear_noptc(start_ptep + i);
+ page_table_check_pte_clear(mm, ptent_addr,
+ pte_mknonnapot(ptent, ptent_addr));
if (pte_dirty(ptent))
target = riscv_pte_mkhwdirty(target);
if (pte_young(ptent))
@@ -136,6 +176,22 @@ static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte)
pte_val(pte_mask_ad(pte)) == pte_val(pte_mask_ad(orig_pte));
}
+static bool napotpte_all_subptes_same(pte_t *ptep, pte_t expected_pte)
+{
+ pte_t *start;
+ unsigned int i, nr;
+
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+
+ for (i = 0; i < nr; i++) {
+ if (!pte_same(READ_ONCE(start[i]), expected_pte))
+ return false;
+ }
+
+ return true;
+}
+
void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte)
{
@@ -271,3 +327,262 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
return napotpte_subpte(orig_ptep, orig_pte);
}
EXPORT_SYMBOL(napotpte_ptep_get_lockless);
+
+static void napotpte_try_unfold_range(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr)
+{
+ unsigned long next;
+ pte_t pte;
+ unsigned int chunk;
+
+ while (nr) {
+ pte = READ_ONCE(*ptep);
+ if (pte_present_napot(pte)) {
+ __napotpte_try_unfold(mm, addr, ptep, pte);
+ next = napot_align_addr(addr) + napotpte_size();
+ chunk = (next - addr) >> PAGE_SHIFT;
+ } else {
+ chunk = 1;
+ }
+
+ if (chunk > nr)
+ chunk = nr;
+
+ ptep += chunk;
+ addr += chunk * PAGE_SIZE;
+ nr -= chunk;
+ }
+}
+
+static void napotpte_try_unfold_partial(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr)
+{
+ pte_t pte;
+
+ if (ptep != napot_align_ptep(ptep) || nr < napotpte_pte_num()) {
+ pte = READ_ONCE(*ptep);
+ if (pte_present_napot(pte))
+ __napotpte_try_unfold(mm, addr, ptep, pte);
+ }
+
+ if (ptep + nr != napot_align_ptep(ptep + nr)) {
+ unsigned long last_addr;
+ pte_t *last_ptep;
+
+ last_addr = addr + PAGE_SIZE * (nr - 1);
+ last_ptep = ptep + nr - 1;
+ pte = READ_ONCE(*last_ptep);
+ if (pte_present_napot(pte))
+ __napotpte_try_unfold(mm, last_addr, last_ptep, pte);
+ }
+}
+
+void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, pte_t pte, unsigned int nr)
+{
+ unsigned long next, end;
+ unsigned long pfn, size, boundary;
+ pgprot_t prot;
+ unsigned int chunk, i;
+ pte_t cur;
+
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ __set_ptes(mm, addr, ptep, pte, nr);
+ return;
+ }
+
+ size = napotpte_size();
+ end = addr + ((unsigned long)nr << PAGE_SHIFT);
+ pfn = pte_pfn(pte);
+ prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
+
+ do {
+ boundary = (addr + size) & ~(size - 1);
+ next = (boundary - 1 < end - 1) ? boundary : end;
+ chunk = (next - addr) >> PAGE_SHIFT;
+
+ cur = pfn_pte(pfn, prot);
+ if (((addr | next | (pfn << PAGE_SHIFT)) & (size - 1)) == 0) {
+ cur = pte_mknapot(cur, napotpte_order());
+ page_table_check_ptes_set(mm, addr, ptep, cur, chunk);
+ for (i = 0; i < chunk; i++)
+ __set_pte_at(mm, ptep + i, cur);
+ } else {
+ __set_ptes(mm, addr, ptep, cur, chunk);
+ }
+
+ addr = next;
+ ptep += chunk;
+ pfn += chunk;
+ } while (addr != end);
+}
+EXPORT_SYMBOL(napotpte_set_ptes);
+
+void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr, int full)
+{
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ __clear_full_ptes(mm, addr, ptep, nr, full);
+ return;
+ }
+
+ /*
+ * Svnapot stores identical napot-encoded entries across the whole block
+ * rather than per-page PFNs, so batch zap paths must unfold the covered
+ * range before the generic MM consumes ordinary per-page PTEs.
+ */
+ napotpte_try_unfold_range(mm, addr, ptep, nr);
+ __clear_full_ptes(mm, addr, ptep, nr, full);
+}
+EXPORT_SYMBOL(napotpte_clear_full_ptes);
+
+pte_t napotpte_get_and_clear_full_ptes(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, int full)
+{
+ if (!napot_hw_supported() || !mm_is_user(mm))
+ return __get_and_clear_full_ptes(mm, addr, ptep, nr, full);
+
+ napotpte_try_unfold_range(mm, addr, ptep, nr);
+
+ return __get_and_clear_full_ptes(mm, addr, ptep, nr, full);
+}
+EXPORT_SYMBOL(napotpte_get_and_clear_full_ptes);
+
+void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, cydp_t flags)
+{
+ struct mm_struct *mm;
+ unsigned long start, end;
+ unsigned int total;
+
+ mm = vma->vm_mm;
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ __clear_young_dirty_ptes(vma, addr, ptep, nr, flags);
+ return;
+ }
+
+ start = addr;
+ end = start + nr * PAGE_SIZE;
+
+ if (pte_present_napot(READ_ONCE(*(ptep + nr - 1))))
+ end = ALIGN(end, napotpte_size());
+
+ if (pte_present_napot(READ_ONCE(*ptep))) {
+ start = napot_align_addr(start);
+ ptep = napot_align_ptep(ptep);
+ }
+
+ total = (end - start) >> PAGE_SHIFT;
+ for (; total; total--, ptep++, start += PAGE_SIZE)
+ napotpte_clear_young_dirty_pte(ptep, flags);
+}
+EXPORT_SYMBOL(napotpte_clear_young_dirty_ptes);
+
+void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr,
+ pte_t *ptep, unsigned int nr)
+{
+ unsigned int i;
+
+ if (!napot_hw_supported() || !mm_is_user(mm)) {
+ for (i = 0; i < nr; i++, ptep++, addr += PAGE_SIZE)
+ __ptep_set_wrprotect(mm, addr, ptep);
+ return;
+ }
+
+ napotpte_try_unfold_partial(mm, addr, ptep, nr);
+
+ for (i = 0; i < nr; i++, ptep++, addr += PAGE_SIZE)
+ __ptep_set_wrprotect(mm, addr, ptep);
+}
+EXPORT_SYMBOL(napotpte_wrprotect_ptes);
+
+int napotpte_ptep_set_access_flags(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep,
+ pte_t entry, int dirty)
+{
+ pte_t raw_pte, napot_pte;
+ pte_t *start;
+ pgprot_t prot;
+ unsigned long start_addr;
+ unsigned int i, nr;
+ bool changed;
+
+ raw_pte = READ_ONCE(*ptep);
+ if (!napot_hw_supported() || !pte_present_napot(raw_pte))
+ return 0;
+
+ prot = pte_pgprot(entry);
+ napot_pte = pfn_pte(pte_pfn(raw_pte), prot);
+ napot_pte = pte_mknapot(napot_pte, napotpte_order());
+
+ if (napotpte_all_subptes_same(ptep, napot_pte))
+ return !riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC);
+
+ if (pte_write(raw_pte) != pte_write(napot_pte)) {
+ __napotpte_try_unfold(vma->vm_mm, address, ptep, raw_pte);
+ entry = pte_mknonnapot(entry, address);
+
+ return __ptep_set_access_flags(vma, address, ptep, entry,
+ dirty);
+ }
+
+ start = napot_align_ptep(ptep);
+ address = napot_align_addr(address);
+ start_addr = address;
+ nr = napotpte_pte_num();
+ changed = false;
+
+ for (i = 0; i < nr; i++, start++, address += PAGE_SIZE) {
+ if (__ptep_set_access_flags(vma, address, start, napot_pte, 0))
+ changed = true;
+ }
+
+ if (changed)
+ flush_tlb_range(vma, start_addr, start_addr + napotpte_size());
+
+ return changed;
+}
+EXPORT_SYMBOL(napotpte_ptep_set_access_flags);
+
+int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ pte_t *start;
+ unsigned int i, nr;
+ int young;
+
+ if (!napot_hw_supported() || !pte_present_napot(READ_ONCE(*ptep)))
+ return 0;
+
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+ young = 0;
+
+ for (i = 0; i < nr; i++)
+ young |= test_and_clear_bit(_PAGE_ACCESSED_OFFSET,
+ &pte_val(start[i]));
+
+ return young;
+}
+EXPORT_SYMBOL(napotpte_ptep_test_and_clear_young);
+
+int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,
+ unsigned long address, pte_t *ptep)
+{
+ unsigned long start_addr;
+ int young;
+
+ young = napotpte_ptep_test_and_clear_young(vma, address, ptep);
+ if (!young)
+ return 0;
+
+ start_addr = napot_align_addr(address);
+ flush_tlb_range(vma, start_addr, start_addr + napotpte_size());
+
+ return young;
+}
+EXPORT_SYMBOL(napotpte_ptep_clear_flush_young);
diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c
index 43bd542c6ff02..50e4c4cc10691 100644
--- a/arch/riscv/mm/pgtable.c
+++ b/arch/riscv/mm/pgtable.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0
#include <asm/pgalloc.h>
+#include <linux/cpufeature.h>
#include <linux/gfp.h>
#include <linux/kernel.h>
#include <linux/pgtable.h>
@@ -32,6 +33,16 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep,
pte_t entry, int dirty)
{
+ pte_t raw_pte;
+
+ entry = pte_mknonnapot(entry, address);
+
+ raw_pte = READ_ONCE(*ptep);
+ if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT) &&
+ pte_present_napot(raw_pte))
+ return napotpte_ptep_set_access_flags(vma, address, ptep, entry,
+ dirty);
+
return __ptep_set_access_flags(vma, address, ptep, entry, dirty);
}
@@ -50,6 +61,13 @@ bool ptep_test_and_clear_young(struct vm_area_struct *vma,
unsigned long address,
pte_t *ptep)
{
+ pte_t raw_pte;
+
+ raw_pte = READ_ONCE(*ptep);
+ if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT) &&
+ pte_present_napot(raw_pte))
+ return napotpte_ptep_test_and_clear_young(vma, address, ptep);
+
return __ptep_test_and_clear_young(vma, address, ptep);
}
EXPORT_SYMBOL_GPL(ptep_test_and_clear_young);
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 05/15] mm: extend pte batch and leaf-size helpers
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Folded mappings such as RISC-V Svnapot need two extra pieces of generic
MM support.
First, architecture pte_batch_hint() implementations need access to the
same batch flags and normalization rules used by folio_pte_batch(), so
folded ranges can participate in batching with semantics consistent with
generic folio comparisons.
Second, leaf-size users may need the original PTE pointer instead of the
public ptep_get() view. For folded mappings, the public helper can return
a logical per-page sub-PTE, while the effective leaf size still depends
on the raw encoded entry in the page table.
Move the batch flag helpers into pgtable.h, extend pte_batch_hint() to
take the batch flags, add __ptep_leaf_size(), and update the generic
callers to use the new interfaces.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/arm64/include/asm/pgtable.h | 9 +++++-
include/linux/pgtable.h | 53 +++++++++++++++++++++++++++++++-
kernel/events/core.c | 2 +-
mm/internal.h | 39 ++---------------------
mm/mincore.c | 2 +-
mm/mremap.c | 2 +-
6 files changed, 65 insertions(+), 42 deletions(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index a2681d7553584..018781a1fcbee 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -5,6 +5,13 @@
#ifndef __ASM_PGTABLE_H
#define __ASM_PGTABLE_H
+#ifndef __ASSEMBLY__
+#ifndef __LINUX_FPB_T
+#define __LINUX_FPB_T
+typedef int __bitwise fpb_t;
+#endif
+#endif
+
#include <asm/bug.h>
#include <asm/proc-fns.h>
@@ -1705,7 +1712,7 @@ static __always_inline void contpte_try_unfold(struct mm_struct *mm,
}
#define pte_batch_hint pte_batch_hint
-static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte)
+static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
{
if (!pte_valid_cont(pte))
return 1;
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 8c093c119e5a8..da14328093a86 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -3,6 +3,35 @@
#define _LINUX_PGTABLE_H
#include <linux/pfn.h>
+
+#ifndef __ASSEMBLY__
+#ifndef __LINUX_FPB_T
+#define __LINUX_FPB_T
+typedef int __bitwise fpb_t;
+#endif
+
+/* Compare PTEs respecting the dirty bit. */
+#define FPB_RESPECT_DIRTY ((__force fpb_t)BIT(0))
+
+/* Compare PTEs respecting the soft-dirty bit. */
+#define FPB_RESPECT_SOFT_DIRTY ((__force fpb_t)BIT(1))
+
+/* Compare PTEs respecting the writable bit. */
+#define FPB_RESPECT_WRITE ((__force fpb_t)BIT(2))
+
+/*
+ * Merge PTE write bits: if any PTE in the batch is writable, modify the
+ * PTE at @ptentp to be writable.
+ */
+#define FPB_MERGE_WRITE ((__force fpb_t)BIT(3))
+
+/*
+ * Merge PTE young and dirty bits: if any PTE in the batch is young or dirty,
+ * modify the PTE at @ptentp to be young or dirty, respectively.
+ */
+#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(4))
+#endif /* !__ASSEMBLY__ */
+
#include <asm/pgtable.h>
#define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT)
@@ -397,6 +426,7 @@ static inline void lazy_mmu_mode_resume(void) {}
* pte_batch_hint - Number of pages that can be added to batch without scanning.
* @ptep: Page table pointer for the entry.
* @pte: Page table entry.
+ * @flags: Flags that control PTE normalization for batch comparisons.
*
* Some architectures know that a set of contiguous ptes all map the same
* contiguous memory with the same permissions. In this case, it can provide a
@@ -407,7 +437,7 @@ static inline void lazy_mmu_mode_resume(void) {}
*
* May be overridden by the architecture, else pte_batch_hint is always 1.
*/
-static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte)
+static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
{
return 1;
}
@@ -1855,6 +1885,7 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
return pmd;
}
#endif
+
#else /* !CONFIG_HAVE_ARCH_SOFT_DIRTY */
static inline int pte_soft_dirty(pte_t pte)
{
@@ -1917,6 +1948,17 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
}
#endif
+static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
+{
+ if (!(flags & FPB_RESPECT_DIRTY))
+ pte = pte_mkclean(pte);
+ if (likely(!(flags & FPB_RESPECT_SOFT_DIRTY)))
+ pte = pte_clear_soft_dirty(pte);
+ if (likely(!(flags & FPB_RESPECT_WRITE)))
+ pte = pte_wrprotect(pte);
+ return pte_mkold(pte);
+}
+
#ifndef __HAVE_PFNMAP_TRACKING
/*
* Interfaces that can be used by architecture code to keep track of
@@ -2414,6 +2456,15 @@ static inline const char *pgtable_level_to_str(enum pgtable_level level)
#define __pte_leaf_size(x,y) pte_leaf_size(y)
#endif
+#ifndef __ptep_leaf_size
+#ifndef __ASSEMBLY__
+static inline unsigned long __ptep_leaf_size(pmd_t pmd, pte_t *ptep, pte_t pte)
+{
+ return __pte_leaf_size(pmd, pte);
+}
+#endif /* !__ASSEMBLY__ */
+#endif
+
/*
* We always define pmd_pfn for all archs as it's used in lots of generic
* code. Now it happens too for pud_pfn (and can happen for larger
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 51c1200ea3fdd..37b860abe9e61 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8551,7 +8551,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr)
pte = ptep_get_lockless(ptep);
if (pte_present(pte))
- size = __pte_leaf_size(pmd, pte);
+ size = __ptep_leaf_size(pmd, ptep, pte);
pte_unmap(ptep);
#endif /* CONFIG_HAVE_GUP_FAST */
diff --git a/mm/internal.h b/mm/internal.h
index 874be94cf2570..aff00a3fdff50 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -343,41 +343,6 @@ static inline int anon_vma_prepare(struct vm_area_struct *vma)
return __anon_vma_prepare(vma);
}
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs respecting the dirty bit. */
-#define FPB_RESPECT_DIRTY ((__force fpb_t)BIT(0))
-
-/* Compare PTEs respecting the soft-dirty bit. */
-#define FPB_RESPECT_SOFT_DIRTY ((__force fpb_t)BIT(1))
-
-/* Compare PTEs respecting the writable bit. */
-#define FPB_RESPECT_WRITE ((__force fpb_t)BIT(2))
-
-/*
- * Merge PTE write bits: if any PTE in the batch is writable, modify the
- * PTE at @ptentp to be writable.
- */
-#define FPB_MERGE_WRITE ((__force fpb_t)BIT(3))
-
-/*
- * Merge PTE young and dirty bits: if any PTE in the batch is young or dirty,
- * modify the PTE at @ptentp to be young or dirty, respectively.
- */
-#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(4))
-
-static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
-{
- if (!(flags & FPB_RESPECT_DIRTY))
- pte = pte_mkclean(pte);
- if (likely(!(flags & FPB_RESPECT_SOFT_DIRTY)))
- pte = pte_clear_soft_dirty(pte);
- if (likely(!(flags & FPB_RESPECT_WRITE)))
- pte = pte_wrprotect(pte);
- return pte_mkold(pte);
-}
-
/**
* folio_pte_batch_flags - detect a PTE batch for a large folio
* @folio: The large folio to detect a PTE batch for.
@@ -430,7 +395,7 @@ static inline unsigned int folio_pte_batch_flags(struct folio *folio,
max_nr = min_t(unsigned long, max_nr,
folio_pfn(folio) + folio_nr_pages(folio) - pte_pfn(pte));
- nr = pte_batch_hint(ptep, pte);
+ nr = pte_batch_hint(ptep, pte, flags);
expected_pte = __pte_batch_clear_ignored(pte_advance_pfn(pte, nr), flags);
ptep = ptep + nr;
@@ -447,7 +412,7 @@ static inline unsigned int folio_pte_batch_flags(struct folio *folio,
any_dirty |= pte_dirty(pte);
}
- cur_nr = pte_batch_hint(ptep, pte);
+ cur_nr = pte_batch_hint(ptep, pte, flags);
expected_pte = pte_advance_pfn(expected_pte, cur_nr);
ptep += cur_nr;
nr += cur_nr;
diff --git a/mm/mincore.c b/mm/mincore.c
index 53b9828037713..e23ef3f270db6 100644
--- a/mm/mincore.c
+++ b/mm/mincore.c
@@ -191,7 +191,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
__mincore_unmapped_range(addr, addr + PAGE_SIZE,
vma, vec);
else if (pte_present(pte)) {
- unsigned int batch = pte_batch_hint(ptep, pte);
+ unsigned int batch = pte_batch_hint(ptep, pte, 0);
if (batch > 1) {
unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
diff --git a/mm/mremap.c b/mm/mremap.c
index 384ef4cc2195b..f8b585e59771d 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -184,7 +184,7 @@ static int mremap_folio_pte_batch(struct vm_area_struct *vma, unsigned long addr
return 1;
/* Avoid expensive folio lookup if we stand no chance of benefit. */
- if (pte_batch_hint(ptep, pte) == 1)
+ if (pte_batch_hint(ptep, pte, 0) == 1)
return 1;
folio = vm_normal_folio(vma, addr, pte);
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 05/15] mm: extend pte batch and leaf-size helpers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Folded mappings such as RISC-V Svnapot need two extra pieces of generic
MM support.
First, architecture pte_batch_hint() implementations need access to the
same batch flags and normalization rules used by folio_pte_batch(), so
folded ranges can participate in batching with semantics consistent with
generic folio comparisons.
Second, leaf-size users may need the original PTE pointer instead of the
public ptep_get() view. For folded mappings, the public helper can return
a logical per-page sub-PTE, while the effective leaf size still depends
on the raw encoded entry in the page table.
Move the batch flag helpers into pgtable.h, extend pte_batch_hint() to
take the batch flags, add __ptep_leaf_size(), and update the generic
callers to use the new interfaces.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/arm64/include/asm/pgtable.h | 9 +++++-
include/linux/pgtable.h | 53 +++++++++++++++++++++++++++++++-
kernel/events/core.c | 2 +-
mm/internal.h | 39 ++---------------------
mm/mincore.c | 2 +-
mm/mremap.c | 2 +-
6 files changed, 65 insertions(+), 42 deletions(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index a2681d7553584..018781a1fcbee 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -5,6 +5,13 @@
#ifndef __ASM_PGTABLE_H
#define __ASM_PGTABLE_H
+#ifndef __ASSEMBLY__
+#ifndef __LINUX_FPB_T
+#define __LINUX_FPB_T
+typedef int __bitwise fpb_t;
+#endif
+#endif
+
#include <asm/bug.h>
#include <asm/proc-fns.h>
@@ -1705,7 +1712,7 @@ static __always_inline void contpte_try_unfold(struct mm_struct *mm,
}
#define pte_batch_hint pte_batch_hint
-static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte)
+static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
{
if (!pte_valid_cont(pte))
return 1;
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 8c093c119e5a8..da14328093a86 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -3,6 +3,35 @@
#define _LINUX_PGTABLE_H
#include <linux/pfn.h>
+
+#ifndef __ASSEMBLY__
+#ifndef __LINUX_FPB_T
+#define __LINUX_FPB_T
+typedef int __bitwise fpb_t;
+#endif
+
+/* Compare PTEs respecting the dirty bit. */
+#define FPB_RESPECT_DIRTY ((__force fpb_t)BIT(0))
+
+/* Compare PTEs respecting the soft-dirty bit. */
+#define FPB_RESPECT_SOFT_DIRTY ((__force fpb_t)BIT(1))
+
+/* Compare PTEs respecting the writable bit. */
+#define FPB_RESPECT_WRITE ((__force fpb_t)BIT(2))
+
+/*
+ * Merge PTE write bits: if any PTE in the batch is writable, modify the
+ * PTE at @ptentp to be writable.
+ */
+#define FPB_MERGE_WRITE ((__force fpb_t)BIT(3))
+
+/*
+ * Merge PTE young and dirty bits: if any PTE in the batch is young or dirty,
+ * modify the PTE at @ptentp to be young or dirty, respectively.
+ */
+#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(4))
+#endif /* !__ASSEMBLY__ */
+
#include <asm/pgtable.h>
#define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT)
@@ -397,6 +426,7 @@ static inline void lazy_mmu_mode_resume(void) {}
* pte_batch_hint - Number of pages that can be added to batch without scanning.
* @ptep: Page table pointer for the entry.
* @pte: Page table entry.
+ * @flags: Flags that control PTE normalization for batch comparisons.
*
* Some architectures know that a set of contiguous ptes all map the same
* contiguous memory with the same permissions. In this case, it can provide a
@@ -407,7 +437,7 @@ static inline void lazy_mmu_mode_resume(void) {}
*
* May be overridden by the architecture, else pte_batch_hint is always 1.
*/
-static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte)
+static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
{
return 1;
}
@@ -1855,6 +1885,7 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
return pmd;
}
#endif
+
#else /* !CONFIG_HAVE_ARCH_SOFT_DIRTY */
static inline int pte_soft_dirty(pte_t pte)
{
@@ -1917,6 +1948,17 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
}
#endif
+static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
+{
+ if (!(flags & FPB_RESPECT_DIRTY))
+ pte = pte_mkclean(pte);
+ if (likely(!(flags & FPB_RESPECT_SOFT_DIRTY)))
+ pte = pte_clear_soft_dirty(pte);
+ if (likely(!(flags & FPB_RESPECT_WRITE)))
+ pte = pte_wrprotect(pte);
+ return pte_mkold(pte);
+}
+
#ifndef __HAVE_PFNMAP_TRACKING
/*
* Interfaces that can be used by architecture code to keep track of
@@ -2414,6 +2456,15 @@ static inline const char *pgtable_level_to_str(enum pgtable_level level)
#define __pte_leaf_size(x,y) pte_leaf_size(y)
#endif
+#ifndef __ptep_leaf_size
+#ifndef __ASSEMBLY__
+static inline unsigned long __ptep_leaf_size(pmd_t pmd, pte_t *ptep, pte_t pte)
+{
+ return __pte_leaf_size(pmd, pte);
+}
+#endif /* !__ASSEMBLY__ */
+#endif
+
/*
* We always define pmd_pfn for all archs as it's used in lots of generic
* code. Now it happens too for pud_pfn (and can happen for larger
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 51c1200ea3fdd..37b860abe9e61 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8551,7 +8551,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr)
pte = ptep_get_lockless(ptep);
if (pte_present(pte))
- size = __pte_leaf_size(pmd, pte);
+ size = __ptep_leaf_size(pmd, ptep, pte);
pte_unmap(ptep);
#endif /* CONFIG_HAVE_GUP_FAST */
diff --git a/mm/internal.h b/mm/internal.h
index 874be94cf2570..aff00a3fdff50 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -343,41 +343,6 @@ static inline int anon_vma_prepare(struct vm_area_struct *vma)
return __anon_vma_prepare(vma);
}
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs respecting the dirty bit. */
-#define FPB_RESPECT_DIRTY ((__force fpb_t)BIT(0))
-
-/* Compare PTEs respecting the soft-dirty bit. */
-#define FPB_RESPECT_SOFT_DIRTY ((__force fpb_t)BIT(1))
-
-/* Compare PTEs respecting the writable bit. */
-#define FPB_RESPECT_WRITE ((__force fpb_t)BIT(2))
-
-/*
- * Merge PTE write bits: if any PTE in the batch is writable, modify the
- * PTE at @ptentp to be writable.
- */
-#define FPB_MERGE_WRITE ((__force fpb_t)BIT(3))
-
-/*
- * Merge PTE young and dirty bits: if any PTE in the batch is young or dirty,
- * modify the PTE at @ptentp to be young or dirty, respectively.
- */
-#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(4))
-
-static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
-{
- if (!(flags & FPB_RESPECT_DIRTY))
- pte = pte_mkclean(pte);
- if (likely(!(flags & FPB_RESPECT_SOFT_DIRTY)))
- pte = pte_clear_soft_dirty(pte);
- if (likely(!(flags & FPB_RESPECT_WRITE)))
- pte = pte_wrprotect(pte);
- return pte_mkold(pte);
-}
-
/**
* folio_pte_batch_flags - detect a PTE batch for a large folio
* @folio: The large folio to detect a PTE batch for.
@@ -430,7 +395,7 @@ static inline unsigned int folio_pte_batch_flags(struct folio *folio,
max_nr = min_t(unsigned long, max_nr,
folio_pfn(folio) + folio_nr_pages(folio) - pte_pfn(pte));
- nr = pte_batch_hint(ptep, pte);
+ nr = pte_batch_hint(ptep, pte, flags);
expected_pte = __pte_batch_clear_ignored(pte_advance_pfn(pte, nr), flags);
ptep = ptep + nr;
@@ -447,7 +412,7 @@ static inline unsigned int folio_pte_batch_flags(struct folio *folio,
any_dirty |= pte_dirty(pte);
}
- cur_nr = pte_batch_hint(ptep, pte);
+ cur_nr = pte_batch_hint(ptep, pte, flags);
expected_pte = pte_advance_pfn(expected_pte, cur_nr);
ptep += cur_nr;
nr += cur_nr;
diff --git a/mm/mincore.c b/mm/mincore.c
index 53b9828037713..e23ef3f270db6 100644
--- a/mm/mincore.c
+++ b/mm/mincore.c
@@ -191,7 +191,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
__mincore_unmapped_range(addr, addr + PAGE_SIZE,
vma, vec);
else if (pte_present(pte)) {
- unsigned int batch = pte_batch_hint(ptep, pte);
+ unsigned int batch = pte_batch_hint(ptep, pte, 0);
if (batch > 1) {
unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
diff --git a/mm/mremap.c b/mm/mremap.c
index 384ef4cc2195b..f8b585e59771d 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -184,7 +184,7 @@ static int mremap_folio_pte_batch(struct vm_area_struct *vma, unsigned long addr
return 1;
/* Avoid expensive folio lookup if we stand no chance of benefit. */
- if (pte_batch_hint(ptep, pte) == 1)
+ if (pte_batch_hint(ptep, pte, 0) == 1)
return 1;
folio = vm_normal_folio(vma, addr, pte);
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 05/15] mm: extend pte batch and leaf-size helpers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Folded mappings such as RISC-V Svnapot need two extra pieces of generic
MM support.
First, architecture pte_batch_hint() implementations need access to the
same batch flags and normalization rules used by folio_pte_batch(), so
folded ranges can participate in batching with semantics consistent with
generic folio comparisons.
Second, leaf-size users may need the original PTE pointer instead of the
public ptep_get() view. For folded mappings, the public helper can return
a logical per-page sub-PTE, while the effective leaf size still depends
on the raw encoded entry in the page table.
Move the batch flag helpers into pgtable.h, extend pte_batch_hint() to
take the batch flags, add __ptep_leaf_size(), and update the generic
callers to use the new interfaces.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/arm64/include/asm/pgtable.h | 9 +++++-
include/linux/pgtable.h | 53 +++++++++++++++++++++++++++++++-
kernel/events/core.c | 2 +-
mm/internal.h | 39 ++---------------------
mm/mincore.c | 2 +-
mm/mremap.c | 2 +-
6 files changed, 65 insertions(+), 42 deletions(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index a2681d7553584..018781a1fcbee 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -5,6 +5,13 @@
#ifndef __ASM_PGTABLE_H
#define __ASM_PGTABLE_H
+#ifndef __ASSEMBLY__
+#ifndef __LINUX_FPB_T
+#define __LINUX_FPB_T
+typedef int __bitwise fpb_t;
+#endif
+#endif
+
#include <asm/bug.h>
#include <asm/proc-fns.h>
@@ -1705,7 +1712,7 @@ static __always_inline void contpte_try_unfold(struct mm_struct *mm,
}
#define pte_batch_hint pte_batch_hint
-static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte)
+static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
{
if (!pte_valid_cont(pte))
return 1;
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 8c093c119e5a8..da14328093a86 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -3,6 +3,35 @@
#define _LINUX_PGTABLE_H
#include <linux/pfn.h>
+
+#ifndef __ASSEMBLY__
+#ifndef __LINUX_FPB_T
+#define __LINUX_FPB_T
+typedef int __bitwise fpb_t;
+#endif
+
+/* Compare PTEs respecting the dirty bit. */
+#define FPB_RESPECT_DIRTY ((__force fpb_t)BIT(0))
+
+/* Compare PTEs respecting the soft-dirty bit. */
+#define FPB_RESPECT_SOFT_DIRTY ((__force fpb_t)BIT(1))
+
+/* Compare PTEs respecting the writable bit. */
+#define FPB_RESPECT_WRITE ((__force fpb_t)BIT(2))
+
+/*
+ * Merge PTE write bits: if any PTE in the batch is writable, modify the
+ * PTE at @ptentp to be writable.
+ */
+#define FPB_MERGE_WRITE ((__force fpb_t)BIT(3))
+
+/*
+ * Merge PTE young and dirty bits: if any PTE in the batch is young or dirty,
+ * modify the PTE at @ptentp to be young or dirty, respectively.
+ */
+#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(4))
+#endif /* !__ASSEMBLY__ */
+
#include <asm/pgtable.h>
#define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT)
@@ -397,6 +426,7 @@ static inline void lazy_mmu_mode_resume(void) {}
* pte_batch_hint - Number of pages that can be added to batch without scanning.
* @ptep: Page table pointer for the entry.
* @pte: Page table entry.
+ * @flags: Flags that control PTE normalization for batch comparisons.
*
* Some architectures know that a set of contiguous ptes all map the same
* contiguous memory with the same permissions. In this case, it can provide a
@@ -407,7 +437,7 @@ static inline void lazy_mmu_mode_resume(void) {}
*
* May be overridden by the architecture, else pte_batch_hint is always 1.
*/
-static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte)
+static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
{
return 1;
}
@@ -1855,6 +1885,7 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
return pmd;
}
#endif
+
#else /* !CONFIG_HAVE_ARCH_SOFT_DIRTY */
static inline int pte_soft_dirty(pte_t pte)
{
@@ -1917,6 +1948,17 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
}
#endif
+static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
+{
+ if (!(flags & FPB_RESPECT_DIRTY))
+ pte = pte_mkclean(pte);
+ if (likely(!(flags & FPB_RESPECT_SOFT_DIRTY)))
+ pte = pte_clear_soft_dirty(pte);
+ if (likely(!(flags & FPB_RESPECT_WRITE)))
+ pte = pte_wrprotect(pte);
+ return pte_mkold(pte);
+}
+
#ifndef __HAVE_PFNMAP_TRACKING
/*
* Interfaces that can be used by architecture code to keep track of
@@ -2414,6 +2456,15 @@ static inline const char *pgtable_level_to_str(enum pgtable_level level)
#define __pte_leaf_size(x,y) pte_leaf_size(y)
#endif
+#ifndef __ptep_leaf_size
+#ifndef __ASSEMBLY__
+static inline unsigned long __ptep_leaf_size(pmd_t pmd, pte_t *ptep, pte_t pte)
+{
+ return __pte_leaf_size(pmd, pte);
+}
+#endif /* !__ASSEMBLY__ */
+#endif
+
/*
* We always define pmd_pfn for all archs as it's used in lots of generic
* code. Now it happens too for pud_pfn (and can happen for larger
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 51c1200ea3fdd..37b860abe9e61 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8551,7 +8551,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr)
pte = ptep_get_lockless(ptep);
if (pte_present(pte))
- size = __pte_leaf_size(pmd, pte);
+ size = __ptep_leaf_size(pmd, ptep, pte);
pte_unmap(ptep);
#endif /* CONFIG_HAVE_GUP_FAST */
diff --git a/mm/internal.h b/mm/internal.h
index 874be94cf2570..aff00a3fdff50 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -343,41 +343,6 @@ static inline int anon_vma_prepare(struct vm_area_struct *vma)
return __anon_vma_prepare(vma);
}
-/* Flags for folio_pte_batch(). */
-typedef int __bitwise fpb_t;
-
-/* Compare PTEs respecting the dirty bit. */
-#define FPB_RESPECT_DIRTY ((__force fpb_t)BIT(0))
-
-/* Compare PTEs respecting the soft-dirty bit. */
-#define FPB_RESPECT_SOFT_DIRTY ((__force fpb_t)BIT(1))
-
-/* Compare PTEs respecting the writable bit. */
-#define FPB_RESPECT_WRITE ((__force fpb_t)BIT(2))
-
-/*
- * Merge PTE write bits: if any PTE in the batch is writable, modify the
- * PTE at @ptentp to be writable.
- */
-#define FPB_MERGE_WRITE ((__force fpb_t)BIT(3))
-
-/*
- * Merge PTE young and dirty bits: if any PTE in the batch is young or dirty,
- * modify the PTE at @ptentp to be young or dirty, respectively.
- */
-#define FPB_MERGE_YOUNG_DIRTY ((__force fpb_t)BIT(4))
-
-static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
-{
- if (!(flags & FPB_RESPECT_DIRTY))
- pte = pte_mkclean(pte);
- if (likely(!(flags & FPB_RESPECT_SOFT_DIRTY)))
- pte = pte_clear_soft_dirty(pte);
- if (likely(!(flags & FPB_RESPECT_WRITE)))
- pte = pte_wrprotect(pte);
- return pte_mkold(pte);
-}
-
/**
* folio_pte_batch_flags - detect a PTE batch for a large folio
* @folio: The large folio to detect a PTE batch for.
@@ -430,7 +395,7 @@ static inline unsigned int folio_pte_batch_flags(struct folio *folio,
max_nr = min_t(unsigned long, max_nr,
folio_pfn(folio) + folio_nr_pages(folio) - pte_pfn(pte));
- nr = pte_batch_hint(ptep, pte);
+ nr = pte_batch_hint(ptep, pte, flags);
expected_pte = __pte_batch_clear_ignored(pte_advance_pfn(pte, nr), flags);
ptep = ptep + nr;
@@ -447,7 +412,7 @@ static inline unsigned int folio_pte_batch_flags(struct folio *folio,
any_dirty |= pte_dirty(pte);
}
- cur_nr = pte_batch_hint(ptep, pte);
+ cur_nr = pte_batch_hint(ptep, pte, flags);
expected_pte = pte_advance_pfn(expected_pte, cur_nr);
ptep += cur_nr;
nr += cur_nr;
diff --git a/mm/mincore.c b/mm/mincore.c
index 53b9828037713..e23ef3f270db6 100644
--- a/mm/mincore.c
+++ b/mm/mincore.c
@@ -191,7 +191,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
__mincore_unmapped_range(addr, addr + PAGE_SIZE,
vma, vec);
else if (pte_present(pte)) {
- unsigned int batch = pte_batch_hint(ptep, pte);
+ unsigned int batch = pte_batch_hint(ptep, pte, 0);
if (batch > 1) {
unsigned int max_nr = (end - addr) >> PAGE_SHIFT;
diff --git a/mm/mremap.c b/mm/mremap.c
index 384ef4cc2195b..f8b585e59771d 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -184,7 +184,7 @@ static int mremap_folio_pte_batch(struct vm_area_struct *vma, unsigned long addr
return 1;
/* Avoid expensive folio lookup if we stand no chance of benefit. */
- if (pte_batch_hint(ptep, pte) == 1)
+ if (pte_batch_hint(ptep, pte, 0) == 1)
return 1;
folio = vm_normal_folio(vma, addr, pte);
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 06/15] riscv: make pte_batch_hint() honor folio batch flags
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot folded mappings may be exposed through a logical per-page PTE
view instead of the raw encoded block entry. Generic folio batching code
already defines which PTE bits are ignored when comparing entries, and
folded mappings need to use the same comparison rules. Otherwise RISC-V
can reject a batch that generic MM would consider equivalent simply
because the hint path compared a different view of the same mapping.
Install a Svnapot-aware pte_batch_hint() implementation. If the caller
passes a logical sub-PTE instead of a raw NAPOT entry, rebuild the
corresponding block PTE first, apply the same folio batch flags as the
generic batching code, and only return a multi-entry hint if the
remaining raw PTEs in the folded block still match.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 26 +++++++++++++
arch/riscv/mm/contpte.c | 66 ++++++++++++++++++++++++++++++++
2 files changed, 92 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 771faabb2b00b..da7ffea0dcd98 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -6,6 +6,13 @@
#ifndef _ASM_RISCV_PGTABLE_H
#define _ASM_RISCV_PGTABLE_H
+#ifndef __ASSEMBLY__
+#ifndef __LINUX_FPB_T
+#define __LINUX_FPB_T
+typedef int __bitwise fpb_t;
+#endif
+#endif
+
#include <linux/mmzone.h>
#include <linux/sizes.h>
@@ -591,6 +598,8 @@ void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,
unsigned int nr, cydp_t flags);
void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, unsigned int nr);
+unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte,
+ fpb_t flags);
int napotpte_ptep_set_access_flags(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep,
pte_t entry, int dirty);
@@ -1033,6 +1042,16 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma,
return napotpte_ptep_clear_flush_young(vma, address, ptep);
}
+#define pte_batch_hint pte_batch_hint
+static inline unsigned int
+pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
+{
+ if (!pte_present(pte))
+ return 1;
+
+ return napotpte_pte_batch_hint_from_pte(ptep, pte, flags);
+}
+
#else /* CONFIG_RISCV_ISA_SVNAPOT */
#define napotpte_ptep_set_access_flags(vma, address, ptep, entry, dirty) \
@@ -1095,6 +1114,13 @@ static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
#define ptep_clear_flush_young __ptep_clear_flush_young
+#define pte_batch_hint pte_batch_hint
+static inline unsigned int
+pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
+{
+ return 1;
+}
+
#endif /* CONFIG_RISCV_ISA_SVNAPOT */
#define set_pte __set_pte
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 3f8a0a2970afb..9ffdffdc2b0c0 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -176,6 +176,32 @@ static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte)
pte_val(pte_mask_ad(pte)) == pte_val(pte_mask_ad(orig_pte));
}
+static inline bool
+napotpte_is_batch_consistent(pte_t pte, pte_t batch_pte, fpb_t flags)
+{
+ return pte_present_napot(pte) &&
+ pte_val(__pte_batch_clear_ignored(pte, flags)) ==
+ pte_val(batch_pte);
+}
+
+static inline pte_t
+napotpte_normalize_batch_pte(pte_t *ptep, pte_t orig_pte, fpb_t flags)
+{
+ unsigned long pfn;
+ pgprot_t prot;
+ unsigned int off;
+
+ if (pte_present_napot(orig_pte))
+ return __pte_batch_clear_ignored(orig_pte, flags);
+
+ off = ptep - napot_align_ptep(ptep);
+ pfn = pte_pfn(orig_pte) - off;
+ prot = __pgprot(pte_protval_no_pfn_no_napot(orig_pte));
+
+ return __pte_batch_clear_ignored(pte_mknapot(pfn_pte(pfn, prot),
+ napotpte_order()), flags);
+}
+
static bool napotpte_all_subptes_same(pte_t *ptep, pte_t expected_pte)
{
pte_t *start;
@@ -328,6 +354,46 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
}
EXPORT_SYMBOL(napotpte_ptep_get_lockless);
+unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte,
+ fpb_t flags)
+{
+ pte_t batch_pte, pte;
+ pte_t *start;
+ unsigned int i, nr, off;
+
+ if (!napot_hw_supported())
+ return 1;
+
+ if (!pte_present_napot(orig_pte) && !pte_present_napot(READ_ONCE(*ptep)))
+ return 1;
+
+ /*
+ * @orig_pte may be either the raw NAPOT entry read from the page
+ * table or the logical sub-PTE returned by ptep_get(). In the latter
+ * case, the public view has the NAPOT bit stripped and the PFN adjusted
+ * by the slot offset within the folded block.
+ *
+ * If the caller passes a logical sub-PTE, rebuild the corresponding
+ * block PTE first. Then apply the same folio batch flags as the generic
+ * batching code, and only return a multi-entry hint if every remaining
+ * raw PTE in the folded block still matches.
+ */
+ batch_pte = napotpte_normalize_batch_pte(ptep, orig_pte, flags);
+
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+ off = ptep - start;
+
+ for (i = off; i < nr; i++) {
+ pte = READ_ONCE(start[i]);
+ if (!napotpte_is_batch_consistent(pte, batch_pte, flags))
+ return 1;
+ }
+
+ return nr - off;
+}
+EXPORT_SYMBOL(napotpte_pte_batch_hint_from_pte);
+
static void napotpte_try_unfold_range(struct mm_struct *mm,
unsigned long addr, pte_t *ptep,
unsigned int nr)
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 06/15] riscv: make pte_batch_hint() honor folio batch flags
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot folded mappings may be exposed through a logical per-page PTE
view instead of the raw encoded block entry. Generic folio batching code
already defines which PTE bits are ignored when comparing entries, and
folded mappings need to use the same comparison rules. Otherwise RISC-V
can reject a batch that generic MM would consider equivalent simply
because the hint path compared a different view of the same mapping.
Install a Svnapot-aware pte_batch_hint() implementation. If the caller
passes a logical sub-PTE instead of a raw NAPOT entry, rebuild the
corresponding block PTE first, apply the same folio batch flags as the
generic batching code, and only return a multi-entry hint if the
remaining raw PTEs in the folded block still match.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 26 +++++++++++++
arch/riscv/mm/contpte.c | 66 ++++++++++++++++++++++++++++++++
2 files changed, 92 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 771faabb2b00b..da7ffea0dcd98 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -6,6 +6,13 @@
#ifndef _ASM_RISCV_PGTABLE_H
#define _ASM_RISCV_PGTABLE_H
+#ifndef __ASSEMBLY__
+#ifndef __LINUX_FPB_T
+#define __LINUX_FPB_T
+typedef int __bitwise fpb_t;
+#endif
+#endif
+
#include <linux/mmzone.h>
#include <linux/sizes.h>
@@ -591,6 +598,8 @@ void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,
unsigned int nr, cydp_t flags);
void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, unsigned int nr);
+unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte,
+ fpb_t flags);
int napotpte_ptep_set_access_flags(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep,
pte_t entry, int dirty);
@@ -1033,6 +1042,16 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma,
return napotpte_ptep_clear_flush_young(vma, address, ptep);
}
+#define pte_batch_hint pte_batch_hint
+static inline unsigned int
+pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
+{
+ if (!pte_present(pte))
+ return 1;
+
+ return napotpte_pte_batch_hint_from_pte(ptep, pte, flags);
+}
+
#else /* CONFIG_RISCV_ISA_SVNAPOT */
#define napotpte_ptep_set_access_flags(vma, address, ptep, entry, dirty) \
@@ -1095,6 +1114,13 @@ static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
#define ptep_clear_flush_young __ptep_clear_flush_young
+#define pte_batch_hint pte_batch_hint
+static inline unsigned int
+pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
+{
+ return 1;
+}
+
#endif /* CONFIG_RISCV_ISA_SVNAPOT */
#define set_pte __set_pte
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 3f8a0a2970afb..9ffdffdc2b0c0 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -176,6 +176,32 @@ static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte)
pte_val(pte_mask_ad(pte)) == pte_val(pte_mask_ad(orig_pte));
}
+static inline bool
+napotpte_is_batch_consistent(pte_t pte, pte_t batch_pte, fpb_t flags)
+{
+ return pte_present_napot(pte) &&
+ pte_val(__pte_batch_clear_ignored(pte, flags)) ==
+ pte_val(batch_pte);
+}
+
+static inline pte_t
+napotpte_normalize_batch_pte(pte_t *ptep, pte_t orig_pte, fpb_t flags)
+{
+ unsigned long pfn;
+ pgprot_t prot;
+ unsigned int off;
+
+ if (pte_present_napot(orig_pte))
+ return __pte_batch_clear_ignored(orig_pte, flags);
+
+ off = ptep - napot_align_ptep(ptep);
+ pfn = pte_pfn(orig_pte) - off;
+ prot = __pgprot(pte_protval_no_pfn_no_napot(orig_pte));
+
+ return __pte_batch_clear_ignored(pte_mknapot(pfn_pte(pfn, prot),
+ napotpte_order()), flags);
+}
+
static bool napotpte_all_subptes_same(pte_t *ptep, pte_t expected_pte)
{
pte_t *start;
@@ -328,6 +354,46 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
}
EXPORT_SYMBOL(napotpte_ptep_get_lockless);
+unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte,
+ fpb_t flags)
+{
+ pte_t batch_pte, pte;
+ pte_t *start;
+ unsigned int i, nr, off;
+
+ if (!napot_hw_supported())
+ return 1;
+
+ if (!pte_present_napot(orig_pte) && !pte_present_napot(READ_ONCE(*ptep)))
+ return 1;
+
+ /*
+ * @orig_pte may be either the raw NAPOT entry read from the page
+ * table or the logical sub-PTE returned by ptep_get(). In the latter
+ * case, the public view has the NAPOT bit stripped and the PFN adjusted
+ * by the slot offset within the folded block.
+ *
+ * If the caller passes a logical sub-PTE, rebuild the corresponding
+ * block PTE first. Then apply the same folio batch flags as the generic
+ * batching code, and only return a multi-entry hint if every remaining
+ * raw PTE in the folded block still matches.
+ */
+ batch_pte = napotpte_normalize_batch_pte(ptep, orig_pte, flags);
+
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+ off = ptep - start;
+
+ for (i = off; i < nr; i++) {
+ pte = READ_ONCE(start[i]);
+ if (!napotpte_is_batch_consistent(pte, batch_pte, flags))
+ return 1;
+ }
+
+ return nr - off;
+}
+EXPORT_SYMBOL(napotpte_pte_batch_hint_from_pte);
+
static void napotpte_try_unfold_range(struct mm_struct *mm,
unsigned long addr, pte_t *ptep,
unsigned int nr)
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 06/15] riscv: make pte_batch_hint() honor folio batch flags
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot folded mappings may be exposed through a logical per-page PTE
view instead of the raw encoded block entry. Generic folio batching code
already defines which PTE bits are ignored when comparing entries, and
folded mappings need to use the same comparison rules. Otherwise RISC-V
can reject a batch that generic MM would consider equivalent simply
because the hint path compared a different view of the same mapping.
Install a Svnapot-aware pte_batch_hint() implementation. If the caller
passes a logical sub-PTE instead of a raw NAPOT entry, rebuild the
corresponding block PTE first, apply the same folio batch flags as the
generic batching code, and only return a multi-entry hint if the
remaining raw PTEs in the folded block still match.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 26 +++++++++++++
arch/riscv/mm/contpte.c | 66 ++++++++++++++++++++++++++++++++
2 files changed, 92 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 771faabb2b00b..da7ffea0dcd98 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -6,6 +6,13 @@
#ifndef _ASM_RISCV_PGTABLE_H
#define _ASM_RISCV_PGTABLE_H
+#ifndef __ASSEMBLY__
+#ifndef __LINUX_FPB_T
+#define __LINUX_FPB_T
+typedef int __bitwise fpb_t;
+#endif
+#endif
+
#include <linux/mmzone.h>
#include <linux/sizes.h>
@@ -591,6 +598,8 @@ void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,
unsigned int nr, cydp_t flags);
void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, unsigned int nr);
+unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte,
+ fpb_t flags);
int napotpte_ptep_set_access_flags(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep,
pte_t entry, int dirty);
@@ -1033,6 +1042,16 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma,
return napotpte_ptep_clear_flush_young(vma, address, ptep);
}
+#define pte_batch_hint pte_batch_hint
+static inline unsigned int
+pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
+{
+ if (!pte_present(pte))
+ return 1;
+
+ return napotpte_pte_batch_hint_from_pte(ptep, pte, flags);
+}
+
#else /* CONFIG_RISCV_ISA_SVNAPOT */
#define napotpte_ptep_set_access_flags(vma, address, ptep, entry, dirty) \
@@ -1095,6 +1114,13 @@ static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
#define ptep_clear_flush_young __ptep_clear_flush_young
+#define pte_batch_hint pte_batch_hint
+static inline unsigned int
+pte_batch_hint(pte_t *ptep, pte_t pte, fpb_t flags)
+{
+ return 1;
+}
+
#endif /* CONFIG_RISCV_ISA_SVNAPOT */
#define set_pte __set_pte
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 3f8a0a2970afb..9ffdffdc2b0c0 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -176,6 +176,32 @@ static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte)
pte_val(pte_mask_ad(pte)) == pte_val(pte_mask_ad(orig_pte));
}
+static inline bool
+napotpte_is_batch_consistent(pte_t pte, pte_t batch_pte, fpb_t flags)
+{
+ return pte_present_napot(pte) &&
+ pte_val(__pte_batch_clear_ignored(pte, flags)) ==
+ pte_val(batch_pte);
+}
+
+static inline pte_t
+napotpte_normalize_batch_pte(pte_t *ptep, pte_t orig_pte, fpb_t flags)
+{
+ unsigned long pfn;
+ pgprot_t prot;
+ unsigned int off;
+
+ if (pte_present_napot(orig_pte))
+ return __pte_batch_clear_ignored(orig_pte, flags);
+
+ off = ptep - napot_align_ptep(ptep);
+ pfn = pte_pfn(orig_pte) - off;
+ prot = __pgprot(pte_protval_no_pfn_no_napot(orig_pte));
+
+ return __pte_batch_clear_ignored(pte_mknapot(pfn_pte(pfn, prot),
+ napotpte_order()), flags);
+}
+
static bool napotpte_all_subptes_same(pte_t *ptep, pte_t expected_pte)
{
pte_t *start;
@@ -328,6 +354,46 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
}
EXPORT_SYMBOL(napotpte_ptep_get_lockless);
+unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte,
+ fpb_t flags)
+{
+ pte_t batch_pte, pte;
+ pte_t *start;
+ unsigned int i, nr, off;
+
+ if (!napot_hw_supported())
+ return 1;
+
+ if (!pte_present_napot(orig_pte) && !pte_present_napot(READ_ONCE(*ptep)))
+ return 1;
+
+ /*
+ * @orig_pte may be either the raw NAPOT entry read from the page
+ * table or the logical sub-PTE returned by ptep_get(). In the latter
+ * case, the public view has the NAPOT bit stripped and the PFN adjusted
+ * by the slot offset within the folded block.
+ *
+ * If the caller passes a logical sub-PTE, rebuild the corresponding
+ * block PTE first. Then apply the same folio batch flags as the generic
+ * batching code, and only return a multi-entry hint if every remaining
+ * raw PTE in the folded block still matches.
+ */
+ batch_pte = napotpte_normalize_batch_pte(ptep, orig_pte, flags);
+
+ start = napot_align_ptep(ptep);
+ nr = napotpte_pte_num();
+ off = ptep - start;
+
+ for (i = off; i < nr; i++) {
+ pte = READ_ONCE(start[i]);
+ if (!napotpte_is_batch_consistent(pte, batch_pte, flags))
+ return 1;
+ }
+
+ return nr - off;
+}
+EXPORT_SYMBOL(napotpte_pte_batch_hint_from_pte);
+
static void napotpte_try_unfold_range(struct mm_struct *mm,
unsigned long addr, pte_t *ptep,
unsigned int nr)
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 07/15] riscv/mm: preserve Svnapot leaf-size semantics for page-table consumers
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot keeps a folded 64KB mapping encoded in the page table, while the
public PTE helpers may expose a logical per-page sub-PTE view to generic
MM users. That logical view is suitable for leaf-style PTE consumers, but
it must not change the effective leaf size reported for the mapping.
Use the original page-table entry for leaf-size queries, so page-table
consumers continue to observe the Svnapot leaf size instead of incorrectly
treating a folded mapping as PAGE_SIZE entries.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index da7ffea0dcd98..e4b9c05014c00 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -580,6 +580,17 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
napot_cont_size(napot_cont_order(pte)) :\
PAGE_SIZE)
+#define __ptep_leaf_size __ptep_leaf_size
+static inline unsigned long __ptep_leaf_size(pmd_t pmd, pte_t *ptep, pte_t pte)
+{
+ pte_t raw_pte = READ_ONCE(*ptep);
+
+ if (pte_present_napot(raw_pte))
+ return pte_leaf_size(raw_pte);
+
+ return PAGE_SIZE;
+}
+
void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte);
void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 07/15] riscv/mm: preserve Svnapot leaf-size semantics for page-table consumers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot keeps a folded 64KB mapping encoded in the page table, while the
public PTE helpers may expose a logical per-page sub-PTE view to generic
MM users. That logical view is suitable for leaf-style PTE consumers, but
it must not change the effective leaf size reported for the mapping.
Use the original page-table entry for leaf-size queries, so page-table
consumers continue to observe the Svnapot leaf size instead of incorrectly
treating a folded mapping as PAGE_SIZE entries.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index da7ffea0dcd98..e4b9c05014c00 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -580,6 +580,17 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
napot_cont_size(napot_cont_order(pte)) :\
PAGE_SIZE)
+#define __ptep_leaf_size __ptep_leaf_size
+static inline unsigned long __ptep_leaf_size(pmd_t pmd, pte_t *ptep, pte_t pte)
+{
+ pte_t raw_pte = READ_ONCE(*ptep);
+
+ if (pte_present_napot(raw_pte))
+ return pte_leaf_size(raw_pte);
+
+ return PAGE_SIZE;
+}
+
void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte);
void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 07/15] riscv/mm: preserve Svnapot leaf-size semantics for page-table consumers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot keeps a folded 64KB mapping encoded in the page table, while the
public PTE helpers may expose a logical per-page sub-PTE view to generic
MM users. That logical view is suitable for leaf-style PTE consumers, but
it must not change the effective leaf size reported for the mapping.
Use the original page-table entry for leaf-size queries, so page-table
consumers continue to observe the Svnapot leaf size instead of incorrectly
treating a folded mapping as PAGE_SIZE entries.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index da7ffea0dcd98..e4b9c05014c00 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -580,6 +580,17 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
napot_cont_size(napot_cont_order(pte)) :\
PAGE_SIZE)
+#define __ptep_leaf_size __ptep_leaf_size
+static inline unsigned long __ptep_leaf_size(pmd_t pmd, pte_t *ptep, pte_t pte)
+{
+ pte_t raw_pte = READ_ONCE(*ptep);
+
+ if (pte_present_napot(raw_pte))
+ return pte_leaf_size(raw_pte);
+
+ return PAGE_SIZE;
+}
+
void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte);
void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 08/15] riscv/mm: avoid redundant Svnapot A/D aggregation
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot mappings encode a 64KB range as a NAPOT PTE block, while the
public getter returns a logical base-page PTE. To preserve that PTE view,
the getter folds accessed and dirty information from the whole NAPOT block
into the returned PTE.
This aggregation is an OR-style operation. Once the returned PTE has both
accessed and dirty state set, later entries in the block cannot change the
result. Avoid reading the rest of the block in that case.
The lockless getter still validates each entry before consuming its A/D
state, so it preserves the self-consistent range requirement while also
avoiding reads that cannot affect the returned PTE.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/mm/contpte.c | 60 ++++++++++++++++++++++++++++++++++++++---
1 file changed, 56 insertions(+), 4 deletions(-)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 9ffdffdc2b0c0..70f9a0668b50c 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -309,10 +309,34 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
cur = READ_ONCE(start[i]);
if (!napotpte_is_consistent(cur, orig_pte))
return napotpte_subpte(ptep, orig_pte);
- if (pte_dirty(cur))
+ if (pte_dirty(cur)) {
pte = riscv_pte_mkhwdirty(pte);
- if (pte_young(cur))
+ for (; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (!napotpte_is_consistent(cur, orig_pte))
+ return napotpte_subpte(ptep, orig_pte);
+ if (pte_young(cur)) {
+ pte = pte_mkyoung(pte);
+ break;
+ }
+ }
+ break;
+ }
+
+ if (pte_young(cur)) {
pte = pte_mkyoung(pte);
+ i++;
+ for (; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (!napotpte_is_consistent(cur, orig_pte))
+ return napotpte_subpte(ptep, orig_pte);
+ if (pte_dirty(cur)) {
+ pte = riscv_pte_mkhwdirty(pte);
+ break;
+ }
+ }
+ break;
+ }
}
return napotpte_subpte(ptep, pte);
@@ -343,11 +367,39 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
if (!napotpte_is_consistent(pte, orig_pte))
goto retry;
- if (pte_dirty(pte))
+ if (pte_dirty(pte)) {
orig_pte = riscv_pte_mkhwdirty(orig_pte);
+ for (; i < nr; i++, ptep++) {
+ pte = READ_ONCE(*ptep);
+
+ if (!napotpte_is_consistent(pte, orig_pte))
+ goto retry;
- if (pte_young(pte))
+ if (pte_young(pte)) {
+ orig_pte = pte_mkyoung(orig_pte);
+ break;
+ }
+ }
+ break;
+ }
+
+ if (pte_young(pte)) {
orig_pte = pte_mkyoung(orig_pte);
+ i++;
+ ptep++;
+ for (; i < nr; i++, ptep++) {
+ pte = READ_ONCE(*ptep);
+
+ if (!napotpte_is_consistent(pte, orig_pte))
+ goto retry;
+
+ if (pte_dirty(pte)) {
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
+ break;
+ }
+ }
+ break;
+ }
}
return napotpte_subpte(orig_ptep, orig_pte);
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 08/15] riscv/mm: avoid redundant Svnapot A/D aggregation
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot mappings encode a 64KB range as a NAPOT PTE block, while the
public getter returns a logical base-page PTE. To preserve that PTE view,
the getter folds accessed and dirty information from the whole NAPOT block
into the returned PTE.
This aggregation is an OR-style operation. Once the returned PTE has both
accessed and dirty state set, later entries in the block cannot change the
result. Avoid reading the rest of the block in that case.
The lockless getter still validates each entry before consuming its A/D
state, so it preserves the self-consistent range requirement while also
avoiding reads that cannot affect the returned PTE.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/mm/contpte.c | 60 ++++++++++++++++++++++++++++++++++++++---
1 file changed, 56 insertions(+), 4 deletions(-)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 9ffdffdc2b0c0..70f9a0668b50c 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -309,10 +309,34 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
cur = READ_ONCE(start[i]);
if (!napotpte_is_consistent(cur, orig_pte))
return napotpte_subpte(ptep, orig_pte);
- if (pte_dirty(cur))
+ if (pte_dirty(cur)) {
pte = riscv_pte_mkhwdirty(pte);
- if (pte_young(cur))
+ for (; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (!napotpte_is_consistent(cur, orig_pte))
+ return napotpte_subpte(ptep, orig_pte);
+ if (pte_young(cur)) {
+ pte = pte_mkyoung(pte);
+ break;
+ }
+ }
+ break;
+ }
+
+ if (pte_young(cur)) {
pte = pte_mkyoung(pte);
+ i++;
+ for (; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (!napotpte_is_consistent(cur, orig_pte))
+ return napotpte_subpte(ptep, orig_pte);
+ if (pte_dirty(cur)) {
+ pte = riscv_pte_mkhwdirty(pte);
+ break;
+ }
+ }
+ break;
+ }
}
return napotpte_subpte(ptep, pte);
@@ -343,11 +367,39 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
if (!napotpte_is_consistent(pte, orig_pte))
goto retry;
- if (pte_dirty(pte))
+ if (pte_dirty(pte)) {
orig_pte = riscv_pte_mkhwdirty(orig_pte);
+ for (; i < nr; i++, ptep++) {
+ pte = READ_ONCE(*ptep);
+
+ if (!napotpte_is_consistent(pte, orig_pte))
+ goto retry;
- if (pte_young(pte))
+ if (pte_young(pte)) {
+ orig_pte = pte_mkyoung(orig_pte);
+ break;
+ }
+ }
+ break;
+ }
+
+ if (pte_young(pte)) {
orig_pte = pte_mkyoung(orig_pte);
+ i++;
+ ptep++;
+ for (; i < nr; i++, ptep++) {
+ pte = READ_ONCE(*ptep);
+
+ if (!napotpte_is_consistent(pte, orig_pte))
+ goto retry;
+
+ if (pte_dirty(pte)) {
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
+ break;
+ }
+ }
+ break;
+ }
}
return napotpte_subpte(orig_ptep, orig_pte);
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 08/15] riscv/mm: avoid redundant Svnapot A/D aggregation
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot mappings encode a 64KB range as a NAPOT PTE block, while the
public getter returns a logical base-page PTE. To preserve that PTE view,
the getter folds accessed and dirty information from the whole NAPOT block
into the returned PTE.
This aggregation is an OR-style operation. Once the returned PTE has both
accessed and dirty state set, later entries in the block cannot change the
result. Avoid reading the rest of the block in that case.
The lockless getter still validates each entry before consuming its A/D
state, so it preserves the self-consistent range requirement while also
avoiding reads that cannot affect the returned PTE.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/mm/contpte.c | 60 ++++++++++++++++++++++++++++++++++++++---
1 file changed, 56 insertions(+), 4 deletions(-)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 9ffdffdc2b0c0..70f9a0668b50c 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -309,10 +309,34 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
cur = READ_ONCE(start[i]);
if (!napotpte_is_consistent(cur, orig_pte))
return napotpte_subpte(ptep, orig_pte);
- if (pte_dirty(cur))
+ if (pte_dirty(cur)) {
pte = riscv_pte_mkhwdirty(pte);
- if (pte_young(cur))
+ for (; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (!napotpte_is_consistent(cur, orig_pte))
+ return napotpte_subpte(ptep, orig_pte);
+ if (pte_young(cur)) {
+ pte = pte_mkyoung(pte);
+ break;
+ }
+ }
+ break;
+ }
+
+ if (pte_young(cur)) {
pte = pte_mkyoung(pte);
+ i++;
+ for (; i < nr; i++) {
+ cur = READ_ONCE(start[i]);
+ if (!napotpte_is_consistent(cur, orig_pte))
+ return napotpte_subpte(ptep, orig_pte);
+ if (pte_dirty(cur)) {
+ pte = riscv_pte_mkhwdirty(pte);
+ break;
+ }
+ }
+ break;
+ }
}
return napotpte_subpte(ptep, pte);
@@ -343,11 +367,39 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
if (!napotpte_is_consistent(pte, orig_pte))
goto retry;
- if (pte_dirty(pte))
+ if (pte_dirty(pte)) {
orig_pte = riscv_pte_mkhwdirty(orig_pte);
+ for (; i < nr; i++, ptep++) {
+ pte = READ_ONCE(*ptep);
+
+ if (!napotpte_is_consistent(pte, orig_pte))
+ goto retry;
- if (pte_young(pte))
+ if (pte_young(pte)) {
+ orig_pte = pte_mkyoung(orig_pte);
+ break;
+ }
+ }
+ break;
+ }
+
+ if (pte_young(pte)) {
orig_pte = pte_mkyoung(orig_pte);
+ i++;
+ ptep++;
+ for (; i < nr; i++, ptep++) {
+ pte = READ_ONCE(*ptep);
+
+ if (!napotpte_is_consistent(pte, orig_pte))
+ goto retry;
+
+ if (pte_dirty(pte)) {
+ orig_pte = riscv_pte_mkhwdirty(orig_pte);
+ break;
+ }
+ }
+ break;
+ }
}
return napotpte_subpte(orig_ptep, orig_pte);
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 09/15] riscv/mm: avoid Svnapot consistency checks in ptep_get()
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
napotpte_ptep_get() builds the returned logical sub-PTE from the
initial PTE observed at the target address. Neighbouring entries in the
NAPOT block are only scanned to gather accessed and dirty state, so they
cannot contribute PFN, permission, present or writable state to the
returned PTE.
Drop the range consistency checks from this ptep_get() path. Callers
that rely on the block-wide A/D view are expected to hold the PTL. There
are generic lockless ptep_get() users, such as userfaultfd_must_wait()
and HMM walks after pte_offset_map(), but those paths use the present,
marker, write and PFN state and do not consume the gathered A/D bits.
Keep the consistency checks in napotpte_ptep_get_lockless(). That API
must return a self-consistent PTE snapshot to lockless walkers and fault
retry paths. napotpte_is_consistent() verifies that every scanned entry
still belongs to the same NAPOT mapping, ignoring A/D bits, and retries
if a concurrent unfold, rewrite or refold tears the range while A/D
state is being gathered.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 3 +++
arch/riscv/mm/contpte.c | 19 +++++++++++++------
2 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index e4b9c05014c00..dd56c20a817bb 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -930,6 +930,9 @@ static __always_inline void napotpte_try_unfold(struct mm_struct *mm,
/*
* Public PTE helpers may transparently handle Svnapot-encoded mappings.
* NAPOT-aware arch users should stick to the private/raw __pte* helpers.
+ * Callers relying on the block-wide A/D view from ptep_get() are
+ * expected to hold the PTL. Lockless callers that require a
+ * self-consistent Svnapot range must use ptep_get_lockless().
*/
#define ptep_get ptep_get
static inline pte_t ptep_get(pte_t *ptep)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 70f9a0668b50c..0f66426b0ecfd 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -294,6 +294,13 @@ EXPORT_SYMBOL(__napotpte_try_unfold);
pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
{
+ /*
+ * Gather access/dirty bits from the whole NAPOT range for the
+ * ptep_get() view. The returned sub-PTE is built from orig_pte;
+ * neighbouring entries only contribute A/D state. Lockless callers
+ * requiring a self-consistent range must use ptep_get_lockless().
+ */
+
pte_t pte, cur;
pte_t *start;
unsigned int i, nr;
@@ -307,14 +314,10 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
for (i = 0; i < nr; i++) {
cur = READ_ONCE(start[i]);
- if (!napotpte_is_consistent(cur, orig_pte))
- return napotpte_subpte(ptep, orig_pte);
if (pte_dirty(cur)) {
pte = riscv_pte_mkhwdirty(pte);
for (; i < nr; i++) {
cur = READ_ONCE(start[i]);
- if (!napotpte_is_consistent(cur, orig_pte))
- return napotpte_subpte(ptep, orig_pte);
if (pte_young(cur)) {
pte = pte_mkyoung(pte);
break;
@@ -328,8 +331,6 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
i++;
for (; i < nr; i++) {
cur = READ_ONCE(start[i]);
- if (!napotpte_is_consistent(cur, orig_pte))
- return napotpte_subpte(ptep, orig_pte);
if (pte_dirty(cur)) {
pte = riscv_pte_mkhwdirty(pte);
break;
@@ -345,6 +346,12 @@ EXPORT_SYMBOL(napotpte_ptep_get);
pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
{
+ /*
+ * ptep_get_lockless() must return a self-consistent PTE without the
+ * PTL. Recheck that the whole NAPOT range still describes the same
+ * mapping, ignoring A/D bits, and retry if a concurrent update tears
+ * the range while A/D state is being gathered.
+ */
pte_t orig_pte, pte;
pte_t *ptep;
unsigned int i, nr;
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 09/15] riscv/mm: avoid Svnapot consistency checks in ptep_get()
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
napotpte_ptep_get() builds the returned logical sub-PTE from the
initial PTE observed at the target address. Neighbouring entries in the
NAPOT block are only scanned to gather accessed and dirty state, so they
cannot contribute PFN, permission, present or writable state to the
returned PTE.
Drop the range consistency checks from this ptep_get() path. Callers
that rely on the block-wide A/D view are expected to hold the PTL. There
are generic lockless ptep_get() users, such as userfaultfd_must_wait()
and HMM walks after pte_offset_map(), but those paths use the present,
marker, write and PFN state and do not consume the gathered A/D bits.
Keep the consistency checks in napotpte_ptep_get_lockless(). That API
must return a self-consistent PTE snapshot to lockless walkers and fault
retry paths. napotpte_is_consistent() verifies that every scanned entry
still belongs to the same NAPOT mapping, ignoring A/D bits, and retries
if a concurrent unfold, rewrite or refold tears the range while A/D
state is being gathered.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 3 +++
arch/riscv/mm/contpte.c | 19 +++++++++++++------
2 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index e4b9c05014c00..dd56c20a817bb 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -930,6 +930,9 @@ static __always_inline void napotpte_try_unfold(struct mm_struct *mm,
/*
* Public PTE helpers may transparently handle Svnapot-encoded mappings.
* NAPOT-aware arch users should stick to the private/raw __pte* helpers.
+ * Callers relying on the block-wide A/D view from ptep_get() are
+ * expected to hold the PTL. Lockless callers that require a
+ * self-consistent Svnapot range must use ptep_get_lockless().
*/
#define ptep_get ptep_get
static inline pte_t ptep_get(pte_t *ptep)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 70f9a0668b50c..0f66426b0ecfd 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -294,6 +294,13 @@ EXPORT_SYMBOL(__napotpte_try_unfold);
pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
{
+ /*
+ * Gather access/dirty bits from the whole NAPOT range for the
+ * ptep_get() view. The returned sub-PTE is built from orig_pte;
+ * neighbouring entries only contribute A/D state. Lockless callers
+ * requiring a self-consistent range must use ptep_get_lockless().
+ */
+
pte_t pte, cur;
pte_t *start;
unsigned int i, nr;
@@ -307,14 +314,10 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
for (i = 0; i < nr; i++) {
cur = READ_ONCE(start[i]);
- if (!napotpte_is_consistent(cur, orig_pte))
- return napotpte_subpte(ptep, orig_pte);
if (pte_dirty(cur)) {
pte = riscv_pte_mkhwdirty(pte);
for (; i < nr; i++) {
cur = READ_ONCE(start[i]);
- if (!napotpte_is_consistent(cur, orig_pte))
- return napotpte_subpte(ptep, orig_pte);
if (pte_young(cur)) {
pte = pte_mkyoung(pte);
break;
@@ -328,8 +331,6 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
i++;
for (; i < nr; i++) {
cur = READ_ONCE(start[i]);
- if (!napotpte_is_consistent(cur, orig_pte))
- return napotpte_subpte(ptep, orig_pte);
if (pte_dirty(cur)) {
pte = riscv_pte_mkhwdirty(pte);
break;
@@ -345,6 +346,12 @@ EXPORT_SYMBOL(napotpte_ptep_get);
pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
{
+ /*
+ * ptep_get_lockless() must return a self-consistent PTE without the
+ * PTL. Recheck that the whole NAPOT range still describes the same
+ * mapping, ignoring A/D bits, and retry if a concurrent update tears
+ * the range while A/D state is being gathered.
+ */
pte_t orig_pte, pte;
pte_t *ptep;
unsigned int i, nr;
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 09/15] riscv/mm: avoid Svnapot consistency checks in ptep_get()
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
napotpte_ptep_get() builds the returned logical sub-PTE from the
initial PTE observed at the target address. Neighbouring entries in the
NAPOT block are only scanned to gather accessed and dirty state, so they
cannot contribute PFN, permission, present or writable state to the
returned PTE.
Drop the range consistency checks from this ptep_get() path. Callers
that rely on the block-wide A/D view are expected to hold the PTL. There
are generic lockless ptep_get() users, such as userfaultfd_must_wait()
and HMM walks after pte_offset_map(), but those paths use the present,
marker, write and PFN state and do not consume the gathered A/D bits.
Keep the consistency checks in napotpte_ptep_get_lockless(). That API
must return a self-consistent PTE snapshot to lockless walkers and fault
retry paths. napotpte_is_consistent() verifies that every scanned entry
still belongs to the same NAPOT mapping, ignoring A/D bits, and retries
if a concurrent unfold, rewrite or refold tears the range while A/D
state is being gathered.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 3 +++
arch/riscv/mm/contpte.c | 19 +++++++++++++------
2 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index e4b9c05014c00..dd56c20a817bb 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -930,6 +930,9 @@ static __always_inline void napotpte_try_unfold(struct mm_struct *mm,
/*
* Public PTE helpers may transparently handle Svnapot-encoded mappings.
* NAPOT-aware arch users should stick to the private/raw __pte* helpers.
+ * Callers relying on the block-wide A/D view from ptep_get() are
+ * expected to hold the PTL. Lockless callers that require a
+ * self-consistent Svnapot range must use ptep_get_lockless().
*/
#define ptep_get ptep_get
static inline pte_t ptep_get(pte_t *ptep)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 70f9a0668b50c..0f66426b0ecfd 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -294,6 +294,13 @@ EXPORT_SYMBOL(__napotpte_try_unfold);
pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
{
+ /*
+ * Gather access/dirty bits from the whole NAPOT range for the
+ * ptep_get() view. The returned sub-PTE is built from orig_pte;
+ * neighbouring entries only contribute A/D state. Lockless callers
+ * requiring a self-consistent range must use ptep_get_lockless().
+ */
+
pte_t pte, cur;
pte_t *start;
unsigned int i, nr;
@@ -307,14 +314,10 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
for (i = 0; i < nr; i++) {
cur = READ_ONCE(start[i]);
- if (!napotpte_is_consistent(cur, orig_pte))
- return napotpte_subpte(ptep, orig_pte);
if (pte_dirty(cur)) {
pte = riscv_pte_mkhwdirty(pte);
for (; i < nr; i++) {
cur = READ_ONCE(start[i]);
- if (!napotpte_is_consistent(cur, orig_pte))
- return napotpte_subpte(ptep, orig_pte);
if (pte_young(cur)) {
pte = pte_mkyoung(pte);
break;
@@ -328,8 +331,6 @@ pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)
i++;
for (; i < nr; i++) {
cur = READ_ONCE(start[i]);
- if (!napotpte_is_consistent(cur, orig_pte))
- return napotpte_subpte(ptep, orig_pte);
if (pte_dirty(cur)) {
pte = riscv_pte_mkhwdirty(pte);
break;
@@ -345,6 +346,12 @@ EXPORT_SYMBOL(napotpte_ptep_get);
pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
{
+ /*
+ * ptep_get_lockless() must return a self-consistent PTE without the
+ * PTL. Recheck that the whole NAPOT range still describes the same
+ * mapping, ignoring A/D bits, and retry if a concurrent update tears
+ * the range while A/D state is being gathered.
+ */
pte_t orig_pte, pte;
pte_t *ptep;
unsigned int i, nr;
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 10/15] mm/gup: add fast-GUP specific lockless PTE helpers
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
fast-GUP snapshots a PTE without holding the PTL, pins the page, and
then revalidates that the PTE did not change. Keep both reads under the
same lockless PTE semantics by defaulting the fast-GUP revalidation to
ptep_get_lockless().
Introduce fast-GUP specific PTE snapshot and revalidation helpers. The
default implementation preserves the existing ptep_get_lockless()
semantics, while allowing architectures to override the helpers when the
public lockless getter provides extra semantics that fast-GUP does not
consume.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
include/linux/pgtable.h | 18 ++++++++++++++++++
mm/gup.c | 6 ++++--
2 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index da14328093a86..62943fcbf7046 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -813,6 +813,24 @@ static inline pte_t ptep_get_lockless(pte_t *ptep)
}
#endif
+#ifndef gup_ptep_get_lockless
+static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp)
+{
+ pte_t pte = ptep_get_lockless(ptep);
+
+ *rawp = pte;
+
+ return pte;
+}
+#endif
+
+#ifndef gup_ptep_revalidate
+static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte)
+{
+ return pte_val(raw_pte) == pte_val(ptep_get_lockless(ptep));
+}
+#endif
+
#ifndef pmdp_get_lockless
static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
{
diff --git a/mm/gup.c b/mm/gup.c
index 99902c15703b0..72fb147193e55 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2842,10 +2842,12 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,
if (!ptep)
return 0;
do {
- pte_t pte = ptep_get_lockless(ptep);
+ pte_t raw_pte, pte;
struct page *page;
struct folio *folio;
+ pte = gup_ptep_get_lockless(ptep, &raw_pte);
+
/*
* Always fallback to ordinary GUP on PROT_NONE-mapped pages:
* pte_access_permitted() better should reject these pages
@@ -2871,7 +2873,7 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,
goto pte_unmap;
if (unlikely(pmd_val(pmd) != pmd_val(pmdp_get_lockless(pmdp))) ||
- unlikely(pte_val(pte) != pte_val(ptep_get_lockless(ptep)))) {
+ unlikely(!gup_ptep_revalidate(ptep, raw_pte))) {
gup_put_folio(folio, 1, flags);
goto pte_unmap;
}
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 10/15] mm/gup: add fast-GUP specific lockless PTE helpers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
fast-GUP snapshots a PTE without holding the PTL, pins the page, and
then revalidates that the PTE did not change. Keep both reads under the
same lockless PTE semantics by defaulting the fast-GUP revalidation to
ptep_get_lockless().
Introduce fast-GUP specific PTE snapshot and revalidation helpers. The
default implementation preserves the existing ptep_get_lockless()
semantics, while allowing architectures to override the helpers when the
public lockless getter provides extra semantics that fast-GUP does not
consume.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
include/linux/pgtable.h | 18 ++++++++++++++++++
mm/gup.c | 6 ++++--
2 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index da14328093a86..62943fcbf7046 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -813,6 +813,24 @@ static inline pte_t ptep_get_lockless(pte_t *ptep)
}
#endif
+#ifndef gup_ptep_get_lockless
+static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp)
+{
+ pte_t pte = ptep_get_lockless(ptep);
+
+ *rawp = pte;
+
+ return pte;
+}
+#endif
+
+#ifndef gup_ptep_revalidate
+static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte)
+{
+ return pte_val(raw_pte) == pte_val(ptep_get_lockless(ptep));
+}
+#endif
+
#ifndef pmdp_get_lockless
static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
{
diff --git a/mm/gup.c b/mm/gup.c
index 99902c15703b0..72fb147193e55 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2842,10 +2842,12 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,
if (!ptep)
return 0;
do {
- pte_t pte = ptep_get_lockless(ptep);
+ pte_t raw_pte, pte;
struct page *page;
struct folio *folio;
+ pte = gup_ptep_get_lockless(ptep, &raw_pte);
+
/*
* Always fallback to ordinary GUP on PROT_NONE-mapped pages:
* pte_access_permitted() better should reject these pages
@@ -2871,7 +2873,7 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,
goto pte_unmap;
if (unlikely(pmd_val(pmd) != pmd_val(pmdp_get_lockless(pmdp))) ||
- unlikely(pte_val(pte) != pte_val(ptep_get_lockless(ptep)))) {
+ unlikely(!gup_ptep_revalidate(ptep, raw_pte))) {
gup_put_folio(folio, 1, flags);
goto pte_unmap;
}
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 10/15] mm/gup: add fast-GUP specific lockless PTE helpers
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
fast-GUP snapshots a PTE without holding the PTL, pins the page, and
then revalidates that the PTE did not change. Keep both reads under the
same lockless PTE semantics by defaulting the fast-GUP revalidation to
ptep_get_lockless().
Introduce fast-GUP specific PTE snapshot and revalidation helpers. The
default implementation preserves the existing ptep_get_lockless()
semantics, while allowing architectures to override the helpers when the
public lockless getter provides extra semantics that fast-GUP does not
consume.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
include/linux/pgtable.h | 18 ++++++++++++++++++
mm/gup.c | 6 ++++--
2 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index da14328093a86..62943fcbf7046 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -813,6 +813,24 @@ static inline pte_t ptep_get_lockless(pte_t *ptep)
}
#endif
+#ifndef gup_ptep_get_lockless
+static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp)
+{
+ pte_t pte = ptep_get_lockless(ptep);
+
+ *rawp = pte;
+
+ return pte;
+}
+#endif
+
+#ifndef gup_ptep_revalidate
+static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte)
+{
+ return pte_val(raw_pte) == pte_val(ptep_get_lockless(ptep));
+}
+#endif
+
#ifndef pmdp_get_lockless
static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
{
diff --git a/mm/gup.c b/mm/gup.c
index 99902c15703b0..72fb147193e55 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2842,10 +2842,12 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,
if (!ptep)
return 0;
do {
- pte_t pte = ptep_get_lockless(ptep);
+ pte_t raw_pte, pte;
struct page *page;
struct folio *folio;
+ pte = gup_ptep_get_lockless(ptep, &raw_pte);
+
/*
* Always fallback to ordinary GUP on PROT_NONE-mapped pages:
* pte_access_permitted() better should reject these pages
@@ -2871,7 +2873,7 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,
goto pte_unmap;
if (unlikely(pmd_val(pmd) != pmd_val(pmdp_get_lockless(pmdp))) ||
- unlikely(pte_val(pte) != pte_val(ptep_get_lockless(ptep)))) {
+ unlikely(!gup_ptep_revalidate(ptep, raw_pte))) {
gup_put_folio(folio, 1, flags);
goto pte_unmap;
}
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 11/15] riscv: mm: avoid Svnapot A/D aggregation in fast-GUP
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
fast-GUP only needs the PTE state for the current virtual address and
revalidates the raw PTE after pinning the page. It does not consume the
block-wide accessed/dirty aggregation provided by the public Svnapot
lockless getter.
Override the fast-GUP PTE helpers to translate a NAPOT PTE into the
current sub-PTE without scanning the whole NAPOT block. Keep the public
ptep_get_lockless() semantics unchanged for other lockless users.
With 64K THP enabled in always mode, fio 4K random direct read
throughput improves by about 10% over the previous implementation on the
same system, from 2.615M to 2.874M IOPS and from 9.97GiB/s to 11.0GiB/s,
while average clat drops from 95.54 usec to 86.97 usec.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 20 ++++++++++++++++++++
arch/riscv/mm/contpte.c | 9 +++++++++
2 files changed, 29 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index dd56c20a817bb..3727c148fa0fc 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -597,6 +597,7 @@ void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte);
pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte);
pte_t napotpte_ptep_get_lockless(pte_t *ptep);
+pte_t napotpte_ptep_get_gup_fast(pte_t *ptep, pte_t orig_pte);
void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte, unsigned int nr);
void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr,
@@ -956,6 +957,25 @@ static inline pte_t ptep_get_lockless(pte_t *ptep)
return napotpte_ptep_get_lockless(ptep);
}
+#define gup_ptep_get_lockless gup_ptep_get_lockless
+static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp)
+{
+ pte_t pte = __ptep_get_lockless(ptep);
+
+ *rawp = pte;
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get_gup_fast(ptep, pte);
+}
+
+#define gup_ptep_revalidate gup_ptep_revalidate
+static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte)
+{
+ return pte_val(raw_pte) == pte_val(__ptep_get_lockless(ptep));
+}
+
#define set_ptes set_ptes
static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pteval, unsigned int nr)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 0f66426b0ecfd..b0b8ff0aade3a 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -413,6 +413,15 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
}
EXPORT_SYMBOL(napotpte_ptep_get_lockless);
+pte_t napotpte_ptep_get_gup_fast(pte_t *orig_ptep, pte_t orig_pte)
+{
+ if (!napot_hw_supported() || !pte_present_napot(orig_pte))
+ return orig_pte;
+
+ return napotpte_subpte(orig_ptep, orig_pte);
+}
+EXPORT_SYMBOL(napotpte_ptep_get_gup_fast);
+
unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte,
fpb_t flags)
{
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 11/15] riscv: mm: avoid Svnapot A/D aggregation in fast-GUP
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
fast-GUP only needs the PTE state for the current virtual address and
revalidates the raw PTE after pinning the page. It does not consume the
block-wide accessed/dirty aggregation provided by the public Svnapot
lockless getter.
Override the fast-GUP PTE helpers to translate a NAPOT PTE into the
current sub-PTE without scanning the whole NAPOT block. Keep the public
ptep_get_lockless() semantics unchanged for other lockless users.
With 64K THP enabled in always mode, fio 4K random direct read
throughput improves by about 10% over the previous implementation on the
same system, from 2.615M to 2.874M IOPS and from 9.97GiB/s to 11.0GiB/s,
while average clat drops from 95.54 usec to 86.97 usec.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 20 ++++++++++++++++++++
arch/riscv/mm/contpte.c | 9 +++++++++
2 files changed, 29 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index dd56c20a817bb..3727c148fa0fc 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -597,6 +597,7 @@ void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte);
pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte);
pte_t napotpte_ptep_get_lockless(pte_t *ptep);
+pte_t napotpte_ptep_get_gup_fast(pte_t *ptep, pte_t orig_pte);
void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte, unsigned int nr);
void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr,
@@ -956,6 +957,25 @@ static inline pte_t ptep_get_lockless(pte_t *ptep)
return napotpte_ptep_get_lockless(ptep);
}
+#define gup_ptep_get_lockless gup_ptep_get_lockless
+static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp)
+{
+ pte_t pte = __ptep_get_lockless(ptep);
+
+ *rawp = pte;
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get_gup_fast(ptep, pte);
+}
+
+#define gup_ptep_revalidate gup_ptep_revalidate
+static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte)
+{
+ return pte_val(raw_pte) == pte_val(__ptep_get_lockless(ptep));
+}
+
#define set_ptes set_ptes
static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pteval, unsigned int nr)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 0f66426b0ecfd..b0b8ff0aade3a 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -413,6 +413,15 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
}
EXPORT_SYMBOL(napotpte_ptep_get_lockless);
+pte_t napotpte_ptep_get_gup_fast(pte_t *orig_ptep, pte_t orig_pte)
+{
+ if (!napot_hw_supported() || !pte_present_napot(orig_pte))
+ return orig_pte;
+
+ return napotpte_subpte(orig_ptep, orig_pte);
+}
+EXPORT_SYMBOL(napotpte_ptep_get_gup_fast);
+
unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte,
fpb_t flags)
{
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 11/15] riscv: mm: avoid Svnapot A/D aggregation in fast-GUP
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
fast-GUP only needs the PTE state for the current virtual address and
revalidates the raw PTE after pinning the page. It does not consume the
block-wide accessed/dirty aggregation provided by the public Svnapot
lockless getter.
Override the fast-GUP PTE helpers to translate a NAPOT PTE into the
current sub-PTE without scanning the whole NAPOT block. Keep the public
ptep_get_lockless() semantics unchanged for other lockless users.
With 64K THP enabled in always mode, fio 4K random direct read
throughput improves by about 10% over the previous implementation on the
same system, from 2.615M to 2.874M IOPS and from 9.97GiB/s to 11.0GiB/s,
while average clat drops from 95.54 usec to 86.97 usec.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 20 ++++++++++++++++++++
arch/riscv/mm/contpte.c | 9 +++++++++
2 files changed, 29 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index dd56c20a817bb..3727c148fa0fc 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -597,6 +597,7 @@ void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte);
pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte);
pte_t napotpte_ptep_get_lockless(pte_t *ptep);
+pte_t napotpte_ptep_get_gup_fast(pte_t *ptep, pte_t orig_pte);
void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pte, unsigned int nr);
void napotpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr,
@@ -956,6 +957,25 @@ static inline pte_t ptep_get_lockless(pte_t *ptep)
return napotpte_ptep_get_lockless(ptep);
}
+#define gup_ptep_get_lockless gup_ptep_get_lockless
+static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp)
+{
+ pte_t pte = __ptep_get_lockless(ptep);
+
+ *rawp = pte;
+
+ if (likely(!pte_present_napot(pte)))
+ return pte;
+
+ return napotpte_ptep_get_gup_fast(ptep, pte);
+}
+
+#define gup_ptep_revalidate gup_ptep_revalidate
+static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte)
+{
+ return pte_val(raw_pte) == pte_val(__ptep_get_lockless(ptep));
+}
+
#define set_ptes set_ptes
static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t pteval, unsigned int nr)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 0f66426b0ecfd..b0b8ff0aade3a 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -413,6 +413,15 @@ pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
}
EXPORT_SYMBOL(napotpte_ptep_get_lockless);
+pte_t napotpte_ptep_get_gup_fast(pte_t *orig_ptep, pte_t orig_pte)
+{
+ if (!napot_hw_supported() || !pte_present_napot(orig_pte))
+ return orig_pte;
+
+ return napotpte_subpte(orig_ptep, orig_pte);
+}
+EXPORT_SYMBOL(napotpte_ptep_get_gup_fast);
+
unsigned int napotpte_pte_batch_hint_from_pte(pte_t *ptep, pte_t orig_pte,
fpb_t flags)
{
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 12/15] arm64: mm: avoid contpte A/D aggregation in fast-GUP
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
fast-GUP only needs the PTE state for the current virtual address and
revalidates the raw PTE after pinning the page. It does not consume the
contpte block-wide accessed/dirty aggregation provided by the public
lockless getter.
Override the fast-GUP PTE helpers to use the raw PTE for the current
entry and raw revalidation. Keep the public ptep_get_lockless() semantics
unchanged for other lockless users.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/arm64/include/asm/pgtable.h | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 018781a1fcbee..d1e08dce9f7c5 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1760,6 +1760,22 @@ static inline pte_t ptep_get_lockless(pte_t *ptep)
return contpte_ptep_get_lockless(ptep);
}
+#define gup_ptep_get_lockless gup_ptep_get_lockless
+static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp)
+{
+ pte_t pte = __ptep_get(ptep);
+
+ *rawp = pte;
+
+ return pte;
+}
+
+#define gup_ptep_revalidate gup_ptep_revalidate
+static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte)
+{
+ return pte_val(raw_pte) == pte_val(__ptep_get(ptep));
+}
+
static inline void set_pte(pte_t *ptep, pte_t pte)
{
/*
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 12/15] arm64: mm: avoid contpte A/D aggregation in fast-GUP
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
fast-GUP only needs the PTE state for the current virtual address and
revalidates the raw PTE after pinning the page. It does not consume the
contpte block-wide accessed/dirty aggregation provided by the public
lockless getter.
Override the fast-GUP PTE helpers to use the raw PTE for the current
entry and raw revalidation. Keep the public ptep_get_lockless() semantics
unchanged for other lockless users.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/arm64/include/asm/pgtable.h | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 018781a1fcbee..d1e08dce9f7c5 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1760,6 +1760,22 @@ static inline pte_t ptep_get_lockless(pte_t *ptep)
return contpte_ptep_get_lockless(ptep);
}
+#define gup_ptep_get_lockless gup_ptep_get_lockless
+static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp)
+{
+ pte_t pte = __ptep_get(ptep);
+
+ *rawp = pte;
+
+ return pte;
+}
+
+#define gup_ptep_revalidate gup_ptep_revalidate
+static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte)
+{
+ return pte_val(raw_pte) == pte_val(__ptep_get(ptep));
+}
+
static inline void set_pte(pte_t *ptep, pte_t pte)
{
/*
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 12/15] arm64: mm: avoid contpte A/D aggregation in fast-GUP
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
fast-GUP only needs the PTE state for the current virtual address and
revalidates the raw PTE after pinning the page. It does not consume the
contpte block-wide accessed/dirty aggregation provided by the public
lockless getter.
Override the fast-GUP PTE helpers to use the raw PTE for the current
entry and raw revalidation. Keep the public ptep_get_lockless() semantics
unchanged for other lockless users.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/arm64/include/asm/pgtable.h | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 018781a1fcbee..d1e08dce9f7c5 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1760,6 +1760,22 @@ static inline pte_t ptep_get_lockless(pte_t *ptep)
return contpte_ptep_get_lockless(ptep);
}
+#define gup_ptep_get_lockless gup_ptep_get_lockless
+static inline pte_t gup_ptep_get_lockless(pte_t *ptep, pte_t *rawp)
+{
+ pte_t pte = __ptep_get(ptep);
+
+ *rawp = pte;
+
+ return pte;
+}
+
+#define gup_ptep_revalidate gup_ptep_revalidate
+static inline bool gup_ptep_revalidate(pte_t *ptep, pte_t raw_pte)
+{
+ return pte_val(raw_pte) == pte_val(__ptep_get(ptep));
+}
+
static inline void set_pte(pte_t *ptep, pte_t pte)
{
/*
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 13/15] riscv/mm: remove redundant TLB flush in napotpte_convert
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
napotpte_convert() switches a NAPOT block between the Svnapot encoding
and the equivalent set of base PTEs. The conversion itself preserves the
effective translations: PFNs and permissions are kept the same while the
page-table representation changes.
Do not issue an SFENCE.VMA in the middle of this representation change.
For partial updates, such as unfolding a writable NAPOT block before
write-protecting one subrange, a flush at conversion time would not be the
operation that enforces the final permission downgrade. Writable base-PTE
translations could still be refilled after the conversion and before the
subrange is write-protected. The required TLB maintenance belongs to the
operation that actually changes the effective mapping.
The RISC-V privileged architecture discussion clarifies that Svnapot is
an encoding of a naturally aligned range. When napotpte_convert() only
switches between the NAPOT encoding and the corresponding set of base PTEs
for the same effective translations, the conversion itself does not
require an extra TLB invalidation beyond the usual page-table update
rules.
Link: https://lists.riscv.org/g/tech-privileged/message/2950
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/mm/contpte.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index b0b8ff0aade3a..8289f39d6e188 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -129,7 +129,7 @@ static pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t target)
{
- unsigned long start_addr, end, ptent_addr;
+ unsigned long start_addr, ptent_addr;
pte_t *start_ptep;
pte_t ptent, pte;
unsigned int i, nr;
@@ -137,7 +137,6 @@ static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
start_addr = napot_align_addr(addr);
start_ptep = napot_align_ptep(ptep);
nr = napotpte_pte_num();
- end = start_addr + napotpte_size();
for (i = 0; i < nr; i++) {
ptent_addr = start_addr + i * PAGE_SIZE;
@@ -150,8 +149,15 @@ static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
target = pte_mkyoung(target);
}
- flush_tlb_mm_range(mm, start_addr, end, PAGE_SIZE);
-
+ /*
+ * This conversion only changes the representation of the same
+ * effective translations: NAPOT encoding <-> equivalent base PTEs.
+ * A flush here would not replace the TLB maintenance required by a
+ * later permission downgrade, because writable base-PTE translations
+ * could be refilled after this conversion and before that downgrade.
+ * Leave TLB invalidation to the operation that changes the effective
+ * mapping.
+ */
page_table_check_ptes_set(mm, start_addr, start_ptep, target, nr);
if (pte_napot(target)) {
for (i = 0; i < nr; i++)
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 13/15] riscv/mm: remove redundant TLB flush in napotpte_convert
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
napotpte_convert() switches a NAPOT block between the Svnapot encoding
and the equivalent set of base PTEs. The conversion itself preserves the
effective translations: PFNs and permissions are kept the same while the
page-table representation changes.
Do not issue an SFENCE.VMA in the middle of this representation change.
For partial updates, such as unfolding a writable NAPOT block before
write-protecting one subrange, a flush at conversion time would not be the
operation that enforces the final permission downgrade. Writable base-PTE
translations could still be refilled after the conversion and before the
subrange is write-protected. The required TLB maintenance belongs to the
operation that actually changes the effective mapping.
The RISC-V privileged architecture discussion clarifies that Svnapot is
an encoding of a naturally aligned range. When napotpte_convert() only
switches between the NAPOT encoding and the corresponding set of base PTEs
for the same effective translations, the conversion itself does not
require an extra TLB invalidation beyond the usual page-table update
rules.
Link: https://lists.riscv.org/g/tech-privileged/message/2950
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/mm/contpte.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index b0b8ff0aade3a..8289f39d6e188 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -129,7 +129,7 @@ static pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t target)
{
- unsigned long start_addr, end, ptent_addr;
+ unsigned long start_addr, ptent_addr;
pte_t *start_ptep;
pte_t ptent, pte;
unsigned int i, nr;
@@ -137,7 +137,6 @@ static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
start_addr = napot_align_addr(addr);
start_ptep = napot_align_ptep(ptep);
nr = napotpte_pte_num();
- end = start_addr + napotpte_size();
for (i = 0; i < nr; i++) {
ptent_addr = start_addr + i * PAGE_SIZE;
@@ -150,8 +149,15 @@ static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
target = pte_mkyoung(target);
}
- flush_tlb_mm_range(mm, start_addr, end, PAGE_SIZE);
-
+ /*
+ * This conversion only changes the representation of the same
+ * effective translations: NAPOT encoding <-> equivalent base PTEs.
+ * A flush here would not replace the TLB maintenance required by a
+ * later permission downgrade, because writable base-PTE translations
+ * could be refilled after this conversion and before that downgrade.
+ * Leave TLB invalidation to the operation that changes the effective
+ * mapping.
+ */
page_table_check_ptes_set(mm, start_addr, start_ptep, target, nr);
if (pte_napot(target)) {
for (i = 0; i < nr; i++)
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 13/15] riscv/mm: remove redundant TLB flush in napotpte_convert
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
napotpte_convert() switches a NAPOT block between the Svnapot encoding
and the equivalent set of base PTEs. The conversion itself preserves the
effective translations: PFNs and permissions are kept the same while the
page-table representation changes.
Do not issue an SFENCE.VMA in the middle of this representation change.
For partial updates, such as unfolding a writable NAPOT block before
write-protecting one subrange, a flush at conversion time would not be the
operation that enforces the final permission downgrade. Writable base-PTE
translations could still be refilled after the conversion and before the
subrange is write-protected. The required TLB maintenance belongs to the
operation that actually changes the effective mapping.
The RISC-V privileged architecture discussion clarifies that Svnapot is
an encoding of a naturally aligned range. When napotpte_convert() only
switches between the NAPOT encoding and the corresponding set of base PTEs
for the same effective translations, the conversion itself does not
require an extra TLB invalidation beyond the usual page-table update
rules.
Link: https://lists.riscv.org/g/tech-privileged/message/2950
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/mm/contpte.c | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index b0b8ff0aade3a..8289f39d6e188 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -129,7 +129,7 @@ static pte_t __get_and_clear_full_ptes(struct mm_struct *mm,
static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
pte_t *ptep, pte_t target)
{
- unsigned long start_addr, end, ptent_addr;
+ unsigned long start_addr, ptent_addr;
pte_t *start_ptep;
pte_t ptent, pte;
unsigned int i, nr;
@@ -137,7 +137,6 @@ static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
start_addr = napot_align_addr(addr);
start_ptep = napot_align_ptep(ptep);
nr = napotpte_pte_num();
- end = start_addr + napotpte_size();
for (i = 0; i < nr; i++) {
ptent_addr = start_addr + i * PAGE_SIZE;
@@ -150,8 +149,15 @@ static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
target = pte_mkyoung(target);
}
- flush_tlb_mm_range(mm, start_addr, end, PAGE_SIZE);
-
+ /*
+ * This conversion only changes the representation of the same
+ * effective translations: NAPOT encoding <-> equivalent base PTEs.
+ * A flush here would not replace the TLB maintenance required by a
+ * later permission downgrade, because writable base-PTE translations
+ * could be refilled after this conversion and before that downgrade.
+ * Leave TLB invalidation to the operation that changes the effective
+ * mapping.
+ */
page_table_check_ptes_set(mm, start_addr, start_ptep, target, nr);
if (pte_napot(target)) {
for (i = 0; i < nr; i++)
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 14/15] riscv/mm: optimize mprotect for Svnapot mappings
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot represents a naturally aligned range as a block of base PTEs, but
generic mprotect() still processes a full block one base PTE at a time.
For a full-block permission change, that repeatedly clears and reinstalls
the same logical mapping and then has to reconstruct the folded Svnapot
block afterwards, making the operation unnecessarily expensive.
Use the batched protection transaction hooks to handle a full Svnapot
block as one operation. The fast path is restricted to user mappings
where hardware Svnapot is supported, the range is exactly one aligned
Svnapot block, and every sub-PTE is consistent with the block PTE. The
generic per-PTE fallback remains in place for partial, unaligned, special,
non-user, or otherwise inconsistent mappings.
libMicro mprotect median latency improves on the full-block cases:
mprot_tw4m (-l 4m -w -t -f /dev/zero): 40.4 us -> 4.3 us
mprot_tw128k (-l 128k -w -t -f /dev/zero): about 15-16% faster
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 8 ++++
arch/riscv/mm/contpte.c | 76 ++++++++++++++++++++++++++++++++
2 files changed, 84 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 3727c148fa0fc..b546e6614c558 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -619,6 +619,14 @@ int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep);
int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep);
+#define modify_prot_start_ptes modify_prot_start_ptes
+pte_t modify_prot_start_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr);
+#define modify_prot_commit_ptes modify_prot_commit_ptes
+void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pte_t *ptep, pte_t old_pte, pte_t pte,
+ unsigned int nr);
#endif
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 8289f39d6e188..6a1156629d115 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -190,6 +190,82 @@ napotpte_is_batch_consistent(pte_t pte, pte_t batch_pte, fpb_t flags)
pte_val(batch_pte);
}
+static bool napotpte_can_modify_prot_block(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, pte_t *raw_pte)
+{
+ pte_t raw, pte;
+ unsigned int i;
+
+ if (!napot_hw_supported() || !mm_is_user(mm))
+ return false;
+
+ if (nr != napotpte_pte_num())
+ return false;
+
+ if (addr != napot_align_addr(addr) || ptep != napot_align_ptep(ptep))
+ return false;
+
+ raw = READ_ONCE(*ptep);
+ if (!pte_present_napot(raw) || pte_special(raw))
+ return false;
+
+ for (i = 0; i < nr; i++) {
+ pte = READ_ONCE(ptep[i]);
+ if (!napotpte_is_consistent(pte, raw))
+ return false;
+ }
+
+ *raw_pte = raw;
+ return true;
+}
+
+pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pte_t *ptep, unsigned int nr)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ unsigned long ptent_addr;
+ pte_t raw, pte, ptent;
+ unsigned int i;
+
+ if (napotpte_can_modify_prot_block(mm, addr, ptep, nr, &raw)) {
+ pte = pte_mknonnapot(raw, addr);
+
+ for (i = 0; i < nr; i++) {
+ ptent_addr = addr + i * PAGE_SIZE;
+ ptent = __ptep_get_and_clear_noptc(ptep + i);
+ page_table_check_pte_clear(mm, ptent_addr,
+ pte_mknonnapot(ptent, ptent_addr));
+ if (pte_dirty(ptent))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(ptent))
+ pte = pte_mkyoung(pte);
+ }
+
+ return pte;
+ }
+
+ pte = __ptep_modify_prot_start(vma, addr, ptep);
+ while (--nr) {
+ ptep++;
+ addr += PAGE_SIZE;
+ ptent = __ptep_modify_prot_start(vma, addr, ptep);
+ if (pte_dirty(ptent))
+ pte = pte_mkdirty(pte);
+ if (pte_young(ptent))
+ pte = pte_mkyoung(pte);
+ }
+
+ return pte;
+}
+
+void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pte_t *ptep, pte_t old_pte, pte_t pte,
+ unsigned int nr)
+{
+ set_ptes(vma->vm_mm, addr, ptep, pte, nr);
+}
+
static inline pte_t
napotpte_normalize_batch_pte(pte_t *ptep, pte_t orig_pte, fpb_t flags)
{
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 14/15] riscv/mm: optimize mprotect for Svnapot mappings
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot represents a naturally aligned range as a block of base PTEs, but
generic mprotect() still processes a full block one base PTE at a time.
For a full-block permission change, that repeatedly clears and reinstalls
the same logical mapping and then has to reconstruct the folded Svnapot
block afterwards, making the operation unnecessarily expensive.
Use the batched protection transaction hooks to handle a full Svnapot
block as one operation. The fast path is restricted to user mappings
where hardware Svnapot is supported, the range is exactly one aligned
Svnapot block, and every sub-PTE is consistent with the block PTE. The
generic per-PTE fallback remains in place for partial, unaligned, special,
non-user, or otherwise inconsistent mappings.
libMicro mprotect median latency improves on the full-block cases:
mprot_tw4m (-l 4m -w -t -f /dev/zero): 40.4 us -> 4.3 us
mprot_tw128k (-l 128k -w -t -f /dev/zero): about 15-16% faster
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 8 ++++
arch/riscv/mm/contpte.c | 76 ++++++++++++++++++++++++++++++++
2 files changed, 84 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 3727c148fa0fc..b546e6614c558 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -619,6 +619,14 @@ int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep);
int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep);
+#define modify_prot_start_ptes modify_prot_start_ptes
+pte_t modify_prot_start_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr);
+#define modify_prot_commit_ptes modify_prot_commit_ptes
+void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pte_t *ptep, pte_t old_pte, pte_t pte,
+ unsigned int nr);
#endif
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 8289f39d6e188..6a1156629d115 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -190,6 +190,82 @@ napotpte_is_batch_consistent(pte_t pte, pte_t batch_pte, fpb_t flags)
pte_val(batch_pte);
}
+static bool napotpte_can_modify_prot_block(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, pte_t *raw_pte)
+{
+ pte_t raw, pte;
+ unsigned int i;
+
+ if (!napot_hw_supported() || !mm_is_user(mm))
+ return false;
+
+ if (nr != napotpte_pte_num())
+ return false;
+
+ if (addr != napot_align_addr(addr) || ptep != napot_align_ptep(ptep))
+ return false;
+
+ raw = READ_ONCE(*ptep);
+ if (!pte_present_napot(raw) || pte_special(raw))
+ return false;
+
+ for (i = 0; i < nr; i++) {
+ pte = READ_ONCE(ptep[i]);
+ if (!napotpte_is_consistent(pte, raw))
+ return false;
+ }
+
+ *raw_pte = raw;
+ return true;
+}
+
+pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pte_t *ptep, unsigned int nr)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ unsigned long ptent_addr;
+ pte_t raw, pte, ptent;
+ unsigned int i;
+
+ if (napotpte_can_modify_prot_block(mm, addr, ptep, nr, &raw)) {
+ pte = pte_mknonnapot(raw, addr);
+
+ for (i = 0; i < nr; i++) {
+ ptent_addr = addr + i * PAGE_SIZE;
+ ptent = __ptep_get_and_clear_noptc(ptep + i);
+ page_table_check_pte_clear(mm, ptent_addr,
+ pte_mknonnapot(ptent, ptent_addr));
+ if (pte_dirty(ptent))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(ptent))
+ pte = pte_mkyoung(pte);
+ }
+
+ return pte;
+ }
+
+ pte = __ptep_modify_prot_start(vma, addr, ptep);
+ while (--nr) {
+ ptep++;
+ addr += PAGE_SIZE;
+ ptent = __ptep_modify_prot_start(vma, addr, ptep);
+ if (pte_dirty(ptent))
+ pte = pte_mkdirty(pte);
+ if (pte_young(ptent))
+ pte = pte_mkyoung(pte);
+ }
+
+ return pte;
+}
+
+void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pte_t *ptep, pte_t old_pte, pte_t pte,
+ unsigned int nr)
+{
+ set_ptes(vma->vm_mm, addr, ptep, pte, nr);
+}
+
static inline pte_t
napotpte_normalize_batch_pte(pte_t *ptep, pte_t orig_pte, fpb_t flags)
{
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 14/15] riscv/mm: optimize mprotect for Svnapot mappings
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
Svnapot represents a naturally aligned range as a block of base PTEs, but
generic mprotect() still processes a full block one base PTE at a time.
For a full-block permission change, that repeatedly clears and reinstalls
the same logical mapping and then has to reconstruct the folded Svnapot
block afterwards, making the operation unnecessarily expensive.
Use the batched protection transaction hooks to handle a full Svnapot
block as one operation. The fast path is restricted to user mappings
where hardware Svnapot is supported, the range is exactly one aligned
Svnapot block, and every sub-PTE is consistent with the block PTE. The
generic per-PTE fallback remains in place for partial, unaligned, special,
non-user, or otherwise inconsistent mappings.
libMicro mprotect median latency improves on the full-block cases:
mprot_tw4m (-l 4m -w -t -f /dev/zero): 40.4 us -> 4.3 us
mprot_tw128k (-l 128k -w -t -f /dev/zero): about 15-16% faster
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 8 ++++
arch/riscv/mm/contpte.c | 76 ++++++++++++++++++++++++++++++++
2 files changed, 84 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 3727c148fa0fc..b546e6614c558 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -619,6 +619,14 @@ int napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep);
int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,
unsigned long address, pte_t *ptep);
+#define modify_prot_start_ptes modify_prot_start_ptes
+pte_t modify_prot_start_ptes(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr);
+#define modify_prot_commit_ptes modify_prot_commit_ptes
+void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pte_t *ptep, pte_t old_pte, pte_t pte,
+ unsigned int nr);
#endif
#ifdef CONFIG_ARCH_HAS_PTE_PROTNONE
diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
index 8289f39d6e188..6a1156629d115 100644
--- a/arch/riscv/mm/contpte.c
+++ b/arch/riscv/mm/contpte.c
@@ -190,6 +190,82 @@ napotpte_is_batch_consistent(pte_t pte, pte_t batch_pte, fpb_t flags)
pte_val(batch_pte);
}
+static bool napotpte_can_modify_prot_block(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ unsigned int nr, pte_t *raw_pte)
+{
+ pte_t raw, pte;
+ unsigned int i;
+
+ if (!napot_hw_supported() || !mm_is_user(mm))
+ return false;
+
+ if (nr != napotpte_pte_num())
+ return false;
+
+ if (addr != napot_align_addr(addr) || ptep != napot_align_ptep(ptep))
+ return false;
+
+ raw = READ_ONCE(*ptep);
+ if (!pte_present_napot(raw) || pte_special(raw))
+ return false;
+
+ for (i = 0; i < nr; i++) {
+ pte = READ_ONCE(ptep[i]);
+ if (!napotpte_is_consistent(pte, raw))
+ return false;
+ }
+
+ *raw_pte = raw;
+ return true;
+}
+
+pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pte_t *ptep, unsigned int nr)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ unsigned long ptent_addr;
+ pte_t raw, pte, ptent;
+ unsigned int i;
+
+ if (napotpte_can_modify_prot_block(mm, addr, ptep, nr, &raw)) {
+ pte = pte_mknonnapot(raw, addr);
+
+ for (i = 0; i < nr; i++) {
+ ptent_addr = addr + i * PAGE_SIZE;
+ ptent = __ptep_get_and_clear_noptc(ptep + i);
+ page_table_check_pte_clear(mm, ptent_addr,
+ pte_mknonnapot(ptent, ptent_addr));
+ if (pte_dirty(ptent))
+ pte = riscv_pte_mkhwdirty(pte);
+ if (pte_young(ptent))
+ pte = pte_mkyoung(pte);
+ }
+
+ return pte;
+ }
+
+ pte = __ptep_modify_prot_start(vma, addr, ptep);
+ while (--nr) {
+ ptep++;
+ addr += PAGE_SIZE;
+ ptent = __ptep_modify_prot_start(vma, addr, ptep);
+ if (pte_dirty(ptent))
+ pte = pte_mkdirty(pte);
+ if (pte_young(ptent))
+ pte = pte_mkyoung(pte);
+ }
+
+ return pte;
+}
+
+void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr,
+ pte_t *ptep, pte_t old_pte, pte_t pte,
+ unsigned int nr)
+{
+ set_ptes(vma->vm_mm, addr, ptep, pte, nr);
+}
+
static inline pte_t
napotpte_normalize_batch_pte(pte_t *ptep, pte_t orig_pte, fpb_t flags)
{
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 15/15] riscv: mm: Request large exec folios for Svnapot
2026-07-16 12:41 ` Yunhui Cui
(?)
@ 2026-07-16 12:41 ` Yunhui Cui
-1 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
The Svnapot MM support in this series allows generic MM paths to operate
on folded 64KB executable mappings while preserving a per-page public PTE
view. To benefit from that more often, executable file-backed mappings
should be faulted in with a folio size that can be folded into a Svnapot
range.
Request 64KB executable folios when Svnapot is available, so executable
text mappings are more likely to be installed as folded Svnapot blocks
and benefit from lower iTLB pressure. Keep the default order-0 behavior
when Svnapot is unavailable.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index b546e6614c558..6801ba135c483 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -296,6 +296,14 @@ static __always_inline bool has_svnapot(void)
return riscv_has_extension_likely(RISCV_ISA_EXT_SVNAPOT);
}
+/*
+ * Request executable file-backed mappings to be read into 64KB folios when
+ * Svnapot is available. A naturally aligned 64KB folio can be folded into a
+ * Svnapot PTE range, reducing iTLB pressure for executable text.
+ */
+#define exec_folio_order() \
+ (has_svnapot() ? NAPOT_CONT64KB_ORDER : 0)
+
static inline unsigned long pte_napot(pte_t pte)
{
return pte_val(pte) & _PAGE_NAPOT;
--
2.39.5
--
kvm-riscv mailing list
kvm-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kvm-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 15/15] riscv: mm: Request large exec folios for Svnapot
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
The Svnapot MM support in this series allows generic MM paths to operate
on folded 64KB executable mappings while preserving a per-page public PTE
view. To benefit from that more often, executable file-backed mappings
should be faulted in with a folio size that can be folded into a Svnapot
range.
Request 64KB executable folios when Svnapot is available, so executable
text mappings are more likely to be installed as folded Svnapot blocks
and benefit from lower iTLB pressure. Keep the default order-0 behavior
when Svnapot is unavailable.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index b546e6614c558..6801ba135c483 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -296,6 +296,14 @@ static __always_inline bool has_svnapot(void)
return riscv_has_extension_likely(RISCV_ISA_EXT_SVNAPOT);
}
+/*
+ * Request executable file-backed mappings to be read into 64KB folios when
+ * Svnapot is available. A naturally aligned 64KB folio can be folded into a
+ * Svnapot PTE range, reducing iTLB pressure for executable text.
+ */
+#define exec_folio_order() \
+ (has_svnapot() ? NAPOT_CONT64KB_ORDER : 0)
+
static inline unsigned long pte_napot(pte_t pte)
{
return pte_val(pte) & _PAGE_NAPOT;
--
2.39.5
^ permalink raw reply related [flat|nested] 57+ messages in thread
* [PATCH v2 15/15] riscv: mm: Request large exec folios for Svnapot
@ 2026-07-16 12:41 ` Yunhui Cui
0 siblings, 0 replies; 57+ messages in thread
From: Yunhui Cui @ 2026-07-16 12:41 UTC (permalink / raw)
To: linux-riscv, linux-mm, linux-kernel
Cc: linux-arch, kvm-riscv, kvm, linux-arm-kernel, linux-efi,
linux-perf-users, kasan-dev, Yunhui Cui
The Svnapot MM support in this series allows generic MM paths to operate
on folded 64KB executable mappings while preserving a per-page public PTE
view. To benefit from that more often, executable file-backed mappings
should be faulted in with a folio size that can be folded into a Svnapot
range.
Request 64KB executable folios when Svnapot is available, so executable
text mappings are more likely to be installed as folded Svnapot blocks
and benefit from lower iTLB pressure. Keep the default order-0 behavior
when Svnapot is unavailable.
Signed-off-by: Yunhui Cui <cuiyunhui@bytedance.com>
---
arch/riscv/include/asm/pgtable.h | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index b546e6614c558..6801ba135c483 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -296,6 +296,14 @@ static __always_inline bool has_svnapot(void)
return riscv_has_extension_likely(RISCV_ISA_EXT_SVNAPOT);
}
+/*
+ * Request executable file-backed mappings to be read into 64KB folios when
+ * Svnapot is available. A naturally aligned 64KB folio can be folded into a
+ * Svnapot PTE range, reducing iTLB pressure for executable text.
+ */
+#define exec_folio_order() \
+ (has_svnapot() ? NAPOT_CONT64KB_ORDER : 0)
+
static inline unsigned long pte_napot(pte_t pte)
{
return pte_val(pte) & _PAGE_NAPOT;
--
2.39.5
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply related [flat|nested] 57+ messages in thread
* Re: [PATCH v2 07/15] riscv/mm: preserve Svnapot leaf-size semantics for page-table consumers
2026-07-16 12:41 ` Yunhui Cui
(?)
(?)
@ 2026-07-16 12:57 ` sashiko-bot
-1 siblings, 0 replies; 57+ messages in thread
From: sashiko-bot @ 2026-07-16 12:57 UTC (permalink / raw)
To: Yunhui Cui; +Cc: linux-perf-users, kvm
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Single-CPU deadlock in `napotpte_ptep_get_lockless` when called from NMI/IRQ context during a partial Svnapot block update.
--
--- Patch [7]: [PATCH v2 07/15] riscv/mm: preserve Svnapot leaf-size semantics for page-table consumers ---
commit 0488a9106519eab47850cf758ecce3d0c6bdf46e
Author: Yunhui Cui <cuiyunhui@bytedance.com>
riscv/mm: preserve Svnapot leaf-size semantics for page-table consumers
Svnapot keeps a folded 64KB mapping encoded in the page table, while the
public PTE helpers may expose a logical per-page sub-PTE view. This patch
uses the original page-table entry for leaf-size queries so consumers
continue to observe the Svnapot leaf size instead of treating a folded
mapping as PAGE_SIZE entries.
> diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
> index da7ffea0dcd98..e4b9c05014c00 100644
> --- a/arch/riscv/include/asm/pgtable.h
> +++ b/arch/riscv/include/asm/pgtable.h
[ ... ]
> @@ -580,6 +580,17 @@ static inline pte_t pte_swp_clear_soft_dirty(pte_t pte)
> napot_cont_size(napot_cont_order(pte)) :\
> PAGE_SIZE)
>
> +#define __ptep_leaf_size __ptep_leaf_size
> +static inline unsigned long __ptep_leaf_size(pmd_t pmd, pte_t *ptep, pte_t pte)
> +{
> + pte_t raw_pte = READ_ONCE(*ptep);
> +
> + if (pte_present_napot(raw_pte))
> + return pte_leaf_size(raw_pte);
> +
> + return PAGE_SIZE;
> +}
[Severity: High]
Can this lead to a single-CPU deadlock in NMI/IRQ context during a partial
Svnapot block update?
If a thread executing napotpte_convert() holds the page table lock (which
disables preemption but leaves NMIs and hardware IRQs enabled) and
sequentially clears PTEs in a 64KB Svnapot block:
arch/riscv/mm/contpte.c:napotpte_convert() {
...
for (i = 0; i < nr; i++) {
ptent_addr = start_addr + i * PAGE_SIZE;
ptent = __ptep_get_and_clear_noptc(start_ptep + i);
...
}
And a Performance Monitoring Interrupt (PMI) interrupts this CPU after some
but not all PTEs are cleared, the PMI handler may invoke
perf_get_pgtable_size(), which eventually invokes the lockless PTE reader.
This would lead to napotpte_ptep_get_lockless() entering a retry loop:
arch/riscv/mm/contpte.c:napotpte_ptep_get_lockless() {
...
retry:
orig_pte = READ_ONCE(*orig_ptep);
if (!pte_present_napot(orig_pte))
return orig_pte;
ptep = napot_align_ptep(orig_ptep);
for (i = 0; i < nr; i++, ptep++) {
pte = READ_ONCE(*ptep);
if (!napotpte_is_consistent(pte, orig_pte))
goto retry;
...
}
Because the NMI/IRQ handler prevents the interrupted thread on the same CPU
from finishing the update, the block remains partially cleared. Does this
cause the consistency check to fail repeatedly, resulting in an infinite
loop and hard lockup?
Userspace could potentially trigger this via perf_event_open by sampling
memory accesses during Svnapot page migrations or unmapping.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1784201104.git.cuiyunhui@bytedance.com?part=7
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v2 03/15] riscv/mm: implement Svnapot contpte read-side helpers
2026-07-16 12:41 ` Yunhui Cui
(?)
(?)
@ 2026-07-16 12:59 ` sashiko-bot
-1 siblings, 0 replies; 57+ messages in thread
From: sashiko-bot @ 2026-07-16 12:59 UTC (permalink / raw)
To: Yunhui Cui; +Cc: linux-perf-users, kvm
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Critical] Missing single-PTE modifier overrides for NAPOT blocks break contiguous encoding, causing Use-After-Free and lockups.
- [High] Unbounded retry loop in `napotpte_ptep_get_lockless()` causes same-CPU deadlock if interrupted by NMI/IRQ.
- [High] `page_table_check` refcount corruption during Svnapot block conversion/clearing.
- [Medium] `ptep_get` and `ptep_get_lockless` strip the `_PAGE_NAPOT` bit, breaking `pte_leaf_size()` for generic MM.
--
--- Patch [3]: [PATCH v2 03/15] riscv/mm: implement Svnapot contpte read-side helpers ---
commit 5e07f7373fe29103de43866f47711b800fb7a33e
Author: Yunhui Cui <cuiyunhui@bytedance.com>
riscv/mm: implement Svnapot contpte read-side helpers
This patch introduces read-side helpers for Svnapot contiguous mappings to
allow generic MM code to operate on the logical sub-PTEs. The public PTE
getters are routed through these helpers.
> diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
> index a0f64e980e981..fce6d04ea65db 100644
> --- a/arch/riscv/include/asm/pgtable.h
> +++ b/arch/riscv/include/asm/pgtable.h
> @@ -706,37 +706,37 @@ static inline pte_t __ptep_get_lockless(pte_t *ptep)
> return __ptep_get(ptep);
> }
>
> +#ifdef CONFIG_RISCV_ISA_SVNAPOT
> +
> +#define ptep_get ptep_get
> +static inline pte_t ptep_get(pte_t *ptep)
> +{
> + pte_t pte = __ptep_get(ptep);
> +
> + if (likely(!pte_present_napot(pte)))
> + return pte;
> +
> + return napotpte_ptep_get(ptep, pte);
> +}
> +
> +#define ptep_get_lockless ptep_get_lockless
> +static inline pte_t ptep_get_lockless(pte_t *ptep)
> +{
> + pte_t pte = __ptep_get_lockless(ptep);
> +
> + if (likely(!pte_present_napot(pte)))
> + return pte;
> +
> + return napotpte_ptep_get_lockless(ptep);
> +}
> +
[Severity: Critical]
Does this implementation miss overriding ptep_get_and_clear and
ptep_modify_prot_start for NAPOT blocks?
If a NAPOT block is partially unmapped or modified via madvise() or
mprotect(), the generic MM code will call ptep_get_and_clear() on a single
PTE. Because these functions are not overridden, they fall back to the raw
__ptep_get_and_clear().
This clears only one PTE, leaving the other 15 with _PAGE_NAPOT still set,
which breaks hardware rules. Additionally, it returns the raw NAPOT PTE where
pte_pfn() evaluates to the base PFN, causing the core MM to drop the
mapcount and refcount on the base sub-page instead of the actual targeted
sub-page. Could this lead to data corruption or a use-after-free?
[ ... ]
> diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
> new file mode 100644
> index 0000000000000..43ea76e424492
> --- /dev/null
> +++ b/arch/riscv/mm/contpte.c
> @@ -0,0 +1,273 @@
[ ... ]
> +static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
> +{
> + unsigned long pfn;
> + pgprot_t prot;
> +
> + if (!pte_present_napot(pte))
> + return pte;
> +
> + pfn = pte_pfn(pte) + (ptep - napot_align_ptep(ptep));
> + prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
> +
> + return pfn_pte(pfn, prot);
> +}
[Severity: Medium]
Since this helper strips the _PAGE_NAPOT bit by using
pte_protval_no_pfn_no_napot(pte), does it break pte_leaf_size() for generic MM?
If pte_leaf_size() expects the Svnapot bit to be present to return the
contiguous block size, returning a subpte without it will cause callers like
perf_get_page_size() to falsely evaluate the mapping as PAGE_SIZE.
> +
> +static inline pte_t __napot_ptep_get_and_clear(struct mm_struct *mm,
> + unsigned long addr, pte_t *ptep)
> +{
> + pte_t pte;
> +
> + pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));
> + page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
> +
> + return pte;
> +}
[Severity: High]
Can this cause page_table_check refcount corruption during Svnapot block
conversion?
pte_mknonnapot() returns the raw PTE unmodified. Because the raw Svnapot PFN
encoded in all 16 PTEs is identically base_pfn | 0x8, pte_pfn() inside
page_table_check_pte_clear() evaluates to the exact same base_pfn for all
16 iterations.
Will this cause the refcount for the base PFN to be decremented 16 times
while leaking the remaining 15 mapped sub-PFNs?
> +
> +static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
> + pte_t *ptep, pte_t target)
> +{
> + unsigned long start_addr, end, ptent_addr;
> + pte_t *start_ptep;
> + pte_t ptent, pte;
> + unsigned int i, nr;
> +
> + start_addr = napot_align_addr(addr);
> + start_ptep = napot_align_ptep(ptep);
> + nr = napotpte_pte_num();
> + end = start_addr + napotpte_size();
> +
> + for (i = 0; i < nr; i++) {
> + ptent_addr = start_addr + i * PAGE_SIZE;
> + ptent = __napot_ptep_get_and_clear(mm, ptent_addr,
> + start_ptep + i);
[Severity: High]
Does this create a preemption window that can lead to a CPU deadlock?
napotpte_convert() modifies NAPOT blocks by clearing their 16 PTEs sequentially
while holding the PTL, but it does not disable local interrupts.
[ ... ]
> +pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
> +{
> + pte_t orig_pte, pte;
> + pte_t *ptep;
> + unsigned int i, nr;
> +
> + if (!napot_hw_supported())
> + return READ_ONCE(*orig_ptep);
> +
> + nr = napotpte_pte_num();
> +
> +retry:
> + orig_pte = READ_ONCE(*orig_ptep);
> + if (!pte_present_napot(orig_pte))
> + return orig_pte;
> +
> + ptep = napot_align_ptep(orig_ptep);
> +
> + for (i = 0; i < nr; i++, ptep++) {
> + pte = READ_ONCE(*ptep);
> +
> + if (!napotpte_is_consistent(pte, orig_pte))
> + goto retry;
[Severity: High]
If a perf NMI interrupts napotpte_convert() on the same CPU while the block is
partially cleared and attempts a lockless page table walk, won't this loop
spin indefinitely?
The getter detects the intermediate inconsistency, jumps to retry, and spins
forever because the writer is interrupted on the exact same CPU and cannot
resume to restore consistency.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1784201104.git.cuiyunhui@bytedance.com?part=3
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v2 01/15] riscv: introduce raw PTE helpers
2026-07-16 12:41 ` Yunhui Cui
(?)
(?)
@ 2026-07-16 12:59 ` sashiko-bot
-1 siblings, 0 replies; 57+ messages in thread
From: sashiko-bot @ 2026-07-16 12:59 UTC (permalink / raw)
To: Yunhui Cui; +Cc: linux-perf-users, kvm
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [Low] The patch declares raw PTE helpers `__ptep_test_and_clear_young` and `__ptep_set_access_flags` but never defines them, causing a linker error.
Pre-existing issues:
- [High] Non-atomic read-modify-write in `__ptep_set_wrprotect` can overwrite concurrent hardware A/D bit updates, causing silent data corruption.
--
--- Patch [1]: [PATCH v2 01/15] riscv: introduce raw PTE helpers ---
commit 3ca72eca5363bad7487aaf256a6d44b7c83506ca
Author: Yunhui Cui <cuiyunhui@bytedance.com>
riscv: introduce raw PTE helpers
Svnapot needs RISC-V to distinguish between two PTE views: the raw encoded
entries used by architecture code, and the logical per-page PTE view that
will later be exposed to generic MM paths. Split the low-level RISC-V PTE
operations from the public helpers so the raw helpers can keep operating on
the hardware encoding.
> diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
> index a810655ce8f9b..8ca5da9534967 100644
> --- a/arch/riscv/include/asm/pgtable.h
> +++ b/arch/riscv/include/asm/pgtable.h
[ ... ]
> #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS /* defined in mm/pgtable.c */
> extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address,
> pte_t *ptep, pte_t entry, int dirty);
> +int __ptep_set_access_flags(struct vm_area_struct *vma,
> + unsigned long address, pte_t *ptep,
> + pte_t entry, int dirty);
> #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG /* defined in mm/pgtable.c */
> bool ptep_test_and_clear_young(struct vm_area_struct *vma,
> - unsigned long address, pte_t *ptep);
> + unsigned long address, pte_t *ptep);
> +bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
> + unsigned long address, pte_t *ptep);
>
[Severity: Low]
Are the newly declared __ptep_test_and_clear_young() and
__ptep_set_access_flags() functions implemented anywhere in this patch?
They appear to be declared here but missing definitions, which could cause a
linker error when __ptep_clear_flush_young() calls them below.
[ ... ]
> #define __HAVE_ARCH_PTEP_SET_WRPROTECT
> -static inline void ptep_set_wrprotect(struct mm_struct *mm,
> - unsigned long address, pte_t *ptep)
> +static inline void __ptep_set_wrprotect(struct mm_struct *mm,
> + unsigned long address, pte_t *ptep)
> {
> pte_t read_pte = READ_ONCE(*ptep);
> /*
> @@ -705,9 +784,11 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm,
> ((pte_val(read_pte) & ~(unsigned long)_PAGE_WRITE) | _PAGE_READ));
> }
[Severity: High]
This is a pre-existing issue, but does this non-atomic read-modify-write
sequence overwrite concurrent hardware A/D bit updates?
If a concurrent hardware A/D bit update (e.g. via the RISC-V Svadu hardware
page table walker) occurs between the READ_ONCE(*ptep) and the
atomic_long_set(), it seems any bit set by the hardware during this window
would be permanently lost. Could this lead to silent data corruption if a
dirty bit is lost on a writable page during writeback?
This would also propagate to huge pages since pmdp_set_wrprotect()
delegates to this helper.
> +#define __ptep_set_wrprotect __ptep_set_wrprotect
> +
> #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH
> -static inline bool ptep_clear_flush_young(struct vm_area_struct *vma,
> - unsigned long address, pte_t *ptep)
> +static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
> + unsigned long address, pte_t *ptep)
> {
> /*
[ ... ]
> - return ptep_test_and_clear_young(vma, address, ptep);
> + return __ptep_test_and_clear_young(vma, address, ptep);
> }
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1784201104.git.cuiyunhui@bytedance.com?part=1
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v2 04/15] riscv/mm: implement Svnapot contpte update helpers
2026-07-16 12:41 ` Yunhui Cui
(?)
(?)
@ 2026-07-16 13:03 ` sashiko-bot
-1 siblings, 0 replies; 57+ messages in thread
From: sashiko-bot @ 2026-07-16 13:03 UTC (permalink / raw)
To: Yunhui Cui; +Cc: linux-perf-users, kvm
Thank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider:
New issues:
- [Critical] Extracting sub-PTEs from a SVNAPOT PTE fails to mask out the NAPOT size encoding bits from the PFN, resulting in incorrect physical address mappings.
- [Critical] napotpte_clear_young_dirty_ptes() unsafely expands its bounds to the entire NAPOT block, clearing dirty bits for unaffected pages and causing data loss during reclaim.
- [High] napotpte_wrprotect_ptes modifies the write permission bit in-place for fully-covered SVNAPOT blocks without unfolding them, temporarily violating the RISC-V SVNAPOT identical-PTE requirement.
- [High] Folding 16 4KB PTEs into a 64KB NAPOT PTE without a full TLB flush creates overlapping TLB entries, leading to hardware Machine Check Exceptions.
- [Medium] napotpte_ptep_clear_flush_young unconditionally issues a full synchronous TLB flush, which destroys the RISC-V page reclaim optimization.
Pre-existing issues:
- [High] Unbounded spin-retry loop in lockless PTE accessor causes livelock if interrupted by an NMI or hardirq.
--
--- Patch [4]: [PATCH v2 04/15] riscv/mm: implement Svnapot contpte update helpers ---
commit 6810ac383bbe56cdae57e157126e14c9d376a982
Author: Yunhui Cui <cuiyunhui@bytedance.com>
Subject: riscv/mm: implement Svnapot contpte update helpers
This commit adds Svnapot-aware contpte update helpers to preserve logical
per-page views for generic MM users. It routes public PTE operations through
these helpers to manage Svnapot folded mappings safely.
> diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
> index fce6d04ea65db..771faabb2b00b 100644
> --- a/arch/riscv/include/asm/pgtable.h
> +++ b/arch/riscv/include/asm/pgtable.h
[ ... ]
> @@ -413,6 +413,30 @@ static inline int pte_special(pte_t pte)
> +#ifdef CONFIG_RISCV_ISA_SVNAPOT
> +static inline pte_t pte_mknonnapot(pte_t pte, unsigned long addr)
> +{
> + unsigned long pfn;
> + unsigned long offset;
> + pgprot_t prot;
> +
> + if (!pte_present_napot(pte))
> + return pte;
> +
> + offset = (addr & (napot_cont_size(napot_cont_order(pte)) - 1)) >>
> + PAGE_SHIFT;
> + pfn = pte_pfn(pte) + offset;
[Severity: Critical]
Does pte_pfn() return the raw PFN including the NAPOT size encoding bit (like
PPN[3] for 64KB blocks)?
If the size bit is not masked out before adding the offset, it appears this
could shift the physical mapping by 32KB and map the wrong physical memory
when breaking down a PTE.
> + prot = __pgprot((pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT);
[ ... ]
> @@ -870,7 +881,169 @@ static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
[ ... ]
> +static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
> + pte_t *ptep, pte_t pteval, unsigned int nr)
> +{
> + pteval = pte_mknonnapot(pteval, addr);
> +
> + if (likely(nr == 1)) {
> + napotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));
> + __set_ptes(mm, addr, ptep, pteval, 1);
> + napotpte_try_fold(mm, addr, ptep, pteval);
[Severity: High]
Can folding 16 4KB PTEs into a NAPOT PTE here without a full TLB flush create
overlapping TLB entries?
Since napotpte_convert() skips the flush expecting the caller to handle it,
the remaining 15 4KB entries might stay cached alongside the new 64KB entry.
On RISC-V, overlapping TLB entries can trigger hardware machine check
exceptions.
> + return;
> + }
> +
> + napotpte_set_ptes(mm, addr, ptep, pteval, nr);
> +}
> diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
> index 43ea76e424492..3f8a0a2970afb 100644
> --- a/arch/riscv/mm/contpte.c
> +++ b/arch/riscv/mm/contpte.c
> @@ -62,6 +62,24 @@ static inline unsigned long pte_protval_no_pfn_no_napot(pte_t pte)
[ ... ]
> +static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)
> +{
> + unsigned long pfn;
> + pgprot_t prot;
> +
> + if (!pte_present_napot(pte))
> + return pte;
> +
> + pfn = pte_pfn(pte) + (ptep - napot_align_ptep(ptep));
[Severity: Critical]
Similar to pte_mknonnapot(), is the NAPOT size encoding bit left unmasked
here when extracting the base PFN?
> + prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
[ ... ]
> +void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,
> + pte_t *ptep, pte_t pte)
> +{
> + pte_t target;
> + pgprot_t prot;
> +
> + if (!napot_hw_supported() || !mm_is_user(mm) ||
> + !pte_present_napot(pte))
> + return;
> +
> + prot = __pgprot(pte_protval_no_pfn_no_napot(pte));
> + target = pfn_pte(pte_pfn(pte), prot);
[Severity: Critical]
Does target need to mask the NAPOT size bit out of pte_pfn(pte) here as well
before generating all 16 base PTEs?
> +
> + napotpte_convert(mm, addr, ptep, target);
> +}
[ ... ]
> +pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
> +{
[ ... ]
> +retry:
> + orig_pte = READ_ONCE(*orig_ptep);
> + if (!pte_present_napot(orig_pte))
> + return orig_pte;
> +
> + ptep = napot_align_ptep(orig_ptep);
> +
> + for (i = 0; i < nr; i++, ptep++) {
> + pte = READ_ONCE(*ptep);
> +
> + if (!napotpte_is_consistent(pte, orig_pte))
> + goto retry;
[Severity: High]
This is a pre-existing issue, but can this unbounded spin-retry loop cause a
hard lockup?
If a page table writer is interrupted by an NMI or hardirq while modifying a
NAPOT block, and napotpte_ptep_get_lockless() is called from that interrupt
context, it appears the NMI will spin forever waiting for the block to become
consistent since the interrupted writer cannot resume.
> +
> + if (pte_dirty(pte)) {
[ ... ]
> +void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,
> + unsigned long addr, pte_t *ptep,
> + unsigned int nr, cydp_t flags)
> +{
> + struct mm_struct *mm;
> + unsigned long start, end;
> + unsigned int total;
> +
> + mm = vma->vm_mm;
> + if (!napot_hw_supported() || !mm_is_user(mm)) {
> + __clear_young_dirty_ptes(vma, addr, ptep, nr, flags);
> + return;
> + }
> +
> + start = addr;
> + end = start + nr * PAGE_SIZE;
> +
> + if (pte_present_napot(READ_ONCE(*(ptep + nr - 1))))
> + end = ALIGN(end, napotpte_size());
> +
> + if (pte_present_napot(READ_ONCE(*ptep))) {
> + start = napot_align_addr(start);
> + ptep = napot_align_ptep(ptep);
> + }
[Severity: Critical]
Could this unbounded expansion of the operating range cause data loss?
If madvise_free_pte_range() passes a restricted nr for a partially split
folio, expanding the bounds to the full 64KB block will clear hardware dirty
bits for unaffected pages outside the requested range.
This could lead to those pages being dropped without writeback during memory
reclaim.
> +
> + total = (end - start) >> PAGE_SHIFT;
> + for (; total; total--, ptep++, start += PAGE_SIZE)
> + napotpte_clear_young_dirty_pte(ptep, flags);
> +}
[ ... ]
> +void napotpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr,
> + pte_t *ptep, unsigned int nr)
> +{
> + unsigned int i;
> +
> + if (!napot_hw_supported() || !mm_is_user(mm)) {
> + for (i = 0; i < nr; i++, ptep++, addr += PAGE_SIZE)
> + __ptep_set_wrprotect(mm, addr, ptep);
> + return;
> + }
> +
> + napotpte_try_unfold_partial(mm, addr, ptep, nr);
> +
> + for (i = 0; i < nr; i++, ptep++, addr += PAGE_SIZE)
> + __ptep_set_wrprotect(mm, addr, ptep);
[Severity: High]
Does modifying the write permission bit in-place for fully-covered blocks
violate the identical-PTE requirement?
If the fully-covered blocks are not unfolded first, iterating sequentially
through __ptep_set_wrprotect() creates a window where PTEs in the same NAPOT
block have mixed write permissions.
This transient state could trigger a machine check exception if the block is
accessed concurrently.
> +}
[ ... ]
> +int napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,
> + unsigned long address, pte_t *ptep)
> +{
> + unsigned long start_addr;
> + int young;
> +
> + young = napotpte_ptep_test_and_clear_young(vma, address, ptep);
> + if (!young)
> + return 0;
> +
> + start_addr = napot_align_addr(address);
> + flush_tlb_range(vma, start_addr, start_addr + napotpte_size());
[Severity: Medium]
Will this unconditional flush_tlb_range() trigger an IPI storm during memory
reclaim?
The base RISC-V implementation deliberately avoids TLB flushes when clearing
the accessed bit to optimize reclaim, and adding a heavyweight flush here
appears to heavily degrade system performance.
> +
> + return young;
> +}
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1784201104.git.cuiyunhui@bytedance.com?part=4
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v2 09/15] riscv/mm: avoid Svnapot consistency checks in ptep_get()
2026-07-16 12:41 ` Yunhui Cui
(?)
(?)
@ 2026-07-16 13:03 ` sashiko-bot
-1 siblings, 0 replies; 57+ messages in thread
From: sashiko-bot @ 2026-07-16 13:03 UTC (permalink / raw)
To: Yunhui Cui; +Cc: linux-perf-users, kvm
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Unbounded `goto retry` loop in `napotpte_ptep_get_lockless()` causes a hard lockup if it interrupts a thread on the same CPU that is concurrently modifying the NAPOT block.
--
--- Patch [9]: [PATCH v2 09/15] riscv/mm: avoid Svnapot consistency checks in ptep_get() ---
commit 6e778d5fb0d9c3254546937eeef1badca713e19a
Author: Yunhui Cui <cuiyunhui@bytedance.com>
riscv/mm: avoid Svnapot consistency checks in ptep_get()
This commit removes range consistency checks from the generic ptep_get() path,
leaving them only in napotpte_ptep_get_lockless(). It also adds comments
explaining the lockless retry behavior.
> diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
> index 70f9a0668b50c..0f66426b0ecfd 100644
> --- a/arch/riscv/mm/contpte.c
> +++ b/arch/riscv/mm/contpte.c
[ ... ]
> @@ -345,6 +346,12 @@ EXPORT_SYMBOL(napotpte_ptep_get);
>
> pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)
> {
> + /*
> + * ptep_get_lockless() must return a self-consistent PTE without the
> + * PTL. Recheck that the whole NAPOT range still describes the same
> + * mapping, ignoring A/D bits, and retry if a concurrent update tears
> + * the range while A/D state is being gathered.
> + */
> pte_t orig_pte, pte;
> pte_t *ptep;
> unsigned int i, nr;
[Severity: High]
Could the unbounded retry loop in napotpte_ptep_get_lockless() cause a hard
lockup regression if it interrupts a modifying thread on the same CPU?
If a thread modifying the NAPOT block is preempted by an NMI (such as a PMU
sampling interrupt) exactly in the middle of an update, and the NMI handler
calls ptep_get_lockless():
arch/riscv/mm/contpte.c:napotpte_ptep_get_lockless() {
...
retry:
orig_pte = READ_ONCE(*orig_ptep);
if (!pte_present_napot(orig_pte))
return orig_pte;
ptep = napot_align_ptep(orig_ptep);
for (i = 0; i < nr; i++, ptep++) {
pte = READ_ONCE(*ptep);
if (!napotpte_is_consistent(pte, orig_pte))
goto retry;
...
}
Will this partially updated block cause napotpte_is_consistent() to fail and
jump to retry?
Since the modifying thread is suspended on the same CPU, it would never be able
to resume and finish the update, resulting in an infinite spin loop.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1784201104.git.cuiyunhui@bytedance.com?part=9
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v2 02/15] riscv: switch arch page-table users to raw PTE helpers
2026-07-16 12:41 ` Yunhui Cui
(?)
(?)
@ 2026-07-16 13:05 ` sashiko-bot
-1 siblings, 0 replies; 57+ messages in thread
From: sashiko-bot @ 2026-07-16 13:05 UTC (permalink / raw)
To: Yunhui Cui; +Cc: kvm, linux-perf-users
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
New issues:
- [High] Svnapot huge page set and clear paths have mismatched page_table_check PFN tracking and use unaligned addresses, leading to immediate BUG_ON panics.
- [Medium] The conversion to raw PTE helpers incorrectly bypasses the page_table_check framework for non-NAPOT hugetlb pages.
Pre-existing issues:
- [Critical] KVM stage-2 page table pages are freed before the stage-2 TLB is flushed, causing a use-after-free race.
- [High] huge_pte_clear() and huge_ptep_get_and_clear() fail to clear all contiguous PTEs for non-present entries like migration or hwpoison entries.
--
--- Patch [2]: [PATCH v2 02/15] riscv: switch arch page-table users to raw PTE helpers ---
commit 3f0c1de9839b5c9daaa76b96222e6c57e5096eda
Author: Yunhui Cui <cuiyunhui@bytedance.com>
Subject: riscv: switch arch page-table users to raw PTE helpers
This patch converts RISC-V internal page-table walkers and mutators to
use raw PTE helpers. It also adds a raw get-and-clear helper and updates
the hugetlb Svnapot paths to operate on contiguous blocks using the raw
helper model.
> diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c
> index c4c3b79567f10..67b2c75f5cec1 100644
> --- a/arch/riscv/kvm/gstage.c
> +++ b/arch/riscv/kvm/gstage.c
[ ... ]
> @@ -371,18 +371,18 @@ bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr,
>
> WARN_ON(addr & (page_size - 1));
>
> - if (!pte_val(ptep_get(ptep)))
> + if (!pte_val(__ptep_get(ptep)))
> return false;
>
> if (ptep_level && !gstage_pte_leaf(ptep)) {
> - next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep));
> + next_ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep));
> next_ptep_level = ptep_level - 1;
> ret = gstage_level_to_page_size(gstage, next_ptep_level, &next_page_size);
> if (ret)
> return false;
>
> if (op == GSTAGE_OP_CLEAR)
> - set_pte(ptep, __pte(0));
> + __set_pte(ptep, __pte(0));
> for (i = 0; i < PTRS_PER_PTE; i++)
> flush |= kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size,
> &next_ptep[i], next_ptep_level, op);
> if (op == GSTAGE_OP_CLEAR)
> put_page(virt_to_page(next_ptep));
[Severity: Critical]
This is a pre-existing issue, but does this unconditionally free the child
page table page before the stage-2 TLB is invalidated?
The page is freed here by calling put_page() before returning the flush
boolean to the caller. This allows a concurrent hardware page table walk
to use the cached TLB directory entry to read from the freed page.
> diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c
> index a6d217112cf46..5ededac02161d 100644
> --- a/arch/riscv/mm/hugetlbpage.c
> +++ b/arch/riscv/mm/hugetlbpage.c
[ ... ]
> @@ -145,28 +145,52 @@ unsigned long hugetlb_mask_last_page(struct hstate *h)
> return 0UL;
> }
>
> +static unsigned long napot_hugetlb_block_addr(pte_t pte, unsigned long addr)
> +{
> + unsigned long order;
> +
> + if (!pte_napot(pte))
> + return addr;
> +
> + order = napot_cont_order(pte);
> +
> + return addr & napot_cont_mask(order);
> +}
> +
> +static pte_t napot_hugetlb_get_and_clear(struct mm_struct *mm,
> + unsigned long addr, pte_t *ptep)
> +{
> + pte_t pte;
> +
> + pte = __ptep_get_and_clear_noptc(ptep);
> + page_table_check_pte_clear(mm, addr, pte_mknonnapot(pte, addr));
> +
> + return pte;
> +}
[Severity: High]
Does this cause a refcount underflow and trigger a BUG_ON in the
page_table_check logic?
When a SVNAPOT huge page is unmapped, get_clear_contig() iteratively calls
napot_hugetlb_get_and_clear() 16 times for a 64KB block. Each iteration calls
page_table_check_pte_clear() with the exact same encoded PTE (base_PFN + 8).
Since set_huge_napot_ptes() tracks the block consecutively, untracking the
same base PFN + 8 sixteen times will underflow the counter and panic.
[ ... ]
> @@ -207,14 +234,31 @@ static void clear_flush(struct mm_struct *mm,
> unsigned long ncontig)
> {
> struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0);
> + pte_t pte = __ptep_get(ptep);
> unsigned long i, saddr = addr;
>
> + addr = napot_hugetlb_block_addr(pte, addr);
> + if (pte_napot(pte))
> + ptep = huge_pte_offset(mm, addr,
> + napot_cont_size(napot_cont_order(pte)));
> + saddr = addr;
> +
> for (i = 0; i < ncontig; i++, addr += pgsize, ptep++)
> - ptep_get_and_clear(mm, addr, ptep);
> + napot_hugetlb_get_and_clear(mm, addr, ptep);
>
> flush_tlb_range(&vma, saddr, addr);
> }
>
> +static void set_huge_napot_ptes(struct mm_struct *mm, unsigned long addr,
> + pte_t *ptep, pte_t pte, unsigned long pte_num)
> +{
> + unsigned long i;
> +
> + page_table_check_ptes_set(mm, addr, ptep, pte, pte_num);
> + for (i = 0; i < pte_num; i++)
> + __set_pte_at(mm, ptep + i, pte);
> +}
[Severity: High]
Can this pass an unaligned faulting address to page_table_check_ptes_set()
and track the wrong PFNs?
When called from huge_ptep_set_access_flags(), addr is the unaligned
faulting address, whereas the clear path block-aligns it. This tracks and
untracks different virtual address ranges.
Additionally, passing pte_num to page_table_check_ptes_set() tracks
consecutive PFNs starting from the raw encoded PFN (base_PFN + 8 for order 4)
instead of the true base PFN.
[ ... ]
> @@ -256,19 +300,18 @@ void set_huge_pte_at(struct mm_struct *mm,
>
> if (!pte_present(pte)) {
> for (i = 0; i < pte_num; i++, ptep++, addr += pgsize)
> - set_ptes(mm, addr, ptep, pte, 1);
> + __set_ptes(mm, addr, ptep, pte, 1);
> return;
> }
>
> if (!pte_napot(pte)) {
> - set_ptes(mm, addr, ptep, pte, 1);
> + __set_ptes(mm, addr, ptep, pte, 1);
> return;
> }
[Severity: Medium]
Does this bypass the page_table_check framework for non-NAPOT 2MB huge pages?
The conversion to the raw __set_ptes() helper removes the internal
page_table_check_ptes_set() call that set_ptes() previously provided. Without
a manual page_table_check call on this fallback path, memory corruption
detection is lost for 2MB PMD-mapped huge pages.
[ ... ]
> @@ -306,14 +348,13 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm,
> unsigned long addr,
> pte_t *ptep, unsigned long sz)
> {
> - size_t pgsize;
> - pte_t orig_pte = ptep_get(ptep);
> + pte_t orig_pte = __ptep_get(ptep);
> int pte_num;
>
> if (!pte_napot(orig_pte))
> - return ptep_get_and_clear(mm, addr, ptep);
> + return __ptep_get_and_clear(mm, addr, ptep);
[Severity: High]
This is a pre-existing issue, but does this fail to clear all contiguous PTEs
for non-present entries like migration or hwpoison entries?
Since non-present entries do not have the hardware _PAGE_NAPOT bit set,
pte_napot() evaluates to false. This fallback path returns early after
clearing only a single 4KB PTE, leaving the remaining 15 stale entries
in the page table.
[Severity: Medium]
Also related to the conversion to raw PTE helpers, does replacing
ptep_get_and_clear() with __ptep_get_and_clear() bypass the page_table_check
framework for non-NAPOT 2MB huge pages?
[ ... ]
> @@ -363,19 +403,16 @@ void huge_pte_clear(struct mm_struct *mm,
> pte_t *ptep,
> unsigned long sz)
> {
> - size_t pgsize;
> - pte_t pte = ptep_get(ptep);
> - int i, pte_num;
> + pte_t pte = __ptep_get(ptep);
> + int pte_num;
>
> if (!pte_napot(pte)) {
> - pte_clear(mm, addr, ptep);
> + __pte_clear(mm, addr, ptep);
> return;
> }
[Severity: High]
This is a pre-existing issue, but does this also fail to clear all contiguous
PTEs for non-present entries (migration or hwpoison) because pte_napot()
evaluates to false?
[Severity: Medium]
Similar to the issue above, does replacing pte_clear() with __pte_clear()
without a manual page_table_check call disable memory corruption tracking
for non-NAPOT 2MB huge pages?
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1784201104.git.cuiyunhui@bytedance.com?part=2
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v2 05/15] mm: extend pte batch and leaf-size helpers
2026-07-16 12:41 ` Yunhui Cui
(?)
(?)
@ 2026-07-16 13:06 ` sashiko-bot
-1 siblings, 0 replies; 57+ messages in thread
From: sashiko-bot @ 2026-07-16 13:06 UTC (permalink / raw)
To: Yunhui Cui; +Cc: linux-perf-users, kvm
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] The new `__ptep_leaf_size` API encourages a double-read TOCTOU race in the lockless page table walker `perf_get_pgtable_size()`, violating generic lockless walk concurrency rules.
- [Low] The `__pte_batch_clear_ignored()` inline function breaks the build for architectures without an MMU (!CONFIG_MMU).
--
--- Patch [5]: [PATCH v2 05/15] mm: extend pte batch and leaf-size helpers ---
commit a6d281b7dea89e7c1eed5117dbe4f8bc7a9fd3f6
Author: Yunhui Cui <cuiyunhui@bytedance.com>
mm: extend pte batch and leaf-size helpers
This commit extends pte_batch_hint() with batch flags and introduces
__ptep_leaf_size() to give leaf-size users access to the original PTE pointer.
It also moves batch flag helpers into pgtable.h.
> diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
> index 8c093c119e5a8..da14328093a86 100644
> --- a/include/linux/pgtable.h
> +++ b/include/linux/pgtable.h
[ ... ]
> @@ -1917,6 +1948,17 @@ static inline pmd_t pmd_swp_clear_soft_dirty(pmd_t pmd)
> }
> #endif
>
> +static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)
> +{
> + if (!(flags & FPB_RESPECT_DIRTY))
> + pte = pte_mkclean(pte);
> + if (likely(!(flags & FPB_RESPECT_SOFT_DIRTY)))
> + pte = pte_clear_soft_dirty(pte);
> + if (likely(!(flags & FPB_RESPECT_WRITE)))
> + pte = pte_wrprotect(pte);
> + return pte_mkold(pte);
> +}
> +
[Severity: Low]
Does moving __pte_batch_clear_ignored() to the global scope in pgtable.h
break the build for !CONFIG_MMU architectures?
The functions pte_mkclean(), pte_wrprotect(), and pte_mkold() are generally
undefined without an MMU. Since this is now placed outside of any CONFIG_MMU
guard, it appears it could cause implicit function declaration errors for
nommu targets.
> diff --git a/kernel/events/core.c b/kernel/events/core.c
> index 51c1200ea3fdd..37b860abe9e61 100644
> --- a/kernel/events/core.c
> +++ b/kernel/events/core.c
> @@ -8551,7 +8551,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr)
>
> pte = ptep_get_lockless(ptep);
> if (pte_present(pte))
> - size = __pte_leaf_size(pmd, pte);
> + size = __ptep_leaf_size(pmd, ptep, pte);
> pte_unmap(ptep);
> #endif /* CONFIG_HAVE_GUP_FAST */
>
[Severity: Medium]
Could the new __ptep_leaf_size() API introduce a Time-Of-Check to
Time-Of-Use race in lockless walkers like perf_get_pgtable_size()?
In perf_get_pgtable_size(), we acquire an atomic snapshot of the PTE using
ptep_get_lockless(), and then pass both the snapshot (pte) and the raw
pointer (ptep) to __ptep_leaf_size().
If an architecture implementation of __ptep_leaf_size() reads the ptep
pointer directly to handle folded mappings, it would perform a second
non-atomic read. A concurrent page table modification could alter the PTE
between the initial ptep_get_lockless() snapshot and the architecture's
read, leading to a mismatched evaluation.
Lockless walkers typically need to derive all information from a single
atomic snapshot to ensure concurrency rules are respected.
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1784201104.git.cuiyunhui@bytedance.com?part=5
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v2 14/15] riscv/mm: optimize mprotect for Svnapot mappings
2026-07-16 12:41 ` Yunhui Cui
(?)
(?)
@ 2026-07-16 13:07 ` sashiko-bot
-1 siblings, 0 replies; 57+ messages in thread
From: sashiko-bot @ 2026-07-16 13:07 UTC (permalink / raw)
To: Yunhui Cui; +Cc: kvm, linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The fallback path corrupts userspace memory tracking by erroneously injecting `_PAGE_SOFT_DIRTY` via `pte_mkdirty()`.
--
--- Patch [14]: [PATCH v2 14/15] riscv/mm: optimize mprotect for Svnapot mappings ---
commit f75eddb541d9ba388068a3cbeef6571534eb416b
Author: Yunhui Cui <cuiyunhui@bytedance.com>
riscv/mm: optimize mprotect for Svnapot mappings
This commit optimizes mprotect operations for Svnapot mappings by handling
full blocks in a single batched operation. A fallback loop is retained for
partial, unaligned, or otherwise inconsistent mappings.
> diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
> index 8289f39d6e188..6a1156629d115 100644
> --- a/arch/riscv/mm/contpte.c
> +++ b/arch/riscv/mm/contpte.c
[ ... ]
> +pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr,
> + pte_t *ptep, unsigned int nr)
> +{
> + struct mm_struct *mm = vma->vm_mm;
> + unsigned long ptent_addr;
> + pte_t raw, pte, ptent;
> + unsigned int i;
> +
> + if (napotpte_can_modify_prot_block(mm, addr, ptep, nr, &raw)) {
[ ... ]
> + if (pte_dirty(ptent))
> + pte = riscv_pte_mkhwdirty(pte);
[ ... ]
> + return pte;
> + }
> +
> + pte = __ptep_modify_prot_start(vma, addr, ptep);
> + while (--nr) {
> + ptep++;
> + addr += PAGE_SIZE;
> + ptent = __ptep_modify_prot_start(vma, addr, ptep);
> + if (pte_dirty(ptent))
> + pte = pte_mkdirty(pte);
[Severity: Medium]
Does this fallback path unintentionally set the soft-dirty bit for userspace
tracking?
On RISC-V, pte_mkdirty() sets both the hardware _PAGE_DIRTY bit and the
userspace-visible _PAGE_SOFT_DIRTY bit. If a Svnapot mapping was previously
written to (hardware dirty) but its soft-dirty bit was explicitly cleared by
userspace (e.g. via /proc/pid/clear_refs), calling mprotect() on a partial
or unaligned segment of this block triggers this fallback path.
Since the fallback path uses pte_mkdirty() to accumulate the dirty state, it
would inadvertently re-set the _PAGE_SOFT_DIRTY bit on the final batched PTE.
Userspace tracking tools like CRIU could subsequently falsely record this
memory as modified.
The fast path correctly uses riscv_pte_mkhwdirty(pte) to avoid this exact
issue. Should this loop also use riscv_pte_mkhwdirty(pte) instead of
pte_mkdirty(pte)?
> + if (pte_young(ptent))
> + pte = pte_mkyoung(pte);
> + }
> +
> + return pte;
> +}
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1784201104.git.cuiyunhui@bytedance.com?part=14
^ permalink raw reply [flat|nested] 57+ messages in thread
* Re: [PATCH v2 13/15] riscv/mm: remove redundant TLB flush in napotpte_convert
2026-07-16 12:41 ` Yunhui Cui
(?)
(?)
@ 2026-07-16 13:17 ` sashiko-bot
-1 siblings, 0 replies; 57+ messages in thread
From: sashiko-bot @ 2026-07-16 13:17 UTC (permalink / raw)
To: Yunhui Cui; +Cc: kvm, linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Removing the TLB flush in napotpte_convert() allows stale 4KB TLB entries to survive when folding base PTEs into a NAPOT block, leading to silent data corruption during partial-block TLB flushes.
--
--- Patch [13]: [PATCH v2 13/15] riscv/mm: remove redundant TLB flush in napotpte_convert ---
commit 63217ab7f83ce36005df21143bf66fc315027a26
Author: Yunhui Cui <cuiyunhui@bytedance.com>
riscv/mm: remove redundant TLB flush in napotpte_convert
This commit removes the TLB flush during the conversion between a NAPOT
block and its equivalent base PTEs in napotpte_convert(). The justification
is that since the effective translations (PFNs and permissions) are unchanged,
TLB invalidation can be deferred to operations that actually change the
effective mapping.
Link: https://lists.riscv.org/g/tech-privileged/message/2950
> diff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c
> index b0b8ff0aade3a..8289f39d6e188 100644
> --- a/arch/riscv/mm/contpte.c
> +++ b/arch/riscv/mm/contpte.c
[ ... ]
> @@ -149,8 +149,15 @@ static void napotpte_convert(struct mm_struct *mm, unsigned long addr,
> target = pte_mkyoung(target);
> }
>
> - flush_tlb_mm_range(mm, start_addr, end, PAGE_SIZE);
> -
[Severity: High]
Does removing this TLB flush allow stale 4KB TLB entries to survive when
folding base PTEs into a NAPOT block?
When set_ptes() maps a single page that completes a 16-page aligned block
backed by a large folio, __napotpte_try_fold() folds the 16 base PTEs into
a 64KB NAPOT block via napotpte_convert(). The CPU may have already cached
separate 4KB TLB entries for the existing 15 pages. Without the TLB flush
in napotpte_convert(), these 4KB TLB entries can survive the folding process.
Could this lead to silent data corruption during partial-block TLB flushes?
Later, if an operation like madvise(MADV_FREE) targets a single 4KB subpage
of this NAPOT block, it calls split_folio(), which splits the underlying
folio but does not alter the NAPOT page table entries. On retry, the loop
processes the now order-0 folio with nr = 1 and calls
clear_young_dirty_ptes() with nr = 1.
Because the PTE still has the NAPOT bit set, napotpte_clear_young_dirty_ptes()
correctly expands the operation to clear the young and dirty bits for all 16
PTEs in the page table to maintain the NAPOT encoding invariant. However,
madvise_free_pte_range() then queues only a 1-page TLB flush:
tlb_remove_tlb_entries(tlb, pte, 1, addr);
If the TLB held a 64KB NAPOT entry, the single-page sfence.vma would
successfully invalidate the entire 64KB block. But because the old 4KB TLB
entries survived the folding, the flush only invalidates the single targeted
4KB entry, leaving the other 15 4KB TLB entries valid.
Since their dirty bits were cleared in the page table but they are still
cached as dirty in the TLB, subsequent writes to these 15 pages will not set
the hardware dirty bit in the page table. Will the kernel later view these
modified pages as clean and discard them without writeback during memory
reclaim, dropping dirty data?
> + /*
> + * This conversion only changes the representation of the same
> + * effective translations: NAPOT encoding <-> equivalent base PTEs.
> + * A flush here would not replace the TLB maintenance required by a
> + * later permission downgrade, because writable base-PTE translations
> + * could be refilled after this conversion and before that downgrade.
> + * Leave TLB invalidation to the operation that changes the effective
> + * mapping.
> + */
> page_table_check_ptes_set(mm, start_addr, start_ptep, target, nr);
> if (pte_napot(target)) {
--
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1784201104.git.cuiyunhui@bytedance.com?part=13
^ permalink raw reply [flat|nested] 57+ messages in thread
end of thread, other threads:[~2026-07-16 13:17 UTC | newest]
Thread overview: 57+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-16 12:41 [PATCH v2 00/15] riscv: add Svnapot PTE folding support Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 01/15] riscv: introduce raw PTE helpers Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:59 ` sashiko-bot
2026-07-16 12:41 ` [PATCH v2 02/15] riscv: switch arch page-table users to " Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 13:05 ` sashiko-bot
2026-07-16 12:41 ` [PATCH v2 03/15] riscv/mm: implement Svnapot contpte read-side helpers Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:59 ` sashiko-bot
2026-07-16 12:41 ` [PATCH v2 04/15] riscv/mm: implement Svnapot contpte update helpers Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 13:03 ` sashiko-bot
2026-07-16 12:41 ` [PATCH v2 05/15] mm: extend pte batch and leaf-size helpers Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 13:06 ` sashiko-bot
2026-07-16 12:41 ` [PATCH v2 06/15] riscv: make pte_batch_hint() honor folio batch flags Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 07/15] riscv/mm: preserve Svnapot leaf-size semantics for page-table consumers Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:57 ` sashiko-bot
2026-07-16 12:41 ` [PATCH v2 08/15] riscv/mm: avoid redundant Svnapot A/D aggregation Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 09/15] riscv/mm: avoid Svnapot consistency checks in ptep_get() Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 13:03 ` sashiko-bot
2026-07-16 12:41 ` [PATCH v2 10/15] mm/gup: add fast-GUP specific lockless PTE helpers Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 11/15] riscv: mm: avoid Svnapot A/D aggregation in fast-GUP Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 12/15] arm64: mm: avoid contpte " Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` [PATCH v2 13/15] riscv/mm: remove redundant TLB flush in napotpte_convert Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 13:17 ` sashiko-bot
2026-07-16 12:41 ` [PATCH v2 14/15] riscv/mm: optimize mprotect for Svnapot mappings Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 13:07 ` sashiko-bot
2026-07-16 12:41 ` [PATCH v2 15/15] riscv: mm: Request large exec folios for Svnapot Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
2026-07-16 12:41 ` Yunhui Cui
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.