* [PATCH v11 01/10] arm64/mm: add vmap_pte_set_huge() and vmap_pte_clear_huge()
2026-11-03 10:16 [PATCH v11 00/10] mm/vmalloc: Speed up ioremap, vmalloc and vmap with contiguous memory Wen Jiang
@ 2026-11-03 10:16 ` Wen Jiang
2026-11-03 10:16 ` [PATCH v11 02/10] powerpc/8xx: add vmap_pte_set_huge() Wen Jiang
` (8 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Wen Jiang @ 2026-11-03 10:16 UTC (permalink / raw)
To: akpm, catalin.marinas, linux-mm, urezki, will
Cc: Xueyuan.chen21, ajd, anshuman.khandual, baohua, chleroy, david,
dev.jain, jiangwen6, leo.yan, linux-arm-kernel, linux-kernel,
linuxppc-dev, maddy, mpe, npiggin, rppt, ryan.roberts
From: Wen Jiang <jiangwen6@xiaomi.com>
vmap installs PTE-level block mappings (PTE_CONT on arm64) by reusing
set_huge_pte_at() and huge_ptep_get_and_clear(), which are HugeTLB
helpers gated by CONFIG_HUGETLB_PAGE. This makes the feature silently
unavailable on CONFIG_HUGETLB_PAGE=n kernels and couples mm/vmalloc.c to
HugeTLB internals it does not otherwise need.
Add vmap_pte_set_huge()/vmap_pte_clear_huge() to arm64, joining the
existing pmd_set_huge()/pud_set_huge() family in mm/mmu.c. They build on
__set_ptes() and __get_and_clear_full_ptes() directly, so they do not
depend on CONFIG_HUGETLB_PAGE.
There is no caller yet: mm/vmalloc.c is converted later in this series,
once the generic fallbacks are in place.
Signed-off-by: Wen Jiang <jiangwen6@xiaomi.com>
Reviewed-by: Barry Song <baohua@kernel.org>
---
arch/arm64/include/asm/pgtable.h | 6 ++++++
arch/arm64/mm/mmu.c | 28 ++++++++++++++++++++++++++++
2 files changed, 34 insertions(+)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index a474c9fb485a..fe138590972a 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1927,6 +1927,12 @@ static inline void clear_young_dirty_ptes(struct vm_area_struct *vma,
#endif /* CONFIG_ARM64_CONTPTE */
+#define __HAVE_ARCH_VMAP_PTE_SET_HUGE
+void vmap_pte_set_huge(pte_t *ptep, unsigned long addr, phys_addr_t phys,
+ pgprot_t prot, unsigned long size);
+#define __HAVE_ARCH_VMAP_PTE_CLEAR_HUGE
+pte_t vmap_pte_clear_huge(pte_t *ptep, unsigned long addr, unsigned long size);
+
#endif /* !__ASSEMBLER__ */
#endif /* __ASM_PGTABLE_H */
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 140520b351e2..2929bec8821c 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -1912,6 +1912,34 @@ int pmd_clear_huge(pmd_t *pmdp)
return 1;
}
+void vmap_pte_set_huge(pte_t *ptep, unsigned long addr, phys_addr_t phys,
+ pgprot_t prot, unsigned long size)
+{
+ unsigned long pfn = __phys_to_pfn(phys);
+ pte_t pte = pte_mkcont(pfn_pte(pfn, prot));
+ unsigned int nr = size >> PAGE_SHIFT;
+
+ VM_WARN_ON(!IS_ALIGNED(size, CONT_PTE_SIZE));
+ VM_WARN_ON(!IS_ALIGNED(addr, CONT_PTE_SIZE));
+ VM_WARN_ON(!IS_ALIGNED(pfn, CONT_PTES));
+
+ /* For vmap, there is no valid -> valid transition */
+ VM_WARN_ON(pte_valid(__ptep_get(ptep)));
+
+ __set_ptes(&init_mm, addr, ptep, pte, nr);
+}
+
+pte_t vmap_pte_clear_huge(pte_t *ptep, unsigned long addr, unsigned long size)
+{
+ /* Kernel mappings only */
+
+ VM_WARN_ON(!IS_ALIGNED(size, CONT_PTE_SIZE));
+ VM_WARN_ON(!IS_ALIGNED(addr, CONT_PTE_SIZE));
+
+ return __get_and_clear_full_ptes(&init_mm, addr, ptep,
+ size >> PAGE_SHIFT, 0);
+}
+
int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
{
pte_t *table;
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v11 02/10] powerpc/8xx: add vmap_pte_set_huge()
2026-11-03 10:16 [PATCH v11 00/10] mm/vmalloc: Speed up ioremap, vmalloc and vmap with contiguous memory Wen Jiang
2026-11-03 10:16 ` [PATCH v11 01/10] arm64/mm: add vmap_pte_set_huge() and vmap_pte_clear_huge() Wen Jiang
@ 2026-11-03 10:16 ` Wen Jiang
2026-11-03 10:16 ` [PATCH v11 03/10] mm/vmalloc: use vmap_pte_set_huge()/vmap_pte_clear_huge() for PTE-level block mappings Wen Jiang
` (7 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Wen Jiang @ 2026-11-03 10:16 UTC (permalink / raw)
To: akpm, catalin.marinas, linux-mm, urezki, will
Cc: Xueyuan.chen21, ajd, anshuman.khandual, baohua, chleroy, david,
dev.jain, jiangwen6, leo.yan, linux-arm-kernel, linux-kernel,
linuxppc-dev, maddy, mpe, npiggin, rppt, ryan.roberts
From: Wen Jiang <jiangwen6@xiaomi.com>
vmap installs PTE-level block mappings (SZ_16K/SZ_512K on powerpc/8xx)
by reusing set_huge_pte_at(), a HugeTLB helper gated by
CONFIG_HUGETLB_PAGE. This makes the feature silently unavailable on
CONFIG_HUGETLB_PAGE=n kernels and couples mm/vmalloc.c to HugeTLB
internals it does not otherwise need.
Add vmap_pte_set_huge() to powerpc/8xx, next to the existing
pmd_clear_huge()/pud_clear_huge() in mm/nohash/8xx.c. It builds the huge
PTE and writes it into the backing cells directly, without going through
set_huge_pte_at(), so it does not depend on CONFIG_HUGETLB_PAGE.
No vmap_pte_clear_huge() is needed: 8xx does not implement
arch_vmap_pte_range_unmap_size(), so the vmap unmap path never takes the
PTE-clearing branch.
In practice PPC_8xx selects HUGETLBFS unconditionally, so
CONFIG_HUGETLB_PAGE=n does not occur there today; this is a decoupling
cleanup rather than a new configuration.
There is no caller yet: mm/vmalloc.c is converted later in this series,
once the generic fallbacks are in place.
Signed-off-by: Wen Jiang <jiangwen6@xiaomi.com>
Reviewed-by: Christophe Leroy (CS GROUP) <chleroy@kernel.org>
---
arch/powerpc/include/asm/nohash/32/pte-8xx.h | 4 +++
arch/powerpc/mm/nohash/8xx.c | 29 ++++++++++++++++++++
2 files changed, 33 insertions(+)
diff --git a/arch/powerpc/include/asm/nohash/32/pte-8xx.h b/arch/powerpc/include/asm/nohash/32/pte-8xx.h
index e2ea8ba9f8ca..6ea854d67a2e 100644
--- a/arch/powerpc/include/asm/nohash/32/pte-8xx.h
+++ b/arch/powerpc/include/asm/nohash/32/pte-8xx.h
@@ -235,6 +235,10 @@ static inline pte_t ptep_get(pte_t *ptep)
}
#endif /* CONFIG_PPC_16K_PAGES */
+#define __HAVE_ARCH_VMAP_PTE_SET_HUGE
+void vmap_pte_set_huge(pte_t *ptep, unsigned long addr, phys_addr_t phys,
+ pgprot_t prot, unsigned long size);
+
#endif
#endif /* __KERNEL__ */
diff --git a/arch/powerpc/mm/nohash/8xx.c b/arch/powerpc/mm/nohash/8xx.c
index a9d3f4729ead..54254182700c 100644
--- a/arch/powerpc/mm/nohash/8xx.c
+++ b/arch/powerpc/mm/nohash/8xx.c
@@ -220,3 +220,32 @@ int pmd_clear_huge(pmd_t *pmd)
{
return 0;
}
+
+/*
+ * vmap PTE-level block mapping. Sets a present kernel mapping directly,
+ * so unlike set_huge_pte_at() it needs neither CONFIG_HUGETLB_PAGE nor
+ * set_pte_filter() (a no-op for non-exec kernel mappings).
+ */
+void vmap_pte_set_huge(pte_t *ptep, unsigned long addr, phys_addr_t phys,
+ pgprot_t prot, unsigned long size)
+{
+ pmd_t *pmdp = pmd_off(&init_mm, addr);
+ pte_t pte = pfn_pte(PHYS_PFN(phys), prot);
+ pte_basic_t val;
+ pte_basic_t *entry = (pte_basic_t *)ptep;
+ int num, i;
+
+ pte = arch_make_huge_pte(pte, ilog2(size), 0);
+ val = pte_val(pte);
+
+ /*
+ * Make sure hardware valid bit is not set. We don't do
+ * tlb flush for this update.
+ */
+ VM_WARN_ON(pte_hw_valid(*ptep) && !pte_protnone(*ptep));
+
+ num = number_of_cells_per_pte(pmdp, val, 1);
+
+ for (i = 0; i < num; i++, entry++, val += SZ_4K)
+ *entry = val;
+}
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v11 03/10] mm/vmalloc: use vmap_pte_set_huge()/vmap_pte_clear_huge() for PTE-level block mappings
2026-11-03 10:16 [PATCH v11 00/10] mm/vmalloc: Speed up ioremap, vmalloc and vmap with contiguous memory Wen Jiang
2026-11-03 10:16 ` [PATCH v11 01/10] arm64/mm: add vmap_pte_set_huge() and vmap_pte_clear_huge() Wen Jiang
2026-11-03 10:16 ` [PATCH v11 02/10] powerpc/8xx: add vmap_pte_set_huge() Wen Jiang
@ 2026-11-03 10:16 ` Wen Jiang
2026-11-03 10:16 ` [PATCH v11 04/10] arm64/hugetlb: drop the init_mm special case in clear_flush() Wen Jiang
` (6 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Wen Jiang @ 2026-11-03 10:16 UTC (permalink / raw)
To: akpm, catalin.marinas, linux-mm, urezki, will
Cc: Xueyuan.chen21, ajd, anshuman.khandual, baohua, chleroy, david,
dev.jain, jiangwen6, leo.yan, linux-arm-kernel, linux-kernel,
linuxppc-dev, maddy, mpe, npiggin, rppt, ryan.roberts
From: Wen Jiang <jiangwen6@xiaomi.com>
vmap installs PTE-level block mappings by reusing set_huge_pte_at() and
huge_ptep_get_and_clear() under #ifdef CONFIG_HUGETLB_PAGE. This makes
the feature silently unavailable on CONFIG_HUGETLB_PAGE=n kernels and
couples mm/vmalloc.c to HugeTLB internals it does not otherwise need.
Now that arm64 and powerpc/8xx provide vmap_pte_set_huge() and
vmap_pte_clear_huge(), add the generic fallbacks next to the existing
pmd/pud_set_huge() family and convert vmap_pte_range() and
vunmap_pte_range() to the new helpers. The CONFIG_HUGETLB_PAGE guards
around the block-mapping paths are dropped, so PTE-level block mappings
now also work on CONFIG_HUGETLB_PAGE=n kernels, and mm/vmalloc.c no
longer includes <linux/hugetlb.h>.
The fallbacks exist only to keep the build working on architectures
without PTE-level block mapping support. They are unreachable there:
the callers only run when arch_vmap_pte_range_map_size() or
arch_vmap_pte_range_unmap_size() return a size other than PAGE_SIZE,
which requires an arch implementation. BUILD_BUG() makes that
explicit rather than silently doing nothing.
Signed-off-by: Wen Jiang <jiangwen6@xiaomi.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
---
include/linux/pgtable.h | 29 +++++++++++++++++++++++++++++
mm/vmalloc.c | 19 ++++++-------------
2 files changed, 35 insertions(+), 13 deletions(-)
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 780fe849ff8b..78689941b637 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -2204,6 +2204,35 @@ static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
}
#endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */
+/*
+ * PTE-level block mappings for vmap.
+ *
+ * vmap_pte_set_huge() only has to be implemented by architectures whose
+ * arch_vmap_pte_range_map_size() can return a size other than PAGE_SIZE.
+ */
+#ifndef __HAVE_ARCH_VMAP_PTE_SET_HUGE
+static inline void vmap_pte_set_huge(pte_t *ptep, unsigned long addr,
+ phys_addr_t phys, pgprot_t prot,
+ unsigned long size)
+{
+ BUILD_BUG();
+}
+#endif
+
+/*
+ * Likewise, vmap_pte_clear_huge() only has to be implemented by architectures
+ * whose arch_vmap_pte_range_unmap_size() can return a size other than
+ * PAGE_SIZE.
+ */
+#ifndef __HAVE_ARCH_VMAP_PTE_CLEAR_HUGE
+static inline pte_t vmap_pte_clear_huge(pte_t *ptep, unsigned long addr,
+ unsigned long size)
+{
+ BUILD_BUG();
+ return __pte(0);
+}
+#endif
+
#ifndef __HAVE_ARCH_FLUSH_PMD_TLB_RANGE
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
/*
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index f7aa4686fe5b..df37c8490ae7 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -38,7 +38,6 @@
#include <linux/rbtree_augmented.h>
#include <linux/overflow.h>
#include <linux/pgtable.h>
-#include <linux/hugetlb.h>
#include <linux/sched/mm.h>
#include <asm/tlbflush.h>
#include <asm/shmparam.h>
@@ -100,7 +99,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
pte_t *pte;
u64 pfn;
struct page *page;
- unsigned long size = PAGE_SIZE;
+ unsigned long size;
if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
return -EINVAL;
@@ -121,17 +120,12 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
BUG();
}
-#ifdef CONFIG_HUGETLB_PAGE
size = arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift);
if (size != PAGE_SIZE) {
- pte_t entry = pfn_pte(pfn, prot);
-
- entry = arch_make_huge_pte(entry, ilog2(size), 0);
- set_huge_pte_at(&init_mm, addr, pte, entry, size);
+ vmap_pte_set_huge(pte, addr, PFN_PHYS(pfn), prot, size);
pfn += PFN_DOWN(size);
continue;
}
-#endif
set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot));
pfn++;
} while (pte += PFN_DOWN(size), addr += size, addr != end);
@@ -391,25 +385,24 @@ static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
{
pte_t *pte;
pte_t ptent;
- unsigned long size = PAGE_SIZE;
+ unsigned long size;
pte = pte_offset_kernel(pmd, addr);
lazy_mmu_mode_enable();
do {
-#ifdef CONFIG_HUGETLB_PAGE
size = arch_vmap_pte_range_unmap_size(addr, pte);
if (size != PAGE_SIZE) {
if (WARN_ON(!IS_ALIGNED(addr, size))) {
addr = ALIGN_DOWN(addr, size);
pte = PTR_ALIGN_DOWN(pte, sizeof(*pte) * (size >> PAGE_SHIFT));
}
- ptent = huge_ptep_get_and_clear(&init_mm, addr, pte, size);
+ ptent = vmap_pte_clear_huge(pte, addr, size);
if (WARN_ON(end - addr < size))
size = end - addr;
- } else
-#endif
+ } else {
ptent = ptep_get_and_clear(&init_mm, addr, pte);
+ }
WARN_ON(!pte_none(ptent) && !pte_present(ptent));
} while (pte += (size >> PAGE_SHIFT), addr += size, addr != end);
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v11 04/10] arm64/hugetlb: drop the init_mm special case in clear_flush()
2026-11-03 10:16 [PATCH v11 00/10] mm/vmalloc: Speed up ioremap, vmalloc and vmap with contiguous memory Wen Jiang
` (2 preceding siblings ...)
2026-11-03 10:16 ` [PATCH v11 03/10] mm/vmalloc: use vmap_pte_set_huge()/vmap_pte_clear_huge() for PTE-level block mappings Wen Jiang
@ 2026-11-03 10:16 ` Wen Jiang
2026-11-03 10:16 ` [PATCH v11 05/10] arm64/vmalloc: allow arch_vmap_pte_range_map_size() to batch multiple CONT_PTE Wen Jiang
` (5 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Wen Jiang @ 2026-11-03 10:16 UTC (permalink / raw)
To: akpm, catalin.marinas, linux-mm, urezki, will
Cc: Xueyuan.chen21, ajd, anshuman.khandual, baohua, chleroy, david,
dev.jain, jiangwen6, leo.yan, linux-arm-kernel, linux-kernel,
linuxppc-dev, maddy, mpe, npiggin, rppt, ryan.roberts
From: Wen Jiang <jiangwen6@xiaomi.com>
clear_flush() calls flush_tlb_kernel_range() when mm is &init_mm. This
was added by commit 06fc959fcff7 ("arm64/mm: Support huge pte-mapped
pages in vmap") because vmap called set_huge_pte_at(&init_mm, ...).
Now that vmap uses vmap_pte_set_huge()/vmap_pte_clear_huge(),
mm/vmalloc.c is no longer a caller of set_huge_pte_at() on arm64. The
only remaining callers passing &init_mm are in arch/powerpc, which uses
its own implementation. Remove the dead branch.
No functional change.
Signed-off-by: Wen Jiang <jiangwen6@xiaomi.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Reviewed-by: Dev Jain <dev.jain@arm.com>
---
arch/arm64/mm/hugetlbpage.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
index 8e799c1fe0aa..ce247b06fe03 100644
--- a/arch/arm64/mm/hugetlbpage.c
+++ b/arch/arm64/mm/hugetlbpage.c
@@ -206,10 +206,7 @@ static void clear_flush(struct mm_struct *mm,
for (i = 0; i < ncontig; i++, addr += pgsize, ptep++)
__ptep_get_and_clear_anysz(mm, addr, ptep, pgsize);
- if (mm == &init_mm)
- flush_tlb_kernel_range(saddr, addr);
- else
- __flush_hugetlb_tlb_range(&vma, saddr, addr, pgsize, TLBF_NOWALKCACHE);
+ __flush_hugetlb_tlb_range(&vma, saddr, addr, pgsize, TLBF_NOWALKCACHE);
}
void set_huge_pte_at(struct mm_struct *mm, unsigned long addr,
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v11 05/10] arm64/vmalloc: allow arch_vmap_pte_range_map_size() to batch multiple CONT_PTE
2026-11-03 10:16 [PATCH v11 00/10] mm/vmalloc: Speed up ioremap, vmalloc and vmap with contiguous memory Wen Jiang
` (3 preceding siblings ...)
2026-11-03 10:16 ` [PATCH v11 04/10] arm64/hugetlb: drop the init_mm special case in clear_flush() Wen Jiang
@ 2026-11-03 10:16 ` Wen Jiang
2026-11-04 10:18 ` sashiko-bot
2026-11-03 10:16 ` [PATCH v11 06/10] mm/vmalloc: extract vmap_set_ptes() to consolidate PTE mapping logic Wen Jiang
` (4 subsequent siblings)
9 siblings, 1 reply; 14+ messages in thread
From: Wen Jiang @ 2026-11-03 10:16 UTC (permalink / raw)
To: akpm, catalin.marinas, linux-mm, urezki, will
Cc: Xueyuan.chen21, ajd, anshuman.khandual, baohua, chleroy, david,
dev.jain, jiangwen6, leo.yan, linux-arm-kernel, linux-kernel,
linuxppc-dev, maddy, mpe, npiggin, rppt, ryan.roberts,
Xueyuan Chen
From: "Barry Song (Xiaomi)" <baohua@kernel.org>
Allow arch_vmap_pte_range_map_size to batch across multiple CONT_PTE
blocks, reducing both PTE setup and TLB flush iterations.
Previously the function returned a single CONT_PTE_SIZE for an aligned
range, so the caller mapped one CONT_PTE block per iteration. Now, for a
CONT_PTE_SIZE-aligned range, it returns the largest CONT_PTE-aligned size
that fits within both the remaining range and the caller's max_page_shift,
so a single vmap_pte_set_huge() call installs several CONT_PTE blocks at
once.
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
Signed-off-by: Wen Jiang <jiangwen6@xiaomi.com>
Tested-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Tested-by: Leo Yan <leo.yan@arm.com>
Reviewed-by: Dev Jain <dev.jain@arm.com>
---
arch/arm64/include/asm/vmalloc.h | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmalloc.h
index 4ec1acd3c1b3..c29704a19e27 100644
--- a/arch/arm64/include/asm/vmalloc.h
+++ b/arch/arm64/include/asm/vmalloc.h
@@ -23,10 +23,12 @@ static inline unsigned long arch_vmap_pte_range_map_size(unsigned long addr,
unsigned long end, u64 pfn,
unsigned int max_page_shift)
{
+ unsigned long size;
+
/*
- * If the block is at least CONT_PTE_SIZE in size, and is naturally
- * aligned in both virtual and physical space, then we can pte-map the
- * block using the PTE_CONT bit for more efficient use of the TLB.
+ * If the range is at least CONT_PTE_SIZE and naturally aligned in both
+ * virtual and physical space, return the largest CONT_PTE-aligned size
+ * that fits, for more efficient use of the TLB.
*/
if (max_page_shift < CONT_PTE_SHIFT)
return PAGE_SIZE;
@@ -40,7 +42,9 @@ static inline unsigned long arch_vmap_pte_range_map_size(unsigned long addr,
if (!IS_ALIGNED(PFN_PHYS(pfn), CONT_PTE_SIZE))
return PAGE_SIZE;
- return CONT_PTE_SIZE;
+ size = min(end - addr, 1UL << max_page_shift);
+ size = round_down(size, CONT_PTE_SIZE);
+ return size;
}
#define arch_vmap_pte_range_unmap_size arch_vmap_pte_range_unmap_size
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v11 05/10] arm64/vmalloc: allow arch_vmap_pte_range_map_size() to batch multiple CONT_PTE
2026-11-03 10:16 ` [PATCH v11 05/10] arm64/vmalloc: allow arch_vmap_pte_range_map_size() to batch multiple CONT_PTE Wen Jiang
@ 2026-11-04 10:18 ` sashiko-bot
0 siblings, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-11-04 10:18 UTC (permalink / raw)
To: Wen Jiang, akpm, catalin.marinas, linux-mm, urezki, will
Cc: Xueyuan Chen, ajd, anshuman.khandual, baohua, chleroy, david,
dev.jain, jiangwen6, leo.yan, maddy, mpe, npiggin, rppt,
ryan.roberts
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] arm64/vmalloc: physical block overrun in arch_vmap_pte_range_map_size()
--
commit 21610101d177ce2dfc10c0e91b5ac55eb277a5f9
Author: Barry Song (Xiaomi) <baohua@kernel.org>
arm64/vmalloc: allow arch_vmap_pte_range_map_size() to batch multiple CONT_PTE
This commit updates arch_vmap_pte_range_map_size() to return the largest
CONT_PTE_SIZE-aligned size that fits within the remaining range and the
caller's max_page_shift, allowing vmap_pte_set_huge() to install multiple
CONT_PTE blocks in a single call.
> diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmalloc.h
> index 4ec1acd3c1b34..c29704a19e276 100644
> --- a/arch/arm64/include/asm/vmalloc.h
> +++ b/arch/arm64/include/asm/vmalloc.h
> @@ -23,10 +23,12 @@ static inline unsigned long arch_vmap_pte_range_map_size(unsigned long addr,
> unsigned long end, u64 pfn,
> unsigned int max_page_shift)
> {
> + unsigned long size;
> +
> /*
> - * If the block is at least CONT_PTE_SIZE in size, and is naturally
> - * aligned in both virtual and physical space, then we can pte-map the
> - * block using the PTE_CONT bit for more efficient use of the TLB.
> + * If the range is at least CONT_PTE_SIZE and naturally aligned in both
> + * virtual and physical space, return the largest CONT_PTE-aligned size
> + * that fits, for more efficient use of the TLB.
> */
> if (max_page_shift < CONT_PTE_SHIFT)
> return PAGE_SIZE;
> @@ -40,7 +42,9 @@ static inline unsigned long arch_vmap_pte_range_map_size(unsigned long addr,
> if (!IS_ALIGNED(PFN_PHYS(pfn), CONT_PTE_SIZE))
> return PAGE_SIZE;
>
> - return CONT_PTE_SIZE;
> + size = min(end - addr, 1UL << max_page_shift);
> + size = round_down(size, CONT_PTE_SIZE);
> + return size;
[Severity: Critical]
As reported by David Carlier on v7 [1], can arch_vmap_pte_range_map_size()
overrun the current physically contiguous block when a PMD boundary splits a
coalesced batch?
In the batched vmap() path introduced later in this series:
vmap()
-> vmap_pages_range()
-> vmap_pages_range_noflush()
-> vmap_pages_range_batched()
-> vmap_pages_range_noflush_walk()
vmap_pages_range_batched() coalesces consecutive blocks that share the same
mapping shift (where shift > CONT_PTE_SHIFT) into a single walk over
[batch_start, batch_end). Within that coalesced range, pages[] is only
physically contiguous within each naturally aligned (1UL << max_page_shift)
block; separate blocks in the same batch are not physically contiguous with
each other.
If [batch_start, batch_end) is only aligned to CONT_PTE_SIZE and crosses a
PMD boundary mid-block, vmap_pages_pmd_range() splits the walk at
pmd_addr_end(addr, end). The first PMD maps the initial CONT_PTE_SIZE-aligned
portion of the block, advancing *nr into the middle of that physical block.
At the start of the next PMD, addr and PFN_PHYS(pfn) are still aligned to
CONT_PTE_SIZE, while end - addr includes the subsequent block(s) in the
batch. Because arch_vmap_pte_range_map_size() only checks
IS_ALIGNED(PFN_PHYS(pfn), CONT_PTE_SIZE) and does not bound size to the
remaining bytes in the current (1UL << max_page_shift) physical block, it can
return up to (1UL << max_page_shift) starting from a mid-block PFN.
Won't vmap_pte_set_huge() then map physical pages past the end of that block
while vmap_pages_pte_range() advances *nr past the start of the next block in
pages[], leaving the tail of pages[] unmapped?
[1] https://lore.kernel.org/r/20260721181729.305137-1-devnexen@gmail.com
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261103101658.766905-1-jiangwenxiaomi@gmail.com?part=5
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v11 06/10] mm/vmalloc: extract vmap_set_ptes() to consolidate PTE mapping logic
2026-11-03 10:16 [PATCH v11 00/10] mm/vmalloc: Speed up ioremap, vmalloc and vmap with contiguous memory Wen Jiang
` (4 preceding siblings ...)
2026-11-03 10:16 ` [PATCH v11 05/10] arm64/vmalloc: allow arch_vmap_pte_range_map_size() to batch multiple CONT_PTE Wen Jiang
@ 2026-11-03 10:16 ` Wen Jiang
2026-11-03 10:16 ` [PATCH v11 07/10] mm/vmalloc: extend page table walk to support larger page_shift sizes and eliminate page table rewalk Wen Jiang
` (3 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Wen Jiang @ 2026-11-03 10:16 UTC (permalink / raw)
To: akpm, catalin.marinas, linux-mm, urezki, will
Cc: Xueyuan.chen21, ajd, anshuman.khandual, baohua, chleroy, david,
dev.jain, jiangwen6, leo.yan, linux-arm-kernel, linux-kernel,
linuxppc-dev, maddy, mpe, npiggin, rppt, ryan.roberts,
Xueyuan Chen
From: Wen Jiang <jiangwen6@xiaomi.com>
Extract the common PTE mapping logic from vmap_pte_range() into a
shared helper vmap_set_ptes(). This handles both CONT_PTE and regular
PTE mappings in a single function, preparing for the next patch which
will extend vmap_pages_pte_range() to also use this helper.
No functional change.
Signed-off-by: Wen Jiang <jiangwen6@xiaomi.com>
Tested-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Tested-by: Leo Yan <leo.yan@arm.com>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Reviewed-by: Dev Jain <dev.jain@arm.com>
---
mm/vmalloc.c | 36 +++++++++++++++++++++++++++---------
1 file changed, 27 insertions(+), 9 deletions(-)
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index df37c8490ae7..e3ea9581e68d 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -92,6 +92,29 @@ struct vfree_deferred {
static DEFINE_PER_CPU(struct vfree_deferred, vfree_deferred);
/*** Page table manipulation functions ***/
+
+/*
+ * Try contiguous mappings at the PTE level for arches which support them, and if
+ * requested by the caller. Fall back to PAGE_SIZE mappings otherwise.
+ *
+ * Return: mapping size.
+ */
+static __always_inline unsigned long vmap_set_ptes(pte_t *ptep,
+ unsigned long addr, unsigned long end, u64 pfn,
+ pgprot_t prot, unsigned int max_page_shift)
+{
+ unsigned long size;
+
+ size = arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift);
+ if (size != PAGE_SIZE) {
+ vmap_pte_set_huge(ptep, addr, PFN_PHYS(pfn), prot, size);
+ return size;
+ }
+
+ set_pte_at(&init_mm, addr, ptep, pfn_pte(pfn, prot));
+ return PAGE_SIZE;
+}
+
static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
phys_addr_t phys_addr, pgprot_t prot,
unsigned int max_page_shift, pgtbl_mod_mask *mask)
@@ -100,6 +123,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
u64 pfn;
struct page *page;
unsigned long size;
+ unsigned int nr_pages;
if (WARN_ON_ONCE(!PAGE_ALIGNED(end - addr)))
return -EINVAL;
@@ -120,15 +144,9 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
BUG();
}
- size = arch_vmap_pte_range_map_size(addr, end, pfn, max_page_shift);
- if (size != PAGE_SIZE) {
- vmap_pte_set_huge(pte, addr, PFN_PHYS(pfn), prot, size);
- pfn += PFN_DOWN(size);
- continue;
- }
- set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot));
- pfn++;
- } while (pte += PFN_DOWN(size), addr += size, addr != end);
+ size = vmap_set_ptes(pte, addr, end, pfn, prot, max_page_shift);
+ nr_pages = PFN_DOWN(size);
+ } while (pte += nr_pages, pfn += nr_pages, addr += size, addr != end);
lazy_mmu_mode_disable();
*mask |= PGTBL_PTE_MODIFIED;
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v11 07/10] mm/vmalloc: extend page table walk to support larger page_shift sizes and eliminate page table rewalk
2026-11-03 10:16 [PATCH v11 00/10] mm/vmalloc: Speed up ioremap, vmalloc and vmap with contiguous memory Wen Jiang
` (5 preceding siblings ...)
2026-11-03 10:16 ` [PATCH v11 06/10] mm/vmalloc: extract vmap_set_ptes() to consolidate PTE mapping logic Wen Jiang
@ 2026-11-03 10:16 ` Wen Jiang
2026-11-03 10:16 ` [PATCH v11 08/10] mm/vmalloc: extract vm_shift() to consolidate mapping shift selection Wen Jiang
` (2 subsequent siblings)
9 siblings, 0 replies; 14+ messages in thread
From: Wen Jiang @ 2026-11-03 10:16 UTC (permalink / raw)
To: akpm, catalin.marinas, linux-mm, urezki, will
Cc: Xueyuan.chen21, ajd, anshuman.khandual, baohua, chleroy, david,
dev.jain, jiangwen6, leo.yan, linux-arm-kernel, linux-kernel,
linuxppc-dev, maddy, mpe, npiggin, rppt, ryan.roberts,
Xueyuan Chen
From: "Barry Song (Xiaomi)" <baohua@kernel.org>
vmap_pages_range_noflush_walk() (formerly vmap_small_pages_range_noflush())
provides a clean interface by taking struct page **pages and mapping them
via direct PTE iteration. This avoids the page table rewalk seen when
using vmap_range_noflush() for page_shift values other than PAGE_SHIFT.
Extend it to support larger page_shift values, and add PMD- and
contiguous-PTE mappings as well.
Rename it to vmap_pages_range_noflush_walk() since it now handles
more than just small pages.
For vmalloc() allocations with VM_ALLOW_HUGE_VMAP, we no longer need to
iterate over pages one by one via vmap_range_noflush(), which would
otherwise lead to page table rewalk. The code is now unified with the
PAGE_SHIFT case by simply calling vmap_pages_range_noflush_walk().
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
Signed-off-by: Wen Jiang <jiangwen6@xiaomi.com>
Tested-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Tested-by: Leo Yan <leo.yan@arm.com>
Reviewed-by: Dev Jain <dev.jain@arm.com>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
---
mm/vmalloc.c | 84 +++++++++++++++++++++++++++++++---------------------
1 file changed, 50 insertions(+), 34 deletions(-)
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index e3ea9581e68d..61f2b6fbbcd9 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -558,8 +558,10 @@ void vunmap_range(unsigned long addr, unsigned long end)
static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr,
unsigned long end, pgprot_t prot, struct page **pages, int *nr,
- pgtbl_mod_mask *mask)
+ pgtbl_mod_mask *mask, unsigned int shift)
{
+ unsigned long pfn, size;
+ unsigned int nr_pages;
int err = 0;
pte_t *pte;
@@ -590,9 +592,10 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr,
break;
}
- set_pte_at(&init_mm, addr, pte, mk_pte(page, prot));
- (*nr)++;
- } while (pte++, addr += PAGE_SIZE, addr != end);
+ pfn = page_to_pfn(page);
+ size = vmap_set_ptes(pte, addr, end, pfn, prot, shift);
+ nr_pages = PFN_DOWN(size);
+ } while (pte += nr_pages, *nr += nr_pages, addr += size, addr != end);
lazy_mmu_mode_disable();
*mask |= PGTBL_PTE_MODIFIED;
@@ -602,60 +605,88 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr,
static int vmap_pages_pmd_range(pud_t *pud, unsigned long addr,
unsigned long end, pgprot_t prot, struct page **pages, int *nr,
- pgtbl_mod_mask *mask)
+ pgtbl_mod_mask *mask, unsigned int shift)
{
pmd_t *pmd;
unsigned long next;
+ int err;
pmd = pmd_alloc_track(&init_mm, pud, addr, mask);
if (!pmd)
return -ENOMEM;
do {
next = pmd_addr_end(addr, end);
- if (vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask))
- return -ENOMEM;
+
+ if (shift >= PMD_SHIFT) {
+ struct page *page = pages[*nr];
+ phys_addr_t phys_addr;
+
+ if (WARN_ON(!page))
+ return -ENOMEM;
+ if (WARN_ON(!pfn_valid(page_to_pfn(page))))
+ return -EINVAL;
+
+ phys_addr = page_to_phys(page);
+
+ if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, shift)) {
+ *mask |= PGTBL_PMD_MODIFIED;
+ *nr += 1U << (PMD_SHIFT - PAGE_SHIFT);
+ continue;
+ }
+ }
+ err = vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask, shift);
+ if (err)
+ return err;
} while (pmd++, addr = next, addr != end);
return 0;
}
static int vmap_pages_pud_range(p4d_t *p4d, unsigned long addr,
unsigned long end, pgprot_t prot, struct page **pages, int *nr,
- pgtbl_mod_mask *mask)
+ pgtbl_mod_mask *mask, unsigned int shift)
{
pud_t *pud;
unsigned long next;
+ int err;
pud = pud_alloc_track(&init_mm, p4d, addr, mask);
if (!pud)
return -ENOMEM;
do {
next = pud_addr_end(addr, end);
- if (vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask))
- return -ENOMEM;
+ err = vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask, shift);
+ if (err)
+ return err;
} while (pud++, addr = next, addr != end);
return 0;
}
static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr,
unsigned long end, pgprot_t prot, struct page **pages, int *nr,
- pgtbl_mod_mask *mask)
+ pgtbl_mod_mask *mask, unsigned int shift)
{
p4d_t *p4d;
unsigned long next;
+ int err;
p4d = p4d_alloc_track(&init_mm, pgd, addr, mask);
if (!p4d)
return -ENOMEM;
do {
next = p4d_addr_end(addr, end);
- if (vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask))
- return -ENOMEM;
+ err = vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask, shift);
+ if (err)
+ return err;
} while (p4d++, addr = next, addr != end);
return 0;
}
-static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end,
- pgprot_t prot, struct page **pages)
+/*
+ * It can take an array of pages which are not all contiguous, but it
+ * may have contiguous chunks, as hinted by @shift.
+ */
+static int vmap_pages_range_noflush_walk(unsigned long addr, unsigned long end,
+ pgprot_t prot, struct page **pages, unsigned int shift)
{
unsigned long start = addr;
pgd_t *pgd;
@@ -670,7 +701,7 @@ static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end,
next = pgd_addr_end(addr, end);
if (pgd_bad(*pgd))
mask |= PGTBL_PGD_MODIFIED;
- err = vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask);
+ err = vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask, shift);
if (err)
break;
} while (pgd++, addr = next, addr != end);
@@ -693,28 +724,13 @@ static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end,
int __vmap_pages_range_noflush(unsigned long addr, unsigned long end,
pgprot_t prot, struct page **pages, unsigned int page_shift)
{
- unsigned int i, nr = (end - addr) >> PAGE_SHIFT;
-
if (WARN_ON_ONCE(page_shift < PAGE_SHIFT))
return -EINVAL;
- if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) ||
- page_shift == PAGE_SHIFT)
- return vmap_small_pages_range_noflush(addr, end, prot, pages);
+ if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC))
+ page_shift = PAGE_SHIFT;
- for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) {
- int err;
-
- err = vmap_range_noflush(addr, addr + (1UL << page_shift),
- page_to_phys(pages[i]), prot,
- page_shift);
- if (err)
- return err;
-
- addr += 1UL << page_shift;
- }
-
- return 0;
+ return vmap_pages_range_noflush_walk(addr, end, prot, pages, page_shift);
}
int vmap_pages_range_noflush(unsigned long addr, unsigned long end,
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v11 08/10] mm/vmalloc: extract vm_shift() to consolidate mapping shift selection
2026-11-03 10:16 [PATCH v11 00/10] mm/vmalloc: Speed up ioremap, vmalloc and vmap with contiguous memory Wen Jiang
` (6 preceding siblings ...)
2026-11-03 10:16 ` [PATCH v11 07/10] mm/vmalloc: extend page table walk to support larger page_shift sizes and eliminate page table rewalk Wen Jiang
@ 2026-11-03 10:16 ` Wen Jiang
2026-11-03 10:16 ` [PATCH v11 09/10] mm/vmalloc: map contiguous pages in batches for vmap() if possible Wen Jiang
2026-11-03 10:16 ` [PATCH v11 10/10] mm/vmalloc: align vm_area so vmap() can batch mappings Wen Jiang
9 siblings, 0 replies; 14+ messages in thread
From: Wen Jiang @ 2026-11-03 10:16 UTC (permalink / raw)
To: akpm, catalin.marinas, linux-mm, urezki, will
Cc: Xueyuan.chen21, ajd, anshuman.khandual, baohua, chleroy, david,
dev.jain, jiangwen6, leo.yan, linux-arm-kernel, linux-kernel,
linuxppc-dev, maddy, mpe, npiggin, rppt, ryan.roberts,
Xueyuan Chen
From: Wen Jiang <jiangwen6@xiaomi.com>
Extract the vmalloc mapping shift selection from
__vmalloc_node_range_noprof() into vm_shift(), preparing for reuse by the
vmap batching path.
No functional change.
Suggested-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Wen Jiang <jiangwen6@xiaomi.com>
Tested-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Tested-by: Leo Yan <leo.yan@arm.com>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Reviewed-by: Dev Jain <dev.jain@arm.com>
---
mm/vmalloc.c | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 61f2b6fbbcd9..ee2c1ec4ff1f 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -3605,6 +3605,14 @@ void vunmap(const void *addr)
}
EXPORT_SYMBOL(vunmap);
+static inline unsigned int vm_shift(pgprot_t prot, unsigned long size)
+{
+ if (arch_vmap_pmd_supported(prot) && size >= PMD_SIZE)
+ return PMD_SHIFT;
+
+ return arch_vmap_pte_supported_shift(size);
+}
+
/**
* vmap - map an array of pages into virtually contiguous space
* @pages: array of page pointers
@@ -4124,11 +4132,7 @@ void *__vmalloc_node_range_noprof(unsigned long size, unsigned long align,
* supporting them.
*/
- if (arch_vmap_pmd_supported(prot) && size >= PMD_SIZE)
- shift = PMD_SHIFT;
- else
- shift = arch_vmap_pte_supported_shift(size);
-
+ shift = vm_shift(prot, size);
align = max(original_align, 1UL << shift);
}
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v11 09/10] mm/vmalloc: map contiguous pages in batches for vmap() if possible
2026-11-03 10:16 [PATCH v11 00/10] mm/vmalloc: Speed up ioremap, vmalloc and vmap with contiguous memory Wen Jiang
` (7 preceding siblings ...)
2026-11-03 10:16 ` [PATCH v11 08/10] mm/vmalloc: extract vm_shift() to consolidate mapping shift selection Wen Jiang
@ 2026-11-03 10:16 ` Wen Jiang
2026-11-04 10:19 ` sashiko-bot
2026-11-03 10:16 ` [PATCH v11 10/10] mm/vmalloc: align vm_area so vmap() can batch mappings Wen Jiang
9 siblings, 1 reply; 14+ messages in thread
From: Wen Jiang @ 2026-11-03 10:16 UTC (permalink / raw)
To: akpm, catalin.marinas, linux-mm, urezki, will
Cc: Xueyuan.chen21, ajd, anshuman.khandual, baohua, chleroy, david,
dev.jain, jiangwen6, leo.yan, linux-arm-kernel, linux-kernel,
linuxppc-dev, maddy, mpe, npiggin, rppt, ryan.roberts,
Xueyuan Chen
From: "Barry Song (Xiaomi)" <baohua@kernel.org>
In many cases, the pages passed to vmap() may include high-order
pages. For example, the systemheap often allocates pages in descending
order: order 8, then 4, then 0. Currently, vmap() iterates over every
page individually—even pages inside a high-order block are handled
one by one.
This patch detects physically contiguous pages (regardless of whether
they are compound or non-compound) by scanning with
num_pages_contiguous(), and maps them as a single contiguous block
whenever possible. The mapping order is determined by taking the
minimum of the contiguous page count and the pfn alignment, allowing
graceful degradation when pfn alignment is less than the contiguous
range.
Pages with the same page_shift are coalesced and mapped via
vmap_pages_range_noflush_walk() to avoid page table rewalk.
As users typically allocate memory in descending orders (e.g.
8 → 4 → 0), once an order-0 page is encountered, we stop scanning
for contiguous pages since subsequent pages are likely order-0 as well.
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
Co-developed-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Wen Jiang <jiangwen6@xiaomi.com>
Tested-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Tested-by: Leo Yan <leo.yan@arm.com>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
---
Sashiko review: flagged a potential NULL pointer dereference -- with
HUGE_VMAP enabled, a NULL entry in the pages array reaches
page_to_pfn()/num_pages_contiguous() in get_vmap_mapping_order() before
the WARN_ON(!page) safeguard in vmap_pages_pte_range() runs.
This is not a bug: vmap()'s contract requires the caller to pass an
array of valid pages. The WARN_ON(!page) is a defensive sanity check,
not part of the API contract -- passing NULL pages has always been a
caller bug. get_vmap_mapping_order() runs on the same pages array that
vmap() received, so a caller passing NULL pages is already broken
regardless of whether we batch.
mm/vmalloc.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 80 insertions(+), 2 deletions(-)
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index ee2c1ec4ff1f..42b3addd20f9 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -3613,6 +3613,85 @@ static inline unsigned int vm_shift(pgprot_t prot, unsigned long size)
return arch_vmap_pte_supported_shift(size);
}
+static inline int get_vmap_mapping_order(struct page **pages,
+ pgprot_t prot, unsigned int nr_pages)
+{
+ unsigned long pfn;
+ unsigned int nr_contig;
+ int order;
+
+ if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP))
+ return 0;
+
+ /* Limit nr_pages by pfn alignment */
+ pfn = page_to_pfn(*pages);
+ if (pfn > 0)
+ nr_pages = min_t(size_t, nr_pages, 1UL << __ffs(pfn));
+
+ nr_contig = num_pages_contiguous(pages, nr_pages);
+ if (nr_contig < 2)
+ return 0;
+
+ order = ilog2(nr_contig);
+
+ if (vm_shift(prot, PAGE_SIZE << order) == PAGE_SHIFT)
+ return 0;
+
+ return order;
+}
+
+static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
+ pgprot_t prot, struct page **pages)
+{
+ const unsigned int nr_pages = (end - addr) >> PAGE_SHIFT;
+ unsigned int prev_shift = 0, batch_idx = 0;
+ unsigned long batch_start = addr, batch_end = addr;
+ int err;
+
+ err = kmsan_vmap_pages_range_noflush(addr, end, prot, pages,
+ PAGE_SHIFT, GFP_KERNEL);
+
+ if (err)
+ goto out;
+
+ for (unsigned int i = 0; i < nr_pages; ) {
+ unsigned int shift = PAGE_SHIFT +
+ get_vmap_mapping_order(pages + i, prot, nr_pages - i);
+
+ if (!i)
+ prev_shift = shift;
+
+ if (shift != prev_shift) {
+ err = vmap_pages_range_noflush_walk(batch_start, batch_end,
+ prot, pages + batch_idx, prev_shift);
+ if (err)
+ goto out;
+ prev_shift = shift;
+ batch_start = batch_end;
+ batch_idx = i;
+ }
+
+ /*
+ * Once we fail to batch pages, we expect to fail batching
+ * for all remaining pages, so just give up.
+ */
+ if (shift == PAGE_SHIFT)
+ break;
+
+ batch_end += 1UL << shift;
+ i += 1U << (shift - PAGE_SHIFT);
+ }
+
+ /* Remaining */
+ if (batch_start < end)
+ err = vmap_pages_range_noflush_walk(batch_start, end, prot,
+ pages + batch_idx, prev_shift);
+
+out:
+ flush_cache_vmap(addr, end);
+ return err;
+}
+
/**
* vmap - map an array of pages into virtually contiguous space
* @pages: array of page pointers
@@ -3656,8 +3735,7 @@ void *vmap(struct page **pages, unsigned int count,
return NULL;
addr = (unsigned long)area->addr;
- if (vmap_pages_range(addr, addr + size, pgprot_nx(prot),
- pages, PAGE_SHIFT) < 0) {
+ if (vmap_pages_range_batched(addr, addr + size, pgprot_nx(prot), pages) < 0) {
vunmap(area->addr);
return NULL;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v11 09/10] mm/vmalloc: map contiguous pages in batches for vmap() if possible
2026-11-03 10:16 ` [PATCH v11 09/10] mm/vmalloc: map contiguous pages in batches for vmap() if possible Wen Jiang
@ 2026-11-04 10:19 ` sashiko-bot
0 siblings, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-11-04 10:19 UTC (permalink / raw)
To: Wen Jiang, akpm, catalin.marinas, linux-mm, urezki, will
Cc: Xueyuan Chen, ajd, anshuman.khandual, baohua, chleroy, david,
dev.jain, jiangwen6, leo.yan, maddy, mpe, npiggin, rppt,
ryan.roberts
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] mm: physical memory overrun in vmap_pages_range_batched() across PMDs
- [High] mm/vmalloc: missing VM_UNINITIALIZED in vmap() during huge PMD promotion
--
commit 1baf52bf8f22bab279e2ffbca6ca7296ae8e5c51
Author: Barry Song (Xiaomi) <baohua@kernel.org>
mm/vmalloc: map contiguous pages in batches for vmap() if possible
This commit optimizes vmap() by scanning the input page array with
num_pages_contiguous() to detect physically contiguous page ranges and
coalescing ranges with the same mapping order into batched calls to
vmap_pages_range_noflush_walk().
> diff --git a/mm/vmalloc.c b/mm/vmalloc.c
> index ee2c1ec4ff1f3..42b3addd20f98 100644
> --- a/mm/vmalloc.c
> +++ b/mm/vmalloc.c
> @@ -3613,6 +3613,85 @@ static inline unsigned int vm_shift(pgprot_t prot, unsigned long size)
> return arch_vmap_pte_supported_shift(size);
> }
>
> +static inline int get_vmap_mapping_order(struct page **pages,
> + pgprot_t prot, unsigned int nr_pages)
> +{
> + unsigned long pfn;
> + unsigned int nr_contig;
> + int order;
> +
> + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP))
> + return 0;
> +
> + /* Limit nr_pages by pfn alignment */
> + pfn = page_to_pfn(*pages);
> + if (pfn > 0)
> + nr_pages = min_t(size_t, nr_pages, 1UL << __ffs(pfn));
> +
> + nr_contig = num_pages_contiguous(pages, nr_pages);
> + if (nr_contig < 2)
> + return 0;
> +
> + order = ilog2(nr_contig);
> +
> + if (vm_shift(prot, PAGE_SIZE << order) == PAGE_SHIFT)
> + return 0;
> +
> + return order;
> +}
> +
> +static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
> + pgprot_t prot, struct page **pages)
> +{
> + const unsigned int nr_pages = (end - addr) >> PAGE_SHIFT;
> + unsigned int prev_shift = 0, batch_idx = 0;
> + unsigned long batch_start = addr, batch_end = addr;
> + int err;
> +
> + err = kmsan_vmap_pages_range_noflush(addr, end, prot, pages,
> + PAGE_SHIFT, GFP_KERNEL);
> +
> + if (err)
> + goto out;
> +
> + for (unsigned int i = 0; i < nr_pages; ) {
> + unsigned int shift = PAGE_SHIFT +
> + get_vmap_mapping_order(pages + i, prot, nr_pages - i);
> +
> + if (!i)
> + prev_shift = shift;
> +
> + if (shift != prev_shift) {
> + err = vmap_pages_range_noflush_walk(batch_start, batch_end,
> + prot, pages + batch_idx, prev_shift);
> + if (err)
> + goto out;
> + prev_shift = shift;
> + batch_start = batch_end;
> + batch_idx = i;
> + }
> +
> + /*
> + * Once we fail to batch pages, we expect to fail batching
> + * for all remaining pages, so just give up.
> + */
> + if (shift == PAGE_SHIFT)
> + break;
> +
> + batch_end += 1UL << shift;
> + i += 1U << (shift - PAGE_SHIFT);
> + }
> +
> + /* Remaining */
> + if (batch_start < end)
> + err = vmap_pages_range_noflush_walk(batch_start, end, prot,
> + pages + batch_idx, prev_shift);
[Severity: Critical]
Can vmap_pages_range_batched() overrun a physical chunk and map unowned
physical memory when a coalesced batch of same-order chunks straddles a PMD
boundary?
In get_vmap_mapping_order(), nr_pages is bounded by physical PFN alignment
and physical contiguity via num_pages_contiguous(), returning the full
contiguity order (for example, order 8, shift 20 for 1MB chunks) without
bounding the order by the virtual address alignment of the current mapping
address.
vmap_pages_range_batched() then coalesces consecutive physically
discontiguous chunks that share the same shift into a single
[batch_start, batch_end) range passed to vmap_pages_range_noflush_walk().
In this commit, get_vm_area_caller() in vmap() only aligns area->addr to
PAGE_SIZE. Even with follow-up commit bba985c9d433 ("mm/vmalloc: align
vm_area so vmap() can batch mappings"), vmap_get_aligned_vm_area() only
aligns area->addr to 1UL << arch_vmap_pte_supported_shift(size)
(CONT_PTE_SIZE, 64KB on 4K-page arm64) whenever size < PMD_SIZE (or when a
PMD_SIZE-aligned KVA allocation falls back), and any preceding smaller-order
batch (such as an order-4 chunk before order-8 chunks) can also leave
batch_start aligned to CONT_PTE_SIZE rather than 1UL << prev_shift.
Looking at the call chain on arm64:
vmap()
vmap_pages_range_batched()
vmap_pages_range_noflush_walk()
vmap_pages_pmd_range()
vmap_pages_pte_range()
vmap_set_ptes()
arch_vmap_pte_range_map_size()
vmap_pte_set_huge()
arch/arm64/include/asm/vmalloc.h:arch_vmap_pte_range_map_size() checks only
CONT_PTE_SIZE alignment and then maps up to 1UL << max_page_shift:
if (!IS_ALIGNED(addr, CONT_PTE_SIZE))
return PAGE_SIZE;
if (!IS_ALIGNED(PFN_PHYS(pfn), CONT_PTE_SIZE))
return PAGE_SIZE;
size = min(end - addr, 1UL << max_page_shift);
size = round_down(size, CONT_PTE_SIZE);
return size;
Consider two physically discontiguous order-8 (1MB, 256-page, prev_shift 20)
chunks at PFNs P0..P0+255 and P1..P1+255 coalesced into a 2MB batch starting
at a CONT_PTE_SIZE-aligned address that is 512KB (128 pages) below a PMD
boundary (for example, batch_start = 0xffff800080180000,
batch_end = 0xffff800080380000 across PMD boundary 0xffff800080200000):
1. In the first PMD [0xffff800080180000, 0xffff800080200000),
vmap_pages_pmd_range() clips next to the PMD boundary (512KB).
arch_vmap_pte_range_map_size() returns min(512KB, 1MB) = 512KB, mapping
pages[0..127] (PFNs P0..P0+127) and advancing *nr to 128.
2. At the start of the second PMD [0xffff800080200000, 0xffff800080380000),
*nr is 128 (pages[128], PFN P0+128), so only 128 contiguous pages (512KB,
PFNs P0+128..P0+255) remain in the first chunk before the physical
discontinuity at pages[256] (PFN P1).
3. Because both addr (0xffff800080200000) and PFN_PHYS(P0+128) are aligned to
CONT_PTE_SIZE (64KB) and end - addr is 1.5MB >= 1MB,
arch_vmap_pte_range_map_size() returns size = min(1.5MB, 1UL << 20) = 1MB
(256 pages).
4. vmap_pte_set_huge() then maps 256 contiguous physical pages starting at
PFN P0+128 (PFNs P0+128..P0+383), mapping 128 pages (512KB,
PFNs P0+256..P0+383) of unowned physical memory past the end of the first
chunk into the vmap() range, while advancing *nr by 256 to 384 and
skipping pages[256..383] of the second chunk.
The same overrun also occurs for two coalesced order-9 (2MB, prev_shift 21)
chunks when batch_start is CONT_PTE_SIZE-aligned but not PMD_SIZE-aligned,
because vmap_try_huge_pmd() fails in both PMDs and falls back to
vmap_pages_pte_range() with shift = 21.
Should get_vmap_mapping_order() also bound the mapping order by the virtual
address alignment of the current page or by the supported architectural map
size?
[ ... ]
> @@ -3656,8 +3735,7 @@ void *vmap(struct page **pages, unsigned int count,
> return NULL;
>
> addr = (unsigned long)area->addr;
> - if (vmap_pages_range(addr, addr + size, pgprot_nx(prot),
> - pages, PAGE_SHIFT) < 0) {
> + if (vmap_pages_range_batched(addr, addr + size, pgprot_nx(prot), pages) < 0) {
> vunmap(area->addr);
> return NULL;
> }
[Severity: High]
Could switching vmap() to vmap_pages_range_batched() without setting
VM_UNINITIALIZED on area expose a race with vread_iter() when a PMD is
promoted to a huge page?
In vmap(), get_vm_area_caller() publishes area in vn->busy without
VM_UNINITIALIZED before vmap_pages_range_batched() populates the page tables.
Previously, vmap() always passed PAGE_SHIFT to vmap_pages_range(), which only
allocated missing page tables and wrote leaf PTEs without freeing existing
page table pages.
With vmap_pages_range_batched(), when get_vmap_mapping_order() returns an
order >= PMD_SHIFT - PAGE_SHIFT over a virtual address range that previously
held 4KB mappings (leaving an empty present PTE table at *pmd after
vunmap_range_noflush()), vmap_pages_pmd_range() calls vmap_try_huge_pmd():
mm/vmalloc.c:vmap_try_huge_pmd() {
...
scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) {
if (!pmd_free_pte_page(pmd, addr))
return 0;
return pmd_set_huge(pmd, phys_addr, prot);
}
...
}
Concurrently, a reader of /proc/kcore executing vread_iter() checks:
mm/vmalloc.c:vread_iter() {
...
if (vm && (vm->flags & VM_UNINITIALIZED))
goto next_va;
...
}
Because VM_UNINITIALIZED is not set by vmap() (unlike
__vmalloc_node_range_noprof() and kho_restore_vmalloc()) and vread_iter()
does not take init_mm.mmap_lock, vread_iter() proceeds into
aligned_vread_iter() -> vmalloc_to_page() -> __vmalloc_to_page():
mm/vmalloc.c:__vmalloc_to_page() {
...
pmd = pmd_offset(pud, addr);
if (pmd_none(*pmd))
return NULL;
if (pmd_leaf(*pmd))
return pmd_page(*pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
if (WARN_ON_ONCE(pmd_bad(*pmd)))
return NULL;
ptep = pte_offset_kernel(pmd, addr);
pte = ptep_get(ptep);
...
}
If pmd_free_pte_page() and pmd_set_huge() run concurrently with
__vmalloc_to_page(), __vmalloc_to_page() can dereference a freed PTE page
table page, hit WARN_ON_ONCE(pmd_bad(*pmd)), or pass a newly installed huge
leaf PMD entry to pte_offset_kernel(pmd, addr).
Should vmap() set VM_UNINITIALIZED when allocating the vm_struct and clear it
via clear_vm_uninitialized_flag() after vmap_pages_range_batched() finishes?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261103101658.766905-1-jiangwenxiaomi@gmail.com?part=9
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v11 10/10] mm/vmalloc: align vm_area so vmap() can batch mappings
2026-11-03 10:16 [PATCH v11 00/10] mm/vmalloc: Speed up ioremap, vmalloc and vmap with contiguous memory Wen Jiang
` (8 preceding siblings ...)
2026-11-03 10:16 ` [PATCH v11 09/10] mm/vmalloc: map contiguous pages in batches for vmap() if possible Wen Jiang
@ 2026-11-03 10:16 ` Wen Jiang
2026-11-04 10:19 ` sashiko-bot
9 siblings, 1 reply; 14+ messages in thread
From: Wen Jiang @ 2026-11-03 10:16 UTC (permalink / raw)
To: akpm, catalin.marinas, linux-mm, urezki, will
Cc: Xueyuan.chen21, ajd, anshuman.khandual, baohua, chleroy, david,
dev.jain, jiangwen6, leo.yan, linux-arm-kernel, linux-kernel,
linuxppc-dev, maddy, mpe, npiggin, rppt, ryan.roberts,
Xueyuan Chen
From: "Barry Song (Xiaomi)" <baohua@kernel.org>
Try to align the vmap virtual address to PMD_SHIFT or a
larger PTE mapping size hinted by the architecture, so
contiguous pages can be batch-mapped when setting PMD or
PTE entries. We do not expect any significant overhead for
this.
Add __get_vm_area_node_aligned_caller() as a wrapper over
__get_vm_area_node() to simplify repeated calls with fixed
arguments.
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
Signed-off-by: Wen Jiang <jiangwen6@xiaomi.com>
Tested-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
Tested-by: Leo Yan <leo.yan@arm.com>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Reviewed-by: Dev Jain <dev.jain@arm.com>
---
Sashiko review: raised two concerns:
- Hardcoding GFP_KERNEL in __get_vm_area_node_aligned_caller() strips the
caller's ability to prevent blocking, reclaim, or warnings.
- The speculative PMD-aligned allocation, on failure, could trigger
spurious vmalloc purges (reclaim_and_purge_vmap_areas(), synchronous
TLB flushes, shrinker storms), a possible DoS for vmap() >= PMD_SIZE.
Both follow existing behavior: GFP_KERNEL matches the adjacent
get_vm_area_caller(), which hardcodes it the same way. The purge concern
is not new -- __vmalloc_node_range() already attempts PMD-aligned
allocation in __get_vm_area_node() with the same fallback through
alloc_vmap_area() overflow/purge, so vmap_get_aligned_vm_area() adds no
new purge behavior.
mm/vmalloc.c | 35 ++++++++++++++++++++++++++++++++++-
1 file changed, 34 insertions(+), 1 deletion(-)
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 42b3addd20f9..e15bcef804fc 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -3361,6 +3361,14 @@ struct vm_struct *get_vm_area_caller(unsigned long size, unsigned long flags,
NUMA_NO_NODE, GFP_KERNEL, caller);
}
+static struct vm_struct *__get_vm_area_node_aligned_caller(unsigned long size,
+ unsigned long align, unsigned long flags, const void *caller)
+{
+ return __get_vm_area_node(size, align, PAGE_SHIFT, flags,
+ VMALLOC_START, VMALLOC_END,
+ NUMA_NO_NODE, GFP_KERNEL, caller);
+}
+
/**
* find_vm_area - find a continuous kernel virtual area
* @addr: base address
@@ -3692,6 +3700,30 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
return err;
}
+static struct vm_struct *vmap_get_aligned_vm_area(unsigned long size,
+ unsigned long flags, pgprot_t prot, const void *caller)
+{
+ struct vm_struct *vm_area;
+ unsigned int shift;
+
+ if (arch_vmap_pmd_supported(prot) && size >= PMD_SIZE) {
+ vm_area = __get_vm_area_node_aligned_caller(size, PMD_SIZE,
+ flags, caller);
+ if (vm_area)
+ return vm_area;
+ }
+
+ shift = arch_vmap_pte_supported_shift(size);
+ if (shift > PAGE_SHIFT) {
+ vm_area = __get_vm_area_node_aligned_caller(size, 1UL << shift,
+ flags, caller);
+ if (vm_area)
+ return vm_area;
+ }
+
+ return __get_vm_area_node_aligned_caller(size, PAGE_SIZE, flags, caller);
+}
+
/**
* vmap - map an array of pages into virtually contiguous space
* @pages: array of page pointers
@@ -3730,7 +3762,8 @@ void *vmap(struct page **pages, unsigned int count,
return NULL;
size = (unsigned long)count << PAGE_SHIFT;
- area = get_vm_area_caller(size, flags, __builtin_return_address(0));
+ area = vmap_get_aligned_vm_area(size, flags, prot,
+ __builtin_return_address(0));
if (!area)
return NULL;
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v11 10/10] mm/vmalloc: align vm_area so vmap() can batch mappings
2026-11-03 10:16 ` [PATCH v11 10/10] mm/vmalloc: align vm_area so vmap() can batch mappings Wen Jiang
@ 2026-11-04 10:19 ` sashiko-bot
0 siblings, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-11-04 10:19 UTC (permalink / raw)
To: Wen Jiang, akpm, catalin.marinas, linux-mm, urezki, will
Cc: Xueyuan Chen, ajd, anshuman.khandual, baohua, chleroy, david,
dev.jain, jiangwen6, leo.yan, maddy, mpe, npiggin, rppt,
ryan.roberts
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] mm/vmalloc: out-of-bounds physical mapping in vmap() across PMD boundary
--
commit bba985c9d433d420f66d2ffcfeb47090bbcf7552
Author: Barry Song (Xiaomi) <baohua@kernel.org>
mm/vmalloc: align vm_area so vmap() can batch mappings
This commit aligns the virtual address area allocated for vmap() to PMD_SIZE
or the architecture-supported PTE batch shift so that contiguous pages can be
mapped in batches.
> diff --git a/mm/vmalloc.c b/mm/vmalloc.c
> index 42b3addd20f98..e15bcef804fc3 100644
> --- a/mm/vmalloc.c
> +++ b/mm/vmalloc.c
[ ... ]
> @@ -3692,6 +3700,30 @@ static int vmap_pages_range_batched(unsigned long addr, unsigned long end,
> return err;
> }
>
> +static struct vm_struct *vmap_get_aligned_vm_area(unsigned long size,
> + unsigned long flags, pgprot_t prot, const void *caller)
> +{
> + struct vm_struct *vm_area;
> + unsigned int shift;
> +
> + if (arch_vmap_pmd_supported(prot) && size >= PMD_SIZE) {
> + vm_area = __get_vm_area_node_aligned_caller(size, PMD_SIZE,
> + flags, caller);
> + if (vm_area)
> + return vm_area;
> + }
> +
> + shift = arch_vmap_pte_supported_shift(size);
> + if (shift > PAGE_SHIFT) {
> + vm_area = __get_vm_area_node_aligned_caller(size, 1UL << shift,
> + flags, caller);
> + if (vm_area)
> + return vm_area;
> + }
[Severity: Critical]
Can this cause vmap() to map out-of-bounds physical pages across a PMD
boundary on arm64 when pages[] contains multiple physically discontiguous
chunks that share a mapping shift larger than CONT_PTE_SHIFT?
In vmap_get_aligned_vm_area(), when size < PMD_SIZE (for example, a 1MB vmap
composed of two 512KB order-7 chunks on 4K-page arm64, or an 8MB vmap composed
of two 4MB order-8 chunks on 16K-page arm64 where PMD_SIZE is 32MB), or when
the PMD_SIZE-aligned allocation falls back, the virtual range is only aligned
to 1UL << arch_vmap_pte_supported_shift(size), which is CONT_PTE_SIZE on
arm64. The same alignment also occurs for batch_start if a PMD-aligned vmap
starts with a CONT_PTE_SIZE chunk followed by larger chunks.
In vmap_pages_range_batched(), get_vmap_mapping_order() can return an order
where shift > CONT_PTE_SHIFT, and consecutive discontiguous chunks with the
same shift are coalesced into a single walk:
mm/vmalloc.c:vmap_pages_range_batched() {
for (unsigned int i = 0; i < nr_pages; ) {
unsigned int shift = PAGE_SHIFT +
get_vmap_mapping_order(pages + i, prot, nr_pages - i);
if (!i)
prev_shift = shift;
if (shift != prev_shift) {
err = vmap_pages_range_noflush_walk(batch_start, batch_end,
prot, pages + batch_idx, prev_shift);
...
}
...
batch_end += 1UL << shift;
i += 1U << (shift - PAGE_SHIFT);
}
}
When [batch_start, batch_end) crosses a PMD boundary at an offset M into the
first chunk (where 0 < M < (1UL << prev_shift) and M is a multiple of
CONT_PTE_SIZE), vmap_pages_pmd_range() splits the walk at pmd_addr_end(addr,
end). The first PMD maps the initial M bytes of chunk 0 and advances *nr to
M >> PAGE_SHIFT.
At the start of the next PMD (addr = PMD_BOUNDARY), *nr points M bytes into
chunk 0 (physical address PA_0 + M), leaving only (1UL << prev_shift) - M
contiguous bytes in chunk 0 before pages[] switches to the discontiguous
physical address of chunk 1. However, vmap_pages_pte_range() still passes the
full shift = prev_shift to vmap_set_ptes():
mm/vmalloc.c:vmap_pages_pte_range() {
do {
struct page *page = pages[*nr];
...
pfn = page_to_pfn(page);
size = vmap_set_ptes(pte, addr, end, pfn, prot, shift);
nr_pages = PFN_DOWN(size);
} while (pte += nr_pages, *nr += nr_pages, addr += size, addr != end);
}
Because addr (at PMD_BOUNDARY) and PFN_PHYS(pfn) (at PA_0 + M) are both
aligned to CONT_PTE_SIZE, and end - addr covers the coalesced second chunk,
arch_vmap_pte_range_map_size() returns the full 1UL << prev_shift:
arch/arm64/include/asm/vmalloc.h:arch_vmap_pte_range_map_size() {
if (!IS_ALIGNED(addr, CONT_PTE_SIZE))
return PAGE_SIZE;
if (!IS_ALIGNED(PFN_PHYS(pfn), CONT_PTE_SIZE))
return PAGE_SIZE;
size = min(end - addr, 1UL << max_page_shift);
size = round_down(size, CONT_PTE_SIZE);
return size;
}
Doesn't vmap_pte_set_huge() then map 1UL << prev_shift contiguous physical
bytes starting at PA_0 + M, overrunning the end of chunk 0 by M bytes into
unowned physical memory while advancing *nr past the first M bytes of chunk 1?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261103101658.766905-1-jiangwenxiaomi@gmail.com?part=10
^ permalink raw reply [flat|nested] 14+ messages in thread