Linux-ARM-Kernel Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v3 00/19] Another attempt at HVO support on arm64
@ 2026-11-22 18:44 James Houghton
  2026-11-22 18:44 ` [PATCH v3 01/19] arm64/pgtable: Clear AF with LDCLR on supported systems James Houghton
                   ` (18 more replies)
  0 siblings, 19 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

Hi everyone,

This is v3 of the series to enable HVO on arm64.

This version of the series is based on mm-new (15f114dac9a7) plus
Muchun's "mm: Unify memmap initialization for compound pages" v3[2].
The first patch from v2 was sent standalone[3] and is currently in
mm-new, so it is not included in this series.

I realize this series might be difficult to reapply later, so I've also
uploaded it here: http://github.com/48ca/linux hvo-v3

As with last time, the last 5 patches are optional, and I've tagged
them as DO-NOT-MERGE, as it's unclear if using the AF trick to do
BBM-less block -> table transitions is architecturally sound without
BBML3. The patches should be mergeable if it is in fact sound.

The biggest changes since v2:

- Replaced "hugetlb: Fully initialize tail struct pages of
  non-pre-HVOed bootmem folios" with "hugetlb: Optimize bootmem folios
  that were not pre-HVOed". Muchun's series now initializes all tail
  struct pages of such folios, but it also drops the boot-time bulk
  optimization pass, assuming that all boot-time HVO folios are
  pre-HVOed. On arm64, HVO support is not known yet when bootmem folios
  are allocated, so they are not pre-HVOed and must be optimized when
  they are gathered. Muchun, I'd appreciate your thoughts on this one.

- Reworked "Leave pages partially HVOed upon restore failure" and "Use
  try_update_vmemmap_pte to update in-use PTEs" for the folio-oriented
  code. Restoring a folio now only remaps PTEs that still point at the
  shared tail page and frees any replacement pages that were not
  consumed, and vmemmap_pages list updates only happen once the PTE
  update has succeeded.

- bbm_through_af now checks the local CPU's ID_AA64MMFR1_EL1 when
  verifying a late CPU, instead of the sanitised value.

- New patch: "arm64/mm: Align vmemmap to maximal folio size". With
  CONFIG_VMEMMAP_OPTIMIZATION, compound_head() relies on head struct
  pages being naturally aligned, which arm64 did not guarantee (e.g.
  16G CONT_PMD folios with 64K pages). This mirrors the changes for
  riscv[4] and LoongArch[5]. (Thanks Muchun and Kefeng!)

- In "Allow architectures to dynamically disallow HVO": the arch hook
  is now the generic arch_support_vmemmap_remap(), and arm64 now
  disables the section-based vmemmap optimization with
  section_vmemmap_optimization_disable() in vmemmap_populate() instead
  of hugetlb_vmemmap_optimize_bootmem_page() checking for arch support.
  (Thanks Muchun!)

And some smaller changes:

- "arm64: Add bbm_through_af capability" now selects
  ARCH_HUGETLB_VMEMMAP_OPTIMIZATION_SYSCTL_RO itself, fixing a build
  failure (BUILD_BUG_ON) when bisecting.

- Renamed ARCH_WANT_HUGETLB_VMEMMAP_RO_AFTER_INIT to
  ARCH_HUGETLB_VMEMMAP_OPTIMIZATION_SYSCTL_RO. It now only makes the
  hugetlb_optimize_vmemmap sysctl read-only; vmemmap_optimize_enabled
  is no longer __ro_after_init.

- Removed an unused __HAVE_ARCH_TRY_UPDATE_VMEMMAP_PTE define.

- hugetlb_vmemmap_optimizable_size() now returns 0 when the architecture
  does not support HVO, so such hstates are no longer reported or
  treated as optimizable.

- Dropped "hugetlb_vmemmap: Disable HVO sysctl if arch doesn't support
  HVO". Now that hugetlb_vmemmap_optimizable_size() checks for arch
  support, the sysctl is not registered on such systems anyway.

- try_update_vmemmap_pte() is now defined outside of the
  CONFIG_ARM64_CONTPTE block, fixing the build with
  CONFIG_ARM64_CONTPTE=n.

- Documented fail_hugetlb_vmemmap_pte= and fail_hugetlb_vmemmap_pmd= in
  kernel-parameters.txt.

- Selftest: stop all worker processes (including in-flight dd/cat) at
  the end of each phase, so that they cannot race with the accounting
  checks, without blocking if a worker does not exit; drain surplus
  hugepages on cleanup; and don't treat "unmovable page" dumps from
  failed memory offlining as errors when running with -m.

v2: https://lore.kernel.org/linux-mm/20261003002123.505555-1-jthoughton@google.com/
v1: https://lore.kernel.org/linux-mm/20260708031129.3503195-1-jthoughton@google.com/

Claude Opus 5.5 was used while drafting this version of the series. Like
the last version, it was used to draft the selftests (hugely helpful),
and in this version of the series, it helped with finding some of the
smaller fixes and drafting some of the new patches. A local Sashiko run
pointed out the bbm_through_af error that is now fixed in this version.

Thanks!

And now for the rest of the background of these patches, updated from
v2's cover letter:

-- Background and Structure --

This patch series uses a trick with the Access Flag on CPUs that support
hardware update of the AF to update vmemmap page table entries without
introducing a time window where CPUs accessing the vmemmap might fault.

By avoiding faults, the HugeTLB vmemmap optimization (HVO) can be
implemented correctly on arm64 in a much more straightforward way than
previously attempted, most recently here[1] (please see [1] for a
breakdown of the other approaches attempted before).

For large-memory systems that allocate most of their available memory to
HugeTLB, HVO saves a huge amount of memory (1.5% of system memory).

This series has a few parts:
  1. Use LDCLR to clear the AF for PTEs (patch 1).
  2. Some preliminary HVO changes to support HVO on Arm (patches 2-8).
  3. arm64 changes to enable HVO (patches 9-12).
  4. Fault injection and a stress test (patches 13-14).
  5. Drop the BBML3 requirement for HVO (patches 15-19).

Part 5 is optional, and whether or not the claims it makes about the Arm
memory model are accurate is not 100% clear.

-- The AF trick --

The trick is that translations with the AF unset cannot be cached in the
TLB (see Rule R_DWZCQ in the Arm ARM), so they can be atomically updated
without needing a full break-before-make sequence.

So the PTE update sequence becomes:
  1. Atomically clear the AF on the existing PTE.
  2. Invalidate the TLB.
  3. cmpxchg the AF=0 PTE with the new PTE. If this fails, goto 1.

If there is a CPU on the system that does not support hardware access
flag updates, clearing the AF is problematic, as those CPUs might fault
on the vmemmap usage. Therefore, HVO compatibility checks all CPUs for
HW AF updates.

-- Application to HVO --

HVO relies on the following page table transitions:
  - When enabling HVO for a page, PMD block entries in the vmemmap are
    shattered into PMD table entries. The first vmemmap page of the
    folio is copied to a newly allocated page and remapped to it, and
    the remaining PTEs are remapped to a shared page of tail struct
    pages (that is, there is an OA change for every PTE).
  - When disabling HVO for a page, the PTEs that map the shared tail
    page are remapped to newly reallocated pages of struct pages. The
    PMD block -> table transition is not undone.

In part 3 of this series, I use the Access Flag trick to do the PTE
updates. We rely on BBML3 for the PMD block -> table transition.

In part 5, I re-use the Access Flag trick to do the PMD block -> table
transition without needing BBML3. For systems that support BBML3, the
logic is unchanged.

Because the in-place updates can fail (if the cmpxchg keeps losing the
race with hardware AF updates), HVO now handles failures at any point:
a failed optimization is rolled back, and if the rollback or a later
restore fails, the folio is left partially optimized (and remains marked
as such) rather than being freed with a broken vmemmap.

-- Late-onlining of CPUs that do not support HW AF --

HVO support is modeled as a system feature (called "BBM through AF"). If
not all early CPUs support the feature, attempts to HVO pages will fail,
and the HVO sysctl will be hidden.

To avoid penalizing systems that have asymmetric support for
BBM-through-AF (i.e., HW AF, as BBML3 cannot be mismatched),
BBM-through-AF will check that HVO is in fact enabled.

Because BBM-through-AF needs to know if HVO will be enabled when system
features are finalized, HVO cannot be dynamically enabled, hence the
patches that do this. (There are other ways to solve this, please see
v1).

-- Litmus test --

The following Herd litmus test demonstrates the PTE update routine:

  AArch64 TTDFaultlessUpdate
  Variant=vmsa
  TTHM=HA
  {
   uint64_t x=1;
   uint64_t y=2;
   [PTE(x)]=(oa:PA(x), af:1);
   0:X0=PTE(x); 1:X0=PTE(x);
   0:X1=x; 1:X1=x;
   pteval_t 0:X2=(oa:PA(x), af:0);
   pteval_t 0:X3=(oa:PA(y), af:1);
  }
   P0                | P1             ;
   LDR X4,[X0]       | L0:            ;
   MOV X5,X4         | LDR X2,[X1]    ;
   CAS X4,X2,[X0]    |                ;
   DSB ISHST         |                ;
   LSR X9,X1,#12     |                ;
   TLBI VAALE1IS,X9  |                ;
   DSB ISH           |                ;
   ISB               |                ;
   CAS X2,X3,[X0]    |                ;
  exists
    0:X5=0:X4 /\ (* First CAS must succeed *)
    (fault(P1:L0) \/ ~(1:X2=2 \/ 1:X2=1))

  (* This test should not contain "Warning-BBM-expected". *)

-- Testing --

I've tested this series with the included selftest, which stresses
the optimization, unoptimizing, and failure cases.

[1] https://lore.kernel.org/linux-arm-kernel/20241107202033.2721681-1-yuzhao@google.com/
[2] https://lore.kernel.org/linux-mm/20261120100856.16929-1-songmuchun@bytedance.com/
[3] https://lore.kernel.org/linux-mm/20261116221557.3128331-1-jthoughton@google.com/
[4] commit 476849b0fba4 ("riscv/mm: align vmemmap to maximal folio size")
[5] commit 2969b42c8f99 ("LoongArch/mm: align vmemmap to maximal folio size")

James Houghton (19):
  arm64/pgtable: Clear AF with LDCLR on supported systems
  hugetlb_vmemmap: Always flush TLB if needed upon PTE remapping
  hugetlb_vmemmap: Leave pages partially HVOed upon restore failure
  hugetlb_vmemmap: Use try_update_vmemmap_pte to update in-use PTEs
  hugetlb_vmemmap: Allow architectures to dynamically disallow HVO
  hugetlb: Optimize bootmem folios that were not pre-HVOed
  hugetlb_vmemmap: Allow architectures to make HVO enablement boot-time
    only
  hugetlb_vmemmap: Expose whether HVO is enabled to architecture code
  arm64: Add bbm_through_af capability
  arm64: Implement try_update_vmemmap_pte using the AF trick
  arm64/mm: Align vmemmap to maximal folio size
  arm64: Support hugetlb vmemmap optimization
  hugetlb_vmemmap: Add fault injection for in-place vmemmap PTE updates
  selftests/mm: Add HugeTLB vmemmap optimization stress test
  hugetlb_vmemmap: Use try_populate_vmemmap_pmd for replacing in-use
    PMDs
  arm64: Implement try_populate_vmemmap_pmd using AF trick
  arm64: Drop BBML3 requirement for HVO
  hugetlb_vmemmap: Add fault injection for in-place vmemmap PMD splits
  selftests/mm: Add HVO pmd-split fault injection tests

 .../admin-guide/kernel-parameters.txt         |   2 +
 Documentation/admin-guide/sysctl/vm.rst       |   3 +
 .../fault-injection/fault-injection.rst       |   8 +
 arch/arm64/Kconfig                            |   2 +
 arch/arm64/include/asm/cpufeature.h           |   5 +
 arch/arm64/include/asm/pgalloc.h              |  53 +++
 arch/arm64/include/asm/pgtable.h              |  73 +++-
 arch/arm64/kernel/cpufeature.c                |  32 ++
 arch/arm64/mm/init.c                          |  18 +-
 arch/arm64/mm/mmu.c                           |   9 +
 arch/arm64/tools/cpucaps                      |   1 +
 arch/loongarch/include/asm/pgalloc.h          |   2 +
 arch/loongarch/include/asm/pgtable.h          |   2 +
 arch/riscv/include/asm/pgalloc.h              |   2 +
 arch/riscv/include/asm/pgtable.h              |   2 +
 arch/x86/include/asm/pgalloc.h                |   2 +
 arch/x86/include/asm/pgtable.h                |   2 +
 fs/Kconfig                                    |   3 +-
 include/linux/hugetlb.h                       |   9 +
 include/linux/pgalloc.h                       |  21 +
 include/linux/pgtable.h                       |  37 ++
 lib/Kconfig.debug                             |   9 +
 mm/Kconfig                                    |   7 +
 mm/hugetlb.c                                  |  11 +
 mm/hugetlb_vmemmap.c                          | 164 +++++++-
 mm/hugetlb_vmemmap.h                          |   2 +
 tools/testing/selftests/mm/Makefile           |   2 +
 .../selftests/mm/hugetlb_vmemmap_stress.sh    | 384 ++++++++++++++++++
 .../selftests/mm/ksft_hugetlb_vmemmap.sh      |   4 +
 tools/testing/selftests/mm/run_vmtests.sh     |   4 +
 30 files changed, 839 insertions(+), 36 deletions(-)
 create mode 100755 tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh
 create mode 100755 tools/testing/selftests/mm/ksft_hugetlb_vmemmap.sh


base-commit: 15f114dac9a7bde1796f0e54e22726fd31cfa8a5
prerequisite-patch-id: 79139d48d73311a6b93f3ef4fcf35e81fa62b439
prerequisite-patch-id: 1decdc13c91eb4c111978d0a81264c80ee150f5c
prerequisite-patch-id: 3e89010bdfb5265dd7629e0ac0cdd4188e1e0fcb
prerequisite-patch-id: a46914935c0d5a34a2fe8fc5894372a6f4d3a9b9
prerequisite-patch-id: f5cc7ae1c18a204b0fdcd3d5a8ca2dc4ab02ab9c
prerequisite-patch-id: fa40c383a05f7713030e8e45f2c1d32b1891d59f
prerequisite-patch-id: e4996c7f982db0996d3a6a223d866ba303ae9b38
prerequisite-patch-id: 6e33595454fa72593f0f87a84190b2960e95d28b
prerequisite-patch-id: b6d144f2e3d6166cd6361f8095f7b1fc723c989a
prerequisite-patch-id: 36ea05703be8c54a4adf89a86457a4c73f16bae5
prerequisite-patch-id: 94bb8e10c55ce80b9eef1314b387a9dd5840faea
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply	[flat|nested] 20+ messages in thread

* [PATCH v3 01/19] arm64/pgtable: Clear AF with LDCLR on supported systems
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 02/19] hugetlb_vmemmap: Always flush TLB if needed upon PTE remapping James Houghton
                   ` (17 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

Replace the cmpxchg loop with atomic64_fetch_andnot_relaxed, which, on
systems with LSE atomics, should end up as LDCLR.

This is somewhat overdue. In recent versions of the architecture, if a
system supports FEAT_LSE, forward progress guarantees for exclusive
monitors and cmpxchg loops are relaxed.

Mechanically, I would love to write test_and_clear_bit_relaxed(), but no
such function exists (test_and_clear_bit() has full ordering, which is
also undesirable). Implementing test_and_clear_bit_relaxed() properly is
a pretty large change; use atomic64 instead, as pte_t contains a u64.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/include/asm/pgtable.h | 14 ++++----------
 1 file changed, 4 insertions(+), 10 deletions(-)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index ce51c1b9be22..4285b7e7a6a7 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1252,17 +1252,11 @@ static inline void __pte_clear(struct mm_struct *mm,
 static inline bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
 		unsigned long address, pte_t *ptep)
 {
-	pte_t old_pte, pte;
+	atomic64_t *pteval = (atomic64_t *)&pte_val(*ptep);
+	s64 af_mask = PTE_AF;
 
-	pte = __ptep_get(ptep);
-	do {
-		old_pte = pte;
-		pte = pte_mkold(pte);
-		pte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep),
-					       pte_val(old_pte), pte_val(pte));
-	} while (pte_val(pte) != pte_val(old_pte));
-
-	return pte_young(pte);
+	/* Atomically clear PTE_AF, checking that it was set before. */
+	return af_mask & atomic64_fetch_andnot_relaxed(af_mask, pteval);
 }
 
 static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 02/19] hugetlb_vmemmap: Always flush TLB if needed upon PTE remapping
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
  2026-11-22 18:44 ` [PATCH v3 01/19] arm64/pgtable: Clear AF with LDCLR on supported systems James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 03/19] hugetlb_vmemmap: Leave pages partially HVOed upon restore failure James Houghton
                   ` (16 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

vmemmap_remap_range() can potentially leave the vmemmap in a partially
optimized state. In this case, a TLB flush should be done if the walk
caller asked for it, even if the optimization was not fully applied.

A follow-up patch will more completely deal with partially-optimized
folios.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 mm/hugetlb_vmemmap.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 09b9fd83c198..7be08c68c2f1 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -163,13 +163,11 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end,
 
 	ret = walk_kernel_page_table_range_lockless(start, end, &vmemmap_remap_ops,
 						    NULL, walk);
-	if (ret)
-		return ret;
 
 	if (walk->remap_pte && !(walk->flags & VMEMMAP_DEFER_TLB_FLUSH))
 		flush_tlb_kernel_range(start, end);
 
-	return 0;
+	return ret;
 }
 
 /*
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 03/19] hugetlb_vmemmap: Leave pages partially HVOed upon restore failure
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
  2026-11-22 18:44 ` [PATCH v3 01/19] arm64/pgtable: Clear AF with LDCLR on supported systems James Houghton
  2026-11-22 18:44 ` [PATCH v3 02/19] hugetlb_vmemmap: Always flush TLB if needed upon PTE remapping James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 04/19] hugetlb_vmemmap: Use try_update_vmemmap_pte to update in-use PTEs James Houghton
                   ` (15 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

Right now it is assumed that the restore routine in vmemmap_remap_free()
will not fail; that is, if we fail to fully optimize a folio, it is
assumed that restoring the folio to its pre-HVO state is guaranteed to
succeed.

Although vmemmap restoration may never fail in practice, properly handle
such failure cases now. This will leave folios in a partially HVOed
state. In a follow-up patch, failure certainly becomes possible.

In the event we end up with partially-HVOed folios, make sure to:
  1. Leave the HVO page flag in place.
  2. Free any unused vmemmap pages.

Always pass the vmemmap_tail page to the restore routine to avoid
leaking the non-optimized vmemmap pages for a partially HVOed page.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 mm/hugetlb_vmemmap.c | 34 ++++++++++++++++++++++++++++------
 1 file changed, 28 insertions(+), 6 deletions(-)

diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 7be08c68c2f1..2e9cf8d39c89 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -274,12 +274,22 @@ static int vmemmap_restore_pte(pte_t *pte, unsigned long addr,
 {
 	const struct page *page = (const struct page *)addr;
 	const struct folio *folio = page_folio(page);
-	struct page *src = pte_page(ptep_get(pte)), *dst;
+	struct page *src = pte_page(ptep_get(pte)), *dst, *tail;
 
 	if (unlikely(!vmemmap_optimizable_pfn(page_to_pfn(page), folio_order(folio))))
 		return 0;
 
-	VM_WARN_ON_ONCE(folio_vmemmap_shared_tail_page(folio) != src);
+	tail = folio_vmemmap_shared_tail_page(folio);
+	if (WARN_ON_ONCE(!tail))
+		return -EINVAL;
+
+	/*
+	 * The folio may be only partially optimized, so restore only PTEs
+	 * currently pointing at the shared tail page.
+	 */
+	if (src != tail)
+		return 0;
+
 	dst = vmemmap_dup_page_from_list(src, walk->vmemmap_pages);
 	set_pte_at(&init_mm, addr, pte, mk_pte(dst, PAGE_KERNEL));
 
@@ -325,6 +335,7 @@ static int vmemmap_pteify_folio(const struct folio *folio, unsigned long flags)
 	return vmemmap_remap_range(start, end, &walk);
 }
 
+#define VMEMMAP_REMAP_INCOMPLETE 1
 /**
  * vmemmap_optimize_folio - optimize the vmemmap backing of a folio
  * @folio:	folio whose vmemmap pages should be optimized.
@@ -340,7 +351,8 @@ static int vmemmap_pteify_folio(const struct folio *folio, unsigned long flags)
  * Note: The caller must hold vmemmap_remap_rwsem. If TLB flushing is deferred,
  * the lock must remain held until the final TLB flush.
  *
- * Return: %0 on success, negative error code otherwise.
+ * Return: %0 on success, VMEMMAP_REMAP_INCOMPLETE if the folio is incompletely
+ *         optimized, negative error code otherwise.
  */
 static int vmemmap_optimize_folio(const struct folio *folio,
 		struct list_head *vmemmap_pages, unsigned long flags)
@@ -372,7 +384,8 @@ static int vmemmap_optimize_folio(const struct folio *folio,
 		return ret;
 
 	end = start + walk.nr_walked * PAGE_SIZE;
-	vmemmap_restore_range(start, end, vmemmap_pages, 0);
+	if (vmemmap_restore_range(start, end, vmemmap_pages, 0))
+		return VMEMMAP_REMAP_INCOMPLETE;
 
 	return ret;
 }
@@ -402,7 +415,12 @@ static int vmemmap_restore_folio(const struct folio *folio, unsigned long flags)
 		return ret;
 
 	guard(rwsem_read)(&vmemmap_remap_rwsem);
-	return vmemmap_restore_range(start, end, &vmemmap_pages, flags);
+	ret = vmemmap_restore_range(start, end, &vmemmap_pages, flags);
+
+	/* Not all pages may have been consumed */
+	free_vmemmap_page_list(&vmemmap_pages);
+
+	return ret;
 }
 
 static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON);
@@ -526,7 +544,11 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
 	 */
 	folio_set_hugetlb_vmemmap_optimized(folio);
 	ret = vmemmap_optimize_folio(folio, vmemmap_pages, flags);
-	if (ret)
+	/*
+	 * If ret == VMEMMAP_REMAP_INCOMPLETE, the folio might be partially
+	 * HVOed. Leave the HVO page folio flag in place.
+	 */
+	if (ret < 0)
 		folio_clear_hugetlb_vmemmap_optimized(folio);
 
 	return ret;
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 04/19] hugetlb_vmemmap: Use try_update_vmemmap_pte to update in-use PTEs
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (2 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 03/19] hugetlb_vmemmap: Leave pages partially HVOed upon restore failure James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 05/19] hugetlb_vmemmap: Allow architectures to dynamically disallow HVO James Houghton
                   ` (14 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

set_pte_at() cannot be used to replace in-use vmemmap PTEs on arm64, so
replace it with a more specific routine, try_update_vmemmap_pte().

try_update_vmemmap_pte() is used for modifying page table entries such
that there is a guarantee that no fault will be taken.

For the generic implementation, please note one difference: set_pte()
does not invoke page_table_check_ptes_set(), but set_pte_at(), the call
we are about to replace, does. However, this is a functional no-op
because page_table_check_ptes_set() does nothing for init_mm PTEs, which
vmemmap PTEs are.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/loongarch/include/asm/pgtable.h |  2 ++
 arch/riscv/include/asm/pgtable.h     |  2 ++
 arch/x86/include/asm/pgtable.h       |  2 ++
 include/linux/pgtable.h              | 22 +++++++++++++++++++
 mm/hugetlb_vmemmap.c                 | 32 +++++++++++++++++++++++-----
 5 files changed, 55 insertions(+), 5 deletions(-)

diff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/asm/pgtable.h
index f87603135131..123eda3f6035 100644
--- a/arch/loongarch/include/asm/pgtable.h
+++ b/arch/loongarch/include/asm/pgtable.h
@@ -633,6 +633,8 @@ static inline long pmd_protnone(pmd_t pmd)
 #define pmd_leaf(pmd)		((pmd_val(pmd) & _PAGE_HUGE) != 0)
 #define pud_leaf(pud)		((pud_val(pud) & _PAGE_HUGE) != 0)
 
+#define ARCH_WANTS_GENERIC_POPULATE_VMEMMAP_PTE
+
 /*
  * We provide our own get_unmapped area to cope with the virtual aliasing
  * constraints placed on us by the cache architecture.
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 2ebd5fa5adc5..c8d05bc12cc8 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -1172,6 +1172,8 @@ static inline pud_t pud_modify(pud_t pud, pgprot_t newprot)
 
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
 
+#define ARCH_WANTS_GENERIC_POPULATE_VMEMMAP_PTE
+
 /*
  * Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
  * are !pte_none() && !pte_present().
diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index 000fc6c1ca37..271cdc61de3a 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -1372,6 +1372,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
 }
 #endif
 
+#define ARCH_WANTS_GENERIC_POPULATE_VMEMMAP_PTE
+
 #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD
 static inline pud_t pudp_establish(struct vm_area_struct *vma,
 		unsigned long address, pud_t *pudp, pud_t pud)
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index bc1353f03b08..ab33df268955 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -458,6 +458,28 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
 #endif
 #define set_pte_at(mm, addr, ptep, pte) set_ptes(mm, addr, ptep, pte, 1)
 
+#ifdef ARCH_WANTS_GENERIC_POPULATE_VMEMMAP_PTE
+/*
+ * try_update_vmemmap_pte - Remap PTEs used by the vmemmap.
+ * @addr: Base address of the remapped PTE.
+ * @ptep: Page table pointer to be overwritten.
+ * @pte: Page table entry to write.
+ *
+ * This function is only to be used to update PTEs that map the vmemmap. The
+ * only valid transitions supported by this function are: leaf-level
+ * (PAGE_SIZE), valid-to-valid. The pfn and prot bits may be changed.
+ *
+ * Implementations of this function must ensure that, while the update is taking
+ * place, CPUs will not fault on the remapped virtual address.
+ */
+static inline int try_update_vmemmap_pte(unsigned long addr, pte_t *ptep,
+					 pte_t pte)
+{
+	set_pte(ptep, pte);
+	return 0;
+}
+#endif
+
 #ifndef __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS
 extern int ptep_set_access_flags(struct vm_area_struct *vma,
 				 unsigned long address, pte_t *ptep,
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 2e9cf8d39c89..b4d0311bcc4a 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -218,15 +218,19 @@ static int alloc_vmemmap_page_list(struct list_head *head,
 	return -ENOMEM;
 }
 
+/*
+ * Copy @src into the first page on @vmemmap_pages and return it. The page is
+ * left on the list; the caller must remove it only once it has successfully
+ * been installed in the vmemmap.
+ */
 static struct page *vmemmap_dup_page_from_list(const struct page *src,
 		struct list_head *vmemmap_pages)
 {
 	struct page *dst = list_first_entry(vmemmap_pages, struct page, lru);
 
-	list_del(&dst->lru);
 	copy_page(page_address(dst), page_address(src));
 
-	/* Ensure updates to @dst page are visible before set_pte_at(). */
+	/* Ensure updates to @dst page are visible before try_update_vmemmap_pte(). */
 	smp_wmb();
 
 	return dst;
@@ -248,6 +252,8 @@ static int vmemmap_optimize_pte(pte_t *pte, unsigned long addr,
 	const struct page *page = (const struct page *)addr;
 	const struct folio *folio = page_folio(page);
 	struct page *src = pte_page(ptep_get(pte)), *dst;
+	bool dup = false;
+	int ret;
 
 	if (likely(vmemmap_optimizable_pfn(page_to_pfn(page), folio_order(folio)))) {
 		dst = folio_vmemmap_shared_tail_page(folio);
@@ -255,16 +261,27 @@ static int vmemmap_optimize_pte(pte_t *pte, unsigned long addr,
 			return -ENOMEM;
 	} else {
 		dst = vmemmap_dup_page_from_list(src, walk->vmemmap_pages);
+		dup = true;
 	}
 
-	list_add(&src->lru, walk->vmemmap_pages);
 	/*
 	 * Map the shared tail page writable intentionally. A lockless get that
 	 * read a non-zero refcount before the remap can still do its cmpxchg
 	 * after it. The refcount is zero, so the cmpxchg fails and leaves the
 	 * page unchanged.
 	 */
-	set_pte_at(&init_mm, addr, pte, mk_pte(dst, PAGE_KERNEL));
+	ret = try_update_vmemmap_pte(addr, pte, mk_pte(dst, PAGE_KERNEL));
+	if (ret)
+		return ret;
+
+	/*
+	 * We successfully overwrote the vmemmap PTE, so the page that was just
+	 * unmapped can be freed, and if we mapped a duplicated page, remove it
+	 * from the list so that it doesn't get freed later.
+	 */
+	if (dup)
+		list_del(&dst->lru);
+	list_add(&src->lru, walk->vmemmap_pages);
 
 	return 0;
 }
@@ -275,6 +292,7 @@ static int vmemmap_restore_pte(pte_t *pte, unsigned long addr,
 	const struct page *page = (const struct page *)addr;
 	const struct folio *folio = page_folio(page);
 	struct page *src = pte_page(ptep_get(pte)), *dst, *tail;
+	int ret;
 
 	if (unlikely(!vmemmap_optimizable_pfn(page_to_pfn(page), folio_order(folio))))
 		return 0;
@@ -291,7 +309,11 @@ static int vmemmap_restore_pte(pte_t *pte, unsigned long addr,
 		return 0;
 
 	dst = vmemmap_dup_page_from_list(src, walk->vmemmap_pages);
-	set_pte_at(&init_mm, addr, pte, mk_pte(dst, PAGE_KERNEL));
+	ret = try_update_vmemmap_pte(addr, pte, mk_pte(dst, PAGE_KERNEL));
+	if (ret)
+		return ret;
+
+	list_del(&dst->lru);
 
 	return 0;
 }
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 05/19] hugetlb_vmemmap: Allow architectures to dynamically disallow HVO
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (3 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 04/19] hugetlb_vmemmap: Use try_update_vmemmap_pte to update in-use PTEs James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 06/19] hugetlb: Optimize bootmem folios that were not pre-HVOed James Houghton
                   ` (13 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

For arm64, only some platforms support HVO. Add an arch hook,
arch_support_vmemmap_remap(), to prevent HVO from taking effect if the
architecture lacks support. The hook is not HugeTLB-specific, so it can
also be used by other users of in-place vmemmap remapping (e.g. FS-DAX).

Check it in hugetlb_vmemmap_optimizable_size(), so that hstates are not
reported or treated as optimizable (e.g. by report_hugepages() and the
HugeTLB allocation and freeing paths) when the architecture does not
support HVO.

It may be the case that arch_support_vmemmap_remap() returns false at
first but eventually returns true. This is the case on arm64, where
support depends on a system-wide CPU capability that is not yet known
when bootmem hugepages are allocated. Such architectures cannot pre-HVO
bootmem hugepages, so they must disable the section-based vmemmap
optimization in their vmemmap population path with
section_vmemmap_optimization_disable(). Their bootmem hugepages are then
optimized when they are gathered instead.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 include/linux/pgtable.h | 15 +++++++++++++++
 mm/hugetlb_vmemmap.h    |  2 ++
 2 files changed, 17 insertions(+)

diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index ab33df268955..8cf44120e2ed 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -480,6 +480,21 @@ static inline int try_update_vmemmap_pte(unsigned long addr, pte_t *ptep,
 }
 #endif
 
+#ifndef arch_support_vmemmap_remap
+/*
+ * arch_support_vmemmap_remap - Whether in-use vmemmap mappings may be remapped.
+ *
+ * Vmemmap optimizations, such as the HugeTLB vmemmap optimization (HVO),
+ * remap vmemmap page table entries while they are in use. Architectures that
+ * can only do this on some systems may override this hook. It may return false
+ * early during boot and true later, once system-wide features are known.
+ */
+static inline bool arch_support_vmemmap_remap(void)
+{
+	return IS_ENABLED(CONFIG_VMEMMAP_OPTIMIZATION);
+}
+#endif
+
 #ifndef __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS
 extern int ptep_set_access_flags(struct vm_area_struct *vma,
 				 unsigned long address, pte_t *ptep,
diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h
index 8b28063d78cc..440477288d19 100644
--- a/mm/hugetlb_vmemmap.h
+++ b/mm/hugetlb_vmemmap.h
@@ -48,6 +48,8 @@ static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate
 
 	if (!is_power_of_2(sizeof(struct page)))
 		return 0;
+	if (!arch_support_vmemmap_remap())
+		return 0;
 	return size > 0 ? size : 0;
 }
 #else
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 06/19] hugetlb: Optimize bootmem folios that were not pre-HVOed
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (4 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 05/19] hugetlb_vmemmap: Allow architectures to dynamically disallow HVO James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 07/19] hugetlb_vmemmap: Allow architectures to make HVO enablement boot-time only James Houghton
                   ` (12 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

Boot-time gigantic folios are expected to be pre-HVOed through the
section-based vmemmap optimization, so the bulk optimization pass for
bootmem folios was dropped.

However, arch_support_vmemmap_remap() may not yet
return true when bootmem folios are allocated (e.g. on arm64, where
support depends on a system-wide CPU capability), while it does return
true by the time the bootmem folios are gathered. Such architectures
disable the section-based vmemmap optimization in their vmemmap
population path (see section_vmemmap_optimization_disable()), so the
folios are not pre-HVOed and would never be optimized.

Optimize such folios when gathering bootmem folios. Their tail struct
pages have all been initialized by memmap_init_compound_page(),
as they were not pre-HVOed. Pre-HVOed folios are skipped by
vmemmap_should_optimize_folio().

Signed-off-by: James Houghton <jthoughton@google.com>
---
 mm/hugetlb.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 72839e98f998..8756c0206922 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3325,6 +3325,17 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h,
 	unsigned long flags;
 	struct folio *folio, *tmp_f;
 
+	/*
+	 * Bootmem folios are normally pre-HVOed through the section-based
+	 * vmemmap optimization. However, an architecture that can only
+	 * determine whether it supports HVO after bootmem allocation disables
+	 * the section-based optimization (see
+	 * section_vmemmap_optimization_disable()), in which case the folios
+	 * were not pre-HVOed and must be optimized now. Pre-HVOed folios are
+	 * skipped.
+	 */
+	hugetlb_vmemmap_optimize_folios(h, folio_list);
+
 	list_for_each_entry_safe(folio, tmp_f, folio_list, lru) {
 		/* Subdivide locks to achieve better parallel performance */
 		spin_lock_irqsave(&hugetlb_lock, flags);
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 07/19] hugetlb_vmemmap: Allow architectures to make HVO enablement boot-time only
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (5 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 06/19] hugetlb: Optimize bootmem folios that were not pre-HVOed James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 08/19] hugetlb_vmemmap: Expose whether HVO is enabled to architecture code James Houghton
                   ` (11 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

Architectures may need to configure the system at boot depending on
whether HVO is enabled. For example, arm64 needs all CPUs to support
hardware updates of the access flag to safely update the vmemmap in
place, and may want to refuse to online CPUs that lack it, but only if
HVO may be used.

Such configuration is only valid if HVO cannot be enabled later at
runtime through the vm.hugetlb_optimize_vmemmap sysctl.

Add CONFIG_ARCH_HUGETLB_VMEMMAP_OPTIMIZATION_SYSCTL_RO for architectures
to select. When selected, the vm.hugetlb_optimize_vmemmap sysctl is
read-only, so HVO can only be enabled or disabled with the
hugetlb_free_vmemmap= kernel command line parameter.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 Documentation/admin-guide/sysctl/vm.rst | 3 +++
 fs/Kconfig                              | 3 ++-
 mm/Kconfig                              | 7 +++++++
 mm/hugetlb_vmemmap.c                    | 3 ++-
 4 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst
index 3cb0138e850c..f29da7a35289 100644
--- a/Documentation/admin-guide/sysctl/vm.rst
+++ b/Documentation/admin-guide/sysctl/vm.rst
@@ -699,6 +699,9 @@ pages.  So, those surplus pages are still optimized until they are no longer
 in use.  You would need to wait for those surplus pages to be released before
 there are no optimized pages in the system.
 
+On some architectures, this knob is read-only, and HVO can only be enabled or
+disabled with the hugetlb_free_vmemmap= kernel command line parameter.
+
 
 nr_hugepages_mempolicy
 ======================
diff --git a/fs/Kconfig b/fs/Kconfig
index 70a1d0760d28..0c1a436280f8 100644
--- a/fs/Kconfig
+++ b/fs/Kconfig
@@ -267,7 +267,8 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON
 	help
 	  The HugeTLB Vmemmap Optimization (HVO) defaults to off. Say Y here to
 	  enable HVO by default. It can be disabled via hugetlb_free_vmemmap=off
-	  (boot command line) or hugetlb_optimize_vmemmap (sysctl).
+	  (boot command line) or hugetlb_optimize_vmemmap (sysctl, if the
+	  architecture allows changing it at runtime).
 endif # HUGETLBFS
 
 config HUGETLB_PAGE
diff --git a/mm/Kconfig b/mm/Kconfig
index 18c0beafd4b8..c1c32d2c5cf8 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -466,6 +466,13 @@ config ARCH_WANT_OPTIMIZE_DAX_VMEMMAP
 config ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP
 	bool
 
+#
+# Select this from the architecture Kconfig if the hugetlb_optimize_vmemmap
+# sysctl must be read-only.
+#
+config ARCH_HUGETLB_VMEMMAP_OPTIMIZATION_SYSCTL_RO
+	bool
+
 config HAVE_MEMBLOCK_PHYS_MAP
 	bool
 
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index b4d0311bcc4a..142f2975846b 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -669,7 +669,8 @@ static const struct ctl_table hugetlb_vmemmap_sysctls[] = {
 		.procname	= "hugetlb_optimize_vmemmap",
 		.data		= &vmemmap_optimize_enabled,
 		.maxlen		= sizeof(vmemmap_optimize_enabled),
-		.mode		= 0644,
+		.mode		= IS_ENABLED(CONFIG_ARCH_HUGETLB_VMEMMAP_OPTIMIZATION_SYSCTL_RO) ?
+				  0444 : 0644,
 		.proc_handler	= proc_dobool,
 	},
 };
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 08/19] hugetlb_vmemmap: Expose whether HVO is enabled to architecture code
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (6 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 07/19] hugetlb_vmemmap: Allow architectures to make HVO enablement boot-time only James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 09/19] arm64: Add bbm_through_af capability James Houghton
                   ` (10 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

Architectures that select
CONFIG_ARCH_HUGETLB_VMEMMAP_OPTIMIZATION_SYSCTL_RO may configure the
system at boot depending on whether HVO is enabled, e.g. arm64 may
require all CPUs to support hardware access flag updates only if HVO is
enabled.

Add hugetlb_vmemmap_optimize_enabled() so that they can check. It is
declared before <asm/hugetlb.h> is included, so that it can also be used
from inline helpers there.

It only reflects the hugetlb_free_vmemmap= kernel command line parameter
(and the sysctl, when writable), not whether the architecture supports
HVO.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 include/linux/hugetlb.h |  9 +++++++++
 mm/hugetlb_vmemmap.c    | 18 ++++++++++++++++++
 2 files changed, 27 insertions(+)

diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 0d5414392d87..b2a6441bcd84 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -22,6 +22,15 @@ struct node;
 
 void free_huge_folio(struct folio *folio);
 
+#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
+bool hugetlb_vmemmap_optimize_enabled(void);
+#else
+static inline bool hugetlb_vmemmap_optimize_enabled(void)
+{
+	return false;
+}
+#endif
+
 #ifdef CONFIG_HUGETLB_PAGE
 
 #include <linux/pagemap.h>
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 142f2975846b..fb6da64bd80d 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -452,6 +452,24 @@ static int __init hugetlb_vmemmap_optimize_param(char *buf)
 }
 early_param("hugetlb_free_vmemmap", hugetlb_vmemmap_optimize_param);
 
+/**
+ * hugetlb_vmemmap_optimize_enabled - whether HVO is enabled
+ *
+ * This only reflects the hugetlb_free_vmemmap= kernel command line parameter
+ * and the vm.hugetlb_optimize_vmemmap sysctl, not whether the architecture
+ * supports HVO (see arch_support_vmemmap_remap()).
+ *
+ * The value is final once early parameters have been parsed only if the
+ * architecture selects CONFIG_ARCH_HUGETLB_VMEMMAP_OPTIMIZATION_SYSCTL_RO;
+ * otherwise it may change at any time through the sysctl.
+ *
+ * Return: true if HVO is enabled.
+ */
+bool hugetlb_vmemmap_optimize_enabled(void)
+{
+	return READ_ONCE(vmemmap_optimize_enabled);
+}
+
 static int __hugetlb_vmemmap_restore_folio(const struct hstate *h,
 					   struct folio *folio, unsigned long flags)
 {
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 09/19] arm64: Add bbm_through_af capability
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (7 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 08/19] hugetlb_vmemmap: Expose whether HVO is enabled to architecture code James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 10/19] arm64: Implement try_update_vmemmap_pte using the AF trick James Houghton
                   ` (9 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

HVO requires BBM Level 3 for the block -> table transitions when
optimizing the vmemmap, and it requires hardware AF management to avoid
taking page faults on the !AF translation that is installed as part of
the RW table -> RO + new OA table transitions.

Call this requirement "BBM through AF"; i.e., all CPUs support BBM-less,
in-place TTD updates. Model this as a system feature, as all CPUs must
support it for the kernel to use it.

It may seem odd to check for HVO support directly in bbm_through_af().
If this weren't done, any system with mismatched HW AF will no longer be
able to start late CPUs that are missing HW AF if all early CPUs have HW
AF (and BBML3).

This depends on HVO enablement not changing after boot, so select
ARCH_HUGETLB_VMEMMAP_OPTIMIZATION_SYSCTL_RO.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/Kconfig                  |  1 +
 arch/arm64/include/asm/cpufeature.h |  5 +++++
 arch/arm64/kernel/cpufeature.c      | 33 +++++++++++++++++++++++++++++
 arch/arm64/tools/cpucaps            |  1 +
 4 files changed, 40 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 9b08b2457a55..4cce60c2de66 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -61,6 +61,7 @@ config ARM64
 	select ARCH_HAVE_ELF_PROT
 	select ARCH_HAVE_NMI_SAFE_CMPXCHG
 	select ARCH_HAVE_TRACE_MMIO_ACCESS
+	select ARCH_HUGETLB_VMEMMAP_OPTIMIZATION_SYSCTL_RO
 	select ARCH_KEEP_MEMBLOCK
 	select ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE
 	select ARCH_USE_CMPXCHG_LOCKREF
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index e3c8a1493dc8..b9106b5a8dcc 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -909,6 +909,11 @@ static inline bool system_supports_bbml3(void)
 	return alternative_has_cap_unlikely(ARM64_HAS_BBML3);
 }
 
+static inline bool system_supports_bbm_through_af(void)
+{
+	return alternative_has_cap_unlikely(ARM64_HAS_BBM_THROUGH_AF);
+}
+
 int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt);
 bool try_emulate_mrs(struct pt_regs *regs, u32 isn);
 
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index c9c9a6889455..d241638b5143 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -76,6 +76,7 @@
 #include <linux/kasan.h>
 #include <linux/percpu.h>
 #include <linux/sched/isolation.h>
+#include <linux/hugetlb.h>
 
 #include <asm/arm_pmuv3.h>
 #include <asm/cpu.h>
@@ -2044,6 +2045,31 @@ static bool has_bbml3(const struct arm64_cpu_capabilities *caps, int scope)
 	return cpu_supports_bbml3();
 }
 
+static bool has_bbm_through_af(const struct arm64_cpu_capabilities *caps, int scope)
+{
+	/* The kernel only enables HW AF updates with CONFIG_ARM64_HW_AFDBM. */
+	if (!IS_ENABLED(CONFIG_ARM64_HW_AFDBM))
+		return false;
+
+	/*
+	 * BBM-through-AF is only needed for HVO. If HVO is not in use, don't
+	 * penalize systems with mismatched support.
+	 */
+	if (!hugetlb_vmemmap_optimize_enabled()) {
+		/* Only valid if HVO cannot be enabled later via the sysctl. */
+		BUILD_BUG_ON(!IS_ENABLED(CONFIG_ARCH_HUGETLB_VMEMMAP_OPTIMIZATION_SYSCTL_RO));
+		return false;
+	}
+
+	/*
+	 * We need BBML3 to support Block -> Table transitions without taking
+	 * faults, and we need HW AF support to support changing the OA without
+	 * taking faults. has_cpuid_feature() checks HAFDBS >= AF, reading the
+	 * local CPU's register when verifying a late CPU.
+	 */
+	return cpu_supports_bbml3() && has_cpuid_feature(caps, scope);
+}
+
 static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused)
 {
 	/*
@@ -2953,6 +2979,13 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.type = ARM64_CPUCAP_EARLY_LOCAL_CPU_FEATURE,
 		.matches = has_bbml3,
 	},
+	{
+		.desc = "BBM-less TTD updates via Access Flag",
+		.capability = ARM64_HAS_BBM_THROUGH_AF,
+		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.matches = has_bbm_through_af,
+		ARM64_CPUID_FIELDS(ID_AA64MMFR1_EL1, HAFDBS, AF)
+	},
 	{
 		.desc = "52-bit Virtual-Physical Addressing for S1 and S2 (LPA2)",
 		.capability = ARM64_HAS_LPA2,
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index 3125695122a0..25292233ecd4 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -15,6 +15,7 @@ HAS_ADDRESS_AUTH_IMP_DEF
 HAS_AMU_EXTN
 HAS_ARMv8_4_TTL
 HAS_BBML3
+HAS_BBM_THROUGH_AF
 HAS_CACHE_DIC
 HAS_CACHE_IDC
 HAS_CNP
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 10/19] arm64: Implement try_update_vmemmap_pte using the AF trick
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (8 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 09/19] arm64: Add bbm_through_af capability James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 11/19] arm64/mm: Align vmemmap to maximal folio size James Houghton
                   ` (8 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

try_update_vmemmap_pte() must modify vmemmap PTEs without introducing a
time window where other CPUs on the system might fault.

Normally a break-before-make sequence is required to avoid conflicts
with cached translations. However, if we can guarantee that the existing
translation cannot be cached, a BBM sequence is not needed.

Translations with the AF unset may not be cached (see Arm ARM Rule
DWZCQ); the implementation of try_update_vmemmap_pte() on arm64 takes
advantage of this fact to replace a PTE without BBM and therefore
without leaving a window open where PE might fault on this translation.

Of course, if some CPUs on the system do not support HW AF management,
clearing the AF will introduce potential faults.
system_supports_bbm_through_af() will return false if any CPUs on the
system do not support HW AF.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/include/asm/pgtable.h | 59 +++++++++++++++++++++++++++++---
 1 file changed, 55 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 4285b7e7a6a7..7421c0a048ff 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1249,14 +1249,24 @@ static inline void __pte_clear(struct mm_struct *mm,
 	__set_pte(ptep, __pte(0));
 }
 
-static inline bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
-		unsigned long address, pte_t *ptep)
+/*
+ * Atomically clear the Accessed flag. Return the old value of the PTE.
+ */
+static inline pte_t __ptep_clear_young(pte_t *ptep)
 {
 	atomic64_t *pteval = (atomic64_t *)&pte_val(*ptep);
 	s64 af_mask = PTE_AF;
 
-	/* Atomically clear PTE_AF, checking that it was set before. */
-	return af_mask & atomic64_fetch_andnot_relaxed(af_mask, pteval);
+	/* Atomically clear PTE_AF. */
+	u64 oldval = atomic64_fetch_andnot_relaxed(af_mask, pteval);
+
+	return __pte(oldval);
+}
+
+static inline bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
+		unsigned long address, pte_t *ptep)
+{
+	return pte_young(__ptep_clear_young(ptep));
 }
 
 static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
@@ -1921,6 +1931,47 @@ static inline void clear_young_dirty_ptes(struct vm_area_struct *vma,
 
 #endif /* CONFIG_ARM64_CONTPTE */
 
+static inline int try_update_vmemmap_pte(unsigned long addr, pte_t *ptep,
+					 const pte_t pte)
+{
+	const int max_attempts = 16;
+	int attempts = 0;
+	pte_t old_pte;
+
+	if (!system_supports_bbm_through_af())
+		return -EOPNOTSUPP;
+
+	/* This routine is only to be used for valid-to-valid transitions. */
+	if (WARN_ON_ONCE(!pte_valid(pte)))
+		return -EINVAL;
+
+	old_pte = __ptep_get(ptep);
+
+	do {
+		if (WARN_ON_ONCE(!pte_valid(old_pte)))
+			return -EINVAL;
+
+		/* We should never get a contiguous PTE here. */
+		if (WARN_ON_ONCE(pte_valid_cont(old_pte)))
+			return -EINVAL;
+
+		if (pte_young(old_pte)) {
+			/* __ptep_clear_young() returns the overwritten PTE */
+			old_pte = pte_mkold(__ptep_clear_young(ptep));
+
+			flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
+		}
+	/*
+	 * Translations without AF cannot be cached, so we can replace
+	 * them without BBM.
+	 */
+	} while (!try_cmpxchg_relaxed(&pte_val(*ptep), &pte_val(old_pte),
+				      pte_val(pte)) &&
+		 ++attempts < max_attempts);
+
+	return attempts == max_attempts ? -EAGAIN : 0;
+}
+
 #define __HAVE_ARCH_VMAP_PTE_SET_HUGE
 void vmap_pte_set_huge(pte_t *ptep, unsigned long addr, phys_addr_t phys,
 		       pgprot_t prot, unsigned long size);
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 11/19] arm64/mm: Align vmemmap to maximal folio size
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (9 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 10/19] arm64: Implement try_update_vmemmap_pte using the AF trick James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 12/19] arm64: Support hugetlb vmemmap optimization James Houghton
                   ` (7 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

With CONFIG_VMEMMAP_OPTIMIZATION, compound_head() derives a tail page's
head by masking its address. This requires the head struct page of every
folio to be naturally aligned to the folio's vmemmap size, up to
MAX_FOLIO_VMEMMAP_ALIGN.

On arm64, the vmemmap is offset by memstart_addr, which is only aligned
to ARM64_MEMSTART_ALIGN: as little as 512M with 64K pages, where 16G
CONT_PMD hugetlb folios exist. VMEMMAP_START itself is sufficiently
aligned for all supported page sizes and VA sizes.

Raise ARM64_MEMSTART_ALIGN to cover MAX_FOLIO_VMEMMAP_ALIGN, and assert
at build time that VMEMMAP_START is suitably aligned. This is a no-op
until arm64 enables HVO, which selects CONFIG_VMEMMAP_OPTIMIZATION.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/mm/init.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 4e7682dfe366..7186d86afc41 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -35,6 +35,7 @@
 #include <linux/kmemleak.h>
 #include <linux/execmem.h>
 #include <linux/arm-rsi-cmds.h>
+#include <linux/vmemmap-optimization.h>
 
 
 #include <asm/boot.h>
@@ -92,11 +93,22 @@ phys_addr_t __ro_after_init arm64_dma_phys_limit;
  * in the virtual address space.
  */
 #if ARM64_MEMSTART_SHIFT < SECTION_SIZE_BITS
-#define ARM64_MEMSTART_ALIGN	(1UL << SECTION_SIZE_BITS)
+#define ARM64_MEMSTART_BASE_ALIGN	(1UL << SECTION_SIZE_BITS)
 #else
-#define ARM64_MEMSTART_ALIGN	(1UL << ARM64_MEMSTART_SHIFT)
+#define ARM64_MEMSTART_BASE_ALIGN	(1UL << ARM64_MEMSTART_SHIFT)
 #endif
 
+/*
+ * With CONFIG_VMEMMAP_OPTIMIZATION, compound_head() masks the address of a
+ * tail page to find its head, so the head struct page of every folio must be
+ * naturally aligned to the folio's vmemmap size. The vmemmap is offset by
+ * memstart_addr, so memstart_addr must be aligned to the largest folio size.
+ */
+#define ARM64_VMEMMAP_OPT_ALIGN						\
+	PFN_PHYS(MAX_FOLIO_VMEMMAP_ALIGN / sizeof(struct page))
+#define ARM64_MEMSTART_ALIGN						\
+	max_t(phys_addr_t, ARM64_MEMSTART_BASE_ALIGN, ARM64_VMEMMAP_OPT_ALIGN)
+
 static void __init arch_reserve_crashkernel(void)
 {
 	unsigned long long crash_base, crash_size, cma_size = 0;
@@ -218,6 +230,8 @@ void __init arm64_memblock_init(void)
 	/*
 	 * Select a suitable value for the base of physical memory.
 	 */
+	BUILD_BUG_ON(MAX_FOLIO_VMEMMAP_ALIGN &&
+		     !IS_ALIGNED(VMEMMAP_START, MAX_FOLIO_VMEMMAP_ALIGN));
 	memstart_addr = round_down(memblock_start_of_DRAM(),
 				   ARM64_MEMSTART_ALIGN);
 
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 12/19] arm64: Support hugetlb vmemmap optimization
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (10 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 11/19] arm64/mm: Align vmemmap to maximal folio size James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 13/19] hugetlb_vmemmap: Add fault injection for in-place vmemmap PTE updates James Houghton
                   ` (6 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

Support HVO if HW AF and BBML3 are supported. These are the required
features for the vmemmap to be modified in place.

On arm64 systems that do not have the required hardware features to
support HVO, the HVO sysctl will not appear, and any attempts to enable
HVO (either via the command line or via
CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON=y) will have no effect.

Because HVO support is only known after the vmemmap has been populated,
disable the section-based vmemmap optimization in vmemmap_populate(), as
LoongArch and powerpc do. Boot-time HugeTLB folios are then optimized
when they are gathered.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/Kconfig               | 1 +
 arch/arm64/include/asm/pgtable.h | 6 ++++++
 arch/arm64/mm/mmu.c              | 9 +++++++++
 mm/hugetlb_vmemmap.c             | 4 ++++
 4 files changed, 20 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 4cce60c2de66..b13618f0ace5 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -95,6 +95,7 @@ config ARM64
 	select ARCH_WANT_FRAME_POINTERS
 	select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36)
 	select ARCH_WANT_LD_ORPHAN_WARN
+	select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if ARM64_HW_AFDBM
 	select ARCH_WANTS_EXECMEM_LATE
 	select ARCH_WANTS_NO_INSTR
 	select ARCH_WANTS_THP_SWAP if ARM64_4K_PAGES
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 7421c0a048ff..5dadaac54bef 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1931,6 +1931,12 @@ static inline void clear_young_dirty_ptes(struct vm_area_struct *vma,
 
 #endif /* CONFIG_ARM64_CONTPTE */
 
+#define arch_support_vmemmap_remap arch_support_vmemmap_remap
+static inline bool arch_support_vmemmap_remap(void)
+{
+	return system_supports_bbm_through_af();
+}
+
 static inline int try_update_vmemmap_pte(unsigned long addr, pte_t *ptep,
 					 const pte_t pte)
 {
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index a61464cf7367..bc3eea16004c 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -30,6 +30,7 @@
 #include <linux/mm_inline.h>
 #include <linux/pagewalk.h>
 #include <linux/stop_machine.h>
+#include <linux/vmemmap-optimization.h>
 
 #include <asm/barrier.h>
 #include <asm/cputype.h>
@@ -1843,6 +1844,14 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
 	/* [start, end] should be within one section */
 	WARN_ON_ONCE(end - start > PAGES_PER_SECTION * sizeof(struct page));
 
+	/*
+	 * Whether HVO is supported is only known once the system-wide CPU
+	 * features have been finalized, which is after the vmemmap has been
+	 * populated, so the section-based vmemmap optimization (pre-HVO) is
+	 * not supported. Boot-time HugeTLB folios are optimized later instead.
+	 */
+	section_vmemmap_optimization_disable(start, end);
+
 	if (!IS_ENABLED(CONFIG_ARM64_4K_PAGES) ||
 	    (end - start < PAGES_PER_SECTION * sizeof(struct page)))
 		return vmemmap_populate_basepages(start, end, node, altmap);
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index fb6da64bd80d..a01377c37c96 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -79,6 +79,10 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
 
 		/* Make pte visible before pmd. See comment in pmd_install(). */
 		smp_wmb();
+		/*
+		 * On arm64, this requires BBML3. Its support has already been
+		 * checked.
+		 */
 		pmd_populate_kernel(&init_mm, pmd, pgtable);
 		/*
 		 * Modern MMUs allow overlapping TLB entries with different mapping
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 13/19] hugetlb_vmemmap: Add fault injection for in-place vmemmap PTE updates
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (11 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 12/19] arm64: Support hugetlb vmemmap optimization James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 14/19] selftests/mm: Add HugeTLB vmemmap optimization stress test James Houghton
                   ` (5 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

try_update_vmemmap_pte() may fail, e.g. on arm64 when the update keeps
racing with hardware access flag updates. HVO handles such failures by
rolling back the optimization, or by leaving folios partially optimized
if the rollback or a later restore fails. These paths are rare to hit
normally.

Add a fault-injection capability, fail_hugetlb_vmemmap_pte, under
CONFIG_FAIL_HUGETLB_VMEMMAP. When a fault is injected, the PTE update
fails with -EAGAIN without touching the page tables, as if the in-place
update had given up.

It can be configured through debugfs or through the
fail_hugetlb_vmemmap_pte= boot option, the latter allowing failures to
be injected when optimizing bootmem folios.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 .../admin-guide/kernel-parameters.txt         |  1 +
 .../fault-injection/fault-injection.rst       |  6 +++
 lib/Kconfig.debug                             |  9 +++++
 mm/hugetlb_vmemmap.c                          | 38 ++++++++++++++++++-
 4 files changed, 52 insertions(+), 2 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 176648941aec..b8a8771631c0 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -1694,6 +1694,7 @@ Kernel parameters
 	fail_usercopy=
 	fail_page_alloc=
 	fail_skb_realloc=
+	fail_hugetlb_vmemmap_pte=
 	fail_make_request=[KNL]
 			General fault injection mechanism.
 			Format: <interval>,<probability>,<space>,<times>
diff --git a/Documentation/fault-injection/fault-injection.rst b/Documentation/fault-injection/fault-injection.rst
index c2d3996b5b40..403206645fa4 100644
--- a/Documentation/fault-injection/fault-injection.rst
+++ b/Documentation/fault-injection/fault-injection.rst
@@ -16,6 +16,11 @@ Available fault injection capabilities
 
   injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
 
+- fail_hugetlb_vmemmap_pte
+
+  injects failures of the in-place vmemmap PTE remaps done by HugeTLB vmemmap
+  optimization. (try_update_vmemmap_pte())
+
 - fail_usercopy
 
   injects failures in user memory access functions. (copy_from_user(), get_user(), ...)
@@ -263,6 +268,7 @@ use the boot option::
 
 	failslab=
 	fail_page_alloc=
+	fail_hugetlb_vmemmap_pte=
 	fail_usercopy=
 	fail_make_request=
 	fail_futex=
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 2d22638ec13a..a6a9bc01a320 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2098,6 +2098,15 @@ config FAIL_PAGE_ALLOC
 	help
 	  Provide fault-injection capability for alloc_pages().
 
+config FAIL_HUGETLB_VMEMMAP
+	bool "Fault-injection capability for HugeTLB vmemmap optimization"
+	depends on FAULT_INJECTION && HUGETLB_PAGE_OPTIMIZE_VMEMMAP
+	help
+	  Provide fault-injection capability for the in-place vmemmap page
+	  table updates done by HugeTLB vmemmap optimization (HVO), i.e.
+	  try_update_vmemmap_pte(). This exercises the rollback and
+	  partially-optimized folio paths.
+
 config FAULT_INJECTION_USERCOPY
 	bool "Fault injection capability for usercopy functions"
 	depends on FAULT_INJECTION
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index a01377c37c96..4cacd7aee719 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -17,6 +17,7 @@
 #include <linux/pgalloc.h>
 #include <linux/rwsem.h>
 #include <linux/vmemmap-optimization.h>
+#include <linux/fault-inject.h>
 
 #include <asm/tlbflush.h>
 #include "hugetlb_vmemmap.h"
@@ -44,6 +45,39 @@ struct vmemmap_remap_walk {
 
 DECLARE_RWSEM(vmemmap_remap_rwsem);
 
+#ifdef CONFIG_FAIL_HUGETLB_VMEMMAP
+static DECLARE_FAULT_ATTR(fail_hugetlb_vmemmap_pte);
+
+static int __init setup_fail_hugetlb_vmemmap_pte(char *str)
+{
+	return setup_fault_attr(&fail_hugetlb_vmemmap_pte, str);
+}
+__setup("fail_hugetlb_vmemmap_pte=", setup_fail_hugetlb_vmemmap_pte);
+
+#ifdef CONFIG_FAULT_INJECTION_DEBUG_FS
+static int __init fail_hugetlb_vmemmap_debugfs(void)
+{
+	fault_create_debugfs_attr("fail_hugetlb_vmemmap_pte", NULL,
+				  &fail_hugetlb_vmemmap_pte);
+	return 0;
+}
+late_initcall(fail_hugetlb_vmemmap_debugfs);
+#endif /* CONFIG_FAULT_INJECTION_DEBUG_FS */
+
+/*
+ * Inject failures as if the in-place update lost a race too many times
+ * (see the arm64 implementations), without touching the page tables.
+ */
+static int hvo_update_vmemmap_pte(unsigned long addr, pte_t *ptep, pte_t pte)
+{
+	if (should_fail(&fail_hugetlb_vmemmap_pte, PAGE_SIZE))
+		return -EAGAIN;
+	return try_update_vmemmap_pte(addr, ptep, pte);
+}
+#else
+#define hvo_update_vmemmap_pte		try_update_vmemmap_pte
+#endif /* CONFIG_FAIL_HUGETLB_VMEMMAP */
+
 static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
 			     struct vmemmap_remap_walk *walk)
 {
@@ -274,7 +308,7 @@ static int vmemmap_optimize_pte(pte_t *pte, unsigned long addr,
 	 * after it. The refcount is zero, so the cmpxchg fails and leaves the
 	 * page unchanged.
 	 */
-	ret = try_update_vmemmap_pte(addr, pte, mk_pte(dst, PAGE_KERNEL));
+	ret = hvo_update_vmemmap_pte(addr, pte, mk_pte(dst, PAGE_KERNEL));
 	if (ret)
 		return ret;
 
@@ -313,7 +347,7 @@ static int vmemmap_restore_pte(pte_t *pte, unsigned long addr,
 		return 0;
 
 	dst = vmemmap_dup_page_from_list(src, walk->vmemmap_pages);
-	ret = try_update_vmemmap_pte(addr, pte, mk_pte(dst, PAGE_KERNEL));
+	ret = hvo_update_vmemmap_pte(addr, pte, mk_pte(dst, PAGE_KERNEL));
 	if (ret)
 		return ret;
 
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 14/19] selftests/mm: Add HugeTLB vmemmap optimization stress test
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (12 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 13/19] hugetlb_vmemmap: Add fault injection for in-place vmemmap PTE updates James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 15/19 DO-NOT-MERGE] hugetlb_vmemmap: Use try_populate_vmemmap_pmd for replacing in-use PMDs James Houghton
                   ` (4 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

Add a script that stresses HugeTLB vmemmap optimization (HVO), in
particular on architectures that update the vmemmap in place while it
may be concurrently accessed:

  - Check that optimizing N folios frees exactly N * (vmemmap pages per
    folio - 1) vmemmap pages (per nr_memmap_pages and
    nr_memmap_boot_pages), and that restoring them gives them back.
  - Repeatedly optimize and restore folios by resizing the hugepage pool,
    while concurrently reading struct pages through /proc/kpageflags and
    /proc/kpagecount, compacting memory, and optionally reading
    page_owner and offlining/onlining memory blocks.
  - Do the same with fail_hugetlb_vmemmap_pte fault injection enabled,
    if available, to exercise the rollback and partially-optimized folio
    paths.

After each phase, the pool must shrink back to its original size and the
memmap accounting must return to its baseline. Finally, the kernel log
must not contain warnings or oopses.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 tools/testing/selftests/mm/Makefile           |   2 +
 .../selftests/mm/hugetlb_vmemmap_stress.sh    | 380 ++++++++++++++++++
 .../selftests/mm/ksft_hugetlb_vmemmap.sh      |   4 +
 tools/testing/selftests/mm/run_vmtests.sh     |   4 +
 4 files changed, 390 insertions(+)
 create mode 100755 tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh
 create mode 100755 tools/testing/selftests/mm/ksft_hugetlb_vmemmap.sh

diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile
index 675ec57d47de..9ec348902743 100644
--- a/tools/testing/selftests/mm/Makefile
+++ b/tools/testing/selftests/mm/Makefile
@@ -150,6 +150,7 @@ TEST_PROGS += ksft_cow.sh
 TEST_PROGS += ksft_gup_test.sh
 TEST_PROGS += ksft_hmm.sh
 TEST_PROGS += ksft_hugetlb.sh
+TEST_PROGS += ksft_hugetlb_vmemmap.sh
 TEST_PROGS += ksft_hugevm.sh
 TEST_PROGS += ksft_kmemleak_confirm.sh
 TEST_PROGS += ksft_kmemleak_dedup.sh
@@ -182,6 +183,7 @@ TEST_FILES += test_hmm.sh
 TEST_FILES += va_high_addr_switch.sh
 TEST_FILES += charge_reserved_hugetlb.sh
 TEST_FILES += hugetlb_reparenting_test.sh
+TEST_FILES += hugetlb_vmemmap_stress.sh
 TEST_FILES += test_page_frag.sh
 TEST_FILES += run_vmtests.sh
 
diff --git a/tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh b/tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh
new file mode 100755
index 000000000000..b64fdb3ed761
--- /dev/null
+++ b/tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh
@@ -0,0 +1,380 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# Stress test for HugeTLB vmemmap optimization (HVO).
+#
+# Phases:
+#   1. accounting: allocate N hugepages, check that nr_memmap_pages +
+#      nr_memmap_boot_pages drops by exactly N * (vmemmap pages per folio - 1),
+#      then free them and check it returns to the baseline.
+#   2. stress: churn the pool (optimize/restore) while concurrently reading
+#      struct pages (/proc/kpageflags, /proc/kpagecount), compacting memory,
+#      and optionally reading page_owner and offlining/onlining memory.
+#   3. pte-inject: like 2, with fail_hugetlb_vmemmap_pte enabled; this hits
+#      both the optimize rollback and the restore (partial HVO) paths.
+#
+# After each phase the pool is drained back to its original size, memmap
+# accounting must be back at the phase's baseline, and the kernel log must
+# not contain warnings/oopses.
+#
+# The fault-injection phases require CONFIG_FAIL_HUGETLB_VMEMMAP and
+# CONFIG_FAULT_INJECTION_DEBUG_FS, and are skipped otherwise.
+
+set -u
+
+KSFT_PASS=0
+KSFT_FAIL=1
+KSFT_SKIP=4
+
+size_kb=
+nr=16
+duration=60
+prob=20
+readers=4
+struct_page_size=64
+do_page_owner=0
+do_hotplug=0
+
+usage() {
+	cat <<EOF
+Usage: $0 [options]
+  -s KB     hugepage size in kB (default: default hugepage size)
+  -n N      number of hugepages to churn (default: $nr)
+  -t SEC    duration of each stress phase (default: $duration)
+  -p PCT    fault-injection probability in percent (default: $prob)
+  -j N      number of struct page reader processes (default: $readers)
+  -S BYTES  sizeof(struct page) (default: $struct_page_size)
+  -o        also read /sys/kernel/debug/page_owner during stress
+  -m        also offline/online memory blocks during stress (dissolves
+            free hugepages, exercising the restore path)
+
+For exact accounting checks, run on a hugepage size whose pool is
+initially empty (e.g. no boot-time reservations of that size).
+EOF
+	exit $KSFT_SKIP
+}
+
+while getopts "s:n:t:p:j:S:omh" opt; do
+	case $opt in
+	s) size_kb=$OPTARG ;;
+	n) nr=$OPTARG ;;
+	t) duration=$OPTARG ;;
+	p) prob=$OPTARG ;;
+	j) readers=$OPTARG ;;
+	S) struct_page_size=$OPTARG ;;
+	o) do_page_owner=1 ;;
+	m) do_hotplug=1 ;;
+	*) usage ;;
+	esac
+done
+
+log() { echo "# $*"; }
+skip() { echo "SKIP: $*"; exit $KSFT_SKIP; }
+
+failures=0
+fail() { echo "FAIL: $*"; failures=$((failures + 1)); }
+pass() { echo "PASS: $*"; }
+
+[ "$(id -u)" -eq 0 ] || skip "must be run as root"
+
+[ -r /proc/sys/vm/hugetlb_optimize_vmemmap ] ||
+	skip "HVO not supported (no vm.hugetlb_optimize_vmemmap)"
+[ "$(cat /proc/sys/vm/hugetlb_optimize_vmemmap)" = 1 ] ||
+	skip "HVO disabled (vm.hugetlb_optimize_vmemmap=0)"
+grep -q '^nr_memmap_pages ' /proc/vmstat ||
+	skip "no nr_memmap_pages in /proc/vmstat"
+
+[ -n "$size_kb" ] || size_kb=$(awk '/^Hugepagesize:/ {print $2}' /proc/meminfo)
+hp_dir=/sys/kernel/mm/hugepages/hugepages-${size_kb}kB
+[ -d "$hp_dir" ] || skip "no ${size_kb}kB hugepages"
+
+page_size=$(getconf PAGESIZE)
+vmemmap_pages=$(( size_kb * 1024 / page_size * struct_page_size / page_size ))
+freed_per_folio=$(( vmemmap_pages - 1 ))
+[ "$freed_per_folio" -gt 0 ] ||
+	skip "${size_kb}kB hugepages are not HVO-optimizable"
+
+dbgfs=/sys/kernel/debug
+mountpoint -q $dbgfs || mount -t debugfs none $dbgfs 2>/dev/null
+
+orig_nr=$(cat "$hp_dir/nr_hugepages")
+target_nr=$(( orig_nr + nr ))
+marker="hvo-stress-$$-$(date +%s)"
+tmpdir=$(mktemp -d)
+pids=()
+
+log "hugepage size ${size_kb}kB, page size $page_size"
+log "struct page size $struct_page_size"
+log "vmemmap pages/folio: $vmemmap_pages ($freed_per_folio freed by HVO)"
+log "pool: $orig_nr initially, churning between $orig_nr and $target_nr"
+[ "$orig_nr" -eq 0 ] ||
+	log "WARNING: pool not initially empty, accounting may be inexact"
+
+memmap_total() {
+	awk '/^nr_memmap_(boot_)?pages / {s += $2} END {print s + 0}' \
+		/proc/vmstat
+}
+
+set_nr() {
+	echo "$1" > "$hp_dir/nr_hugepages" 2>/dev/null
+	cat "$hp_dir/nr_hugepages"
+}
+
+# Shrink the pool back to orig_nr. Restore may transiently fail (e.g. with
+# fault injection active), so retry for a while.
+drain() {
+	local i cur surplus
+
+	for i in $(seq 20); do
+		# Pages whose vmemmap could not be restored are kept as free
+		# surplus pages, which shrinking nr_hugepages does not free.
+		# Writing the current size converts them back to persistent
+		# pages first.
+		set_nr "$(cat "$hp_dir/nr_hugepages")" > /dev/null
+		cur=$(set_nr "$orig_nr")
+		[ "$cur" -eq "$orig_nr" ] && return 0
+		sleep 1
+	done
+	surplus=$(cat "$hp_dir/surplus_hugepages")
+	fail "pool stuck at $cur hugepages ($surplus surplus)," \
+		"expected $orig_nr"
+	return 1
+}
+
+fa_dir() { echo "$dbgfs/$1"; }
+
+fa_enable() {
+	local d
+	d=$(fa_dir "$1")
+	echo 0 > "$d/verbose"
+	echo N > "$d/task-filter"
+	echo 1 > "$d/interval"
+	echo 1000000 > "$d/times"
+	echo "$prob" > "$d/probability"
+}
+
+# Disable and print the number of injected failures.
+fa_disable() {
+	local d left
+	d=$(fa_dir "$1")
+	echo 0 > "$d/probability"
+	left=$(cat "$d/times")
+	echo 0 > "$d/times"
+	echo $(( 1000000 - left ))
+}
+
+check_dmesg() {
+	local bad pat
+
+	pat='WARNING:|BUG[: ]|Oops|Unable to handle kernel'
+	pat+='|Internal error|KASAN:|UBSAN:'
+	pat+='|list_(add|del) corruption|page dumped because'
+	bad=$(dmesg | sed -n "/$marker/,\$p" | grep -E "$pat")
+	# Offlining a memory block that contains unmovable pages fails and
+	# dumps the offending page; that is expected with -m.
+	[ "$do_hotplug" -eq 1 ] &&
+		bad=$(echo "$bad" | grep -v 'page dumped because: unmovable page')
+	if [ -n "$bad" ]; then
+		fail "kernel log reports problems:"
+		echo "$bad" | head -20 | sed 's/^/#   /'
+	fi
+}
+
+## Workers
+
+churn() {
+	while :; do
+		set_nr "$target_nr" > /dev/null
+		set_nr "$orig_nr" > /dev/null
+	done
+}
+
+kpage_reader() {
+	while :; do
+		dd if=/proc/kpageflags of=/dev/null bs=4M status=none
+		dd if=/proc/kpagecount of=/dev/null bs=4M status=none
+	done
+}
+
+compactor() {
+	while :; do
+		echo 1 > /proc/sys/vm/compact_memory
+		sleep 1
+	done
+}
+
+page_owner_reader() {
+	while :; do
+		cat $dbgfs/page_owner > /dev/null
+	done
+}
+
+hotplugger() {
+	local blk state removable
+
+	while :; do
+		for blk in /sys/devices/system/memory/memory*; do
+			state=$(cat "$blk/state" 2>/dev/null)
+			removable=$(cat "$blk/removable" 2>/dev/null || echo 1)
+			[ "$state" = online ] || continue
+			[ "$removable" = 1 ] || continue
+			echo "$blk" >> "$tmpdir/hotplug"
+			# A signal aborts a pending offline_pages(). Use
+			# --foreground so that the writer stays in the worker's
+			# process group and is killed by stop_workers().
+			timeout --foreground 10 \
+				sh -c "echo offline > $blk/state" 2>/dev/null
+			echo online > "$blk/state" 2>/dev/null
+			sleep 1
+		done
+	done
+}
+
+start_workers() {
+	local i
+
+	# Start each worker in its own process group (job control), so that
+	# stop_workers() can kill the commands it is running as well.
+	set -m
+	churn & pids+=($!)
+	for i in $(seq "$readers"); do
+		kpage_reader & pids+=($!)
+	done
+	compactor & pids+=($!)
+	if [ "$do_page_owner" -eq 1 ]; then
+		if [ -r $dbgfs/page_owner ]; then
+			page_owner_reader & pids+=($!)
+		else
+			log "page_owner not available, not reading it"
+		fi
+	fi
+	[ "$do_hotplug" -eq 1 ] && { hotplugger & pids+=($!); }
+	set +m
+}
+
+# Stop the workers, including any command they are running (e.g. a dd of
+# /proc/kpageflags), so that nothing touches struct pages after a phase ends.
+stop_workers() {
+	local pid i alive
+
+	[ "${#pids[@]}" -gt 0 ] || return 0
+	for pid in "${pids[@]}"; do
+		kill -TERM -- "-$pid" 2>/dev/null
+	done
+	for i in $(seq 30); do
+		alive=0
+		for pid in "${pids[@]}"; do
+			kill -0 -- "-$pid" 2>/dev/null && alive=1
+		done
+		[ "$alive" -eq 0 ] && break
+		sleep 1
+	done
+	if [ "$alive" -eq 1 ]; then
+		log "WARNING: workers still running, killing them"
+		for pid in "${pids[@]}"; do
+			kill -KILL -- "-$pid" 2>/dev/null
+		done
+	fi
+	# Reap the workers only once they are gone, so that this cannot block.
+	wait "${pids[@]}" 2>/dev/null
+	pids=()
+}
+
+cleanup() {
+	local blk t
+
+	stop_workers
+	for t in fail_hugetlb_vmemmap_pte; do
+		[ -d "$(fa_dir $t)" ] && fa_disable $t > /dev/null
+	done
+	if [ -f "$tmpdir/hotplug" ]; then
+		sort -u "$tmpdir/hotplug" | while read -r blk; do
+			[ "$(cat "$blk/state")" = online ] ||
+				echo online > "$blk/state" 2>/dev/null
+		done
+	fi
+	# drain() also frees free surplus pages left by failed restores.
+	drain > /dev/null
+	rm -rf "$tmpdir"
+}
+trap cleanup EXIT
+trap 'exit $KSFT_FAIL' INT TERM
+
+# Run a stress phase. $1: name, $2: fault attr to enable ("" for none).
+stress_phase() {
+	local name=$1 fa=$2 base after injected
+
+	if [ -n "$fa" ] && [ ! -d "$(fa_dir "$fa")" ]; then
+		echo "SKIP: $name (no $(fa_dir "$fa"))"
+		return
+	fi
+
+	log "phase $name: ${duration}s"
+	base=$(memmap_total)
+	[ -n "$fa" ] && fa_enable "$fa"
+	start_workers
+	sleep "$duration"
+	stop_workers
+	if [ -n "$fa" ]; then
+		injected=$(fa_disable "$fa")
+		log "$name: injected $injected failures"
+		[ "$injected" -gt 0 ] ||
+			log "WARNING: $name: no failures injected"
+	fi
+
+	drain || return
+	after=$(memmap_total)
+	if [ "$after" -ne "$base" ]; then
+		fail "$name: memmap pages $after after drain, expected $base"
+	else
+		pass "$name"
+	fi
+}
+
+accounting_phase() {
+	local base got added after expect
+
+	log "phase accounting"
+	base=$(memmap_total)
+	got=$(set_nr "$target_nr")
+	added=$(( got - orig_nr ))
+	if [ "$added" -le 0 ]; then
+		fail "accounting: could not allocate any ${size_kb}kB hugepages"
+		return
+	fi
+	[ "$added" -eq "$nr" ] || log "accounting: only allocated $added of $nr"
+
+	after=$(memmap_total)
+	expect=$(( base - added * freed_per_folio ))
+	if [ "$after" -ne "$expect" ]; then
+		fail "accounting: memmap pages $after after optimizing" \
+			"$added folios, expected $expect (baseline $base)"
+	else
+		pass "accounting: optimize freed $(( base - after ))" \
+			"vmemmap pages"
+	fi
+
+	drain || return
+	after=$(memmap_total)
+	if [ "$after" -ne "$base" ]; then
+		fail "accounting: memmap pages $after after restore," \
+			"expected $base"
+	else
+		pass "accounting: restore"
+	fi
+}
+
+echo "$marker" > /dev/kmsg
+
+accounting_phase
+stress_phase stress ""
+stress_phase pte-inject fail_hugetlb_vmemmap_pte
+
+check_dmesg
+
+if [ "$failures" -ne 0 ]; then
+	echo "FAILED: $failures check(s)"
+	exit $KSFT_FAIL
+fi
+echo "OK"
+exit $KSFT_PASS
diff --git a/tools/testing/selftests/mm/ksft_hugetlb_vmemmap.sh b/tools/testing/selftests/mm/ksft_hugetlb_vmemmap.sh
new file mode 100755
index 000000000000..905b75b2cfb4
--- /dev/null
+++ b/tools/testing/selftests/mm/ksft_hugetlb_vmemmap.sh
@@ -0,0 +1,4 @@
+#!/bin/sh -e
+# SPDX-License-Identifier: GPL-2.0
+
+./run_vmtests.sh -t hugetlb_vmemmap
diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh
index 973cb715b326..f439affe5d70 100755
--- a/tools/testing/selftests/mm/run_vmtests.sh
+++ b/tools/testing/selftests/mm/run_vmtests.sh
@@ -79,6 +79,8 @@ separated by spaces:
 	test transparent huge pages
 - hugetlb
 	test hugetlbfs huge pages
+- hugetlb_vmemmap
+	test the hugetlb vmemmap optimization
 - migration
 	invoke move_pages(2) to exercise the migration entry code
 	paths in the kernel
@@ -327,6 +329,8 @@ echo "$enable_soft_offline" > /proc/sys/vm/enable_soft_offline
 CATEGORY="hugetlb" run_test ./hugetlb-read-hwpoison
 fi
 
+CATEGORY="hugetlb_vmemmap" run_test ./hugetlb_vmemmap_stress.sh
+
 if [ $VADDR64 -ne 0 ]; then
 	# va high address boundary switch test
 	CATEGORY="hugevm" run_test bash ./va_high_addr_switch.sh
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 15/19 DO-NOT-MERGE] hugetlb_vmemmap: Use try_populate_vmemmap_pmd for replacing in-use PMDs
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (13 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 14/19] selftests/mm: Add HugeTLB vmemmap optimization stress test James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 16/19 DO-NOT-MERGE] arm64: Implement try_populate_vmemmap_pmd using AF trick James Houghton
                   ` (3 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

This routine is to be used for updating in-use, leaf-level PMDs in the
vmemmap without introducing a window where vmemmap accesses might fault.

Because try_populate_vmemmap_pmd() can fail, the split_page() call needs
to be moved to after the PMD update. Because both split_page() and the
PMD update are done under the page table lock, this rearrangement is
safe.

Only leaf-level PMD to non-leaf PMD (mapping the same physical pages) is
the only transition that needs to be supported.

Collapsing a non-leaf PMD to a leaf-level PMD is not used by HVO, so
try_populate_vmemmap_pmd() need not support it.

This patch allows arm64's implementation to replaced with one that does
not require BBML3.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/include/asm/pgalloc.h     |  8 ++++++++
 arch/loongarch/include/asm/pgalloc.h |  2 ++
 arch/riscv/include/asm/pgalloc.h     |  2 ++
 arch/x86/include/asm/pgalloc.h       |  2 ++
 include/linux/pgalloc.h              | 21 +++++++++++++++++++++
 mm/hugetlb_vmemmap.c                 | 25 ++++++++++++++-----------
 6 files changed, 49 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h
index 1b4509d3382c..0263329fb7b3 100644
--- a/arch/arm64/include/asm/pgalloc.h
+++ b/arch/arm64/include/asm/pgalloc.h
@@ -121,4 +121,12 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep)
 		       PMD_TYPE_TABLE | PMD_TABLE_AF | PMD_TABLE_PXN);
 }
 
+static inline int try_populate_vmemmap_pmd(unsigned long addr, pmd_t *pmdp,
+					   pte_t *pgtable)
+{
+	/* BBLM3 is required. Its presence has been checked. */
+	pmd_populate_kernel(&init_mm, pmdp, pgtable);
+	return 0;
+}
+
 #endif
diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h
index 06a36da302d3..e37c94e6d929 100644
--- a/arch/loongarch/include/asm/pgalloc.h
+++ b/arch/loongarch/include/asm/pgalloc.h
@@ -21,6 +21,8 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, pgtable_t pte)
 	set_pmd(pmd, __pmd((unsigned long)page_address(pte)));
 }
 
+#define ARCH_WANTS_GENERIC_POPULATE_VMEMMAP_PMD
+
 #ifndef __PAGETABLE_PMD_FOLDED
 
 static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h
index 0133f7ba41d7..ad76df0b811a 100644
--- a/arch/riscv/include/asm/pgalloc.h
+++ b/arch/riscv/include/asm/pgalloc.h
@@ -31,6 +31,8 @@ static inline void pmd_populate(struct mm_struct *mm,
 	set_pmd(pmd, __pmd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE));
 }
 
+#define ARCH_WANTS_GENERIC_POPULATE_VMEMMAP_PMD
+
 #ifndef __PAGETABLE_PMD_FOLDED
 static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
 {
diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h
index c88691b15f3c..f8d24b97cbf5 100644
--- a/arch/x86/include/asm/pgalloc.h
+++ b/arch/x86/include/asm/pgalloc.h
@@ -82,6 +82,8 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,
 	set_pmd(pmd, __pmd(((pteval_t)pfn << PAGE_SHIFT) | _PAGE_TABLE));
 }
 
+#define ARCH_WANTS_GENERIC_POPULATE_VMEMMAP_PMD
+
 #if CONFIG_PGTABLE_LEVELS > 2
 extern void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd);
 
diff --git a/include/linux/pgalloc.h b/include/linux/pgalloc.h
index 9174fa59bbc5..1dbecb748028 100644
--- a/include/linux/pgalloc.h
+++ b/include/linux/pgalloc.h
@@ -26,4 +26,25 @@
 			arch_sync_kernel_mappings(addr, addr);		\
 	} while (0)
 
+#ifdef ARCH_WANTS_GENERIC_POPULATE_VMEMMAP_PMD
+/*
+ * try_populate_vmemmap_pmd - Populate a PMD that is in use by the vmemmap.
+ * @addr: Base address of the remapped PMD.
+ * @pmdp: Page table pointer to be overwritten.
+ * @pgtable: Pointer to the page table that the new PMD will point to.
+ *
+ * This function is only to be used to update PMDs that map the vmemmap to
+ * point to a page of already-populated PTEs that map the same pages.
+ *
+ * Implementations of this function must ensure that, while the update is taking
+ * place, CPUs will not fault on the remapped virtual address range.
+ */
+static inline int try_populate_vmemmap_pmd(unsigned long addr, pmd_t *pmdp,
+					   pte_t *pgtable)
+{
+	pmd_populate_kernel(&init_mm, pmdp, pgtable);
+	return 0;
+}
+#endif
+
 #endif /* _LINUX_PGALLOC_H */
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 4cacd7aee719..d3230588f31f 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -82,6 +82,7 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
 			     struct vmemmap_remap_walk *walk)
 {
 	pmd_t __pmd;
+	int ret;
 	int i;
 	unsigned long addr = start;
 	pte_t *pgtable;
@@ -101,8 +102,15 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
 		set_pte_at(&init_mm, addr, pte, entry);
 	}
 
+	ret = 0;
 	spin_lock(&init_mm.page_table_lock);
 	if (likely(pmd_leaf(*pmd))) {
+		/* Make pte visible before pmd. See comment in pmd_install(). */
+		smp_wmb();
+		ret = try_populate_vmemmap_pmd(start, pmd, pgtable);
+		if (ret)
+			goto free;
+
 		/*
 		 * Higher order allocations from buddy allocator must be able to
 		 * be treated as independent small pages (as they can be freed
@@ -111,13 +119,6 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
 		if (!PageReserved(head))
 			split_page(head, get_order(PMD_SIZE));
 
-		/* Make pte visible before pmd. See comment in pmd_install(). */
-		smp_wmb();
-		/*
-		 * On arm64, this requires BBML3. Its support has already been
-		 * checked.
-		 */
-		pmd_populate_kernel(&init_mm, pmd, pgtable);
 		/*
 		 * Modern MMUs allow overlapping TLB entries with different mapping
 		 * granularities only if they map the same physical address. So
@@ -125,12 +126,14 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
 		 */
 		if (walk->remap_pte || !(walk->flags & VMEMMAP_DEFER_TLB_FLUSH))
 			flush_tlb_kernel_range(start, start + PMD_SIZE);
-	} else {
-		pte_free_kernel(&init_mm, pgtable);
+		goto out;
 	}
-	spin_unlock(&init_mm.page_table_lock);
 
-	return 0;
+free:
+	pte_free_kernel(&init_mm, pgtable);
+out:
+	spin_unlock(&init_mm.page_table_lock);
+	return ret;
 }
 
 static int vmemmap_pmd_entry(pmd_t *pmd, unsigned long addr,
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 16/19 DO-NOT-MERGE] arm64: Implement try_populate_vmemmap_pmd using AF trick
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (14 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 15/19 DO-NOT-MERGE] hugetlb_vmemmap: Use try_populate_vmemmap_pmd for replacing in-use PMDs James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 17/19 DO-NOT-MERGE] arm64: Drop BBML3 requirement for HVO James Houghton
                   ` (2 subsequent siblings)
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

This routine is to be used for updating in-use, leaf-level PMDs in the
vmemmap without introducing a window where vmemmap accesses might fault.

Implementing this on arm64 requires some care: use the same access flag
trick that is used for vmemmap PTE updates. HAFT is not needed and the
TLB flushing routine remains identical, as we are not overwriting a
non-leaf PMD.

For systems that support BBML3, there is no need to use the AF
trick.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/include/asm/pgalloc.h | 51 ++++++++++++++++++++++++++++++--
 1 file changed, 48 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h
index 0263329fb7b3..7e304f01ce6d 100644
--- a/arch/arm64/include/asm/pgalloc.h
+++ b/arch/arm64/include/asm/pgalloc.h
@@ -124,9 +124,54 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep)
 static inline int try_populate_vmemmap_pmd(unsigned long addr, pmd_t *pmdp,
 					   pte_t *pgtable)
 {
-	/* BBLM3 is required. Its presence has been checked. */
-	pmd_populate_kernel(&init_mm, pmdp, pgtable);
-	return 0;
+	const int max_attempts = 16;
+	int attempts = 0;
+	pmd_t old_pmd, new_pmd;
+
+	if (!system_supports_bbm_through_af())
+		return -EOPNOTSUPP;
+
+	if (system_supports_bbml3()) {
+		/*
+		 * BBML3 allows block->table transitions if the PTEs underneath
+		 * do not conflict with existing, potentially cached
+		 * translations.
+		 */
+		pmd_populate_kernel(&init_mm, pmdp, pgtable);
+		return 0;
+	}
+
+	new_pmd = __pmd(__phys_to_pmd_val(__pa(pgtable)) |
+			PMD_TYPE_TABLE | PMD_TABLE_AF | PMD_TABLE_UXN);
+
+	old_pmd = pmdp_get(pmdp);
+
+	do {
+		if (WARN_ON_ONCE(!pmd_valid(old_pmd)))
+			return -EINVAL;
+
+		if (WARN_ON_ONCE(!pmd_leaf(old_pmd)))
+			return -EINVAL;
+
+		/* We should never get a contiguous PMD here. */
+		if (WARN_ON_ONCE(pmd_cont(old_pmd)))
+			return -EINVAL;
+
+		if (pmd_young(old_pmd)) {
+			/* __ptep_clear_young() returns the overwritten PTE */
+			old_pmd = pte_pmd(pte_mkold(__ptep_clear_young((pte_t *)pmdp)));
+
+			flush_tlb_kernel_range(addr, addr + PMD_SIZE);
+		}
+	/*
+	 * Translations without AF cannot be cached, so we can replace
+	 * them without BBM.
+	 */
+	} while (!try_cmpxchg_relaxed(&pmd_val(*pmdp), &pmd_val(old_pmd),
+				      pmd_val(new_pmd)) &&
+		 ++attempts < max_attempts);
+
+	return attempts == max_attempts ? -EAGAIN : 0;
 }
 
 #endif
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 17/19 DO-NOT-MERGE] arm64: Drop BBML3 requirement for HVO
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (15 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 16/19 DO-NOT-MERGE] arm64: Implement try_populate_vmemmap_pmd using AF trick James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 18/19 DO-NOT-MERGE] hugetlb_vmemmap: Add fault injection for in-place vmemmap PMD splits James Houghton
  2026-11-22 18:44 ` [PATCH v3 19/19 DO-NOT-MERGE] selftests/mm: Add HVO pmd-split fault injection tests James Houghton
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

With PMD updates being done with the Access Flag trick, BBML3 is no
longer needed.

Dropping this requirement vastly widens the pool of systems this
optimization is valid for.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/kernel/cpufeature.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index d241638b5143..6281769f4df9 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -2062,12 +2062,11 @@ static bool has_bbm_through_af(const struct arm64_cpu_capabilities *caps, int sc
 	}
 
 	/*
-	 * We need BBML3 to support Block -> Table transitions without taking
-	 * faults, and we need HW AF support to support changing the OA without
-	 * taking faults. has_cpuid_feature() checks HAFDBS >= AF, reading the
-	 * local CPU's register when verifying a late CPU.
+	 * We need HW AF support to support changing the vmemmap mapping level
+	 * and OA without taking faults. has_cpuid_feature() checks HAFDBS >= AF,
+	 * reading the local CPU's register when verifying a late CPU.
 	 */
-	return cpu_supports_bbml3() && has_cpuid_feature(caps, scope);
+	return has_cpuid_feature(caps, scope);
 }
 
 static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused)
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 18/19 DO-NOT-MERGE] hugetlb_vmemmap: Add fault injection for in-place vmemmap PMD splits
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (16 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 17/19 DO-NOT-MERGE] arm64: Drop BBML3 requirement for HVO James Houghton
@ 2026-11-22 18:44 ` James Houghton
  2026-11-22 18:44 ` [PATCH v3 19/19 DO-NOT-MERGE] selftests/mm: Add HVO pmd-split fault injection tests James Houghton
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

Like try_update_vmemmap_pte(), try_populate_vmemmap_pmd() may fail,
e.g. on arm64 when replacing the block mapping keeps racing with
hardware access flag updates. HVO handles such failures by not
optimizing the folio.

Add a fault-injection capability, fail_hugetlb_vmemmap_pmd, alongside
fail_hugetlb_vmemmap_pte. When a fault is injected, the PMD split fails
with -EAGAIN without touching the page tables.

It can be configured through debugfs or through the
fail_hugetlb_vmemmap_pmd= boot option. Since vmemmap PMDs are only split
once, the boot option is the most effective way to exercise this path.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 .../admin-guide/kernel-parameters.txt         |  1 +
 .../fault-injection/fault-injection.rst       |  8 +++++---
 lib/Kconfig.debug                             |  4 ++--
 mm/hugetlb_vmemmap.c                          | 20 ++++++++++++++++++-
 4 files changed, 27 insertions(+), 6 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index b8a8771631c0..251bd0f4afa3 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -1695,6 +1695,7 @@ Kernel parameters
 	fail_page_alloc=
 	fail_skb_realloc=
 	fail_hugetlb_vmemmap_pte=
+	fail_hugetlb_vmemmap_pmd=
 	fail_make_request=[KNL]
 			General fault injection mechanism.
 			Format: <interval>,<probability>,<space>,<times>
diff --git a/Documentation/fault-injection/fault-injection.rst b/Documentation/fault-injection/fault-injection.rst
index 403206645fa4..5cfabea811da 100644
--- a/Documentation/fault-injection/fault-injection.rst
+++ b/Documentation/fault-injection/fault-injection.rst
@@ -16,10 +16,11 @@ Available fault injection capabilities
 
   injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
 
-- fail_hugetlb_vmemmap_pte
+- fail_hugetlb_vmemmap_pte, fail_hugetlb_vmemmap_pmd
 
-  injects failures of the in-place vmemmap PTE remaps done by HugeTLB vmemmap
-  optimization. (try_update_vmemmap_pte())
+  injects failures of the in-place vmemmap PTE remaps and PMD splits done by
+  HugeTLB vmemmap optimization. (try_update_vmemmap_pte(),
+  try_populate_vmemmap_pmd())
 
 - fail_usercopy
 
@@ -269,6 +270,7 @@ use the boot option::
 	failslab=
 	fail_page_alloc=
 	fail_hugetlb_vmemmap_pte=
+	fail_hugetlb_vmemmap_pmd=
 	fail_usercopy=
 	fail_make_request=
 	fail_futex=
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index a6a9bc01a320..68244c0e4a81 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2104,8 +2104,8 @@ config FAIL_HUGETLB_VMEMMAP
 	help
 	  Provide fault-injection capability for the in-place vmemmap page
 	  table updates done by HugeTLB vmemmap optimization (HVO), i.e.
-	  try_update_vmemmap_pte(). This exercises the rollback and
-	  partially-optimized folio paths.
+	  try_update_vmemmap_pte() and try_populate_vmemmap_pmd(). This
+	  exercises the rollback and partially-optimized folio paths.
 
 config FAULT_INJECTION_USERCOPY
 	bool "Fault injection capability for usercopy functions"
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index d3230588f31f..95a42f59c16b 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -47,6 +47,7 @@ DECLARE_RWSEM(vmemmap_remap_rwsem);
 
 #ifdef CONFIG_FAIL_HUGETLB_VMEMMAP
 static DECLARE_FAULT_ATTR(fail_hugetlb_vmemmap_pte);
+static DECLARE_FAULT_ATTR(fail_hugetlb_vmemmap_pmd);
 
 static int __init setup_fail_hugetlb_vmemmap_pte(char *str)
 {
@@ -54,11 +55,19 @@ static int __init setup_fail_hugetlb_vmemmap_pte(char *str)
 }
 __setup("fail_hugetlb_vmemmap_pte=", setup_fail_hugetlb_vmemmap_pte);
 
+static int __init setup_fail_hugetlb_vmemmap_pmd(char *str)
+{
+	return setup_fault_attr(&fail_hugetlb_vmemmap_pmd, str);
+}
+__setup("fail_hugetlb_vmemmap_pmd=", setup_fail_hugetlb_vmemmap_pmd);
+
 #ifdef CONFIG_FAULT_INJECTION_DEBUG_FS
 static int __init fail_hugetlb_vmemmap_debugfs(void)
 {
 	fault_create_debugfs_attr("fail_hugetlb_vmemmap_pte", NULL,
 				  &fail_hugetlb_vmemmap_pte);
+	fault_create_debugfs_attr("fail_hugetlb_vmemmap_pmd", NULL,
+				  &fail_hugetlb_vmemmap_pmd);
 	return 0;
 }
 late_initcall(fail_hugetlb_vmemmap_debugfs);
@@ -74,8 +83,17 @@ static int hvo_update_vmemmap_pte(unsigned long addr, pte_t *ptep, pte_t pte)
 		return -EAGAIN;
 	return try_update_vmemmap_pte(addr, ptep, pte);
 }
+
+static int hvo_populate_vmemmap_pmd(unsigned long addr, pmd_t *pmdp,
+				    pte_t *pgtable)
+{
+	if (should_fail(&fail_hugetlb_vmemmap_pmd, PMD_SIZE))
+		return -EAGAIN;
+	return try_populate_vmemmap_pmd(addr, pmdp, pgtable);
+}
 #else
 #define hvo_update_vmemmap_pte		try_update_vmemmap_pte
+#define hvo_populate_vmemmap_pmd	try_populate_vmemmap_pmd
 #endif /* CONFIG_FAIL_HUGETLB_VMEMMAP */
 
 static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
@@ -107,7 +125,7 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
 	if (likely(pmd_leaf(*pmd))) {
 		/* Make pte visible before pmd. See comment in pmd_install(). */
 		smp_wmb();
-		ret = try_populate_vmemmap_pmd(start, pmd, pgtable);
+		ret = hvo_populate_vmemmap_pmd(start, pmd, pgtable);
 		if (ret)
 			goto free;
 
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

* [PATCH v3 19/19 DO-NOT-MERGE] selftests/mm: Add HVO pmd-split fault injection tests
  2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
                   ` (17 preceding siblings ...)
  2026-11-22 18:44 ` [PATCH v3 18/19 DO-NOT-MERGE] hugetlb_vmemmap: Add fault injection for in-place vmemmap PMD splits James Houghton
@ 2026-11-22 18:44 ` James Houghton
  18 siblings, 0 replies; 20+ messages in thread
From: James Houghton @ 2026-11-22 18:44 UTC (permalink / raw)
  To: Will Deacon, Catalin Marinas, Muchun Song, Oscar Salvador,
	Andrew Morton
  Cc: Kefeng Wang, Nikos Nikoleris, Linu Cherian, Mark Rutland,
	David Hildenbrand, Ryan Roberts, Nanyong Sun, Yu Zhao,
	Frank van der Linden, David Rientjes, James Houghton,
	linux-kernel, linux-arm-kernel, linux-mm

Run the stress test with PMD splitting failure injection enabled (if
supported). Keep in mind that vmemmap PMDs are only ever split, never
collapsed back to PMDs, so run this step first, before the main stress
test.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh b/tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh
index b64fdb3ed761..d570df02a28d 100755
--- a/tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh
+++ b/tools/testing/selftests/mm/hugetlb_vmemmap_stress.sh
@@ -4,13 +4,16 @@
 # Stress test for HugeTLB vmemmap optimization (HVO).
 #
 # Phases:
-#   1. accounting: allocate N hugepages, check that nr_memmap_pages +
+#   1. pmd-inject: churn the pool with fail_hugetlb_vmemmap_pmd enabled.
+#      Runs first because vmemmap PMDs are only split once, so later phases
+#      would leave few PMD splits to fail.
+#   2. accounting: allocate N hugepages, check that nr_memmap_pages +
 #      nr_memmap_boot_pages drops by exactly N * (vmemmap pages per folio - 1),
 #      then free them and check it returns to the baseline.
-#   2. stress: churn the pool (optimize/restore) while concurrently reading
+#   3. stress: churn the pool (optimize/restore) while concurrently reading
 #      struct pages (/proc/kpageflags, /proc/kpagecount), compacting memory,
 #      and optionally reading page_owner and offlining/onlining memory.
-#   3. pte-inject: like 2, with fail_hugetlb_vmemmap_pte enabled; this hits
+#   4. pte-inject: like 2, with fail_hugetlb_vmemmap_pte enabled; this hits
 #      both the optimize rollback and the restore (partial HVO) paths.
 #
 # After each phase the pool is drained back to its original size, memmap
@@ -284,7 +287,7 @@ cleanup() {
 	local blk t
 
 	stop_workers
-	for t in fail_hugetlb_vmemmap_pte; do
+	for t in fail_hugetlb_vmemmap_pte fail_hugetlb_vmemmap_pmd; do
 		[ -d "$(fa_dir $t)" ] && fa_disable $t > /dev/null
 	done
 	if [ -f "$tmpdir/hotplug" ]; then
@@ -366,6 +369,7 @@ accounting_phase() {
 
 echo "$marker" > /dev/kmsg
 
+stress_phase pmd-inject fail_hugetlb_vmemmap_pmd
 accounting_phase
 stress_phase stress ""
 stress_phase pte-inject fail_hugetlb_vmemmap_pte
-- 
2.56.1.809.ga0b61035b5-goog



^ permalink raw reply related	[flat|nested] 20+ messages in thread

end of thread, other threads:[~2026-11-22 18:47 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
2026-11-22 18:44 ` [PATCH v3 01/19] arm64/pgtable: Clear AF with LDCLR on supported systems James Houghton
2026-11-22 18:44 ` [PATCH v3 02/19] hugetlb_vmemmap: Always flush TLB if needed upon PTE remapping James Houghton
2026-11-22 18:44 ` [PATCH v3 03/19] hugetlb_vmemmap: Leave pages partially HVOed upon restore failure James Houghton
2026-11-22 18:44 ` [PATCH v3 04/19] hugetlb_vmemmap: Use try_update_vmemmap_pte to update in-use PTEs James Houghton
2026-11-22 18:44 ` [PATCH v3 05/19] hugetlb_vmemmap: Allow architectures to dynamically disallow HVO James Houghton
2026-11-22 18:44 ` [PATCH v3 06/19] hugetlb: Optimize bootmem folios that were not pre-HVOed James Houghton
2026-11-22 18:44 ` [PATCH v3 07/19] hugetlb_vmemmap: Allow architectures to make HVO enablement boot-time only James Houghton
2026-11-22 18:44 ` [PATCH v3 08/19] hugetlb_vmemmap: Expose whether HVO is enabled to architecture code James Houghton
2026-11-22 18:44 ` [PATCH v3 09/19] arm64: Add bbm_through_af capability James Houghton
2026-11-22 18:44 ` [PATCH v3 10/19] arm64: Implement try_update_vmemmap_pte using the AF trick James Houghton
2026-11-22 18:44 ` [PATCH v3 11/19] arm64/mm: Align vmemmap to maximal folio size James Houghton
2026-11-22 18:44 ` [PATCH v3 12/19] arm64: Support hugetlb vmemmap optimization James Houghton
2026-11-22 18:44 ` [PATCH v3 13/19] hugetlb_vmemmap: Add fault injection for in-place vmemmap PTE updates James Houghton
2026-11-22 18:44 ` [PATCH v3 14/19] selftests/mm: Add HugeTLB vmemmap optimization stress test James Houghton
2026-11-22 18:44 ` [PATCH v3 15/19 DO-NOT-MERGE] hugetlb_vmemmap: Use try_populate_vmemmap_pmd for replacing in-use PMDs James Houghton
2026-11-22 18:44 ` [PATCH v3 16/19 DO-NOT-MERGE] arm64: Implement try_populate_vmemmap_pmd using AF trick James Houghton
2026-11-22 18:44 ` [PATCH v3 17/19 DO-NOT-MERGE] arm64: Drop BBML3 requirement for HVO James Houghton
2026-11-22 18:44 ` [PATCH v3 18/19 DO-NOT-MERGE] hugetlb_vmemmap: Add fault injection for in-place vmemmap PMD splits James Houghton
2026-11-22 18:44 ` [PATCH v3 19/19 DO-NOT-MERGE] selftests/mm: Add HVO pmd-split fault injection tests James Houghton

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox