[PATCH 00/18] Another attempt at HVO support on arm64

[PATCH 00/18] Another attempt at HVO support on arm64

[James Houghton]

Hi everyone,

This patch series uses a trick with the Access Flag on CPUs that support
hardware update of the AF to update vmemmap page table entries without
introducing a time window where CPUs accessing the vmemmap might fault.

By avoiding faults, the HugeTLB vmemmap optimization (HVO) can be
implemented correctly on arm64 in a much more straightforward way than
previously attempted, most recently here[1] (please see [1] for a
breakdown of the other approaches attempted before).

For large-memory systems that allocate most of their available memory to
HugeTLB, HVO saves a huge amount of memory (1.5% of system memory).

This series has four parts:
  1. Some preparatory changes (patches 1-3)
  1. Bare minimum HVO support (patches 4-10)
  2. Drop BBML2_NOABORT requirement for HVO (patches 11-13)
  3. Drop the user-configurable Kconfig for HVO (patches 14-18)

Parts 3 and 4 are technically optional. More details below.

The main functional caveat with this series is that bootmem HugeTLB
pages are not "pre-HVOed". They will be HVOed, but because at pre-HVO
time SMP CPUs have not been enabled, we cannot query for full system
support.

This series is based on 7.2-rc2 (0e35b9b6ec0f).

This series almost 100% cleanly applies to mm-new, which has some of
Muchun's HVO patches, with one trivial conflict. I imagine this series
will conflict pretty heavily with some of Muchun's other patches[2].

-- The AF trick --

The trick is that translations with the AF unset cannot be cached in the
TLB (see Rule R_DWZCQ in the Arm ARM), so they can be atomically updated
without needing a full break-before-make sequence.

So the PTE update sequence becomes:
  1. Atomically clear the AF on the existing PTE.
  2. Invalidate the TLB.
  3. cmpxchg the AF=0 PTE with the new PTE. If this fails, goto 1.

If there is a CPU on the system that does not support hardware access
flag updates, clearing the AF is problematic, as those CPUs might fault
on the vmemmap usage. Therefore, HVO compatbility checks all CPUs for HW
AF updates.

-- Application to HVO --

HVO relies on the following page table transitions:
  - When enabling HVO for a page, PMD block entries in the vmemmap are
    shattered into PMD table entries. The first PTE remains mapped
    normally (RW mapping to a real page of struct pages), but the
    remaining PTEs in the vmemmap are mapped read-only to a shared
    page of struct pages (that is, there is an OA change and a
    permissions change).
  - When disabling HVO for a page, the RO PTEs are remapped back to RW
    PTEs that point to newly reallocated pages of struct pages. The
    PMD block -> table transition is not undone.

In patches 4-10 of this series, I use the Access Flag trick to do the PTE
OA and permissions updates. We rely on BBML2_NOABORT for the PMD block
-> table transition.

In patches 11-13, I re-use the Access Flag trick to do the PMD block ->
table transition without needing BBML2_NOABORT. For systems that support
BBML2_NOABORT, the logic is unchanged.

I am aware of Linu's BBML3 patches; I've opted not to rebase onto them
for now, but I am happy to do so later.

-- Late-onlining of CPUs that do not support HW AF --

One of the complications with this series is how to handle late-onlining
of CPUs that do not have HW AF when HVO is in use. Naively, if HVO (HW
AF) is supported on all boot CPUs and the kernel is compiled with HVO
support, late CPUs that do not support HVO will not be onlined. This is
a regression.

This series provides two ways of dealing with this. First, add a
default-off Kconfig for users to enable HVO support, avoiding the
regression. This is not ideal.

Patches 14-18 get rid of the new Kconfig by allowing onlining of
HVO-incompatible late CPUs as long as HVO is not actively in use.

-- Litmus test --

The following Herd litmus test demonstrates the PTE update routine:

  AArch64 TTDFaultlessUpdate
  Variant=vmsa
  TTHM=HA
  {
   uint64_t x=1;
   uint64_t y=2;
   [PTE(x)]=(oa:PA(x), af:1);
   0:X0=PTE(x); 1:X0=PTE(x);
   0:X1=x; 1:X1=x;
   pteval_t 0:X2=(oa:PA(x), af:0);
   pteval_t 0:X3=(oa:PA(y), af:1);
  }
   P0                | P1             ;
   LDR X4,[X0]       | L0:            ;
   MOV X5,X4         | LDR X2,[X1]    ;
   CAS X4,X2,[X0]    |                ;
   DSB ISHST         |                ;
   LSR X9,X1,#12     |                ;
   TLBI VAALE1IS,X9  |                ;
   DSB ISH           |                ;
   ISB               |                ;
   CAS X2,X3,[X0]    |                ;
  exists
    0:X5=0:X4 /\ (* First CAS must succeed *)
    (fault(P1:L0) \/ ~(1:X2=2 \/ 1:X2=1))

  (* This test should not violate BBM requirements. *)

This test must be run with herdtools with Nikos's changes[3] to more
accurately model BBM requirements. When tried, the output will notably
*not* contain the "Warning-BBM-expected" flag.

-- Testing --

I haven't yet done extensive testing of this series. HVO is correctly
freeing pages on my system, and the hugetlb-vmemmap test passes. Freeing
HVOed HugeTLB pages also seems to function normally.

[1] https://lore.kernel.org/linux-arm-kernel/20241107202033.2721681-1-yuzhao@google.com/
[2] https://lore.kernel.org/linux-mm/20260513130542.35604-1-songmuchun@bytedance.com/
[3] https://github.com/herd/herdtools7/pull/1864

James Houghton (18):
  hugetlb_vmemmap: Always flush TLB if needed upon PTE remapping
  hugetlb_vmemmap: Move vmemmap_get_tail up
  hugetlb_vmemmap: Leave pages partially HVOed upon restore failure
  hugetlb_vmemmap: Use try_update_vmemmap_pte to update in-use PTEs
  hugetlb_vmemmap: Allow architectures not to allow HVO at runtime
  arm64: Rename cpu_has_hw_af to system_has_hw_af
  arm64: Add system_supports_hvo
  arm64: Implement try_update_vmemmap_pte using the AF trick
  arm64: Prevent HVO if the HVO system feature is not enabled
  arm64: Support hugetlb vmemmap optimization
  hugetlb_vmemmap: Use try_populate_vmemmap_pmd for replacing in-use
    PMDs
  arm64: Implement try_populate_vmemmap_pmd using AF trick
  arm64: Drop BBML2_NOABORT requirement for HVO
  hugetlb_vmemmap: Rename mm/hugetlb_vmemmap.h to
    mm/hugetlb_vmemmap_internal.h
  hugetlb_vmemmap: Add a way to permanently disable HVO when needed
  arm64: Allow "optional" CPU features to be required sometimes
  arm64: Permit onlining of HVO-incompatible late CPUs if HVO is not in
    use
  arm64: Remove user-selectable HVO Kconfig

 MAINTAINERS                                   |   3 +-
 arch/arm64/Kconfig                            |   1 +
 arch/arm64/include/asm/cpucaps.h              |   2 +
 arch/arm64/include/asm/cpufeature.h           |  39 ++-
 arch/arm64/include/asm/hugetlb.h              |   7 +
 arch/arm64/include/asm/pgalloc.h              |  54 ++++
 arch/arm64/include/asm/pgtable.h              |  57 ++++-
 arch/arm64/kernel/cpufeature.c                |  43 ++++
 arch/arm64/tools/cpucaps                      |   1 +
 arch/loongarch/include/asm/pgalloc.h          |   8 +
 arch/loongarch/include/asm/pgtable.h          |   8 +
 arch/riscv/include/asm/pgalloc.h              |   8 +
 arch/riscv/include/asm/pgtable.h              |   8 +
 arch/x86/include/asm/pgalloc.h                |   8 +
 arch/x86/include/asm/pgtable.h                |   8 +
 include/asm-generic/hugetlb.h                 |   7 +
 include/linux/hugetlb_vmemmap.h               |  20 ++
 include/linux/pgalloc.h                       |  20 ++
 include/linux/pgtable.h                       |  21 ++
 mm/hugetlb.c                                  |   2 +-
 mm/hugetlb_sysfs.c                            |   2 +-
 mm/hugetlb_vmemmap.c                          | 237 +++++++++++++-----
 ...b_vmemmap.h => hugetlb_vmemmap_internal.h} |   6 +-
 mm/sparse-vmemmap.c                           |   2 +-
 24 files changed, 489 insertions(+), 83 deletions(-)
 create mode 100644 include/linux/hugetlb_vmemmap.h
 rename mm/{hugetlb_vmemmap.h => hugetlb_vmemmap_internal.h} (95%)


base-commit: 0e35b9b6ec0ffcc5e23cbdec09f5c622ad532b53
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 01/18] hugetlb_vmemmap: Always flush TLB if needed upon PTE remapping

[James Houghton]

vmemmap_remap_range() can potentially leave the vmemmap in a partially
optimized state. In this case, a TLB flush should be done if the walk
caller asked for it, even if the optimization was not fully applied.

A follow-up patch will more completely deal with partially-optimized
folios.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 mm/hugetlb_vmemmap.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 133b46dfb09f..da29954e9533 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -164,13 +164,11 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end,
 	ret = walk_kernel_page_table_range(start, end, &vmemmap_remap_ops,
 				    NULL, walk);
 	mmap_read_unlock(&init_mm);
-	if (ret)
-		return ret;
 
 	if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH))
 		flush_tlb_kernel_range(start, end);
 
-	return 0;
+	return ret;
 }
 
 /*
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 02/18] hugetlb_vmemmap: Move vmemmap_get_tail up

[James Houghton]

In a follow-up patch, it will be used in
vmemmap_remap_alloc().

No functional change intended.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 mm/hugetlb_vmemmap.c | 52 ++++++++++++++++++++++----------------------
 1 file changed, 26 insertions(+), 26 deletions(-)

diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index da29954e9533..8d4e8ee51fdd 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -355,6 +355,32 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end,
 	return -ENOMEM;
 }
 
+static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone)
+{
+	const unsigned int idx = order - VMEMMAP_TAIL_MIN_ORDER;
+	struct page *tail, *p;
+	int node = zone_to_nid(zone);
+
+	tail = READ_ONCE(zone->vmemmap_tails[idx]);
+	if (likely(tail))
+		return tail;
+
+	tail = alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0);
+	if (!tail)
+		return NULL;
+
+	p = page_to_virt(tail);
+	for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++)
+		init_compound_tail(p + i, NULL, order, zone);
+
+	if (cmpxchg(&zone->vmemmap_tails[idx], NULL, tail)) {
+		__free_page(tail);
+		tail = READ_ONCE(zone->vmemmap_tails[idx]);
+	}
+
+	return tail;
+}
+
 /**
  * vmemmap_remap_alloc - remap the vmemmap virtual address range [@start, end)
  *			 to the page which is from the @vmemmap_pages
@@ -491,32 +517,6 @@ static bool vmemmap_should_optimize_folio(const struct hstate *h, struct folio *
 	return true;
 }
 
-static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone)
-{
-	const unsigned int idx = order - VMEMMAP_TAIL_MIN_ORDER;
-	struct page *tail, *p;
-	int node = zone_to_nid(zone);
-
-	tail = READ_ONCE(zone->vmemmap_tails[idx]);
-	if (likely(tail))
-		return tail;
-
-	tail = alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0);
-	if (!tail)
-		return NULL;
-
-	p = page_to_virt(tail);
-	for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++)
-		init_compound_tail(p + i, NULL, order, zone);
-
-	if (cmpxchg(&zone->vmemmap_tails[idx], NULL, tail)) {
-		__free_page(tail);
-		tail = READ_ONCE(zone->vmemmap_tails[idx]);
-	}
-
-	return tail;
-}
-
 static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
 					    struct folio *folio,
 					    struct list_head *vmemmap_pages,
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 03/18] hugetlb_vmemmap: Leave pages partially HVOed upon restore failure

[James Houghton]

Right now it is assumed that the restore routine in vmemmap_remap_free()
will not fail; that is, if we fail to fully optimize a folio, it is
assumed that restoring the folio to its pre-HVO state is guaranteed to
succeed.

Although vmemmap restoration may never fail in practice, properly handle
such failure cases now. This will leave folios in a partially HVOed
state. In a follow-up patch, failure certainly becomes possible.

In the event we end up with partially-HVOed folios, make sure to:
  1. Leave the HVO page flag in place.
  2. Free any unused vmemmap pages.

Always pass the vmemmap_tail page to the restore routine to avoid
leaking the non-optimized vmemmap pages for a partially HVOed page.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 mm/hugetlb_vmemmap.c | 49 ++++++++++++++++++++++++++++++++++----------
 1 file changed, 38 insertions(+), 11 deletions(-)

diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 8d4e8ee51fdd..5fee01bf5b34 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -236,12 +236,15 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr,
 {
 	struct page *src = pte_page(ptep_get(pte)), *dst;
 
+	if (WARN_ON_ONCE(!walk->vmemmap_tail))
+		return;
+
 	/*
-	 * When rolling back vmemmap_remap_free(), keep the copied head page
+	 * When restoring a partially-HVOed page, keep the copied head page
 	 * mapping and restore only PTEs currently pointing at the shared tail
 	 * page.
 	 */
-	if (walk->vmemmap_tail && walk->vmemmap_tail != src)
+	if (walk->vmemmap_tail != src)
 		return;
 
 	VM_WARN_ON_ONCE(PageHead((const struct page *)addr));
@@ -277,6 +280,7 @@ static int vmemmap_remap_split(unsigned long start, unsigned long end)
 	return vmemmap_remap_range(start, end, &walk);
 }
 
+static const int VMEMMAP_REMAP_INCOMPLETE = 1;
 /**
  * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end)
  *			to use @vmemmap_head/tail, then free vmemmap which
@@ -291,7 +295,8 @@ static int vmemmap_remap_split(unsigned long start, unsigned long end)
  *		responsibility to free pages.
  * @flags:	modifications to vmemmap_remap_walk flags
  *
- * Return: %0 on success, negative error code otherwise.
+ * Return: %0 on success, VMEMMAP_REMAP_INCOMPLETE if the page is incompletely
+ *         optimized, negative error code otherwise.
  */
 static int vmemmap_remap_free(unsigned long start, unsigned long end,
 			      struct page *vmemmap_head,
@@ -326,7 +331,8 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end,
 		.flags		= 0,
 	};
 
-	vmemmap_remap_range(start, end, &walk);
+	if (vmemmap_remap_range(start, end, &walk))
+		return VMEMMAP_REMAP_INCOMPLETE;
 
 	return ret;
 }
@@ -385,6 +391,8 @@ static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone)
  * vmemmap_remap_alloc - remap the vmemmap virtual address range [@start, end)
  *			 to the page which is from the @vmemmap_pages
  *			 respectively.
+ * @h:		the hstate for the folio whose vmemmap is getting remapped
+ * @folio:	the folio whose vmemmap is getting remapped
  * @start:	start address of the vmemmap virtual address range that we want
  *		to remap.
  * @end:	end address of the vmemmap virtual address range that we want to
@@ -393,20 +401,35 @@ static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone)
  *
  * Return: %0 on success, negative error code otherwise.
  */
-static int vmemmap_remap_alloc(unsigned long start, unsigned long end,
+static int vmemmap_remap_alloc(const struct hstate *h, struct folio *folio,
+			       unsigned long start, unsigned long end,
 			       unsigned long flags)
 {
 	LIST_HEAD(vmemmap_pages);
-	struct vmemmap_remap_walk walk = {
+	struct vmemmap_remap_walk walk;
+	struct page *vmemmap_tail;
+	int ret;
+
+	vmemmap_tail = vmemmap_get_tail(h->order, folio_zone(folio));
+	if (WARN_ON_ONCE(!vmemmap_tail))
+		return -ENOMEM;
+
+	if (alloc_vmemmap_page_list(start, end, &vmemmap_pages))
+		return -ENOMEM;
+
+	walk = (struct vmemmap_remap_walk) {
 		.remap_pte	= vmemmap_restore_pte,
+		.vmemmap_tail	= vmemmap_tail,
 		.vmemmap_pages	= &vmemmap_pages,
 		.flags		= flags,
 	};
 
-	if (alloc_vmemmap_page_list(start, end, &vmemmap_pages))
-		return -ENOMEM;
+	ret = vmemmap_remap_range(start, end, &walk);
 
-	return vmemmap_remap_range(start, end, &walk);
+	/* Not all pages may have been consumed */
+	free_vmemmap_page_list(&vmemmap_pages);
+
+	return ret;
 }
 
 static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON);
@@ -439,7 +462,7 @@ static int __hugetlb_vmemmap_restore_folio(const struct hstate *h,
 	 * When a HugeTLB page is freed to the buddy allocator, previously
 	 * discarded vmemmap pages must be allocated and remapping.
 	 */
-	ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, flags);
+	ret = vmemmap_remap_alloc(h, folio, vmemmap_start, vmemmap_end, flags);
 	if (!ret)
 		folio_clear_hugetlb_vmemmap_optimized(folio);
 
@@ -572,7 +595,11 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
 				 vmemmap_head, vmemmap_tail,
 				 vmemmap_pages, flags);
 out:
-	if (ret)
+	/*
+	 * If ret == VMEMMAP_REMAP_INCOMPLETE, the folio might be partially
+	 * HVOed. Leave the HVO page folio flag in place.
+	 */
+	if (ret < 0)
 		folio_clear_hugetlb_vmemmap_optimized(folio);
 
 	return ret;
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 04/18] hugetlb_vmemmap: Use try_update_vmemmap_pte to update in-use PTEs

[James Houghton]

set_pte_at() cannot be used to replace in-use vmemmap PTEs on arm64, so
replace it with a more specific routine, try_update_vmemmap_pte().

try_update_vmemmap_pte() is used for modifying page table entries such
that there is a guarantee that no fault will be taken.

For the x86, riscv, and loongarch implementations, please note one
difference: set_pte() does not invoke page_table_check_ptes_set(), but
set_pte_at(), the call we are about to replace, does. However, this is a
functional no-op because page_table_check_ptes_set() does nothing for
init_mm PTEs, which vmemmap PTEs are.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/loongarch/include/asm/pgtable.h |  8 ++++
 arch/riscv/include/asm/pgtable.h     |  8 ++++
 arch/x86/include/asm/pgtable.h       |  8 ++++
 include/linux/pgtable.h              | 21 +++++++++++
 mm/hugetlb_vmemmap.c                 | 56 +++++++++++++++++++---------
 5 files changed, 84 insertions(+), 17 deletions(-)

diff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/asm/pgtable.h
index 223528c04d73..e7b65056ef73 100644
--- a/arch/loongarch/include/asm/pgtable.h
+++ b/arch/loongarch/include/asm/pgtable.h
@@ -638,6 +638,14 @@ static inline long pmd_protnone(pmd_t pmd)
 #define pmd_leaf(pmd)		((pmd_val(pmd) & _PAGE_HUGE) != 0)
 #define pud_leaf(pud)		((pud_val(pud) & _PAGE_HUGE) != 0)
 
+#define __HAVE_ARCH_TRY_UPDATE_VMEMMAP_PTE
+static inline int try_update_vmemmap_pte(unsigned long addr, pte_t *ptep,
+					 pte_t pte)
+{
+	set_pte(ptep, pte);
+	return 0;
+}
+
 /*
  * We provide our own get_unmapped area to cope with the virtual aliasing
  * constraints placed on us by the cache architecture.
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 5d5756bda82e..a11c569ee2f0 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -1162,6 +1162,14 @@ static inline pud_t pud_modify(pud_t pud, pgprot_t newprot)
 
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
 
+#define __HAVE_ARCH_TRY_UPDATE_VMEMMAP_PTE
+static inline int try_update_vmemmap_pte(unsigned long addr, pte_t *ptep,
+					 pte_t pte)
+{
+	set_pte(ptep, pte);
+	return 0;
+}
+
 /*
  * Encode/decode swap entries and swap PTEs. Swap PTEs are all PTEs that
  * are !pte_none() && !pte_present().
diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index ac295ca6c92f..974151564071 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -1372,6 +1372,14 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
 }
 #endif
 
+#define __HAVE_ARCH_TRY_UPDATE_VMEMMAP_PTE
+static inline int try_update_vmemmap_pte(unsigned long addr, pte_t *ptep,
+					 pte_t pte)
+{
+	set_pte(ptep, pte);
+	return 0;
+}
+
 #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD
 static inline pud_t pudp_establish(struct vm_area_struct *vma,
 		unsigned long address, pud_t *pudp, pud_t pud)
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index 2981e386da7b..e1abccf76bb4 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -457,6 +457,27 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
 #endif
 #define set_pte_at(mm, addr, ptep, pte) set_ptes(mm, addr, ptep, pte, 1)
 
+#ifndef __HAVE_ARCH_TRY_UPDATE_VMEMMAP_PTE
+/*
+ * try_update_vmemmap_pte - Remap PTEs used by the vmemmap.
+ * @addr: Base address of the remapped PTE.
+ * @ptep: Page table pointer to be overwritten.
+ * @pte: Page table entry to write.
+ *
+ * This function is only to be used to update PTEs that map the vmemmap. The
+ * only valid transitions supported by this function are: leaf-level
+ * (PAGE_SIZE), valid-to-valid. The pfn and prot bits may be changed.
+ *
+ * Implementations of this function must ensure that, while the update is taking
+ * place, CPUs will not fault on the remapped virtual address.
+ */
+static inline int try_update_vmemmap_pte(unsigned long addr, pte_t *ptep,
+					 pte_t pte)
+{
+	return -EOPNOTSUPP;
+}
+#endif
+
 #ifndef __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS
 extern int ptep_set_access_flags(struct vm_area_struct *vma,
 				 unsigned long address, pte_t *ptep,
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 5fee01bf5b34..3951c043164a 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -34,7 +34,7 @@
  *			operations.
  */
 struct vmemmap_remap_walk {
-	void			(*remap_pte)(pte_t *pte, unsigned long addr,
+	int			(*remap_pte)(pte_t *pte, unsigned long addr,
 					     struct vmemmap_remap_walk *walk);
 
 	unsigned long		nr_walked;
@@ -141,11 +141,13 @@ static int vmemmap_pte_entry(pte_t *pte, unsigned long addr,
 			     unsigned long next, struct mm_walk *walk)
 {
 	struct vmemmap_remap_walk *vmemmap_walk = walk->private;
+	int ret = 0;
 
-	vmemmap_walk->remap_pte(pte, addr, vmemmap_walk);
-	vmemmap_walk->nr_walked++;
+	ret = vmemmap_walk->remap_pte(pte, addr, vmemmap_walk);
+	if (!ret)
+		vmemmap_walk->nr_walked++;
 
-	return 0;
+	return ret;
 }
 
 static const struct mm_walk_ops vmemmap_remap_ops = {
@@ -197,18 +199,20 @@ static void free_vmemmap_page_list(struct list_head *list)
 		free_vmemmap_page(page);
 }
 
-static void vmemmap_remap_pte(pte_t *pte, unsigned long addr,
-			      struct vmemmap_remap_walk *walk)
+static int vmemmap_remap_pte(pte_t *pte, unsigned long addr,
+			     struct vmemmap_remap_walk *walk)
 {
 	struct page *page = pte_page(ptep_get(pte));
 	pte_t entry;
+	bool head;
+	int ret;
+
+	head = walk->nr_walked == 0 && walk->vmemmap_head;
 
 	/* Remapping the head page requires r/w */
-	if (unlikely(walk->nr_walked == 0 && walk->vmemmap_head)) {
+	if (unlikely(head)) {
 		VM_WARN_ON_ONCE(!PageHead((const struct page *)addr));
 
-		list_del(&walk->vmemmap_head->lru);
-
 		/*
 		 * Makes sure that preceding stores to the page contents from
 		 * vmemmap_remap_free() become visible before the set_pte_at()
@@ -227,17 +231,30 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr,
 		entry = mk_pte(walk->vmemmap_tail, PAGE_KERNEL_RO);
 	}
 
+	ret = try_update_vmemmap_pte(addr, pte, entry);
+	if (ret)
+		return ret;
+
+	/* We successfully overwrote the vmemmap PTE, so we can free
+	 * the vmemmap page that was just unmapped, and if we mapped
+	 * the new head page, remove it from the list so that it
+	 * doesn't get freed later.
+	 */
 	list_add(&page->lru, walk->vmemmap_pages);
-	set_pte_at(&init_mm, addr, pte, entry);
+	if (head)
+		list_del(&walk->vmemmap_head->lru);
+
+	return 0;
 }
 
-static void vmemmap_restore_pte(pte_t *pte, unsigned long addr,
-				struct vmemmap_remap_walk *walk)
+static int vmemmap_restore_pte(pte_t *pte, unsigned long addr,
+			       struct vmemmap_remap_walk *walk)
 {
 	struct page *src = pte_page(ptep_get(pte)), *dst;
+	int ret;
 
 	if (WARN_ON_ONCE(!walk->vmemmap_tail))
-		return;
+		return -EINVAL;
 
 	/*
 	 * When restoring a partially-HVOed page, keep the copied head page
@@ -245,20 +262,25 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr,
 	 * page.
 	 */
 	if (walk->vmemmap_tail != src)
-		return;
+		return 0;
 
 	VM_WARN_ON_ONCE(PageHead((const struct page *)addr));
 
 	dst = list_first_entry(walk->vmemmap_pages, struct page, lru);
-	list_del(&dst->lru);
 	copy_page(page_to_virt(dst), page_to_virt(src));
 
 	/*
 	 * Makes sure that preceding stores to the page contents become visible
-	 * before the set_pte_at() write.
+	 * before the try_update_vmemmap_pte() write.
 	 */
 	smp_wmb();
-	set_pte_at(&init_mm, addr, pte, mk_pte(dst, PAGE_KERNEL));
+
+	ret = try_update_vmemmap_pte(addr, pte, mk_pte(dst, PAGE_KERNEL));
+	if (ret)
+		return ret;
+
+	list_del(&dst->lru);
+	return 0;
 }
 
 /**
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 05/18] hugetlb_vmemmap: Allow architectures not to allow HVO at runtime

[James Houghton]

Even if HVO is enabled, architectures may at run-time determine that it
cannot be used.

It may be the case that arch_supports_hugetlb_vmemmap_optimization()
returns false at first but eventually returns true. This is the case on
arm64. When this happens, bootmem hugepages won't be immediately HVOed,
but they will be optimized in prep_and_add_bootmem_folios(), part of
hugetlb_init().

Signed-off-by: James Houghton <jthoughton@google.com>
---
 include/asm-generic/hugetlb.h |  7 +++++++
 mm/hugetlb_vmemmap.c          | 12 ++++++++++++
 2 files changed, 19 insertions(+)

diff --git a/include/asm-generic/hugetlb.h b/include/asm-generic/hugetlb.h
index e1a2e1b7c8e7..c4796c54b702 100644
--- a/include/asm-generic/hugetlb.h
+++ b/include/asm-generic/hugetlb.h
@@ -128,4 +128,11 @@ static inline bool gigantic_page_runtime_supported(void)
 }
 #endif /* __HAVE_ARCH_GIGANTIC_PAGE_RUNTIME_SUPPORTED */
 
+#ifndef __HAVE_ARCH_HVO_SUPPORTED
+static inline bool arch_hugetlb_vmemmap_optimization_supported(void)
+{
+	return IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP);
+}
+#endif /* __HAVE_ARCH_HVO_SUPPORTED */
+
 #endif /* _ASM_GENERIC_HUGETLB_H */
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 3951c043164a..21377f795293 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -17,6 +17,7 @@
 #include <linux/pagewalk.h>
 #include <linux/pgalloc.h>
 
+#include "linux/hugetlb.h"
 #include <asm/tlbflush.h>
 #include "hugetlb_vmemmap.h"
 #include "internal.h"
@@ -556,6 +557,9 @@ static bool vmemmap_should_optimize_folio(const struct hstate *h, struct folio *
 	if (!READ_ONCE(vmemmap_optimize_enabled))
 		return false;
 
+	if (!arch_hugetlb_vmemmap_optimization_supported())
+		return false;
+
 	if (!hugetlb_vmemmap_optimizable(h))
 		return false;
 
@@ -764,6 +768,14 @@ static bool vmemmap_should_optimize_bootmem_page(struct huge_bootmem_page *m)
 	if (!READ_ONCE(vmemmap_optimize_enabled))
 		return false;
 
+	/*
+	 * Architectures may return false here but true by the time
+	 * hugetlb_init() is called. In this case, although the folios will
+	 * not be pre-HVOed, they will be optimized in hugetlb_init().
+	 */
+	if (!arch_hugetlb_vmemmap_optimization_supported())
+		return false;
+
 	if (!hugetlb_vmemmap_optimizable(m->hstate))
 		return false;
 
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 06/18] arm64: Rename cpu_has_hw_af to system_has_hw_af

[James Houghton]

In a follow-up patch, a new system feature will need to check for the
presence of HW AF on each CPU individually.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/include/asm/cpufeature.h | 26 ++++++++++++++++++++------
 arch/arm64/include/asm/pgtable.h    |  4 ++--
 2 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index a57870fa96db..e818ad1b56e2 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -910,22 +910,36 @@ static inline u32 id_aa64mmfr0_parange_to_phys_shift(int parange)
 }
 
 /* Check whether hardware update of the Access flag is supported */
-static inline bool cpu_has_hw_af(void)
+static inline bool supports_hw_af(int scope)
 {
 	u64 mmfr1;
 
 	if (!IS_ENABLED(CONFIG_ARM64_HW_AFDBM))
 		return false;
 
-	/*
-	 * Use cached version to avoid emulated msr operation on KVM
-	 * guests.
-	 */
-	mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
+	if (scope == SCOPE_SYSTEM) {
+		/*
+		 * Use cached version to avoid emulated msr operation on KVM
+		 * guests.
+		 */
+		mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
+	} else {
+		mmfr1 = read_cpuid(ID_AA64MMFR1_EL1);
+	}
 	return cpuid_feature_extract_unsigned_field(mmfr1,
 						ID_AA64MMFR1_EL1_HAFDBS_SHIFT);
 }
 
+static inline bool system_has_hw_af(void)
+{
+	return supports_hw_af(SCOPE_SYSTEM);
+}
+
+static inline bool cpu_has_hw_af(void)
+{
+	return supports_hw_af(SCOPE_LOCAL_CPU);
+}
+
 static inline bool cpu_has_pan(void)
 {
 	u64 mmfr1 = read_cpuid(ID_AA64MMFR1_EL1);
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 27689c62bd25..5f21d3a738ee 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1598,7 +1598,7 @@ static inline void update_mmu_cache_range(struct vm_fault *vmf,
  * page after fork() + CoW for pfn mappings. We don't always have a
  * hardware-managed access flag on arm64.
  */
-#define arch_has_hw_pte_young		cpu_has_hw_af
+#define arch_has_hw_pte_young		system_has_hw_af
 
 #ifdef CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG
 #define arch_has_hw_nonleaf_pmd_young	system_supports_haft
@@ -1608,7 +1608,7 @@ static inline void update_mmu_cache_range(struct vm_fault *vmf,
  * Experimentally, it's cheap to set the access flag in hardware and we
  * benefit from prefaulting mappings as 'old' to start with.
  */
-#define arch_wants_old_prefaulted_pte	cpu_has_hw_af
+#define arch_wants_old_prefaulted_pte	system_has_hw_af
 
 /*
  * Request exec memory is read into pagecache in at least 64K folios. This size
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 07/18] arm64: Add system_supports_hvo

[James Houghton]

Track HVO compatibility as its own system feature. If every boot CPU
supports it, later onlining a CPU that doesn't support it won't be
allowed.

HVO requires BBM Level 2 no abort for the block -> table transitions
when optimizing the vmemmap, and it requires hardware AF management to
avoid taking page faults on the !AF translation that is installed as
part of the RW table -> RO + new OA table transitions.

Another possibility would be to disable HVO if an incompatible CPU were
onlined; I haven't done this.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/include/asm/cpucaps.h    |  2 ++
 arch/arm64/include/asm/cpufeature.h |  5 +++++
 arch/arm64/kernel/cpufeature.c      | 16 ++++++++++++++++
 arch/arm64/tools/cpucaps            |  1 +
 4 files changed, 24 insertions(+)

diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 25c61cda901c..6db3ef827f86 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -75,6 +75,8 @@ cpucap_is_possible(const unsigned int cap)
 		return IS_ENABLED(CONFIG_HW_PERF_EVENTS);
 	case ARM64_HAS_LSUI:
 		return IS_ENABLED(CONFIG_ARM64_LSUI);
+	case ARM64_HVO_COMPATIBLE:
+		return IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP);
 	}
 
 	return true;
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index e818ad1b56e2..a62c284962cf 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -885,6 +885,11 @@ static inline bool system_supports_bbml2_noabort(void)
 	return alternative_has_cap_unlikely(ARM64_HAS_BBML2_NOABORT);
 }
 
+static inline bool system_supports_hvo(void)
+{
+	return alternative_has_cap_unlikely(ARM64_HVO_COMPATIBLE);
+}
+
 int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt);
 bool try_emulate_mrs(struct pt_regs *regs, u32 isn);
 
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 9a22df0c5120..3b9224b99a5f 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -2172,6 +2172,16 @@ static bool has_bbml2_noabort(const struct arm64_cpu_capabilities *caps, int sco
 	return cpu_supports_bbml2_noabort();
 }
 
+static bool hvo_compatible(const struct arm64_cpu_capabilities *caps, int scope)
+{
+	/*
+	 * We need BBML2 to support Block -> Table transitions without taking
+	 * faults, and we need HW AF support to support changing the OA without
+	 * taking faults.
+	 */
+	return cpu_supports_bbml2_noabort() && supports_hw_af(scope);
+}
+
 static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused)
 {
 	/*
@@ -3067,6 +3077,12 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.type = ARM64_CPUCAP_EARLY_LOCAL_CPU_FEATURE,
 		.matches = has_bbml2_noabort,
 	},
+	{
+		.desc = "HugeTLB Vmemmap Optimization Support",
+		.capability = ARM64_HVO_COMPATIBLE,
+		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.matches = hvo_compatible,
+	},
 	{
 		.desc = "52-bit Virtual Addressing for KVM (LPA2)",
 		.capability = ARM64_HAS_LPA2,
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index 9b85a84f6fd4..630953d59c52 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -70,6 +70,7 @@ HAS_VIRT_HOST_EXTN
 HAS_WFXT
 HAS_XNX
 HAFT
+HVO_COMPATIBLE
 HW_DBM
 KVM_HVHE
 KVM_PROTECTED_MODE
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 08/18] arm64: Implement try_update_vmemmap_pte using the AF trick

[James Houghton]

try_update_vmemmap_pte() must modify vmemmap PTEs without introducing a
time window where other CPUs on the system might fault.

Normally a break-before-make sequence is required to avoid conflicts
with cached translations. However, if we can guarantee that the existing
translation cannot be cached, a BBM sequence is not needed.

Translations with the AF unset may not be cached (see Arm ARM Rule
DWZCQ); the implementation of try_update_vmemmap_pte() on arm64 takes
advantage of this fact to replace a PTE without BBM and therefore
without leaving a window open where PE might fault on this translation.

Of course, if some CPUs on the system do not support HW AF management,
clearing the AF will introduce potential faults. system_supports_hvo()
will return false if any CPUs on the system do not support HW AF.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/include/asm/pgtable.h | 53 ++++++++++++++++++++++++++++++--
 1 file changed, 50 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 5f21d3a738ee..7b11aa41d0a0 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1302,8 +1302,7 @@ static inline void __pte_clear(struct mm_struct *mm,
 	__set_pte(ptep, __pte(0));
 }
 
-static inline bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
-		unsigned long address, pte_t *ptep)
+static inline pte_t __ptep_clear_young(pte_t *ptep)
 {
 	pte_t old_pte, pte;
 
@@ -1315,7 +1314,13 @@ static inline bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
 					       pte_val(old_pte), pte_val(pte));
 	} while (pte_val(pte) != pte_val(old_pte));
 
-	return pte_young(pte);
+	return pte;
+}
+
+static inline bool __ptep_test_and_clear_young(struct vm_area_struct *vma,
+		unsigned long address, pte_t *ptep)
+{
+	return pte_young(__ptep_clear_young(ptep));
 }
 
 static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
@@ -1793,6 +1798,48 @@ static inline void pte_clear(struct mm_struct *mm,
 	__pte_clear(mm, addr, ptep);
 }
 
+#define __HAVE_ARCH_TRY_UPDATE_VMEMMAP_PTE
+static inline int try_update_vmemmap_pte(unsigned long addr, pte_t *ptep,
+					 const pte_t pte)
+{
+	const int max_attempts = 16;
+	int attempts = 0;
+	pte_t old_pte;
+
+	if (!system_supports_hvo())
+		return -EOPNOTSUPP;
+
+	/* This routine is only to be used for valid-to-valid transitions. */
+	if (WARN_ON_ONCE(!pte_valid(pte)))
+		return -EINVAL;
+
+	old_pte = __ptep_get(ptep);
+
+	do {
+		if (WARN_ON_ONCE(!pte_valid(old_pte)))
+			return -EINVAL;
+
+		/* We should never get a contiguous PTE here. */
+		if (WARN_ON_ONCE(pte_valid_cont(old_pte)))
+			return -EINVAL;
+
+		if (pte_young(old_pte)) {
+			/* __ptep_clear_young() returns the overwritten PTE */
+			old_pte = pte_mkold(__ptep_clear_young(ptep));
+
+			flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
+		}
+	/*
+	 * Translations without AF cannot be cached, so we can replace
+	 * them without BBM.
+	 */
+	} while (!try_cmpxchg_relaxed(&pte_val(*ptep), &pte_val(old_pte),
+				      pte_val(pte)) &&
+		 ++attempts < max_attempts);
+
+	return attempts == max_attempts ? -EAGAIN : 0;
+}
+
 #define clear_full_ptes clear_full_ptes
 static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr,
 				pte_t *ptep, unsigned int nr, int full)
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 09/18] arm64: Prevent HVO if the HVO system feature is not enabled

[James Houghton]

The HVO system feature checks that all CPUs support HW AF and
BBML2_NOABORT, both of which are needed for HVO (as of this patch).

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/include/asm/hugetlb.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h
index d038ff14d16c..73af7911fd83 100644
--- a/arch/arm64/include/asm/hugetlb.h
+++ b/arch/arm64/include/asm/hugetlb.h
@@ -11,6 +11,7 @@
 #define __ASM_HUGETLB_H
 
 #include <asm/cacheflush.h>
+#include <asm/cpufeature.h>
 #include <asm/mte.h>
 #include <asm/page.h>
 
@@ -65,6 +66,12 @@ extern void huge_ptep_modify_prot_commit(struct vm_area_struct *vma,
 					 unsigned long addr, pte_t *ptep,
 					 pte_t old_pte, pte_t new_pte);
 
+#define __HAVE_ARCH_HVO_SUPPORTED
+static inline bool arch_hugetlb_vmemmap_optimization_supported(void)
+{
+	return system_supports_hvo();
+}
+
 #include <asm-generic/hugetlb.h>
 
 static inline void __flush_hugetlb_tlb_range(struct vm_area_struct *vma,
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 10/18] arm64: Support hugetlb vmemmap optimization

[James Houghton]

Now that HVO is architecturally sound for arm64, enable support. Add a
new Kconfig for this support, as it comes with a regression, described
below.

Add a comment about BBML2_NOABORT where HVO is relying on it.

If a late-onlined CPU does not support HVO (HW AF and BBML2_NOABORT)
while all boot CPUs do support HVO, the late CPU won't be onlined after
this commit when a kernel is compiled with
HUGETLB_PAGE_OPTIMIZE_VMEMMAP=y.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/Kconfig   | 12 ++++++++++++
 mm/hugetlb_vmemmap.c |  4 ++++
 2 files changed, 16 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index b3afe0688919..cf4bd6bd7f11 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -259,6 +259,7 @@ config ARM64
 	select USER_STACKTRACE_SUPPORT
 	select VDSO_GETRANDOM
 	select VMAP_STACK
+	select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if ARM64_WANT_OPTIMIZE_HUGETLB_VMEMMAP
 	help
 	  ARM 64-bit (AArch64) Linux support.
 
@@ -1607,6 +1608,17 @@ config ARCH_SPARSEMEM_ENABLE
 	def_bool y
 	select SPARSEMEM_VMEMMAP_ENABLE
 
+config ARM64_WANT_OPTIMIZE_HUGETLB_VMEMMAP
+	bool "Enable support for HugeTLB Vmemmap Optimization (HVO)"
+	default n
+	depends on HUGETLB_PAGE && SPARSEMEM_VMEMMAP && ARM64_HW_AFDBM
+	help
+	  Enable support for HVO. When enabled, late-onlining of CPUs that do
+	  not support features required by HVO will not be onlined if all boot
+	  CPUs do have such support.
+
+	  If unsure, say N
+
 config HW_PERF_EVENTS
 	def_bool y
 	depends on ARM_PMU
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 21377f795293..977249e22ed0 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -86,6 +86,10 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
 
 		/* Make pte visible before pmd. See comment in pmd_install(). */
 		smp_wmb();
+		/*
+		 * On arm64, this requires BBML2_NOABORT. Its support has
+		 * already been checked.
+		 */
 		pmd_populate_kernel(&init_mm, pmd, pgtable);
 		if (!(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH))
 			flush_tlb_kernel_range(start, start + PMD_SIZE);
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 11/18] hugetlb_vmemmap: Use try_populate_vmemmap_pmd for replacing in-use PMDs

[James Houghton]

This routine is to be used for updating in-use, leaf-level PMDs in the
vmemmap without introducing a window where vmemmap accesses might fault.

Because try_populate_vmemmap_pmd() can fail, the split_page() call needs
to be moved to after the PMD update. Because both split_page() and the
PMD update are done under the page table lock, this rearrangement is
safe.

Only leaf-level PMD to non-leaf PMD (mapping the same physical pages) is
the only transition that needs to be supported.

Collapsing a non-leaf PMD to a leaf-level PMD is not used by HVO, so
try_populate_vmemmap_pmd() need not support it.

This patch allows arm64's implementation to replaced with one that does
not require BBML2_NOABORT.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/include/asm/pgalloc.h     |  9 +++++++++
 arch/loongarch/include/asm/pgalloc.h |  8 ++++++++
 arch/riscv/include/asm/pgalloc.h     |  8 ++++++++
 arch/x86/include/asm/pgalloc.h       |  8 ++++++++
 include/linux/pgalloc.h              | 20 ++++++++++++++++++++
 mm/hugetlb_vmemmap.c                 | 28 ++++++++++++++++------------
 6 files changed, 69 insertions(+), 12 deletions(-)

diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h
index 1b4509d3382c..c8946250d431 100644
--- a/arch/arm64/include/asm/pgalloc.h
+++ b/arch/arm64/include/asm/pgalloc.h
@@ -121,4 +121,13 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep)
 		       PMD_TYPE_TABLE | PMD_TABLE_AF | PMD_TABLE_PXN);
 }
 
+#define __HAVE_ARCH_TRY_POPULATE_VMEMMAP_PMD
+static inline int try_populate_vmemmap_pmd(pmd_t *pmdp, pte_t *pgtable,
+					   unsigned long addr)
+{
+	/* BBML2_NOABORT is required. Its presence has been checked. */
+	pmd_populate_kernel(&init_mm, pmdp, pgtable);
+	return 0;
+}
+
 #endif
diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h
index 248f62d0b590..9322d962c9cc 100644
--- a/arch/loongarch/include/asm/pgalloc.h
+++ b/arch/loongarch/include/asm/pgalloc.h
@@ -24,6 +24,14 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, pgtable_t pte)
 	set_pmd(pmd, __pmd((unsigned long)page_address(pte)));
 }
 
+#define __HAVE_ARCH_TRY_POPULATE_VMEMMAP_PMD
+static inline int try_populate_vmemmap_pmd(pmd_t *pmdp, pte_t *pgtable,
+					   unsigned long addr)
+{
+	pmd_populate_kernel(&init_mm, pmdp, pgtable);
+	return 0;
+}
+
 #ifndef __PAGETABLE_PMD_FOLDED
 
 static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h
index 770ce18a7328..8dbbe62b2574 100644
--- a/arch/riscv/include/asm/pgalloc.h
+++ b/arch/riscv/include/asm/pgalloc.h
@@ -31,6 +31,14 @@ static inline void pmd_populate(struct mm_struct *mm,
 	set_pmd(pmd, __pmd((pfn << _PAGE_PFN_SHIFT) | _PAGE_TABLE));
 }
 
+#define __HAVE_ARCH_TRY_POPULATE_VMEMMAP_PMD
+static inline int try_populate_vmemmap_pmd(pmd_t *pmdp, pte_t *pgtable,
+					   unsigned long addr)
+{
+	pmd_populate_kernel(&init_mm, pmdp, pgtable);
+	return 0;
+}
+
 #ifndef __PAGETABLE_PMD_FOLDED
 static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
 {
diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h
index c88691b15f3c..47b56bbd6236 100644
--- a/arch/x86/include/asm/pgalloc.h
+++ b/arch/x86/include/asm/pgalloc.h
@@ -82,6 +82,14 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,
 	set_pmd(pmd, __pmd(((pteval_t)pfn << PAGE_SHIFT) | _PAGE_TABLE));
 }
 
+#define __HAVE_ARCH_TRY_POPULATE_VMEMMAP_PMD
+static inline int try_populate_vmemmap_pmd(pmd_t *pmdp, pte_t *pgtable,
+					   unsigned long addr)
+{
+	pmd_populate_kernel(&init_mm, pmdp, pgtable);
+	return 0;
+}
+
 #if CONFIG_PGTABLE_LEVELS > 2
 extern void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd);
 
diff --git a/include/linux/pgalloc.h b/include/linux/pgalloc.h
index 9174fa59bbc5..ed446d95ca37 100644
--- a/include/linux/pgalloc.h
+++ b/include/linux/pgalloc.h
@@ -26,4 +26,24 @@
 			arch_sync_kernel_mappings(addr, addr);		\
 	} while (0)
 
+#ifndef __HAVE_ARCH_TRY_POPULATE_VMEMMAP_PMD
+/*
+ * try_populate_vmemmap_pmd - Populate a PMD that is in use by the vmemmap.
+ * @addr: Base address of the remapped PMD.
+ * @pmdp: Page table pointer to be overwritten.
+ * @pgtable: Pointer to the page table that the new PMD will point to.
+ *
+ * This function is only to be used to update PMDs that map the vmemmap to
+ * point to a page of already-populated PTEs that map the same pages.
+ *
+ * Implementations of this function must ensure that, while the update is taking
+ * place, CPUs will not fault on the remapped virtual address range.
+ */
+static inline int try_populate_vmemmap_pmd(pmd_t *pmdp, pte_t *pgtable,
+					   unsigned long addr)
+{
+	return -EOPNOTSUPP;
+}
+#endif
+
 #endif /* _LINUX_PGALLOC_H */
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index 977249e22ed0..b445febac0d2 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -55,6 +55,7 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
 			     struct vmemmap_remap_walk *walk)
 {
 	pmd_t __pmd;
+	int ret;
 	int i;
 	unsigned long addr = start;
 	pte_t *pgtable;
@@ -74,8 +75,15 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
 		set_pte_at(&init_mm, addr, pte, entry);
 	}
 
+	ret = 0;
 	spin_lock(&init_mm.page_table_lock);
 	if (likely(pmd_leaf(*pmd))) {
+		/* Make pte visible before pmd. See comment in pmd_install(). */
+		smp_wmb();
+		ret = try_populate_vmemmap_pmd(pmd, pgtable, start);
+		if (ret)
+			goto free;
+
 		/*
 		 * Higher order allocations from buddy allocator must be able to
 		 * be treated as independent small pages (as they can be freed
@@ -84,21 +92,17 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
 		if (!PageReserved(head))
 			split_page(head, get_order(PMD_SIZE));
 
-		/* Make pte visible before pmd. See comment in pmd_install(). */
-		smp_wmb();
-		/*
-		 * On arm64, this requires BBML2_NOABORT. Its support has
-		 * already been checked.
-		 */
-		pmd_populate_kernel(&init_mm, pmd, pgtable);
 		if (!(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH))
 			flush_tlb_kernel_range(start, start + PMD_SIZE);
-	} else {
-		pte_free_kernel(&init_mm, pgtable);
-	}
-	spin_unlock(&init_mm.page_table_lock);
+	} else
+		goto free;
 
-	return 0;
+out:
+	spin_unlock(&init_mm.page_table_lock);
+	return ret;
+free:
+	pte_free_kernel(&init_mm, pgtable);
+	goto out;
 }
 
 static int vmemmap_pmd_entry(pmd_t *pmd, unsigned long addr,
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 12/18] arm64: Implement try_populate_vmemmap_pmd using AF trick

[James Houghton]

This routine is to be used for updating in-use, leaf-level PMDs in the
vmemmap without introducing a window where vmemmap accesses might fault.

Implementing this on arm64 requires some care: use the same access flag
trick that is used for vmemmap PTE updates. HAFT is not needed and the
TLB flushing routine remains identical, as we are not overwriting a
non-leaf PMD.

For systems that support BBML2_NOABORT, there is no need to use the AF
trick.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/include/asm/pgalloc.h | 51 ++++++++++++++++++++++++++++++--
 1 file changed, 48 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h
index c8946250d431..26b05f8b70cd 100644
--- a/arch/arm64/include/asm/pgalloc.h
+++ b/arch/arm64/include/asm/pgalloc.h
@@ -125,9 +125,54 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep)
 static inline int try_populate_vmemmap_pmd(pmd_t *pmdp, pte_t *pgtable,
 					   unsigned long addr)
 {
-	/* BBML2_NOABORT is required. Its presence has been checked. */
-	pmd_populate_kernel(&init_mm, pmdp, pgtable);
-	return 0;
+	const int max_attempts = 16;
+	int attempts = 0;
+	pmd_t old_pmd, new_pmd;
+
+	if (!system_supports_hvo())
+		return -EOPNOTSUPP;
+
+	if (system_supports_bbml2_noabort()) {
+		/*
+		 * BBML2_NOABORT allows block->table transitions if the PTEs
+		 * underneath do not conflict with existing, potentially cached
+		 * translations.
+		 */
+		pmd_populate_kernel(&init_mm, pmdp, pgtable);
+		return 0;
+	}
+
+	new_pmd = __pmd(__phys_to_pmd_val(__pa(pgtable)) |
+			PMD_TYPE_TABLE | PMD_TABLE_AF | PMD_TABLE_UXN);
+
+	old_pmd = pmdp_get(pmdp);
+
+	do {
+		if (WARN_ON_ONCE(!pmd_valid(old_pmd)))
+			return -EINVAL;
+
+		if (WARN_ON_ONCE(!pmd_leaf(old_pmd)))
+			return -EINVAL;
+
+		/* We should never get a contiguous PMD here. */
+		if (WARN_ON_ONCE(pmd_cont(old_pmd)))
+			return -EINVAL;
+
+		if (pmd_young(old_pmd)) {
+			/* __ptep_clear_young() returns the overwritten PTE */
+			old_pmd = pte_pmd(pte_mkold(__ptep_clear_young((pte_t *)pmdp)));
+
+			flush_tlb_kernel_range(addr, addr + PMD_SIZE);
+		}
+	/*
+	 * Translations without AF cannot be cached, so we can replace
+	 * them without BBM.
+	 */
+	} while (!try_cmpxchg_relaxed(&pmd_val(*pmdp), &pmd_val(old_pmd),
+				      pmd_val(new_pmd)) &&
+		 ++attempts < max_attempts);
+
+	return attempts == max_attempts ? -EAGAIN : 0;
 }
 
 #endif
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 13/18] arm64: Drop BBML2_NOABORT requirement for HVO

[James Houghton]

With PMD updates being done with the Access Flag trick, BBML2_NOABORT is
no longer needed.

Dropping this requirement vastly widens the pool of systems this
optimization is valid for.

Intentionally leave HVO as opt-in, as enabling HVO still introduces the
following regression: if all boot CPUs support HW AF, then any
late-onlined CPUs that do not will not be onlined.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/kernel/cpufeature.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 3b9224b99a5f..c13433316e80 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -2175,11 +2175,10 @@ static bool has_bbml2_noabort(const struct arm64_cpu_capabilities *caps, int sco
 static bool hvo_compatible(const struct arm64_cpu_capabilities *caps, int scope)
 {
 	/*
-	 * We need BBML2 to support Block -> Table transitions without taking
-	 * faults, and we need HW AF support to support changing the OA without
-	 * taking faults.
+	 * We need HW AF support to support changing vmemmap mapping level and
+	 * OA without taking faults.
 	 */
-	return cpu_supports_bbml2_noabort() && supports_hw_af(scope);
+	return supports_hw_af(scope);
 }
 
 static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused)
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 14/18] hugetlb_vmemmap: Rename mm/hugetlb_vmemmap.h to mm/hugetlb_vmemmap_internal.h

[James Houghton]

This avoids the confusion with the new header that will be added in late
commit, at include/linux/hugetlb_vmemmap.h.

The file rename in this patch isn't really required, but the header
guard (now named _MM_HUGETLB_VMEMMAP_INTERNAL_H) does need a rename.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 MAINTAINERS                                          | 2 +-
 mm/hugetlb.c                                         | 2 +-
 mm/hugetlb_sysfs.c                                   | 2 +-
 mm/hugetlb_vmemmap.c                                 | 4 ++--
 mm/{hugetlb_vmemmap.h => hugetlb_vmemmap_internal.h} | 6 +++---
 mm/sparse-vmemmap.c                                  | 2 +-
 6 files changed, 9 insertions(+), 9 deletions(-)
 rename mm/{hugetlb_vmemmap.h => hugetlb_vmemmap_internal.h} (95%)

diff --git a/MAINTAINERS b/MAINTAINERS
index f37a81950e25..ff47013ce995 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12021,7 +12021,7 @@ F:	mm/hugetlb_cma.h
 F:	mm/hugetlb_sysctl.c
 F:	mm/hugetlb_sysfs.c
 F:	mm/hugetlb_vmemmap.c
-F:	mm/hugetlb_vmemmap.h
+F:	mm/hugetlb_vmemmap_internal.h
 F:	tools/testing/selftests/cgroup/test_hugetlb_memcg.c
 
 HVA ST MEDIA DRIVER
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 571212b80835..298673a6c57c 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -47,7 +47,7 @@
 #include <linux/node.h>
 #include <linux/page_owner.h>
 #include "internal.h"
-#include "hugetlb_vmemmap.h"
+#include "hugetlb_vmemmap_internal.h"
 #include "hugetlb_cma.h"
 #include "hugetlb_internal.h"
 #include <linux/page-isolation.h>
diff --git a/mm/hugetlb_sysfs.c b/mm/hugetlb_sysfs.c
index 79ece91406bf..83a95f5142f5 100644
--- a/mm/hugetlb_sysfs.c
+++ b/mm/hugetlb_sysfs.c
@@ -8,7 +8,7 @@
 #include <linux/page_owner.h>
 #include <linux/page-isolation.h>
 
-#include "hugetlb_vmemmap.h"
+#include "hugetlb_vmemmap_internal.h"
 #include "hugetlb_internal.h"
 
 #define HSTATE_ATTR_RO(_name) \
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index b445febac0d2..a490a34998d9 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -16,10 +16,10 @@
 #include <linux/mmdebug.h>
 #include <linux/pagewalk.h>
 #include <linux/pgalloc.h>
+#include <linux/hugetlb.h>
 
-#include "linux/hugetlb.h"
 #include <asm/tlbflush.h>
-#include "hugetlb_vmemmap.h"
+#include "hugetlb_vmemmap_internal.h"
 #include "internal.h"
 
 /**
diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap_internal.h
similarity index 95%
rename from mm/hugetlb_vmemmap.h
rename to mm/hugetlb_vmemmap_internal.h
index 18b490825215..6b28780014bc 100644
--- a/mm/hugetlb_vmemmap.h
+++ b/mm/hugetlb_vmemmap_internal.h
@@ -6,8 +6,8 @@
  *
  *     Author: Muchun Song <songmuchun@bytedance.com>
  */
-#ifndef _LINUX_HUGETLB_VMEMMAP_H
-#define _LINUX_HUGETLB_VMEMMAP_H
+#ifndef _MM_HUGETLB_VMEMMAP_INTERNAL_H
+#define _MM_HUGETLB_VMEMMAP_INTERNAL_H
 #include <linux/hugetlb.h>
 #include <linux/io.h>
 #include <linux/memblock.h>
@@ -95,4 +95,4 @@ static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h)
 {
 	return hugetlb_vmemmap_optimizable_size(h) != 0;
 }
-#endif /* _LINUX_HUGETLB_VMEMMAP_H */
+#endif /* _MM_HUGETLB_VMEMMAP_INTERNAL_H */
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index 99e2be39671b..94d5da9f8ab7 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -32,7 +32,7 @@
 #include <asm/dma.h>
 #include <asm/tlbflush.h>
 
-#include "hugetlb_vmemmap.h"
+#include "hugetlb_vmemmap_internal.h"
 
 /*
  * Flags for vmemmap_populate_range and friends.
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 15/18] hugetlb_vmemmap: Add a way to permanently disable HVO when needed

[James Houghton]

hugetlb_vmemmap_optimization_try_disable() is used by callers to
permanently disable HVO if it becomes impossible to use.

If HVO is already actively in use,
hugetlb_vmemmap_optimization_try_disable() returns false to inform the
caller to take appropriate action (e.g., for arm64, not to online an
incompatible late CPU).

Signed-off-by: James Houghton <jthoughton@google.com>
---
 MAINTAINERS                     |  1 +
 include/linux/hugetlb_vmemmap.h | 20 +++++++++++++++++
 mm/hugetlb_vmemmap.c            | 40 +++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+)
 create mode 100644 include/linux/hugetlb_vmemmap.h

diff --git a/MAINTAINERS b/MAINTAINERS
index ff47013ce995..55bd4c09f494 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12013,6 +12013,7 @@ F:	Documentation/mm/hugetlbfs_reserv.rst
 F:	Documentation/mm/vmemmap_dedup.rst
 F:	fs/hugetlbfs/
 F:	include/linux/hugetlb.h
+F:	include/linux/hugetlb_vmemmap.h
 F:	include/trace/events/hugetlbfs.h
 F:	mm/hugetlb.c
 F:	mm/hugetlb_cgroup.c
diff --git a/include/linux/hugetlb_vmemmap.h b/include/linux/hugetlb_vmemmap.h
new file mode 100644
index 000000000000..a671eb4a4ff4
--- /dev/null
+++ b/include/linux/hugetlb_vmemmap.h
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_HUGETLB_VMEMMAP_H
+#define _LINUX_HUGETLB_VMEMMAP_H
+
+#include <linux/types.h>
+
+#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
+
+bool hugetlb_vmemmap_optimization_try_disable(void);
+
+#else /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */
+
+static inline bool hugetlb_vmemmap_optimization_try_disable(void)
+{
+	return true;
+}
+
+#endif
+
+#endif /* _LINUX_HUGETLB_VMEMMAP_H */
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index a490a34998d9..7e7f8579cc2a 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -17,6 +17,7 @@
 #include <linux/pagewalk.h>
 #include <linux/pgalloc.h>
 #include <linux/hugetlb.h>
+#include <linux/hugetlb_vmemmap.h>
 
 #include <asm/tlbflush.h>
 #include "hugetlb_vmemmap_internal.h"
@@ -463,6 +464,39 @@ static int vmemmap_remap_alloc(const struct hstate *h, struct folio *folio,
 	return ret;
 }
 
+enum hugetlb_hvo_status {
+	HVO_INACTIVE = 0,
+	HVO_ACTIVE,
+	HVO_PERMANENTLY_INACTIVE,
+};
+static enum hugetlb_hvo_status hvo_status = HVO_INACTIVE;
+
+static bool hugetlb_hvo_status_try_set(enum hugetlb_hvo_status status)
+{
+	enum hugetlb_hvo_status old;
+
+	old = READ_ONCE(hvo_status);
+
+retry:
+	/* The current setting is what we want. */
+	if (old == status)
+		return true;
+
+	/* The current setting cannot be changed. */
+	if (old != HVO_INACTIVE)
+		return false;
+
+	if (!try_cmpxchg_relaxed(&hvo_status, &old, status))
+		goto retry;
+
+	return true;
+}
+
+bool hugetlb_vmemmap_optimization_try_disable(void)
+{
+	return hugetlb_hvo_status_try_set(HVO_PERMANENTLY_INACTIVE);
+}
+
 static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON);
 static int __init hugetlb_vmemmap_optimize_param(char *buf)
 {
@@ -571,6 +605,9 @@ static bool vmemmap_should_optimize_folio(const struct hstate *h, struct folio *
 	if (!hugetlb_vmemmap_optimizable(h))
 		return false;
 
+	if (!hugetlb_hvo_status_try_set(HVO_ACTIVE))
+		return false;
+
 	return true;
 }
 
@@ -809,6 +846,9 @@ static bool vmemmap_should_optimize_bootmem_page(struct huge_bootmem_page *m)
 	    !IS_ALIGNED(psize, pmd_vmemmap_size))
 		return false;
 
+	if (!hugetlb_hvo_status_try_set(HVO_ACTIVE))
+		return false;
+
 	return true;
 }
 
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 16/18] arm64: Allow "optional" CPU features to be required sometimes

[James Houghton]

The HugeTLB vmemmap optimization feature should be optional if HVO is
not in fact in use. Otherwise, on systems where the boot CPUs support
HVO (HW AF) but the late CPUs do not, simply compiling with HVO support
will prevent late-onlining.

It is desirable to always compile in support for HVO without regressing
late-onlining of CPUs on systems where HVO is not in use at run-time.

If HVO is in fact in use at late-online time, HVO must have a way to
prevent such incompatible CPUs from being onlined.

Add the late_cpu_enable() callback to provide this "sometimes required"
functionality.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/include/asm/cpufeature.h | 8 ++++++++
 arch/arm64/kernel/cpufeature.c      | 8 ++++++++
 2 files changed, 16 insertions(+)

diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index a62c284962cf..d1504804f2be 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -367,6 +367,14 @@ struct arm64_cpu_capabilities {
 	 * routine must check it before taking any action.
 	 */
 	void (*cpu_enable)(const struct arm64_cpu_capabilities *cap);
+	/*
+	 * In rare cases, capabilities are *sometimes* optional for late CPUs.
+	 * This callback allows a capability to prevent onlining of
+	 * incompatible CPUs when the capability is in fact required.
+	 *
+	 * Returns true iff onlining the CPU is permitted.
+	 */
+	bool (*late_cpu_enable)(const struct arm64_cpu_capabilities *cap);
 	union {
 		struct {	/* To be used for erratum handling only */
 			struct midr_range midr_range;
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index c13433316e80..7e2c134f3b55 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -3684,6 +3684,14 @@ static void verify_local_cpu_caps(u16 scope_mask)
 			 */
 			if (!cpu_has_cap && !cpucap_late_cpu_optional(caps))
 				break;
+
+			/*
+			 * Some optional features may in fact be required due
+			 * to particular runtime conditions.
+			 */
+			if (caps->late_cpu_enable && !caps->late_cpu_enable(caps))
+				break;
+
 			/*
 			 * We have to issue cpu_enable() irrespective of
 			 * whether the CPU has it or not, as it is enabeld
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 17/18] arm64: Permit onlining of HVO-incompatible late CPUs if HVO is not in use

[James Houghton]

First, HVO must have ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU so that
incompatible CPUs can be onlined if HVO is not in use. The
late_cpu_enable() callback is used to check if HVO is truly in use.

If the late CPU is incompatible with HVO and HVO is *not* in use, HVO
becomes permanently disabled.

If the late CPU is incompatible with HVO and HVO *is* in use, the CPU
is not onlined.

If the late CPU is compatible with HVO, there is no issue.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/kernel/cpufeature.c | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 7e2c134f3b55..65bb3c7c9595 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -76,6 +76,7 @@
 #include <linux/kasan.h>
 #include <linux/percpu.h>
 #include <linux/sched/isolation.h>
+#include <linux/hugetlb_vmemmap.h>
 
 #include <asm/arm_pmuv3.h>
 #include <asm/cpu.h>
@@ -2181,6 +2182,22 @@ static bool hvo_compatible(const struct arm64_cpu_capabilities *caps, int scope)
 	return supports_hw_af(scope);
 }
 
+static bool late_cpu_enable_hvo(const struct arm64_cpu_capabilities *__unused)
+{
+#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
+	if (cpu_has_hw_af())
+		return true;
+
+	/*
+	 * If the CPU does not support HW AF, we cannot online it if HVO is
+	 * currently in use.
+	 */
+	return hugetlb_vmemmap_optimization_try_disable();
+#else
+	return true;
+#endif
+}
+
 static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused)
 {
 	/*
@@ -3079,8 +3096,11 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 	{
 		.desc = "HugeTLB Vmemmap Optimization Support",
 		.capability = ARM64_HVO_COMPATIBLE,
-		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.type = ARM64_CPUCAP_SCOPE_SYSTEM |
+			ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU |
+			ARM64_CPUCAP_PERMITTED_FOR_LATE_CPU,
 		.matches = hvo_compatible,
+		.late_cpu_enable = late_cpu_enable_hvo,
 	},
 	{
 		.desc = "52-bit Virtual Addressing for KVM (LPA2)",
-- 
2.55.0.795.g602f6c329a-goog

[PATCH 18/18] arm64: Remove user-selectable HVO Kconfig

[James Houghton]

The Kconfig is not ideal; it is niche and difficult to understand for
users. The Kconfig was added to avoid the following regression: if HVO
is compiled in and all boot CPUs support it, any late CPUs that do not
will not be onlined.

The implementation now avoids this regression as long as HVO is not
actively in use. Therefore, drop the Kconfig.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/Kconfig | 13 +------------
 1 file changed, 1 insertion(+), 12 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index cf4bd6bd7f11..2a59155f68dc 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -259,7 +259,7 @@ config ARM64
 	select USER_STACKTRACE_SUPPORT
 	select VDSO_GETRANDOM
 	select VMAP_STACK
-	select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if ARM64_WANT_OPTIMIZE_HUGETLB_VMEMMAP
+	select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if ARM64_HW_AFDBM
 	help
 	  ARM 64-bit (AArch64) Linux support.
 
@@ -1608,17 +1608,6 @@ config ARCH_SPARSEMEM_ENABLE
 	def_bool y
 	select SPARSEMEM_VMEMMAP_ENABLE
 
-config ARM64_WANT_OPTIMIZE_HUGETLB_VMEMMAP
-	bool "Enable support for HugeTLB Vmemmap Optimization (HVO)"
-	default n
-	depends on HUGETLB_PAGE && SPARSEMEM_VMEMMAP && ARM64_HW_AFDBM
-	help
-	  Enable support for HVO. When enabled, late-onlining of CPUs that do
-	  not support features required by HVO will not be onlined if all boot
-	  CPUs do have such support.
-
-	  If unsure, say N
-
 config HW_PERF_EVENTS
 	def_bool y
 	depends on ARM_PMU
-- 
2.55.0.795.g602f6c329a-goog

Re: [PATCH 00/18] Another attempt at HVO support on arm64

[Muchun Song]

> On Jul 8, 2026, at 11:11, James Houghton <jthoughton@google.com> wrote:
> 
> Hi everyone,
> 
> This patch series uses a trick with the Access Flag on CPUs that support
> hardware update of the AF to update vmemmap page table entries without
> introducing a time window where CPUs accessing the vmemmap might fault.
> 
> By avoiding faults, the HugeTLB vmemmap optimization (HVO) can be
> implemented correctly on arm64 in a much more straightforward way than
> previously attempted, most recently here[1] (please see [1] for a
> breakdown of the other approaches attempted before).

Hi,

Thanks for your ongoing efforts on supporting HVO on arm64. This is truly
one of the most useful features that arm64 has been missing for a while.

> 
> For large-memory systems that allocate most of their available memory to
> HugeTLB, HVO saves a huge amount of memory (1.5% of system memory).
> 
> This series has four parts:
>  1. Some preparatory changes (patches 1-3)
>  1. Bare minimum HVO support (patches 4-10)

It looks like there is a typo in the number here.

>  2. Drop BBML2_NOABORT requirement for HVO (patches 11-13)
>  3. Drop the user-configurable Kconfig for HVO (patches 14-18)
> 
> Parts 3 and 4 are technically optional. More details below.
> 
> The main functional caveat with this series is that bootmem HugeTLB
> pages are not "pre-HVOed". They will be HVOed, but because at pre-HVO
> time SMP CPUs have not been enabled, we cannot query for full system
> support.

Do you mean that the support for AF might vary across different CPUs?
I'm not that familiar with arm64, so it seems a bit strange to me that
such basic hardware features can differ so much from one CPU to another.

> 
> This series is based on 7.2-rc2 (0e35b9b6ec0f).
> 
> This series almost 100% cleanly applies to mm-new, which has some of
> Muchun's HVO patches, with one trivial conflict. I imagine this series
> will conflict pretty heavily with some of Muchun's other patches[2].

You know, as we iterate on HVO, the codebase is becoming increasingly
complex and difficult to maintain. Aside from the work in [2] that you
mentioned, I actually have some local patches aimed at refactoring
hugetlb_vmemmap.c as well to reduce this maintenance complexity. I really
want to avoid piling more code on top of it right now, as that would
only make things worse.

Therefore, my personal suggestion is that if this approach is
theoretically validated on arm64, it it better to wait until the HVO
refactoring is complete, and then review your work based on the clean
codebase. For now, what we need to focus on is evaluating the feasibility
and soundness of the approach itself. What do you think?

> 
> -- The AF trick --
> 
> The trick is that translations with the AF unset cannot be cached in the
> TLB (see Rule R_DWZCQ in the Arm ARM), so they can be atomically updated
> without needing a full break-before-make sequence.
> 
> So the PTE update sequence becomes:
>  1. Atomically clear the AF on the existing PTE.
>  2. Invalidate the TLB.
>  3. cmpxchg the AF=0 PTE with the new PTE. If this fails, goto 1.
> 
> If there is a CPU on the system that does not support hardware access
> flag updates, clearing the AF is problematic, as those CPUs might fault
> on the vmemmap usage. Therefore, HVO compatbility checks all CPUs for HW
> AF updates.

Sorry, I am not familiar with arm64, so I will leave it to the arm64
maintainers (or experts) to verify the feasibility here. Once the viability
of the approach is confirmed, I will look into the HVO-related implementation
details and consider how to support this feature on top of the refactored
code in the future.

> 
> -- Application to HVO --
> 
> HVO relies on the following page table transitions:
>  - When enabling HVO for a page, PMD block entries in the vmemmap are
>    shattered into PMD table entries. The first PTE remains mapped
>    normally (RW mapping to a real page of struct pages), but the
>    remaining PTEs in the vmemmap are mapped read-only to a shared
>    page of struct pages (that is, there is an OA change and a
>    permissions change).

Just to clarify, does OA mean Output Address? Please spell it out for
readers who are less familiar with this context.

Muchun,
Thanks.

>  - When disabling HVO for a page, the RO PTEs are remapped back to RW
>    PTEs that point to newly reallocated pages of struct pages. The
>    PMD block -> table transition is not undone.
> 
> In patches 4-10 of this series, I use the Access Flag trick to do the PTE
> OA and permissions updates. We rely on BBML2_NOABORT for the PMD block
> -> table transition.
> 
> In patches 11-13, I re-use the Access Flag trick to do the PMD block ->
> table transition without needing BBML2_NOABORT. For systems that support
> BBML2_NOABORT, the logic is unchanged.
> 
> I am aware of Linu's BBML3 patches; I've opted not to rebase onto them
> for now, but I am happy to do so later.
> 
> -- Late-onlining of CPUs that do not support HW AF --
> 
> One of the complications with this series is how to handle late-onlining
> of CPUs that do not have HW AF when HVO is in use. Naively, if HVO (HW
> AF) is supported on all boot CPUs and the kernel is compiled with HVO
> support, late CPUs that do not support HVO will not be onlined. This is
> a regression.
> 
> This series provides two ways of dealing with this. First, add a
> default-off Kconfig for users to enable HVO support, avoiding the
> regression. This is not ideal.
> 
> Patches 14-18 get rid of the new Kconfig by allowing onlining of
> HVO-incompatible late CPUs as long as HVO is not actively in use.
> 
> -- Litmus test --
> 
> The following Herd litmus test demonstrates the PTE update routine:
> 
>  AArch64 TTDFaultlessUpdate
>  Variant=vmsa
>  TTHM=HA
>  {
>   uint64_t x=1;
>   uint64_t y=2;
>   [PTE(x)]=(oa:PA(x), af:1);
>   0:X0=PTE(x); 1:X0=PTE(x);
>   0:X1=x; 1:X1=x;
>   pteval_t 0:X2=(oa:PA(x), af:0);
>   pteval_t 0:X3=(oa:PA(y), af:1);
>  }
>   P0                | P1             ;
>   LDR X4,[X0]       | L0:            ;
>   MOV X5,X4         | LDR X2,[X1]    ;
>   CAS X4,X2,[X0]    |                ;
>   DSB ISHST         |                ;
>   LSR X9,X1,#12     |                ;
>   TLBI VAALE1IS,X9  |                ;
>   DSB ISH           |                ;
>   ISB               |                ;
>   CAS X2,X3,[X0]    |                ;
>  exists
>    0:X5=0:X4 /\ (* First CAS must succeed *)
>    (fault(P1:L0) \/ ~(1:X2=2 \/ 1:X2=1))
> 
>  (* This test should not violate BBM requirements. *)
> 
> This test must be run with herdtools with Nikos's changes[3] to more
> accurately model BBM requirements. When tried, the output will notably
> *not* contain the "Warning-BBM-expected" flag.
> 
> -- Testing --
> 
> I haven't yet done extensive testing of this series. HVO is correctly
> freeing pages on my system, and the hugetlb-vmemmap test passes. Freeing
> HVOed HugeTLB pages also seems to function normally.
> 
> [1] https://lore.kernel.org/linux-arm-kernel/20241107202033.2721681-1-yuzhao@google.com/
> [2] https://lore.kernel.org/linux-mm/20260513130542.35604-1-songmuchun@bytedance.com/
> [3] https://github.com/herd/herdtools7/pull/1864
> 
> James Houghton (18):
>  hugetlb_vmemmap: Always flush TLB if needed upon PTE remapping
>  hugetlb_vmemmap: Move vmemmap_get_tail up
>  hugetlb_vmemmap: Leave pages partially HVOed upon restore failure
>  hugetlb_vmemmap: Use try_update_vmemmap_pte to update in-use PTEs
>  hugetlb_vmemmap: Allow architectures not to allow HVO at runtime
>  arm64: Rename cpu_has_hw_af to system_has_hw_af
>  arm64: Add system_supports_hvo
>  arm64: Implement try_update_vmemmap_pte using the AF trick
>  arm64: Prevent HVO if the HVO system feature is not enabled
>  arm64: Support hugetlb vmemmap optimization
>  hugetlb_vmemmap: Use try_populate_vmemmap_pmd for replacing in-use
>    PMDs
>  arm64: Implement try_populate_vmemmap_pmd using AF trick
>  arm64: Drop BBML2_NOABORT requirement for HVO
>  hugetlb_vmemmap: Rename mm/hugetlb_vmemmap.h to
>    mm/hugetlb_vmemmap_internal.h
>  hugetlb_vmemmap: Add a way to permanently disable HVO when needed
>  arm64: Allow "optional" CPU features to be required sometimes
>  arm64: Permit onlining of HVO-incompatible late CPUs if HVO is not in
>    use
>  arm64: Remove user-selectable HVO Kconfig
> 
> MAINTAINERS                                   |   3 +-
> arch/arm64/Kconfig                            |   1 +
> arch/arm64/include/asm/cpucaps.h              |   2 +
> arch/arm64/include/asm/cpufeature.h           |  39 ++-
> arch/arm64/include/asm/hugetlb.h              |   7 +
> arch/arm64/include/asm/pgalloc.h              |  54 ++++
> arch/arm64/include/asm/pgtable.h              |  57 ++++-
> arch/arm64/kernel/cpufeature.c                |  43 ++++
> arch/arm64/tools/cpucaps                      |   1 +
> arch/loongarch/include/asm/pgalloc.h          |   8 +
> arch/loongarch/include/asm/pgtable.h          |   8 +
> arch/riscv/include/asm/pgalloc.h              |   8 +
> arch/riscv/include/asm/pgtable.h              |   8 +
> arch/x86/include/asm/pgalloc.h                |   8 +
> arch/x86/include/asm/pgtable.h                |   8 +
> include/asm-generic/hugetlb.h                 |   7 +
> include/linux/hugetlb_vmemmap.h               |  20 ++
> include/linux/pgalloc.h                       |  20 ++
> include/linux/pgtable.h                       |  21 ++
> mm/hugetlb.c                                  |   2 +-
> mm/hugetlb_sysfs.c                            |   2 +-
> mm/hugetlb_vmemmap.c                          | 237 +++++++++++++-----
> ...b_vmemmap.h => hugetlb_vmemmap_internal.h} |   6 +-
> mm/sparse-vmemmap.c                           |   2 +-
> 24 files changed, 489 insertions(+), 83 deletions(-)
> create mode 100644 include/linux/hugetlb_vmemmap.h
> rename mm/{hugetlb_vmemmap.h => hugetlb_vmemmap_internal.h} (95%)
> 
> 
> base-commit: 0e35b9b6ec0ffcc5e23cbdec09f5c622ad532b53
> -- 
> 2.55.0.795.g602f6c329a-goog
> 

Re: [PATCH 00/18] Another attempt at HVO support on arm64

[James Houghton]

On Wed, Jul 8, 2026 at 1:41 AM Muchun Song <muchun.song@linux.dev> wrote:
> > On Jul 8, 2026, at 11:11, James Houghton <jthoughton@google.com> wrote:
> >
> > Hi everyone,
> >
> > This patch series uses a trick with the Access Flag on CPUs that support
> > hardware update of the AF to update vmemmap page table entries without
> > introducing a time window where CPUs accessing the vmemmap might fault.
> >
> > By avoiding faults, the HugeTLB vmemmap optimization (HVO) can be
> > implemented correctly on arm64 in a much more straightforward way than
> > previously attempted, most recently here[1] (please see [1] for a
> > breakdown of the other approaches attempted before).
>
> Hi,
>
> Thanks for your ongoing efforts on supporting HVO on arm64. This is truly
> one of the most useful features that arm64 has been missing for a while.

Hi Muchun,

Thanks for taking a look so quickly. :)

> >
> > For large-memory systems that allocate most of their available memory to
> > HugeTLB, HVO saves a huge amount of memory (1.5% of system memory).
> >
> > This series has four parts:
> >  1. Some preparatory changes (patches 1-3)
> >  1. Bare minimum HVO support (patches 4-10)
>
> It looks like there is a typo in the number here.

Oh yes, so there is. I'll fix it in the next version of the cover letter.

> >  2. Drop BBML2_NOABORT requirement for HVO (patches 11-13)
> >  3. Drop the user-configurable Kconfig for HVO (patches 14-18)
> >
> > Parts 3 and 4 are technically optional. More details below.
> >
> > The main functional caveat with this series is that bootmem HugeTLB
> > pages are not "pre-HVOed". They will be HVOed, but because at pre-HVO
> > time SMP CPUs have not been enabled, we cannot query for full system
> > support.
>
> Do you mean that the support for AF might vary across different CPUs?
> I'm not that familiar with arm64, so it seems a bit strange to me that
> such basic hardware features can differ so much from one CPU to another.

Yes, hardware updates of the Access Flag is a per-CPU feature. It is
available for a CPU to use if TCR_EL1.HA is set. TCR_EL1 is a system
register; each CPU has its own. (Linux will always enable HW AF for a
CPU when it is onlined[1] if support is advertised, so we simply need
to check if support is advertised to know that it is in fact enabled.)

These days it is not uncommon for a system to have two (or more?)
different core implementations, like with "fast" cores and "efficient"
cores.

[1] See the CONFIG_ARM64_HW_AFDBM bits in arch/arm64/mm/proc.S

> > This series is based on 7.2-rc2 (0e35b9b6ec0f).
> >
> > This series almost 100% cleanly applies to mm-new, which has some of
> > Muchun's HVO patches, with one trivial conflict. I imagine this series
> > will conflict pretty heavily with some of Muchun's other patches[2].
>
> You know, as we iterate on HVO, the codebase is becoming increasingly
> complex and difficult to maintain. Aside from the work in [2] that you
> mentioned, I actually have some local patches aimed at refactoring
> hugetlb_vmemmap.c as well to reduce this maintenance complexity. I really
> want to avoid piling more code on top of it right now, as that would
> only make things worse.
>
> Therefore, my personal suggestion is that if this approach is
> theoretically validated on arm64, it it better to wait until the HVO
> refactoring is complete, and then review your work based on the clean
> codebase. For now, what we need to focus on is evaluating the feasibility
> and soundness of the approach itself. What do you think?

I agree. The first order of business should be for the Arm folks to
tell me if they think this approach is unsound (though I've been
chatting with Will about this for quite some time :)).

I'm happy to rebase on top of your refactoring work when it is ready.

> >
> > -- The AF trick --
> >
> > The trick is that translations with the AF unset cannot be cached in the
> > TLB (see Rule R_DWZCQ in the Arm ARM), so they can be atomically updated
> > without needing a full break-before-make sequence.
> >
> > So the PTE update sequence becomes:
> >  1. Atomically clear the AF on the existing PTE.
> >  2. Invalidate the TLB.
> >  3. cmpxchg the AF=0 PTE with the new PTE. If this fails, goto 1.
> >
> > If there is a CPU on the system that does not support hardware access
> > flag updates, clearing the AF is problematic, as those CPUs might fault
> > on the vmemmap usage. Therefore, HVO compatbility checks all CPUs for HW
> > AF updates.
>
> Sorry, I am not familiar with arm64, so I will leave it to the arm64
> maintainers (or experts) to verify the feasibility here. Once the viability
> of the approach is confirmed, I will look into the HVO-related implementation
> details and consider how to support this feature on top of the refactored
> code in the future.

Sounds good, thanks Muchun.

I'm hoping you can look at patches 1-3 anyway. I think they're the
right thing to do, even without the arm64 changes.

> > -- Application to HVO --
> >
> > HVO relies on the following page table transitions:
> >  - When enabling HVO for a page, PMD block entries in the vmemmap are
> >    shattered into PMD table entries. The first PTE remains mapped
> >    normally (RW mapping to a real page of struct pages), but the
> >    remaining PTEs in the vmemmap are mapped read-only to a shared
> >    page of struct pages (that is, there is an OA change and a
> >    permissions change).
>
> Just to clarify, does OA mean Output Address? Please spell it out for
> readers who are less familiar with this context.

That's right, sorry about that. I'll spell it out more clearly in
future versions of the series.

Thanks, Muchun!

Re: [PATCH 00/18] Another attempt at HVO support on arm64

[Muchun Song]

> On Jul 9, 2026, at 00:49, James Houghton <jthoughton@google.com> wrote:
> 
> On Wed, Jul 8, 2026 at 1:41 AM Muchun Song <muchun.song@linux.dev> wrote:
>>> On Jul 8, 2026, at 11:11, James Houghton <jthoughton@google.com> wrote:
>>> 
>>> Hi everyone,
>>> 
>>> This patch series uses a trick with the Access Flag on CPUs that support
>>> hardware update of the AF to update vmemmap page table entries without
>>> introducing a time window where CPUs accessing the vmemmap might fault.
>>> 
>>> By avoiding faults, the HugeTLB vmemmap optimization (HVO) can be
>>> implemented correctly on arm64 in a much more straightforward way than
>>> previously attempted, most recently here[1] (please see [1] for a
>>> breakdown of the other approaches attempted before).
>> 
>> Hi,
>> 
>> Thanks for your ongoing efforts on supporting HVO on arm64. This is truly
>> one of the most useful features that arm64 has been missing for a while.
> 
> Hi Muchun,

Hi,

> 
> Thanks for taking a look so quickly. :)
> 
>>> 
>>> For large-memory systems that allocate most of their available memory to
>>> HugeTLB, HVO saves a huge amount of memory (1.5% of system memory).
>>> 
>>> This series has four parts:
>>> 1. Some preparatory changes (patches 1-3)
>>> 1. Bare minimum HVO support (patches 4-10)
>> 
>> It looks like there is a typo in the number here.
> 
> Oh yes, so there is. I'll fix it in the next version of the cover letter.
> 
>>> 2. Drop BBML2_NOABORT requirement for HVO (patches 11-13)
>>> 3. Drop the user-configurable Kconfig for HVO (patches 14-18)
>>> 
>>> Parts 3 and 4 are technically optional. More details below.
>>> 
>>> The main functional caveat with this series is that bootmem HugeTLB
>>> pages are not "pre-HVOed". They will be HVOed, but because at pre-HVO
>>> time SMP CPUs have not been enabled, we cannot query for full system
>>> support.
>> 
>> Do you mean that the support for AF might vary across different CPUs?
>> I'm not that familiar with arm64, so it seems a bit strange to me that
>> such basic hardware features can differ so much from one CPU to another.
> 
> Yes, hardware updates of the Access Flag is a per-CPU feature. It is
> available for a CPU to use if TCR_EL1.HA is set. TCR_EL1 is a system
> register; each CPU has its own. (Linux will always enable HW AF for a
> CPU when it is onlined[1] if support is advertised, so we simply need
> to check if support is advertised to know that it is in fact enabled.)
> 
> These days it is not uncommon for a system to have two (or more?)
> different core implementations, like with "fast" cores and "efficient"
> cores.
> 
> [1] See the CONFIG_ARM64_HW_AFDBM bits in arch/arm64/mm/proc.S

Thanks for your detailed explanation. When enabling HVO via the cmdline,
can we simply prevent CPUs that do not support AF from coming online?
Would implementing it this way be much simpler? In practice, developers
definitely know whether the current system is suitable for enabling HVO.
If some CPUs do not support AF, they would just need to evaluate the
trade-off between memory savings and having fewer online CPUs than expected.

For scenarios where HVO is enabled via sysctl, we simply need to check
if all CPUs support AF. If any do not, the system should return an error.

Then, we can proceed with the Pre-HVO.

> 
>>> This series is based on 7.2-rc2 (0e35b9b6ec0f).
>>> 
>>> This series almost 100% cleanly applies to mm-new, which has some of
>>> Muchun's HVO patches, with one trivial conflict. I imagine this series
>>> will conflict pretty heavily with some of Muchun's other patches[2].
>> 
>> You know, as we iterate on HVO, the codebase is becoming increasingly
>> complex and difficult to maintain. Aside from the work in [2] that you
>> mentioned, I actually have some local patches aimed at refactoring
>> hugetlb_vmemmap.c as well to reduce this maintenance complexity. I really
>> want to avoid piling more code on top of it right now, as that would
>> only make things worse.
>> 
>> Therefore, my personal suggestion is that if this approach is
>> theoretically validated on arm64, it it better to wait until the HVO
>> refactoring is complete, and then review your work based on the clean
>> codebase. For now, what we need to focus on is evaluating the feasibility
>> and soundness of the approach itself. What do you think?
> 
> I agree. The first order of business should be for the Arm folks to
> tell me if they think this approach is unsound (though I've been
> chatting with Will about this for quite some time :)).
> 
> I'm happy to rebase on top of your refactoring work when it is ready.
> 
>>> 
>>> -- The AF trick --
>>> 
>>> The trick is that translations with the AF unset cannot be cached in the
>>> TLB (see Rule R_DWZCQ in the Arm ARM), so they can be atomically updated
>>> without needing a full break-before-make sequence.
>>> 
>>> So the PTE update sequence becomes:
>>> 1. Atomically clear the AF on the existing PTE.
>>> 2. Invalidate the TLB.
>>> 3. cmpxchg the AF=0 PTE with the new PTE. If this fails, goto 1.
>>> 
>>> If there is a CPU on the system that does not support hardware access
>>> flag updates, clearing the AF is problematic, as those CPUs might fault
>>> on the vmemmap usage. Therefore, HVO compatbility checks all CPUs for HW
>>> AF updates.
>> 
>> Sorry, I am not familiar with arm64, so I will leave it to the arm64
>> maintainers (or experts) to verify the feasibility here. Once the viability
>> of the approach is confirmed, I will look into the HVO-related implementation
>> details and consider how to support this feature on top of the refactored
>> code in the future.
> 
> Sounds good, thanks Muchun.
> 
> I'm hoping you can look at patches 1-3 anyway. I think they're the
> right thing to do, even without the arm64 changes.

Yes, I've looked it over. I think that making modifications across
several areas will be much simpler on top of my refactored codebase.
For example:

 - In patch 2: There is no longer a need to move vmemmap_get_tail().
 - In patch 3: Handling partially-HVOed pages becomes straightforward
               because vmemmap_restore_pte() no longer relies on
               ->vmemmap_tail to detect whether a restoration is needed.
 - In patch 4: I have updated the ->remap_pte callback to return an int
               as well, making it easy to adapt your changes.

Overall, the implementation is significantly cleaner, so I won't list
every detail here.

Muchun,
Thanks.

> 
>>> -- Application to HVO --
>>> 
>>> HVO relies on the following page table transitions:
>>> - When enabling HVO for a page, PMD block entries in the vmemmap are
>>>   shattered into PMD table entries. The first PTE remains mapped
>>>   normally (RW mapping to a real page of struct pages), but the
>>>   remaining PTEs in the vmemmap are mapped read-only to a shared
>>>   page of struct pages (that is, there is an OA change and a
>>>   permissions change).
>> 
>> Just to clarify, does OA mean Output Address? Please spell it out for
>> readers who are less familiar with this context.
> 
> That's right, sorry about that. I'll spell it out more clearly in
> future versions of the series.
> 
> Thanks, Muchun!

Re: [PATCH 00/18] Another attempt at HVO support on arm64

[David Hildenbrand (Arm)]

> -- The AF trick --
> 
> The trick is that translations with the AF unset cannot be cached in the
> TLB (see Rule R_DWZCQ in the Arm ARM), so they can be atomically updated
> without needing a full break-before-make sequence.
> 
> So the PTE update sequence becomes:
>   1. Atomically clear the AF on the existing PTE.
>   2. Invalidate the TLB.
>   3. cmpxchg the AF=0 PTE with the new PTE. If this fails, goto 1.
> 
> If there is a CPU on the system that does not support hardware access
> flag updates, clearing the AF is problematic, as those CPUs might fault
> on the vmemmap usage. Therefore, HVO compatbility checks all CPUs for HW
> AF updates.

Pretty nice trick, I hope this will fly :)

-- 
Cheers,

David

Re: [PATCH 00/18] Another attempt at HVO support on arm64

[James Houghton]

On Thu, Jul 9, 2026 at 2:55 AM Muchun Song <muchun.song@linux.dev> wrote:
> > On Jul 9, 2026, at 00:49, James Houghton <jthoughton@google.com> wrote:
> >
> > On Wed, Jul 8, 2026 at 1:41 AM Muchun Song <muchun.song@linux.dev> wrote:
> >> Do you mean that the support for AF might vary across different CPUs?
> >> I'm not that familiar with arm64, so it seems a bit strange to me that
> >> such basic hardware features can differ so much from one CPU to another.
> >
> > Yes, hardware updates of the Access Flag is a per-CPU feature. It is
> > available for a CPU to use if TCR_EL1.HA is set. TCR_EL1 is a system
> > register; each CPU has its own. (Linux will always enable HW AF for a
> > CPU when it is onlined[1] if support is advertised, so we simply need
> > to check if support is advertised to know that it is in fact enabled.)
> >
> > These days it is not uncommon for a system to have two (or more?)
> > different core implementations, like with "fast" cores and "efficient"
> > cores.
> >
> > [1] See the CONFIG_ARM64_HW_AFDBM bits in arch/arm64/mm/proc.S
>
> Thanks for your detailed explanation. When enabling HVO via the cmdline,
> can we simply prevent CPUs that do not support AF from coming online?
> Would implementing it this way be much simpler? In practice, developers
> definitely know whether the current system is suitable for enabling HVO.
> If some CPUs do not support AF, they would just need to evaluate the
> trade-off between memory savings and having fewer online CPUs than expected.
>
> For scenarios where HVO is enabled via sysctl, we simply need to check
> if all CPUs support AF. If any do not, the system should return an error.
>
> Then, we can proceed with the Pre-HVO.

I don't think it makes sense to try to implement pre-HVO.

We cannot do HVO if any boot CPUs do not support HW AF, as HW AF will
be required to free the HugeTLB pages later, which we should continue
to support. Pre-HVO (today anyway) happens before all boot CPUs are
onlined. IMO it is not okay to prevent boot CPUs from onlining.

Let's say for a moment that HVO cannot be toggled at run-time, then
the best we can do is:
- If a user does not specify hugetlb_free_vmemmap=1, we can always
allow onlining of late CPUs
- If a user specifies hugetlb_free_vmemmap=1 but not all boot CPUs
support HW AF, we can always allow onlining of late CPUs.
- If a user specifies hugetlb_free_vmemmap=1 and all boot CPUs support
HW AF, we must not allow onlining incompatible late CPUs.

HVO compatibility has to be modeled as an Arm system feature. To
support conditional onlining of late CPUs based on whether or not "HVO
is being used", we still need the extra cpufeature.c logic.

So if HVO cannot be toggled at run-time, we can simplify the
definition of "is HVO being used?" (for the purposes of determining if
late CPUs can be onlined). We can simplify it from what it is now
("are there any optimized pages?") to
"vmemmap_optimize_enabled==true?". That allows us to drop patch 15
(which then needs a slight change to patch 17), but that's about it.
So this is a slight simplification, which is nice.

If you (or the Arm folks) feel strongly, I'm happy to write this simplification.

> > I'm hoping you can look at patches 1-3 anyway. I think they're the
> > right thing to do, even without the arm64 changes.
>
> Yes, I've looked it over. I think that making modifications across
> several areas will be much simpler on top of my refactored codebase.
> For example:
>
>  - In patch 2: There is no longer a need to move vmemmap_get_tail().
>  - In patch 3: Handling partially-HVOed pages becomes straightforward
>                because vmemmap_restore_pte() no longer relies on
>                ->vmemmap_tail to detect whether a restoration is needed.
>  - In patch 4: I have updated the ->remap_pte callback to return an int
>                as well, making it easy to adapt your changes.
>
> Overall, the implementation is significantly cleaner, so I won't list
> every detail here.

Thanks! Good to know I can basically drop those patches. :)

Please CC me on these HVO simplifications when you send them out, thanks!

Re: [PATCH 00/18] Another attempt at HVO support on arm64

[Muchun Song]

> On Jul 10, 2026, at 03:04, James Houghton <jthoughton@google.com> wrote:
> 
> On Thu, Jul 9, 2026 at 2:55 AM Muchun Song <muchun.song@linux.dev> wrote:
>>> On Jul 9, 2026, at 00:49, James Houghton <jthoughton@google.com> wrote:
>>> 
>>> On Wed, Jul 8, 2026 at 1:41 AM Muchun Song <muchun.song@linux.dev> wrote:
>>>> Do you mean that the support for AF might vary across different CPUs?
>>>> I'm not that familiar with arm64, so it seems a bit strange to me that
>>>> such basic hardware features can differ so much from one CPU to another.
>>> 
>>> Yes, hardware updates of the Access Flag is a per-CPU feature. It is
>>> available for a CPU to use if TCR_EL1.HA is set. TCR_EL1 is a system
>>> register; each CPU has its own. (Linux will always enable HW AF for a
>>> CPU when it is onlined[1] if support is advertised, so we simply need
>>> to check if support is advertised to know that it is in fact enabled.)
>>> 
>>> These days it is not uncommon for a system to have two (or more?)
>>> different core implementations, like with "fast" cores and "efficient"
>>> cores.
>>> 
>>> [1] See the CONFIG_ARM64_HW_AFDBM bits in arch/arm64/mm/proc.S
>> 
>> Thanks for your detailed explanation. When enabling HVO via the cmdline,
>> can we simply prevent CPUs that do not support AF from coming online?
>> Would implementing it this way be much simpler? In practice, developers
>> definitely know whether the current system is suitable for enabling HVO.
>> If some CPUs do not support AF, they would just need to evaluate the
>> trade-off between memory savings and having fewer online CPUs than expected.
>> 
>> For scenarios where HVO is enabled via sysctl, we simply need to check
>> if all CPUs support AF. If any do not, the system should return an error.
>> 
>> Then, we can proceed with the Pre-HVO.
> 
> I don't think it makes sense to try to implement pre-HVO.
> 
> We cannot do HVO if any boot CPUs do not support HW AF, as HW AF will
> be required to free the HugeTLB pages later, which we should continue
> to support. Pre-HVO (today anyway) happens before all boot CPUs are
> onlined. IMO it is not okay to prevent boot CPUs from onlining.

When the boot CPU starts, it first checks if the hardware (the current
boot CPU) supports AF (The selection of the Boot CPU is critical here
If the user really wants to enable HVO). If AF is supported and HVO is
enabled via cmdline, we perform a Pre-HVO. Subsequently, any CPUs that
do not support AF will be prevented from coming online.

If the boot CPU does not support AF, enabling HVO is not permitted—neither
via cmdline nor sysctl. All CPUs are allowed to be online.

Once the system has started, there is only one way to enable HVO, and
that is through sysctl. 1) If the current system includes any CPUs that do
not support AF, enabling HVO is not permitted (unless the user chooses to
take these CPUs that don't support AF offline). 2) If all online CPUs in the
current system support AF, then HVO can be enabled. In this case, the
system must also block any CPUs that do not support AF from coming online
in the future. Of course, if a user specifically wants those CPUs to be
allowed to come online, they can choose to disable HVO first (At the same time,
this means that the HVO-optimized HugeTLB must be freed first as well).

My proposal assumes that even though the system supports different types of
CPUs, their support for AF (I suppose this is a very basic feature nowadays)
is generally the same. I believe that for servers (Only servers stand to
benefit more from HVO), there shouldn't be a mix of those that support AF
and those that don't. At the very least, such a situation would be extremely
uncommon. I believe this may be acceptable. That said, if we go this route,
is there a way to simplify the code implementation even further?. Of course,
this is just my personal speculation. Please let me know if I've missed
anything.

> 
> Let's say for a moment that HVO cannot be toggled at run-time, then
> the best we can do is:
> - If a user does not specify hugetlb_free_vmemmap=1, we can always
> allow onlining of late CPUs
> - If a user specifies hugetlb_free_vmemmap=1 but not all boot CPUs
> support HW AF, we can always allow onlining of late CPUs.
> - If a user specifies hugetlb_free_vmemmap=1 and all boot CPUs support
> HW AF, we must not allow onlining incompatible late CPUs.
> 
> HVO compatibility has to be modeled as an Arm system feature. To
> support conditional onlining of late CPUs based on whether or not "HVO
> is being used", we still need the extra cpufeature.c logic.
> 
> So if HVO cannot be toggled at run-time, we can simplify the
> definition of "is HVO being used?" (for the purposes of determining if
> late CPUs can be onlined). We can simplify it from what it is now
> ("are there any optimized pages?") to
> "vmemmap_optimize_enabled==true?". That allows us to drop patch 15
> (which then needs a slight change to patch 17), but that's about it.
> So this is a slight simplification, which is nice.
> 
> If you (or the Arm folks) feel strongly, I'm happy to write this simplification.
> 
>>> I'm hoping you can look at patches 1-3 anyway. I think they're the
>>> right thing to do, even without the arm64 changes.
>> 
>> Yes, I've looked it over. I think that making modifications across
>> several areas will be much simpler on top of my refactored codebase.
>> For example:
>> 
>> - In patch 2: There is no longer a need to move vmemmap_get_tail().
>> - In patch 3: Handling partially-HVOed pages becomes straightforward
>>               because vmemmap_restore_pte() no longer relies on
>>               ->vmemmap_tail to detect whether a restoration is needed.
>> - In patch 4: I have updated the ->remap_pte callback to return an int
>>               as well, making it easy to adapt your changes.
>> 
>> Overall, the implementation is significantly cleaner, so I won't list
>> every detail here.
> 
> Thanks! Good to know I can basically drop those patches. :)
> 
> Please CC me on these HVO simplifications when you send them out, thanks!

No problem. Will do.

Thanks.

Re: [PATCH 00/18] Another attempt at HVO support on arm64

[Muchun Song]

> On Jul 8, 2026, at 11:11, James Houghton <jthoughton@google.com> wrote:
> 
> Hi everyone,
> 
> This patch series uses a trick with the Access Flag on CPUs that support
> hardware update of the AF to update vmemmap page table entries without
> introducing a time window where CPUs accessing the vmemmap might fault.

Hi James,

I was just looking into the Arm ARM specification regarding the TLB caching
behavior to better understand the "AF trick" used here, and came across a few
sections that got me a bit confused.

Specifically, Section D8.5.1 [1] states Rule RDWZCQ:

	"Descriptors with AF set to zero can never be cached in a TLB. For more
	 information about when translation table entries are permitted to be
	 cached in a TLB, see Translation Lookaside Buffers."

This indeed seems to support the core assumption of the AF trick. However, when
following the reference to Section D8.16 "Translation Lookaside Buffers" [2], it
defines the following rule for permitted caching (RSQBCS):

	"When address translation is enabled, a translation table entry for an
	 in-context translation regime that does not cause a Translation fault,
	 an Address size fault, or an Access flag fault is permitted to be cached
	 in a TLB or intermediate TLB caching structure as the result of an
	 explicit or speculative access."

This is exactly where my confusion lies, as these two descriptions in the spec
feel directly contradictory when FEAT_HAF(hardware management of the Access flag)
is enabled.

On one hand, Section D8.5.1 [1] explicitly uses the word "never" for AF=0
descriptors. On the other hand, under FEAT_HAF, an entry with AF=0 does not cause
an Access flag fault anymore, because the hardware is capable of updating the AF
automatically. This technically makes it eligible for speculative caching under
Rule RSQBCS in Section D8.16 [2].

My question is: which part of the spec reflects the actual architectural intent
when FEAT_HAF is enabled? Is an entry with AF=0 permitted to be speculatively
cached in the TLB by the hardware under FEAT_HAF, or does the "never" in RDWZCQ
still hold absolute priority?

Please let me know if I have misread the specification or missed some overriding
constraints here.

Best regards,
Muchun

References:
[1] https://developer.arm.com/documentation/ddi0487/mb/-Part-D-The-AArch64-System-Level-Architecture/-Chapter-D8-The-AArch64-Virtual-Memory-System-Architecture/-D8-5-Hardware-updates-to-the-translation-tables/-D8-5-1-The-Access-flag
[2] https://developer.arm.com/documentation/ddi0487/mb/-Part-D-The-AArch64-System-Level-Architecture/-Chapter-D8-The-AArch64-Virtual-Memory-System-Architecture/-D8-16-Translation-Lookaside-Buffers?lang=en#mdsec_translation_lookaside_buffers

> 
> By avoiding faults, the HugeTLB vmemmap optimization (HVO) can be
> implemented correctly on arm64 in a much more straightforward way than
> previously attempted, most recently here[1] (please see [1] for a
> breakdown of the other approaches attempted before).
> 
> For large-memory systems that allocate most of their available memory to
> HugeTLB, HVO saves a huge amount of memory (1.5% of system memory).
> 
> This series has four parts:
>  1. Some preparatory changes (patches 1-3)
>  1. Bare minimum HVO support (patches 4-10)
>  2. Drop BBML2_NOABORT requirement for HVO (patches 11-13)
>  3. Drop the user-configurable Kconfig for HVO (patches 14-18)
> 
> Parts 3 and 4 are technically optional. More details below.
> 
> The main functional caveat with this series is that bootmem HugeTLB
> pages are not "pre-HVOed". They will be HVOed, but because at pre-HVO
> time SMP CPUs have not been enabled, we cannot query for full system
> support.
> 
> This series is based on 7.2-rc2 (0e35b9b6ec0f).
> 
> This series almost 100% cleanly applies to mm-new, which has some of
> Muchun's HVO patches, with one trivial conflict. I imagine this series
> will conflict pretty heavily with some of Muchun's other patches[2].
> 
> -- The AF trick --
> 
> The trick is that translations with the AF unset cannot be cached in the
> TLB (see Rule R_DWZCQ in the Arm ARM), so they can be atomically updated
> without needing a full break-before-make sequence.
> 
> So the PTE update sequence becomes:
>  1. Atomically clear the AF on the existing PTE.
>  2. Invalidate the TLB.
>  3. cmpxchg the AF=0 PTE with the new PTE. If this fails, goto 1.
> 
> If there is a CPU on the system that does not support hardware access
> flag updates, clearing the AF is problematic, as those CPUs might fault
> on the vmemmap usage. Therefore, HVO compatbility checks all CPUs for HW
> AF updates.
> 
> -- Application to HVO --
> 
> HVO relies on the following page table transitions:
>  - When enabling HVO for a page, PMD block entries in the vmemmap are
>    shattered into PMD table entries. The first PTE remains mapped
>    normally (RW mapping to a real page of struct pages), but the
>    remaining PTEs in the vmemmap are mapped read-only to a shared
>    page of struct pages (that is, there is an OA change and a
>    permissions change).
>  - When disabling HVO for a page, the RO PTEs are remapped back to RW
>    PTEs that point to newly reallocated pages of struct pages. The
>    PMD block -> table transition is not undone.
> 
> In patches 4-10 of this series, I use the Access Flag trick to do the PTE
> OA and permissions updates. We rely on BBML2_NOABORT for the PMD block
> -> table transition.
> 
> In patches 11-13, I re-use the Access Flag trick to do the PMD block ->
> table transition without needing BBML2_NOABORT. For systems that support
> BBML2_NOABORT, the logic is unchanged.
> 
> I am aware of Linu's BBML3 patches; I've opted not to rebase onto them
> for now, but I am happy to do so later.
> 
> -- Late-onlining of CPUs that do not support HW AF --
> 
> One of the complications with this series is how to handle late-onlining
> of CPUs that do not have HW AF when HVO is in use. Naively, if HVO (HW
> AF) is supported on all boot CPUs and the kernel is compiled with HVO
> support, late CPUs that do not support HVO will not be onlined. This is
> a regression.
> 
> This series provides two ways of dealing with this. First, add a
> default-off Kconfig for users to enable HVO support, avoiding the
> regression. This is not ideal.
> 
> Patches 14-18 get rid of the new Kconfig by allowing onlining of
> HVO-incompatible late CPUs as long as HVO is not actively in use.
> 
> -- Litmus test --
> 
> The following Herd litmus test demonstrates the PTE update routine:
> 
>  AArch64 TTDFaultlessUpdate
>  Variant=vmsa
>  TTHM=HA
>  {
>   uint64_t x=1;
>   uint64_t y=2;
>   [PTE(x)]=(oa:PA(x), af:1);
>   0:X0=PTE(x); 1:X0=PTE(x);
>   0:X1=x; 1:X1=x;
>   pteval_t 0:X2=(oa:PA(x), af:0);
>   pteval_t 0:X3=(oa:PA(y), af:1);
>  }
>   P0                | P1             ;
>   LDR X4,[X0]       | L0:            ;
>   MOV X5,X4         | LDR X2,[X1]    ;
>   CAS X4,X2,[X0]    |                ;
>   DSB ISHST         |                ;
>   LSR X9,X1,#12     |                ;
>   TLBI VAALE1IS,X9  |                ;
>   DSB ISH           |                ;
>   ISB               |                ;
>   CAS X2,X3,[X0]    |                ;
>  exists
>    0:X5=0:X4 /\ (* First CAS must succeed *)
>    (fault(P1:L0) \/ ~(1:X2=2 \/ 1:X2=1))
> 
>  (* This test should not violate BBM requirements. *)
> 
> This test must be run with herdtools with Nikos's changes[3] to more
> accurately model BBM requirements. When tried, the output will notably
> *not* contain the "Warning-BBM-expected" flag.
> 
> -- Testing --
> 
> I haven't yet done extensive testing of this series. HVO is correctly
> freeing pages on my system, and the hugetlb-vmemmap test passes. Freeing
> HVOed HugeTLB pages also seems to function normally.
> 
> [1] https://lore.kernel.org/linux-arm-kernel/20241107202033.2721681-1-yuzhao@google.com/
> [2] https://lore.kernel.org/linux-mm/20260513130542.35604-1-songmuchun@bytedance.com/
> [3] https://github.com/herd/herdtools7/pull/1864
> 
> James Houghton (18):
>  hugetlb_vmemmap: Always flush TLB if needed upon PTE remapping
>  hugetlb_vmemmap: Move vmemmap_get_tail up
>  hugetlb_vmemmap: Leave pages partially HVOed upon restore failure
>  hugetlb_vmemmap: Use try_update_vmemmap_pte to update in-use PTEs
>  hugetlb_vmemmap: Allow architectures not to allow HVO at runtime
>  arm64: Rename cpu_has_hw_af to system_has_hw_af
>  arm64: Add system_supports_hvo
>  arm64: Implement try_update_vmemmap_pte using the AF trick
>  arm64: Prevent HVO if the HVO system feature is not enabled
>  arm64: Support hugetlb vmemmap optimization
>  hugetlb_vmemmap: Use try_populate_vmemmap_pmd for replacing in-use
>    PMDs
>  arm64: Implement try_populate_vmemmap_pmd using AF trick
>  arm64: Drop BBML2_NOABORT requirement for HVO
>  hugetlb_vmemmap: Rename mm/hugetlb_vmemmap.h to
>    mm/hugetlb_vmemmap_internal.h
>  hugetlb_vmemmap: Add a way to permanently disable HVO when needed
>  arm64: Allow "optional" CPU features to be required sometimes
>  arm64: Permit onlining of HVO-incompatible late CPUs if HVO is not in
>    use
>  arm64: Remove user-selectable HVO Kconfig
> 
> MAINTAINERS                                   |   3 +-
> arch/arm64/Kconfig                            |   1 +
> arch/arm64/include/asm/cpucaps.h              |   2 +
> arch/arm64/include/asm/cpufeature.h           |  39 ++-
> arch/arm64/include/asm/hugetlb.h              |   7 +
> arch/arm64/include/asm/pgalloc.h              |  54 ++++
> arch/arm64/include/asm/pgtable.h              |  57 ++++-
> arch/arm64/kernel/cpufeature.c                |  43 ++++
> arch/arm64/tools/cpucaps                      |   1 +
> arch/loongarch/include/asm/pgalloc.h          |   8 +
> arch/loongarch/include/asm/pgtable.h          |   8 +
> arch/riscv/include/asm/pgalloc.h              |   8 +
> arch/riscv/include/asm/pgtable.h              |   8 +
> arch/x86/include/asm/pgalloc.h                |   8 +
> arch/x86/include/asm/pgtable.h                |   8 +
> include/asm-generic/hugetlb.h                 |   7 +
> include/linux/hugetlb_vmemmap.h               |  20 ++
> include/linux/pgalloc.h                       |  20 ++
> include/linux/pgtable.h                       |  21 ++
> mm/hugetlb.c                                  |   2 +-
> mm/hugetlb_sysfs.c                            |   2 +-
> mm/hugetlb_vmemmap.c                          | 237 +++++++++++++-----
> ...b_vmemmap.h => hugetlb_vmemmap_internal.h} |   6 +-
> mm/sparse-vmemmap.c                           |   2 +-
> 24 files changed, 489 insertions(+), 83 deletions(-)
> create mode 100644 include/linux/hugetlb_vmemmap.h
> rename mm/{hugetlb_vmemmap.h => hugetlb_vmemmap_internal.h} (95%)
> 
> 
> base-commit: 0e35b9b6ec0ffcc5e23cbdec09f5c622ad532b53
> -- 
> 2.55.0.795.g602f6c329a-goog
>