Linux-mm Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: James Houghton <jthoughton@google.com>
To: Will Deacon <will@kernel.org>,
	Catalin Marinas <catalin.marinas@arm.com>,
	 Muchun Song <muchun.song@linux.dev>,
	Oscar Salvador <osalvador@suse.de>,
	 Andrew Morton <akpm@linux-foundation.org>
Cc: Kefeng Wang <wangkefeng.wang@huawei.com>,
	Nikos Nikoleris <nikos.nikoleris@arm.com>,
	 Linu Cherian <linu.cherian@arm.com>,
	Mark Rutland <mark.rutland@arm.com>,
	 David Hildenbrand <david@kernel.org>,
	Ryan Roberts <ryan.roberts@arm.com>,
	 Nanyong Sun <sunnanyong@huawei.com>, Yu Zhao <yuzhao@google.com>,
	 Frank van der Linden <fvdl@google.com>,
	David Rientjes <rientjes@google.com>,
	 James Houghton <jthoughton@google.com>,
	linux-kernel@vger.kernel.org,
	 linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org
Subject: [PATCH v3 12/19] arm64: Support hugetlb vmemmap optimization
Date: Sun, 22 Nov 2026 18:44:46 +0000	[thread overview]
Message-ID: <20261122184453.2668189-13-jthoughton@google.com> (raw)
In-Reply-To: <20261122184453.2668189-1-jthoughton@google.com>

Support HVO if HW AF and BBML3 are supported. These are the required
features for the vmemmap to be modified in place.

On arm64 systems that do not have the required hardware features to
support HVO, the HVO sysctl will not appear, and any attempts to enable
HVO (either via the command line or via
CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON=y) will have no effect.

Because HVO support is only known after the vmemmap has been populated,
disable the section-based vmemmap optimization in vmemmap_populate(), as
LoongArch and powerpc do. Boot-time HugeTLB folios are then optimized
when they are gathered.

Signed-off-by: James Houghton <jthoughton@google.com>
---
 arch/arm64/Kconfig               | 1 +
 arch/arm64/include/asm/pgtable.h | 6 ++++++
 arch/arm64/mm/mmu.c              | 9 +++++++++
 mm/hugetlb_vmemmap.c             | 4 ++++
 4 files changed, 20 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 4cce60c2de66..b13618f0ace5 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -95,6 +95,7 @@ config ARM64
 	select ARCH_WANT_FRAME_POINTERS
 	select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36)
 	select ARCH_WANT_LD_ORPHAN_WARN
+	select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if ARM64_HW_AFDBM
 	select ARCH_WANTS_EXECMEM_LATE
 	select ARCH_WANTS_NO_INSTR
 	select ARCH_WANTS_THP_SWAP if ARM64_4K_PAGES
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 7421c0a048ff..5dadaac54bef 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1931,6 +1931,12 @@ static inline void clear_young_dirty_ptes(struct vm_area_struct *vma,
 
 #endif /* CONFIG_ARM64_CONTPTE */
 
+#define arch_support_vmemmap_remap arch_support_vmemmap_remap
+static inline bool arch_support_vmemmap_remap(void)
+{
+	return system_supports_bbm_through_af();
+}
+
 static inline int try_update_vmemmap_pte(unsigned long addr, pte_t *ptep,
 					 const pte_t pte)
 {
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index a61464cf7367..bc3eea16004c 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -30,6 +30,7 @@
 #include <linux/mm_inline.h>
 #include <linux/pagewalk.h>
 #include <linux/stop_machine.h>
+#include <linux/vmemmap-optimization.h>
 
 #include <asm/barrier.h>
 #include <asm/cputype.h>
@@ -1843,6 +1844,14 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
 	/* [start, end] should be within one section */
 	WARN_ON_ONCE(end - start > PAGES_PER_SECTION * sizeof(struct page));
 
+	/*
+	 * Whether HVO is supported is only known once the system-wide CPU
+	 * features have been finalized, which is after the vmemmap has been
+	 * populated, so the section-based vmemmap optimization (pre-HVO) is
+	 * not supported. Boot-time HugeTLB folios are optimized later instead.
+	 */
+	section_vmemmap_optimization_disable(start, end);
+
 	if (!IS_ENABLED(CONFIG_ARM64_4K_PAGES) ||
 	    (end - start < PAGES_PER_SECTION * sizeof(struct page)))
 		return vmemmap_populate_basepages(start, end, node, altmap);
diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
index fb6da64bd80d..a01377c37c96 100644
--- a/mm/hugetlb_vmemmap.c
+++ b/mm/hugetlb_vmemmap.c
@@ -79,6 +79,10 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,
 
 		/* Make pte visible before pmd. See comment in pmd_install(). */
 		smp_wmb();
+		/*
+		 * On arm64, this requires BBML3. Its support has already been
+		 * checked.
+		 */
 		pmd_populate_kernel(&init_mm, pmd, pgtable);
 		/*
 		 * Modern MMUs allow overlapping TLB entries with different mapping
-- 
2.56.1.809.ga0b61035b5-goog



  parent reply	other threads:[~2026-11-22 18:45 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-22 18:44 [PATCH v3 00/19] Another attempt at HVO support on arm64 James Houghton
2026-11-22 18:44 ` [PATCH v3 01/19] arm64/pgtable: Clear AF with LDCLR on supported systems James Houghton
2026-11-22 18:44 ` [PATCH v3 02/19] hugetlb_vmemmap: Always flush TLB if needed upon PTE remapping James Houghton
2026-11-22 18:44 ` [PATCH v3 03/19] hugetlb_vmemmap: Leave pages partially HVOed upon restore failure James Houghton
2026-11-22 18:44 ` [PATCH v3 04/19] hugetlb_vmemmap: Use try_update_vmemmap_pte to update in-use PTEs James Houghton
2026-11-22 18:44 ` [PATCH v3 05/19] hugetlb_vmemmap: Allow architectures to dynamically disallow HVO James Houghton
2026-11-22 18:44 ` [PATCH v3 06/19] hugetlb: Optimize bootmem folios that were not pre-HVOed James Houghton
2026-11-22 18:44 ` [PATCH v3 07/19] hugetlb_vmemmap: Allow architectures to make HVO enablement boot-time only James Houghton
2026-11-22 18:44 ` [PATCH v3 08/19] hugetlb_vmemmap: Expose whether HVO is enabled to architecture code James Houghton
2026-11-22 18:44 ` [PATCH v3 09/19] arm64: Add bbm_through_af capability James Houghton
2026-11-22 18:44 ` [PATCH v3 10/19] arm64: Implement try_update_vmemmap_pte using the AF trick James Houghton
2026-11-22 18:44 ` [PATCH v3 11/19] arm64/mm: Align vmemmap to maximal folio size James Houghton
2026-11-22 18:44 ` James Houghton [this message]
2026-11-22 18:44 ` [PATCH v3 13/19] hugetlb_vmemmap: Add fault injection for in-place vmemmap PTE updates James Houghton
2026-11-22 18:44 ` [PATCH v3 14/19] selftests/mm: Add HugeTLB vmemmap optimization stress test James Houghton
2026-11-22 18:44 ` [PATCH v3 15/19 DO-NOT-MERGE] hugetlb_vmemmap: Use try_populate_vmemmap_pmd for replacing in-use PMDs James Houghton
2026-11-22 18:44 ` [PATCH v3 16/19 DO-NOT-MERGE] arm64: Implement try_populate_vmemmap_pmd using AF trick James Houghton
2026-11-22 18:44 ` [PATCH v3 17/19 DO-NOT-MERGE] arm64: Drop BBML3 requirement for HVO James Houghton
2026-11-22 18:44 ` [PATCH v3 18/19 DO-NOT-MERGE] hugetlb_vmemmap: Add fault injection for in-place vmemmap PMD splits James Houghton
2026-11-22 18:44 ` [PATCH v3 19/19 DO-NOT-MERGE] selftests/mm: Add HVO pmd-split fault injection tests James Houghton

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261122184453.2668189-13-jthoughton@google.com \
    --to=jthoughton@google.com \
    --cc=akpm@linux-foundation.org \
    --cc=catalin.marinas@arm.com \
    --cc=david@kernel.org \
    --cc=fvdl@google.com \
    --cc=linu.cherian@arm.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mark.rutland@arm.com \
    --cc=muchun.song@linux.dev \
    --cc=nikos.nikoleris@arm.com \
    --cc=osalvador@suse.de \
    --cc=rientjes@google.com \
    --cc=ryan.roberts@arm.com \
    --cc=sunnanyong@huawei.com \
    --cc=wangkefeng.wang@huawei.com \
    --cc=will@kernel.org \
    --cc=yuzhao@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox