[PATCH v3 0/2] mm: optimize HugeTLB bootmem hugepage init

[PATCH v3 0/2] mm: optimize HugeTLB bootmem hugepage init

[Li Zhe]

This series speeds up the fallback initialization path for HugeTLB tail
struct pages during boot-time hugepage allocation, when HugeTLB vmemmap
optimization (HVO) is unavailable or fails.

Patch 1 factors the PFN-dependent template refresh from the ZONE_DEVICE
template-copy path into memmap_init_refresh_pfn_fields().  The
ZONE_DEVICE path keeps its memcpy_nontemporal() copy primitive.

Patch 2 adds a HugeTLB-specific copy helper in
hugetlb_folio_init_tail_vmemmap(): initialize the first tail page
normally, then initialize the remaining tail pages by copying that page
with memcpy() and refreshing only PFN-dependent fields in the destination
page.

In the earlier discussion, Mike asked whether the same kind of
template-based initialization also helps the ordinary struct page
initialization path:

https://lore.kernel.org/all/akfB6Qe-jKDEMUxz@kernel.org/

I tested that direction as well, but did not see a measurable benefit
for ordinary struct page initialization. The measurable improvement I
could find is in the HugeTLB bootmem hugepage tail path, where gigantic
pages can require initializing a large number of tail struct pages when
HVO does not cover the allocation.

Tested on a v7.4-rc2 base with 4K base pages and 64 boot-time 1G
hugepages, measured with a temporary jiffies-based printk.  HVO is
unavailable on arm64; on x86-64, hugetlb_free_vmemmap=off was used to
exercise the fallback path.

  arch    base      with this patch  reduction
  arm64   360.8 ms  256.0 ms         29.05%
  x86-64  171.0 ms  165.0 ms          3.51%

I also checked memcpy_nontemporal() for the HugeTLB helper, but it did
not show a similar optimization to the PMEM/ZONE_DEVICE template-copy
path.  One likely reason is that HugeTLB initializes tail struct pages in
per-hugepage chunks, while the ZONE_DEVICE path is closer to a long
streaming initialization over a device range.  In this HugeTLB fallback
path, the measured x86-64 result was better with memcpy(), so this series
keeps memcpy_nontemporal() in the ZONE_DEVICE helper and uses memcpy() in
the HugeTLB helper.

The temporary measurement instrumentation is not part of this series.

Li Zhe (2):
  mm/mm_init: factor out memmap_init_refresh_pfn_fields()
  mm/hugetlb: optimize bootmem hugepage tail vmemmap init

 mm/hugetlb.c | 23 ++++++++++++++++++-----
 mm/mm_init.c | 11 ++++++-----
 mm/mm_init.h | 11 +++++++++++
 3 files changed, 35 insertions(+), 10 deletions(-)

---
v2: https://lore.kernel.org/all/20261110114859.76343-1-lizhe.67@bytedance.com/
v1: https://lore.kernel.org/all/20261105120615.16817-1-lizhe.67@bytedance.com/#t

Changes in v3:
- Share only the PFN-dependent template refresh.  Keep
  memcpy_nontemporal() in the ZONE_DEVICE helper and use memcpy() in
  the HugeTLB helper.
- Copy HugeTLB tail pages from the first initialized tail page and
  refresh PFN-dependent fields in the destination, avoiding a local
  template in the HugeTLB helper.
- Restore the empty-range guard in hugetlb_folio_init_tail_vmemmap()
  after Sashiko reported a valid powerpc 8xx case.

Changes in v2:
- Add a comment for the template-before-copy ordering, as suggested by
  Muchun Song.
- Drop the empty-range guard in hugetlb_folio_init_tail_vmemmap()
  as suggested by Balbir Singh and Mike Rapoport.
-- 
2.20.1

[PATCH v3 1/2] mm/mm_init: factor out memmap_init_refresh_pfn_fields()

[Li Zhe]

ZONE_DEVICE memmap initialization already uses a reusable struct page
template when initializing compound tail pages.  The template mostly
contains invariant state, but a few fields still depend on the target PFN
and must be refreshed before copying the template into each destination
struct page.

Factor that refresh into memmap_init_refresh_pfn_fields() so other
boot-time memmap initialization paths can reuse the PFN-dependent part
while keeping their own copy primitive.

Define the helper as static inline in mm/mm_init.h, since it is small
and only shared by mm/ initialization code.

No functional change intended.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 mm/mm_init.c | 11 ++++++-----
 mm/mm_init.h | 11 +++++++++++
 2 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/mm/mm_init.c b/mm/mm_init.c
index d99af80ec752..33a9d8dd044c 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -1076,11 +1076,12 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn,
 static void zone_device_page_init_from_template(struct page *page,
 		unsigned long pfn, struct page *template)
 {
-	set_page_section_from_pfn(template, pfn);
-#ifdef WANT_PAGE_VIRTUAL
-	if (!is_highmem_idx(ZONE_DEVICE))
-		set_page_address(template, __va(pfn << PAGE_SHIFT));
-#endif
+	/*
+	 * Refresh PFN-dependent fields in the template before the
+	 * non-temporal copy, so the destination page does not need cached
+	 * stores after memcpy_nontemporal().
+	 */
+	memmap_init_refresh_pfn_fields(template, pfn, ZONE_DEVICE);
 	memcpy_nontemporal(page, template, sizeof(*page));
 }
 
diff --git a/mm/mm_init.h b/mm/mm_init.h
index 0034e37dbabd..7e1d6897b844 100644
--- a/mm/mm_init.h
+++ b/mm/mm_init.h
@@ -112,4 +112,15 @@ void memblock_free_all(void);
 void __meminit __init_single_page(struct page *page, unsigned long pfn,
 				unsigned long zone, int nid);
 
+static inline void memmap_init_refresh_pfn_fields(struct page *page,
+						  unsigned long pfn,
+						  unsigned long zone)
+{
+	set_page_section_from_pfn(page, pfn);
+#ifdef WANT_PAGE_VIRTUAL
+	if (!is_highmem_idx(zone))
+		set_page_address(page, __va(pfn << PAGE_SHIFT));
+#endif
+}
+
 #endif /* __MM_MM_INIT_H */
-- 
2.20.1

[PATCH v3 2/2] mm/hugetlb: optimize bootmem hugepage tail vmemmap init

[Li Zhe]

Boot-time HugeTLB allocation has to initialize tail struct pages itself
when HugeTLB vmemmap optimization (HVO) is unavailable or fails.  For
gigantic pages, this can cover a large number of tail pages.  Most state
installed by __init_single_page() and prep_compound_tail() is identical
for every tail page in the same huge page; only PFN-dependent fields need
refreshing.

Initialize the first tail page normally, then initialize the remaining
tails with hugetlb_init_tail_page().  This copies the already
initialized first tail page with memcpy() and refreshes only the
PFN-dependent fields in the destination page, while avoiding the full
__init_single_page() and prep_compound_tail() sequence for state that is
identical across tail pages in the same huge page.

The fallback path was measured on a v7.4-rc2 base with a temporary
jiffies-based printk modeled after the existing timing in
memmap_init_zone_device(), using 4K base pages and 64 boot-time 1G
hugepages.  HVO is unavailable on arm64; on x86-64,
hugetlb_free_vmemmap=off was used to exercise the fallback path.

  arch    base      with this patch  reduction
  arm64   360.8 ms  256.0 ms         29.05%
  x86-64  171.0 ms  165.0 ms          3.51%

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 mm/hugetlb.c | 23 ++++++++++++++++++-----
 1 file changed, 18 insertions(+), 5 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 630c96cd1e88..13efadf64023 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3214,6 +3214,14 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid)
 	return true;
 }
 
+static void __init hugetlb_init_tail_page(struct page *page, unsigned long pfn,
+					  enum zone_type zone,
+					  const struct page *source)
+{
+	memcpy(page, source, sizeof(*page));
+	memmap_init_refresh_pfn_fields(page, pfn, zone);
+}
+
 /* Initialize [start_page:end_page_number] tail struct pages of a hugepage */
 static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
 					struct hstate *h,
@@ -3231,11 +3239,16 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
 	 * As we marked all tail pages with memblock_reserved_mark_noinit(),
 	 * we must initialize them ourselves here.
 	 */
-	for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
-		__init_single_page(page, pfn, zone, nid);
-		prep_compound_tail(page, &folio->page, order);
-		set_page_count(page, 0);
-	}
+	if (start_page_number >= end_page_number)
+		return;
+
+	pfn = head_pfn + start_page_number;
+	__init_single_page(page, pfn, zone, nid);
+	prep_compound_tail(page, &folio->page, order);
+	set_page_count(page, 0);
+
+	for (pfn++; pfn < end_pfn; pfn++)
+		hugetlb_init_tail_page(pfn_to_page(pfn), pfn, zone, page);
 }
 
 static void __init hugetlb_folio_init_vmemmap(struct folio *folio,
-- 
2.20.1

Re: [PATCH v3 0/2] mm: optimize HugeTLB bootmem hugepage init

[Mike Rapoport]

Hi,

On Thu, Nov 12, 2026 at 07:28:14PM +0800, Li Zhe wrote:
> This series speeds up the fallback initialization path for HugeTLB tail
> struct pages during boot-time hugepage allocation, when HugeTLB vmemmap
> optimization (HVO) is unavailable or fails.
> 
> Patch 1 factors the PFN-dependent template refresh from the ZONE_DEVICE
> template-copy path into memmap_init_refresh_pfn_fields().  The
> ZONE_DEVICE path keeps its memcpy_nontemporal() copy primitive.
> 
> Patch 2 adds a HugeTLB-specific copy helper in
> hugetlb_folio_init_tail_vmemmap(): initialize the first tail page
> normally, then initialize the remaining tail pages by copying that page
> with memcpy() and refreshing only PFN-dependent fields in the destination
> page.
> 
> Tested on a v7.4-rc2 base with 4K base pages and 64 boot-time 1G
> hugepages, measured with a temporary jiffies-based printk.  HVO is
> unavailable on arm64; on x86-64, hugetlb_free_vmemmap=off was used to
> exercise the fallback path.
> 
>   arch    base      with this patch  reduction
>   arm64   360.8 ms  256.0 ms         29.05%
>   x86-64  171.0 ms  165.0 ms          3.51%

Thanks for the numbers!
 
The changes are fine, but since they are going to be merged into Muchun's
series, I'll wait for that for the detailed review.

> Li Zhe (2):
>   mm/mm_init: factor out memmap_init_refresh_pfn_fields()
>   mm/hugetlb: optimize bootmem hugepage tail vmemmap init
> 
>  mm/hugetlb.c | 23 ++++++++++++++++++-----
>  mm/mm_init.c | 11 ++++++-----
>  mm/mm_init.h | 11 +++++++++++
>  3 files changed, 35 insertions(+), 10 deletions(-)

-- 
Sincerely yours,
Mike.