[PATCH v2 0/2] mm: optimize HugeTLB bootmem hugepage init

[PATCH v2 0/2] mm: optimize HugeTLB bootmem hugepage init

[Li Zhe]

This series speeds up the fallback initialization path for HugeTLB tail
struct pages during boot-time hugepage allocation, when HugeTLB vmemmap
optimization (HVO) is unavailable or fails.

Patch 1 factors the existing ZONE_DEVICE template-copy operation into
memmap_init_page_from_template().

Patch 2 reuses that helper in hugetlb_folio_init_tail_vmemmap():
initialize the first tail page normally, copy it into a local template,
then initialize the remaining tail pages by refreshing only PFN-dependent
fields before copying the template.

In the earlier discussion, Mike asked whether the same kind of
template-based initialization also helps the ordinary struct page
initialization path:

https://lore.kernel.org/all/akfB6Qe-jKDEMUxz@kernel.org/

I tested that direction as well, but did not see a measurable benefit
for ordinary struct page initialization. The measurable improvement I
could find is in the HugeTLB bootmem hugepage tail path, where gigantic
pages can require initializing a large number of tail struct pages when
HVO does not cover the allocation.

Tested on an arm64 QEMU/KVM guest with 4K base pages and 64 boot-time 1G
hugepages. HVO is unavailable on arm64 in v7.4-rc1, so the fallback tail
initialization path is exercised.

Results averaged over five boots, measured with a temporary
jiffies-based printk:

  base (v7.4-rc1):  318.4 ms
  with this patch:  202.4 ms

This is a 36.43% reduction in the measured fallback tail initialization
time. The temporary measurement instrumentation is not part of this
series.

Li Zhe (2):
  mm/mm_init: factor out memmap_init_page_from_template()
  mm/hugetlb: optimize bootmem hugepage tail vmemmap init

 mm/hugetlb.c | 20 +++++++++++++++-----
 mm/mm_init.c | 34 ++++++++++++++++++++--------------
 mm/mm_init.h |  2 ++
 3 files changed, 37 insertions(+), 19 deletions(-)

---
v1: https://lore.kernel.org/all/20261105120615.16817-1-lizhe.67@bytedance.com/#t

Changes in v2:
- Add a comment for the template-before-copy ordering, as suggested by
  Muchun Song.
- Drop the empty-range guard in hugetlb_folio_init_tail_vmemmap()
  as suggested by Balbir Singh and Mike Rapoport; current callers are
  expected to pass a non-empty range.

-- 
2.20.1

[PATCH v2 1/2] mm/mm_init: factor out memmap_init_page_from_template()

[Li Zhe]

ZONE_DEVICE memmap initialization already uses a reusable struct page
template when initializing compound tail pages.  The template mostly
contains invariant state, but a few fields still depend on the target PFN
and must be refreshed before copying the template into each destination
struct page.

Factor that operation into memmap_init_page_from_template() so other
boot-time memmap initialization paths can reuse the same helper.

No functional change intended.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 mm/mm_init.c | 34 ++++++++++++++++++++--------------
 mm/mm_init.h |  2 ++
 2 files changed, 22 insertions(+), 14 deletions(-)

diff --git a/mm/mm_init.c b/mm/mm_init.c
index d99af80ec752..905222772548 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -625,6 +625,22 @@ void __meminit __init_single_page(struct page *page, unsigned long pfn,
 	vmemmap_optimization_verify_zone(page);
 }
 
+void __meminit memmap_init_page_from_template(struct page *page,
+		unsigned long pfn, unsigned long zone, struct page *template)
+{
+	/*
+	 * Refresh PFN-dependent fields in the template before the
+	 * non-temporal copy, so the destination page does not need cached
+	 * stores after memcpy_nontemporal().
+	 */
+	set_page_section_from_pfn(template, pfn);
+#ifdef WANT_PAGE_VIRTUAL
+	if (!is_highmem_idx(zone))
+		set_page_address(template, __va(pfn << PAGE_SHIFT));
+#endif
+	memcpy_nontemporal(page, template, sizeof(*page));
+}
+
 #ifdef CONFIG_NUMA
 /*
  * During memory init memblocks map pfns to nids. The search is expensive and
@@ -1073,17 +1089,6 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn,
 	}
 }
 
-static void zone_device_page_init_from_template(struct page *page,
-		unsigned long pfn, struct page *template)
-{
-	set_page_section_from_pfn(template, pfn);
-#ifdef WANT_PAGE_VIRTUAL
-	if (!is_highmem_idx(ZONE_DEVICE))
-		set_page_address(template, __va(pfn << PAGE_SHIFT));
-#endif
-	memcpy_nontemporal(page, template, sizeof(*page));
-}
-
 /*
  * With compound page geometry and when struct pages are stored in ram most
  * tail pages are reused. Consequently, the amount of unique struct pages to
@@ -1136,8 +1141,8 @@ static void __ref memmap_init_compound(struct page *head,
 
 	/* Initialize the remaining tail pages from template. */
 	for (pfn = head_pfn + 2; pfn < end_pfn; pfn++)
-		zone_device_page_init_from_template(pfn_to_page(pfn), pfn,
-						    &template);
+		memmap_init_page_from_template(pfn_to_page(pfn), pfn,
+					       ZONE_DEVICE, &template);
 	prep_compound_head(head, order);
 }
 
@@ -1192,7 +1197,8 @@ void __ref memmap_init_zone_device(struct zone *zone,
 	for (; pfn < end_pfn; pfn += pfns_per_compound) {
 		page = pfn_to_page(pfn);
 
-		zone_device_page_init_from_template(page, pfn, &template);
+		memmap_init_page_from_template(page, pfn, ZONE_DEVICE,
+					       &template);
 
 		if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
 			cond_resched();
diff --git a/mm/mm_init.h b/mm/mm_init.h
index 0034e37dbabd..22c465122b47 100644
--- a/mm/mm_init.h
+++ b/mm/mm_init.h
@@ -111,5 +111,7 @@ void memblock_free_all(void);
 
 void __meminit __init_single_page(struct page *page, unsigned long pfn,
 				unsigned long zone, int nid);
+void __meminit memmap_init_page_from_template(struct page *page,
+		unsigned long pfn, unsigned long zone, struct page *template);
 
 #endif /* __MM_MM_INIT_H */
-- 
2.20.1

[PATCH v2 2/2] mm/hugetlb: optimize bootmem hugepage tail vmemmap init

[Li Zhe]

Boot-time HugeTLB allocation has to initialize tail struct pages itself
when HugeTLB vmemmap optimization (HVO) is unavailable or fails.  For
gigantic pages, this can cover a large number of tail pages.  Most state
installed by __init_single_page() and prep_compound_tail() is identical
for every tail page in the same huge page; only PFN-dependent fields need
refreshing.

Initialize the first tail page normally, copy it into a local template,
then initialize the remaining tails from that template with
memmap_init_page_from_template().  This avoids repeatedly doing the full
__init_single_page() and prep_compound_tail() sequence for state that is
identical across tail pages in the same huge page.

On an arm64 QEMU/KVM guest using 4K pages and 64 boot-time 1G hugepages
(HVO unavailable on arm64 in v7.4-rc1), the measured fallback tail
initialization time improved by 36.43%, measured with a temporary
jiffies-based printk modeled after the existing timing in
memmap_init_zone_device():

  base (v7.4-rc1):  318.4 ms
  with this patch:  202.4 ms

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Acked-by: Balbir Singh <balbirs@nvidia.com>
---
 mm/hugetlb.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 630c96cd1e88..3f6731161d77 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3223,6 +3223,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
 	enum zone_type zone = folio_zonenum(folio);
 	int nid = folio_nid(folio);
 	struct page *page = folio_page(folio, start_page_number);
+	struct page template;
 	unsigned long head_pfn = folio_pfn(folio);
 	unsigned long pfn, end_pfn = head_pfn + end_page_number;
 	unsigned int order = huge_page_order(h);
@@ -3231,11 +3232,20 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
 	 * As we marked all tail pages with memblock_reserved_mark_noinit(),
 	 * we must initialize them ourselves here.
 	 */
-	for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
-		__init_single_page(page, pfn, zone, nid);
-		prep_compound_tail(page, &folio->page, order);
-		set_page_count(page, 0);
-	}
+	pfn = head_pfn + start_page_number;
+
+	/*
+	 * Seed a template from the first real tail page and refresh only
+	 * PFN-dependent fields before copying it.
+	 */
+	__init_single_page(page, pfn, zone, nid);
+	prep_compound_tail(page, &folio->page, order);
+	set_page_count(page, 0);
+	memcpy(&template, page, sizeof(template));
+
+	for (pfn++; pfn < end_pfn; pfn++)
+		memmap_init_page_from_template(pfn_to_page(pfn), pfn, zone,
+					       &template);
 }
 
 static void __init hugetlb_folio_init_vmemmap(struct folio *folio,
-- 
2.20.1

Re: [PATCH v2 2/2] mm/hugetlb: optimize bootmem hugepage tail vmemmap init

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [High] mm/hugetlb: out-of-bounds write in
  hugetlb_folio_init_tail_vmemmap()
--

commit 5be89b38412d58045fd600596787f54c7559d32c
Author: Li Zhe <lizhe.67@bytedance.com>

mm/hugetlb: optimize bootmem hugepage tail vmemmap init

This commit optimizes bootmem hugepage tail struct page initialization by
seeding a template from the first tail page and initializing the remaining
tail pages via memmap_init_page_from_template().

> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index cb2de24affdc4..6ae5d2679e3aa 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -3239,6 +3239,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
>  	enum zone_type zone = folio_zonenum(folio);
>  	int nid = folio_nid(folio);
>  	struct page *page = folio_page(folio, start_page_number);
> +	struct page template;
>  	unsigned long head_pfn = folio_pfn(folio);
>  	unsigned long pfn, end_pfn = head_pfn + end_page_number;
>  	unsigned int order = huge_page_order(h);
> @@ -3247,11 +3248,20 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
>  	 * As we marked all tail pages with memblock_reserved_mark_noinit(),
>  	 * we must initialize them ourselves here.
>  	 */
> -	for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
> -		__init_single_page(page, pfn, zone, nid);
> -		prep_compound_tail(page, &folio->page, order);
> -		set_page_count(page, 0);
> -	}
> +	pfn = head_pfn + start_page_number;
> +
> +	/*
> +	 * Seed a template from the first real tail page and refresh only
> +	 * PFN-dependent fields before copying it.
> +	 */
> +	__init_single_page(page, pfn, zone, nid);
> +	prep_compound_tail(page, &folio->page, order);
> +	set_page_count(page, 0);
> +	memcpy(&template, page, sizeof(template));
> +
> +	for (pfn++; pfn < end_pfn; pfn++)
> +		memmap_init_page_from_template(pfn_to_page(pfn), pfn, zone,
> +					       &template);

[Severity: High]
Can peeling the first iteration out of the loop overwrite the struct page
past the end of folio when start_page_number >= end_page_number?

Previously, the loop was a no-op when pfn >= end_pfn.  Now, when HVO does
not optimize a gigantic folio, prep_and_add_bootmem_folios() calls
hugetlb_folio_init_tail_vmemmap(folio, h,
VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES, pages_per_huge_page(h)) after
gather_bootmem_prealloc_node() has already initialized the first
VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES pages.

On configurations where a gigantic page has pages_per_huge_page(h) <=
VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES (for example, powerpc
mpc885_ads_defconfig with 16 KiB pages, MAX_PAGE_ORDER == 8, and 8 MiB
gigantic hugepages of order 9, where both values are 512), this
unconditionally initializes folio_page(folio, start_page_number) and marks
the adjacent physical page's struct page as a compound tail of folio,
corrupting mem_map.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261110114859.76343-1-lizhe.67@bytedance.com?part=2

Re: [PATCH v2 2/2] mm/hugetlb: optimize bootmem hugepage tail vmemmap init

[Li Zhe]

On 11/10/26 9:35 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential
> issue to consider:
> - [High] mm/hugetlb: out-of-bounds write in
>    hugetlb_folio_init_tail_vmemmap()
> --
>
> commit 5be89b38412d58045fd600596787f54c7559d32c
> Author: Li Zhe <lizhe.67@bytedance.com>
>
> mm/hugetlb: optimize bootmem hugepage tail vmemmap init
>
> This commit optimizes bootmem hugepage tail struct page initialization by
> seeding a template from the first tail page and initializing the remaining
> tail pages via memmap_init_page_from_template().
>
>> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
>> index cb2de24affdc4..6ae5d2679e3aa 100644
>> --- a/mm/hugetlb.c
>> +++ b/mm/hugetlb.c
>> @@ -3239,6 +3239,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
>>   	enum zone_type zone = folio_zonenum(folio);
>>   	int nid = folio_nid(folio);
>>   	struct page *page = folio_page(folio, start_page_number);
>> +	struct page template;
>>   	unsigned long head_pfn = folio_pfn(folio);
>>   	unsigned long pfn, end_pfn = head_pfn + end_page_number;
>>   	unsigned int order = huge_page_order(h);
>> @@ -3247,11 +3248,20 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
>>   	 * As we marked all tail pages with memblock_reserved_mark_noinit(),
>>   	 * we must initialize them ourselves here.
>>   	 */
>> -	for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
>> -		__init_single_page(page, pfn, zone, nid);
>> -		prep_compound_tail(page, &folio->page, order);
>> -		set_page_count(page, 0);
>> -	}
>> +	pfn = head_pfn + start_page_number;
>> +
>> +	/*
>> +	 * Seed a template from the first real tail page and refresh only
>> +	 * PFN-dependent fields before copying it.
>> +	 */
>> +	__init_single_page(page, pfn, zone, nid);
>> +	prep_compound_tail(page, &folio->page, order);
>> +	set_page_count(page, 0);
>> +	memcpy(&template, page, sizeof(template));
>> +
>> +	for (pfn++; pfn < end_pfn; pfn++)
>> +		memmap_init_page_from_template(pfn_to_page(pfn), pfn, zone,
>> +					       &template);
> [Severity: High]
> Can peeling the first iteration out of the loop overwrite the struct page
> past the end of folio when start_page_number >= end_page_number?
>
> Previously, the loop was a no-op when pfn >= end_pfn.  Now, when HVO does
> not optimize a gigantic folio, prep_and_add_bootmem_folios() calls
> hugetlb_folio_init_tail_vmemmap(folio, h,
> VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES, pages_per_huge_page(h)) after
> gather_bootmem_prealloc_node() has already initialized the first
> VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES pages.
>
> On configurations where a gigantic page has pages_per_huge_page(h) <=
> VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES (for example, powerpc
> mpc885_ads_defconfig with 16 KiB pages, MAX_PAGE_ORDER == 8, and 8 MiB
> gigantic hugepages of order 9, where both values are 512), this
> unconditionally initializes folio_page(folio, start_page_number) and marks
> the adjacent physical page's struct page as a compound tail of folio,
> corrupting mem_map.


Thanks for the report. This looks valid.

I will restore the empty-range check before seeding the template in v3.

Thanks,
Zhe

Re: [PATCH v2 2/2] mm/hugetlb: optimize bootmem hugepage tail vmemmap init

[Muchun Song]

On 2026/11/11 11:14, Li Zhe wrote:
> On 11/10/26 9:35 PM, sashiko-bot@kernel.org wrote:
>> Thank you for your contribution! Sashiko AI review found 1 potential
>> issue to consider:
>> - [High] mm/hugetlb: out-of-bounds write in
>>     hugetlb_folio_init_tail_vmemmap()
>> --
>>
>> commit 5be89b38412d58045fd600596787f54c7559d32c
>> Author: Li Zhe <lizhe.67@bytedance.com>
>>
>> mm/hugetlb: optimize bootmem hugepage tail vmemmap init
>>
>> This commit optimizes bootmem hugepage tail struct page initialization by
>> seeding a template from the first tail page and initializing the remaining
>> tail pages via memmap_init_page_from_template().
>>
>>> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
>>> index cb2de24affdc4..6ae5d2679e3aa 100644
>>> --- a/mm/hugetlb.c
>>> +++ b/mm/hugetlb.c
>>> @@ -3239,6 +3239,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
>>>    	enum zone_type zone = folio_zonenum(folio);
>>>    	int nid = folio_nid(folio);
>>>    	struct page *page = folio_page(folio, start_page_number);
>>> +	struct page template;
>>>    	unsigned long head_pfn = folio_pfn(folio);
>>>    	unsigned long pfn, end_pfn = head_pfn + end_page_number;
>>>    	unsigned int order = huge_page_order(h);
>>> @@ -3247,11 +3248,20 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
>>>    	 * As we marked all tail pages with memblock_reserved_mark_noinit(),
>>>    	 * we must initialize them ourselves here.
>>>    	 */
>>> -	for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
>>> -		__init_single_page(page, pfn, zone, nid);
>>> -		prep_compound_tail(page, &folio->page, order);
>>> -		set_page_count(page, 0);
>>> -	}
>>> +	pfn = head_pfn + start_page_number;
>>> +
>>> +	/*
>>> +	 * Seed a template from the first real tail page and refresh only
>>> +	 * PFN-dependent fields before copying it.
>>> +	 */
>>> +	__init_single_page(page, pfn, zone, nid);
>>> +	prep_compound_tail(page, &folio->page, order);
>>> +	set_page_count(page, 0);
>>> +	memcpy(&template, page, sizeof(template));
>>> +
>>> +	for (pfn++; pfn < end_pfn; pfn++)
>>> +		memmap_init_page_from_template(pfn_to_page(pfn), pfn, zone,
>>> +					       &template);
>> [Severity: High]
>> Can peeling the first iteration out of the loop overwrite the struct page
>> past the end of folio when start_page_number >= end_page_number?
>>
>> Previously, the loop was a no-op when pfn >= end_pfn.  Now, when HVO does
>> not optimize a gigantic folio, prep_and_add_bootmem_folios() calls
>> hugetlb_folio_init_tail_vmemmap(folio, h,
>> VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES, pages_per_huge_page(h)) after
>> gather_bootmem_prealloc_node() has already initialized the first
>> VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES pages.
>>
>> On configurations where a gigantic page has pages_per_huge_page(h) <=
>> VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES (for example, powerpc
>> mpc885_ads_defconfig with 16 KiB pages, MAX_PAGE_ORDER == 8, and 8 MiB
>> gigantic hugepages of order 9, where both values are 512), this
>> unconditionally initializes folio_page(folio, start_page_number) and marks
>> the adjacent physical page's struct page as a compound tail of folio,
>> corrupting mem_map.
>
> Thanks for the report. This looks valid.
>
> I will restore the empty-range check before seeding the template in v3.

Go ahead. I will end up removing this empty-range check here [1].

[1] 
https://lore.kernel.org/all/20261109095807.41547-8-songmuchun@bytedance.com/

>
> Thanks,
> Zhe

Re: [PATCH v2 2/2] mm/hugetlb: optimize bootmem hugepage tail vmemmap init

[Li Zhe]

On 11/11/26 1:28 PM, Muchun Song wrote:
>
>
> On 2026/11/11 11:14, Li Zhe wrote:
>> On 11/10/26 9:35 PM, sashiko-bot@kernel.org wrote:
>>> Thank you for your contribution! Sashiko AI review found 1 potential
>>> issue to consider:
>>> - [High] mm/hugetlb: out-of-bounds write in
>>>     hugetlb_folio_init_tail_vmemmap()
>>> -- 
>>>
>>> commit 5be89b38412d58045fd600596787f54c7559d32c
>>> Author: Li Zhe <lizhe.67@bytedance.com>
>>>
>>> mm/hugetlb: optimize bootmem hugepage tail vmemmap init
>>>
>>> This commit optimizes bootmem hugepage tail struct page 
>>> initialization by
>>> seeding a template from the first tail page and initializing the 
>>> remaining
>>> tail pages via memmap_init_page_from_template().
>>>
>>>> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
>>>> index cb2de24affdc4..6ae5d2679e3aa 100644
>>>> --- a/mm/hugetlb.c
>>>> +++ b/mm/hugetlb.c
>>>> @@ -3239,6 +3239,7 @@ static void __init 
>>>> hugetlb_folio_init_tail_vmemmap(struct folio *folio,
>>>>        enum zone_type zone = folio_zonenum(folio);
>>>>        int nid = folio_nid(folio);
>>>>        struct page *page = folio_page(folio, start_page_number);
>>>> +    struct page template;
>>>>        unsigned long head_pfn = folio_pfn(folio);
>>>>        unsigned long pfn, end_pfn = head_pfn + end_page_number;
>>>>        unsigned int order = huge_page_order(h);
>>>> @@ -3247,11 +3248,20 @@ static void __init 
>>>> hugetlb_folio_init_tail_vmemmap(struct folio *folio,
>>>>         * As we marked all tail pages with 
>>>> memblock_reserved_mark_noinit(),
>>>>         * we must initialize them ourselves here.
>>>>         */
>>>> -    for (pfn = head_pfn + start_page_number; pfn < end_pfn; 
>>>> page++, pfn++) {
>>>> -        __init_single_page(page, pfn, zone, nid);
>>>> -        prep_compound_tail(page, &folio->page, order);
>>>> -        set_page_count(page, 0);
>>>> -    }
>>>> +    pfn = head_pfn + start_page_number;
>>>> +
>>>> +    /*
>>>> +     * Seed a template from the first real tail page and refresh only
>>>> +     * PFN-dependent fields before copying it.
>>>> +     */
>>>> +    __init_single_page(page, pfn, zone, nid);
>>>> +    prep_compound_tail(page, &folio->page, order);
>>>> +    set_page_count(page, 0);
>>>> +    memcpy(&template, page, sizeof(template));
>>>> +
>>>> +    for (pfn++; pfn < end_pfn; pfn++)
>>>> +        memmap_init_page_from_template(pfn_to_page(pfn), pfn, zone,
>>>> +                           &template);
>>> [Severity: High]
>>> Can peeling the first iteration out of the loop overwrite the struct 
>>> page
>>> past the end of folio when start_page_number >= end_page_number?
>>>
>>> Previously, the loop was a no-op when pfn >= end_pfn.  Now, when HVO 
>>> does
>>> not optimize a gigantic folio, prep_and_add_bootmem_folios() calls
>>> hugetlb_folio_init_tail_vmemmap(folio, h,
>>> VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES, pages_per_huge_page(h)) after
>>> gather_bootmem_prealloc_node() has already initialized the first
>>> VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES pages.
>>>
>>> On configurations where a gigantic page has pages_per_huge_page(h) <=
>>> VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES (for example, powerpc
>>> mpc885_ads_defconfig with 16 KiB pages, MAX_PAGE_ORDER == 8, and 8 MiB
>>> gigantic hugepages of order 9, where both values are 512), this
>>> unconditionally initializes folio_page(folio, start_page_number) and 
>>> marks
>>> the adjacent physical page's struct page as a compound tail of folio,
>>> corrupting mem_map.
>>
>> Thanks for the report. This looks valid.
>>
>> I will restore the empty-range check before seeding the template in v3.
>
> Go ahead. I will end up removing this empty-range check here [1].
>
> [1] 
> https://lore.kernel.org/all/20261109095807.41547-8-songmuchun@bytedance.com/


Thanks. I will keep the check in v3 for this series.

Thanks,
Zhe

>
>>
>> Thanks,
>> Zhe
>

Re: [PATCH v2 1/2] mm/mm_init: factor out memmap_init_page_from_template()

[Muchun Song]

> On Nov 10, 2026, at 19:48, Li Zhe <lizhe.67@bytedance.com> wrote:
> 
> ZONE_DEVICE memmap initialization already uses a reusable struct page
> template when initializing compound tail pages.  The template mostly
> contains invariant state, but a few fields still depend on the target PFN
> and must be refreshed before copying the template into each destination
> struct page.
> 
> Factor that operation into memmap_init_page_from_template() so other
> boot-time memmap initialization paths can reuse the same helper.
> 
> No functional change intended.
> 
> Signed-off-by: Li Zhe <lizhe.67@bytedance.com>

Reviewed-by: Muchun Song (ByteDance) <muchun.song@linux.dev>

Thanks.

Re: [PATCH v2 0/2] mm: optimize HugeTLB bootmem hugepage init

[Mike Rapoport]

Hi,

On Tue, Nov 10, 2026 at 07:48:57PM +0800, Li Zhe wrote:
> This series speeds up the fallback initialization path for HugeTLB tail
> struct pages during boot-time hugepage allocation, when HugeTLB vmemmap
> optimization (HVO) is unavailable or fails.
> 
> Patch 1 factors the existing ZONE_DEVICE template-copy operation into
> memmap_init_page_from_template().
> 
> Patch 2 reuses that helper in hugetlb_folio_init_tail_vmemmap():
> initialize the first tail page normally, copy it into a local template,
> then initialize the remaining tail pages by refreshing only PFN-dependent
> fields before copying the template.
> 
> In the earlier discussion, Mike asked whether the same kind of
> template-based initialization also helps the ordinary struct page
> initialization path:
> 
> https://lore.kernel.org/all/akfB6Qe-jKDEMUxz@kernel.org/
> 
> I tested that direction as well, but did not see a measurable benefit
> for ordinary struct page initialization. The measurable improvement I
> could find is in the HugeTLB bootmem hugepage tail path, where gigantic
> pages can require initializing a large number of tail struct pages when
> HVO does not cover the allocation.
> 
> Tested on an arm64 QEMU/KVM guest with 4K base pages and 64 boot-time 1G
> hugepages. HVO is unavailable on arm64 in v7.4-rc1, so the fallback tail
> initialization path is exercised.
> 
> Results averaged over five boots, measured with a temporary
> jiffies-based printk:
> 
>   base (v7.4-rc1):  318.4 ms
>   with this patch:  202.4 ms

Are there numbers for x86 without HVO as well?
We need to be sure this does not regress that use case.
 
> This is a 36.43% reduction in the measured fallback tail initialization
> time. The temporary measurement instrumentation is not part of this
> series.
> 
> Li Zhe (2):
>   mm/mm_init: factor out memmap_init_page_from_template()
>   mm/hugetlb: optimize bootmem hugepage tail vmemmap init
> 
>  mm/hugetlb.c | 20 +++++++++++++++-----
>  mm/mm_init.c | 34 ++++++++++++++++++++--------------
>  mm/mm_init.h |  2 ++
>  3 files changed, 37 insertions(+), 19 deletions(-)
> 
> ---
> v1: https://lore.kernel.org/all/20261105120615.16817-1-lizhe.67@bytedance.com/#t
> 
> Changes in v2:
> - Add a comment for the template-before-copy ordering, as suggested by
>   Muchun Song.
> - Drop the empty-range guard in hugetlb_folio_init_tail_vmemmap()
>   as suggested by Balbir Singh and Mike Rapoport; current callers are
>   expected to pass a non-empty range.
> 
> -- 
> 2.20.1

-- 
Sincerely yours,
Mike.

Re: [PATCH v2 2/2] mm/hugetlb: optimize bootmem hugepage tail vmemmap init

[Mike Rapoport]

On Wed, Nov 11, 2026 at 11:14:46AM +0800, Li Zhe wrote:
> On 11/10/26 9:35 PM, sashiko-bot@kernel.org wrote:
> > Thank you for your contribution! Sashiko AI review found 1 potential
> > issue to consider:
> > - [High] mm/hugetlb: out-of-bounds write in
> >    hugetlb_folio_init_tail_vmemmap()
> > --
> >
> > commit 5be89b38412d58045fd600596787f54c7559d32c
> > Author: Li Zhe <lizhe.67@bytedance.com>
> >
> > mm/hugetlb: optimize bootmem hugepage tail vmemmap init
> >
> > This commit optimizes bootmem hugepage tail struct page initialization by
> > seeding a template from the first tail page and initializing the remaining
> > tail pages via memmap_init_page_from_template().
> >
> >> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> >> index cb2de24affdc4..6ae5d2679e3aa 100644
> >> --- a/mm/hugetlb.c
> >> +++ b/mm/hugetlb.c
> >> @@ -3239,6 +3239,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
> >>   	enum zone_type zone = folio_zonenum(folio);
> >>   	int nid = folio_nid(folio);
> >>   	struct page *page = folio_page(folio, start_page_number);
> >> +	struct page template;
> >>   	unsigned long head_pfn = folio_pfn(folio);
> >>   	unsigned long pfn, end_pfn = head_pfn + end_page_number;
> >>   	unsigned int order = huge_page_order(h);
> >> @@ -3247,11 +3248,20 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
> >>   	 * As we marked all tail pages with memblock_reserved_mark_noinit(),
> >>   	 * we must initialize them ourselves here.
> >>   	 */
> >> -	for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
> >> -		__init_single_page(page, pfn, zone, nid);
> >> -		prep_compound_tail(page, &folio->page, order);
> >> -		set_page_count(page, 0);
> >> -	}
> >> +	pfn = head_pfn + start_page_number;
> >> +
> >> +	/*
> >> +	 * Seed a template from the first real tail page and refresh only
> >> +	 * PFN-dependent fields before copying it.
> >> +	 */
> >> +	__init_single_page(page, pfn, zone, nid);
> >> +	prep_compound_tail(page, &folio->page, order);
> >> +	set_page_count(page, 0);
> >> +	memcpy(&template, page, sizeof(template));
> >> +
> >> +	for (pfn++; pfn < end_pfn; pfn++)
> >> +		memmap_init_page_from_template(pfn_to_page(pfn), pfn, zone,
> >> +					       &template);
> > [Severity: High]
> > Can peeling the first iteration out of the loop overwrite the struct page
> > past the end of folio when start_page_number >= end_page_number?
> >
> > Previously, the loop was a no-op when pfn >= end_pfn.  Now, when HVO does
> > not optimize a gigantic folio, prep_and_add_bootmem_folios() calls
> > hugetlb_folio_init_tail_vmemmap(folio, h,
> > VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES, pages_per_huge_page(h)) after
> > gather_bootmem_prealloc_node() has already initialized the first
> > VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES pages.
> >
> > On configurations where a gigantic page has pages_per_huge_page(h) <=
> > VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES (for example, powerpc
> > mpc885_ads_defconfig with 16 KiB pages, MAX_PAGE_ORDER == 8, and 8 MiB
> > gigantic hugepages of order 9, where both values are 512), this
> > unconditionally initializes folio_page(folio, start_page_number) and marks
> > the adjacent physical page's struct page as a compound tail of folio,
> > corrupting mem_map.
> 
> Thanks for the report. This looks valid.

I love how sashiko finds issue on very niche systems.

But yeah, ppc8xx actually uses hugetlb :(

> I will restore the empty-range check before seeding the template in v3.
> 
> Thanks,
> Zhe
> 

-- 
Sincerely yours,
Mike.

Re: [PATCH v2 2/2] mm/hugetlb: optimize bootmem hugepage tail vmemmap init

[Mike Rapoport]

Hi,
On Wed, Nov 11, 2026 at 01:28:50PM +0800, Muchun Song wrote:
> 
> On 2026/11/11 11:14, Li Zhe wrote:
> > On 11/10/26 9:35 PM, sashiko-bot@kernel.org wrote:
> > > Thank you for your contribution! Sashiko AI review found 1 potential
> > > issue to consider:
> > > - [High] mm/hugetlb: out-of-bounds write in
> > >     hugetlb_folio_init_tail_vmemmap()
> > > --
> > > 
> > > commit 5be89b38412d58045fd600596787f54c7559d32c
> > > Author: Li Zhe <lizhe.67@bytedance.com>
> > > 
> > > mm/hugetlb: optimize bootmem hugepage tail vmemmap init
> > > 
> > > This commit optimizes bootmem hugepage tail struct page initialization by
> > > seeding a template from the first tail page and initializing the remaining
> > > tail pages via memmap_init_page_from_template().
> > > 
> > > > diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> > > > index cb2de24affdc4..6ae5d2679e3aa 100644
> > > > --- a/mm/hugetlb.c
> > > > +++ b/mm/hugetlb.c
> > > > @@ -3239,6 +3239,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
> > > >    	enum zone_type zone = folio_zonenum(folio);
> > > >    	int nid = folio_nid(folio);
> > > >    	struct page *page = folio_page(folio, start_page_number);
> > > > +	struct page template;
> > > >    	unsigned long head_pfn = folio_pfn(folio);
> > > >    	unsigned long pfn, end_pfn = head_pfn + end_page_number;
> > > >    	unsigned int order = huge_page_order(h);
> > > > @@ -3247,11 +3248,20 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
> > > >    	 * As we marked all tail pages with memblock_reserved_mark_noinit(),
> > > >    	 * we must initialize them ourselves here.
> > > >    	 */
> > > > -	for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
> > > > -		__init_single_page(page, pfn, zone, nid);
> > > > -		prep_compound_tail(page, &folio->page, order);
> > > > -		set_page_count(page, 0);
> > > > -	}
> > > > +	pfn = head_pfn + start_page_number;
> > > > +
> > > > +	/*
> > > > +	 * Seed a template from the first real tail page and refresh only
> > > > +	 * PFN-dependent fields before copying it.
> > > > +	 */
> > > > +	__init_single_page(page, pfn, zone, nid);
> > > > +	prep_compound_tail(page, &folio->page, order);
> > > > +	set_page_count(page, 0);
> > > > +	memcpy(&template, page, sizeof(template));
> > > > +
> > > > +	for (pfn++; pfn < end_pfn; pfn++)
> > > > +		memmap_init_page_from_template(pfn_to_page(pfn), pfn, zone,
> > > > +					       &template);
> > > [Severity: High]
> > > Can peeling the first iteration out of the loop overwrite the struct page
> > > past the end of folio when start_page_number >= end_page_number?
> > > 
> > > Previously, the loop was a no-op when pfn >= end_pfn.  Now, when HVO does
> > > not optimize a gigantic folio, prep_and_add_bootmem_folios() calls
> > > hugetlb_folio_init_tail_vmemmap(folio, h,
> > > VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES, pages_per_huge_page(h)) after
> > > gather_bootmem_prealloc_node() has already initialized the first
> > > VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES pages.
> > > 
> > > On configurations where a gigantic page has pages_per_huge_page(h) <=
> > > VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES (for example, powerpc
> > > mpc885_ads_defconfig with 16 KiB pages, MAX_PAGE_ORDER == 8, and 8 MiB
> > > gigantic hugepages of order 9, where both values are 512), this
> > > unconditionally initializes folio_page(folio, start_page_number) and marks
> > > the adjacent physical page's struct page as a compound tail of folio,
> > > corrupting mem_map.
> > 
> > Thanks for the report. This looks valid.
> > 
> > I will restore the empty-range check before seeding the template in v3.
> 
> Go ahead. I will end up removing this empty-range check here [1].
> 
> [1]
> https://lore.kernel.org/all/20261109095807.41547-8-songmuchun@bytedance.com/
 
Maybe you'll pick these two patches into your series since you anyway
change the same areas?

> > Thanks,
> > Zhe
> 

-- 
Sincerely yours,
Mike.

Re: [PATCH v2 2/2] mm/hugetlb: optimize bootmem hugepage tail vmemmap init

[Muchun Song]

On 2026/11/11 16:16, Mike Rapoport wrote:
> Hi,
> On Wed, Nov 11, 2026 at 01:28:50PM +0800, Muchun Song wrote:
>> On 2026/11/11 11:14, Li Zhe wrote:
>>> On 11/10/26 9:35 PM, sashiko-bot@kernel.org wrote:
>>>> Thank you for your contribution! Sashiko AI review found 1 potential
>>>> issue to consider:
>>>> - [High] mm/hugetlb: out-of-bounds write in
>>>>      hugetlb_folio_init_tail_vmemmap()
>>>> --
>>>>
>>>> commit 5be89b38412d58045fd600596787f54c7559d32c
>>>> Author: Li Zhe <lizhe.67@bytedance.com>
>>>>
>>>> mm/hugetlb: optimize bootmem hugepage tail vmemmap init
>>>>
>>>> This commit optimizes bootmem hugepage tail struct page initialization by
>>>> seeding a template from the first tail page and initializing the remaining
>>>> tail pages via memmap_init_page_from_template().
>>>>
>>>>> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
>>>>> index cb2de24affdc4..6ae5d2679e3aa 100644
>>>>> --- a/mm/hugetlb.c
>>>>> +++ b/mm/hugetlb.c
>>>>> @@ -3239,6 +3239,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
>>>>>     	enum zone_type zone = folio_zonenum(folio);
>>>>>     	int nid = folio_nid(folio);
>>>>>     	struct page *page = folio_page(folio, start_page_number);
>>>>> +	struct page template;
>>>>>     	unsigned long head_pfn = folio_pfn(folio);
>>>>>     	unsigned long pfn, end_pfn = head_pfn + end_page_number;
>>>>>     	unsigned int order = huge_page_order(h);
>>>>> @@ -3247,11 +3248,20 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio,
>>>>>     	 * As we marked all tail pages with memblock_reserved_mark_noinit(),
>>>>>     	 * we must initialize them ourselves here.
>>>>>     	 */
>>>>> -	for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) {
>>>>> -		__init_single_page(page, pfn, zone, nid);
>>>>> -		prep_compound_tail(page, &folio->page, order);
>>>>> -		set_page_count(page, 0);
>>>>> -	}
>>>>> +	pfn = head_pfn + start_page_number;
>>>>> +
>>>>> +	/*
>>>>> +	 * Seed a template from the first real tail page and refresh only
>>>>> +	 * PFN-dependent fields before copying it.
>>>>> +	 */
>>>>> +	__init_single_page(page, pfn, zone, nid);
>>>>> +	prep_compound_tail(page, &folio->page, order);
>>>>> +	set_page_count(page, 0);
>>>>> +	memcpy(&template, page, sizeof(template));
>>>>> +
>>>>> +	for (pfn++; pfn < end_pfn; pfn++)
>>>>> +		memmap_init_page_from_template(pfn_to_page(pfn), pfn, zone,
>>>>> +					       &template);
>>>> [Severity: High]
>>>> Can peeling the first iteration out of the loop overwrite the struct page
>>>> past the end of folio when start_page_number >= end_page_number?
>>>>
>>>> Previously, the loop was a no-op when pfn >= end_pfn.  Now, when HVO does
>>>> not optimize a gigantic folio, prep_and_add_bootmem_folios() calls
>>>> hugetlb_folio_init_tail_vmemmap(folio, h,
>>>> VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES, pages_per_huge_page(h)) after
>>>> gather_bootmem_prealloc_node() has already initialized the first
>>>> VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES pages.
>>>>
>>>> On configurations where a gigantic page has pages_per_huge_page(h) <=
>>>> VMEMMAP_OPTIMIZATION_NR_STRUCT_PAGES (for example, powerpc
>>>> mpc885_ads_defconfig with 16 KiB pages, MAX_PAGE_ORDER == 8, and 8 MiB
>>>> gigantic hugepages of order 9, where both values are 512), this
>>>> unconditionally initializes folio_page(folio, start_page_number) and marks
>>>> the adjacent physical page's struct page as a compound tail of folio,
>>>> corrupting mem_map.
>>> Thanks for the report. This looks valid.
>>>
>>> I will restore the empty-range check before seeding the template in v3.
>> Go ahead. I will end up removing this empty-range check here [1].
>>
>> [1]
>> https://lore.kernel.org/all/20261109095807.41547-8-songmuchun@bytedance.com/
>   
> Maybe you'll pick these two patches into your series since you anyway
> change the same areas?

Good point! I'd like to do this.

Thanks,
Muchun

>>> Thanks,
>>> Zhe

Re: [PATCH v2 0/2] mm: optimize HugeTLB bootmem hugepage init

[Li Zhe]

On 11/11/26 4:12 PM, Mike Rapoport wrote:
> Hi,
>
> On Tue, Nov 10, 2026 at 07:48:57PM +0800, Li Zhe wrote:
>> This series speeds up the fallback initialization path for HugeTLB tail
>> struct pages during boot-time hugepage allocation, when HugeTLB vmemmap
>> optimization (HVO) is unavailable or fails.
>>
>> Patch 1 factors the existing ZONE_DEVICE template-copy operation into
>> memmap_init_page_from_template().
>>
>> Patch 2 reuses that helper in hugetlb_folio_init_tail_vmemmap():
>> initialize the first tail page normally, copy it into a local template,
>> then initialize the remaining tail pages by refreshing only PFN-dependent
>> fields before copying the template.
>>
>> In the earlier discussion, Mike asked whether the same kind of
>> template-based initialization also helps the ordinary struct page
>> initialization path:
>>
>> https://lore.kernel.org/all/akfB6Qe-jKDEMUxz@kernel.org/
>>
>> I tested that direction as well, but did not see a measurable benefit
>> for ordinary struct page initialization. The measurable improvement I
>> could find is in the HugeTLB bootmem hugepage tail path, where gigantic
>> pages can require initializing a large number of tail struct pages when
>> HVO does not cover the allocation.
>>
>> Tested on an arm64 QEMU/KVM guest with 4K base pages and 64 boot-time 1G
>> hugepages. HVO is unavailable on arm64 in v7.4-rc1, so the fallback tail
>> initialization path is exercised.
>>
>> Results averaged over five boots, measured with a temporary
>> jiffies-based printk:
>>
>>    base (v7.4-rc1):  318.4 ms
>>    with this patch:  202.4 ms
> Are there numbers for x86 without HVO as well?
> We need to be sure this does not regress that use case.


Thanks for the feedback.

I did some additional x86 testing today, but the result was different
from the similar x86 test I did earlier on linux-7.2-rc1. After
replacing memcpy_nontemporal() with regular memcpy(), the result looked
better. I need to re-check the old test code, since I may have mixed up
which copy primitive was used there.

Muchun, could you please hold off folding this series into your patchset
until I send v3?

Thanks,
Zhe

>   
>> This is a 36.43% reduction in the measured fallback tail initialization
>> time. The temporary measurement instrumentation is not part of this
>> series.
>>
>> Li Zhe (2):
>>    mm/mm_init: factor out memmap_init_page_from_template()
>>    mm/hugetlb: optimize bootmem hugepage tail vmemmap init
>>
>>   mm/hugetlb.c | 20 +++++++++++++++-----
>>   mm/mm_init.c | 34 ++++++++++++++++++++--------------
>>   mm/mm_init.h |  2 ++
>>   3 files changed, 37 insertions(+), 19 deletions(-)
>>
>> ---
>> v1: https://lore.kernel.org/all/20261105120615.16817-1-lizhe.67@bytedance.com/#t
>>
>> Changes in v2:
>> - Add a comment for the template-before-copy ordering, as suggested by
>>    Muchun Song.
>> - Drop the empty-range guard in hugetlb_folio_init_tail_vmemmap()
>>    as suggested by Balbir Singh and Mike Rapoport; current callers are
>>    expected to pass a non-empty range.
>>
>> -- 
>> 2.20.1

Re: [PATCH v2 0/2] mm: optimize HugeTLB bootmem hugepage init

[Muchun Song]

> On Nov 11, 2026, at 19:24, Li Zhe <lizhe.67@bytedance.com> wrote:
> 
> On 11/11/26 4:12 PM, Mike Rapoport wrote:
>> Hi,
>> 
>> On Tue, Nov 10, 2026 at 07:48:57PM +0800, Li Zhe wrote:
>>> This series speeds up the fallback initialization path for HugeTLB tail
>>> struct pages during boot-time hugepage allocation, when HugeTLB vmemmap
>>> optimization (HVO) is unavailable or fails.
>>> 
>>> Patch 1 factors the existing ZONE_DEVICE template-copy operation into
>>> memmap_init_page_from_template().
>>> 
>>> Patch 2 reuses that helper in hugetlb_folio_init_tail_vmemmap():
>>> initialize the first tail page normally, copy it into a local template,
>>> then initialize the remaining tail pages by refreshing only PFN-dependent
>>> fields before copying the template.
>>> 
>>> In the earlier discussion, Mike asked whether the same kind of
>>> template-based initialization also helps the ordinary struct page
>>> initialization path:
>>> 
>>> https://lore.kernel.org/all/akfB6Qe-jKDEMUxz@kernel.org/
>>> 
>>> I tested that direction as well, but did not see a measurable benefit
>>> for ordinary struct page initialization. The measurable improvement I
>>> could find is in the HugeTLB bootmem hugepage tail path, where gigantic
>>> pages can require initializing a large number of tail struct pages when
>>> HVO does not cover the allocation.
>>> 
>>> Tested on an arm64 QEMU/KVM guest with 4K base pages and 64 boot-time 1G
>>> hugepages. HVO is unavailable on arm64 in v7.4-rc1, so the fallback tail
>>> initialization path is exercised.
>>> 
>>> Results averaged over five boots, measured with a temporary
>>> jiffies-based printk:
>>> 
>>>   base (v7.4-rc1):  318.4 ms
>>>   with this patch:  202.4 ms
>> Are there numbers for x86 without HVO as well?
>> We need to be sure this does not regress that use case.
> 
> 
> Thanks for the feedback.
> 
> I did some additional x86 testing today, but the result was different
> from the similar x86 test I did earlier on linux-7.2-rc1. After
> replacing memcpy_nontemporal() with regular memcpy(), the result looked
> better. I need to re-check the old test code, since I may have mixed up
> which copy primitive was used there.
> 
> Muchun, could you please hold off folding this series into your patchset
> until I send v3?

Sure.

Thanks.

> 
> Thanks,
> Zhe
> 
>> 
>>> This is a 36.43% reduction in the measured fallback tail initialization
>>> time. The temporary measurement instrumentation is not part of this
>>> series.
>>> 
>>> Li Zhe (2):
>>>   mm/mm_init: factor out memmap_init_page_from_template()
>>>   mm/hugetlb: optimize bootmem hugepage tail vmemmap init
>>> 
>>>  mm/hugetlb.c | 20 +++++++++++++++-----
>>>  mm/mm_init.c | 34 ++++++++++++++++++++--------------
>>>  mm/mm_init.h |  2 ++
>>>  3 files changed, 37 insertions(+), 19 deletions(-)
>>> 
>>> ---
>>> v1: https://lore.kernel.org/all/20261105120615.16817-1-lizhe.67@bytedance.com/#t
>>> 
>>> Changes in v2:
>>> - Add a comment for the template-before-copy ordering, as suggested by
>>>   Muchun Song.
>>> - Drop the empty-range guard in hugetlb_folio_init_tail_vmemmap()
>>>   as suggested by Balbir Singh and Mike Rapoport; current callers are
>>>   expected to pass a non-empty range.
>>> 
>>> -- 
>>> 2.20.1