[PATCH v6 0/8] mm: optimize zone-device memmap initialization

[PATCH v6 0/8] mm: optimize zone-device memmap initialization

[Li Zhe]

memmap_init_zone_device() can take a noticeable amount of time when large
pmem namespaces are bound or rebound, because it initializes nearly
identical struct page descriptors one PFN at a time. This series reduces
that ZONE_DEVICE memmap initialization overhead by reusing prepared
struct page templates and, on x86, using memcpy_nt() for the template
copy path.

The main target is large fsdax/devdax pmem configurations, where the
cost of initializing the memmap shows up directly in nd_pmem/dax_pmem
bind and rebind latency.

Patches 1-3 are preparatory cleanups and helper extraction. Patches 4-5
add the template-copy fast path for head pages and compound tails.
Patches 6-8 introduce memcpy_nt()/memcpy_nt_drain(), extend the x86
fixed-size memcpy_flushcache() inline cases used by that helper, and
switch the template-copy path over to memcpy_nt().

The fast path remains disabled when the page_ref_set tracepoint is
active, and sanitized builds stay on the slow path so their instrumented
stores are preserved. Architectures without a specialized memcpy_nt()
backend continue to fall back to memcpy().

Tested in a VM with a 100 GB fsdax namespace device configured with
map=dev and a 100 GB devdax namespace (align=2097152) on Intel Ice Lake
server.

Test procedure:
Rebind the nd_pmem and dax_pmem driver 30 times and collect the memmap
initialization time from the pr_debug() output of
memmap_init_zone_device().

Base(v7.2-rc1):
  First binding for nd_pmem driver: 1456 ms
  Average of subsequent rebinds: 244.28 ms

  First binding for dax_pmem driver: 1462 ms
  Average of subsequent rebinds: 273.31 ms

With this series applied:
  First binding for nd_pmem driver: 1272 ms
  Average of subsequent rebinds: 96.79 ms

  First binding for dax_pmem driver: 1354 ms
  Average of subsequent rebinds: 119.04 ms

This reduces the average rebind time by about 60.4% for nd_pmem and
56.4% for dax_pmem.

As an additional data point, I also ran a smaller set of measurements on
the same physical x86_64 host with a 100 GB PMEM region created via the
memmap= kernel command line, configured as fsdax and devdax namespaces
with map=dev and 2 MiB alignment.

For brevity, the individual patches keep only the VM results rather than
including a second set of physical-host measurements throughout the
series. The physical-host numbers below are included only as
supplemental evidence that the same optimization also provides a similar
benefit on a non-virtualized system.

Test procedure:
Reconfigure the namespace mode, rebind the nd_pmem or dax_pmem driver
once, and collect the memmap initialization time from the pr_debug()
output of memmap_init_zone_device().

Base (v7.2-rc1):
  nd_pmem / fsdax: 179 ms
  dax_pmem / devdax: 264 ms

With this series applied:
  nd_pmem / fsdax: 82 ms
  dax_pmem / devdax: 113 ms

This reduces the measured rebind time by about 54.2% for nd_pmem and
57.2% for dax_pmem on that setup, which is broadly consistent with the
VM results above.

As another supplemental data point, I also measured the test_hmm.ko
module on the same physical x86_64 host, using the test_hmm.ko setup
from the previous discussion that times ten 64 GB
memremap_pages()/memunmap_pages() iterations during module insertion[1].
By default, module insertion initializes two DEVICE_PRIVATE dmirror
devices, so two avg memremap values are reported; each value is the
average for one 64 GB chunk.

This is not the primary target workload of the series, but it exercises
the same large ZONE_DEVICE memmap initialization path and shows the same
direction of improvement.

Base (v7.2-rc1):
  avg memremap reported during module insertion: 116689362 ns, 116539263 ns

With this series applied:
  avg memremap reported during module insertion: 54607108 ns, 54458236 ns

This corresponds to about a 53.2% reduction based on the mean of the
reported values, which is again consistent with the pmem bind/rebind
results above.

[1] https://lore.kernel.org/all/aiEoByaQdRR3xtM5@nvdebian.thelocal/

Li Zhe (8):
  mm: fix stale ZONE_DEVICE refcount comment
  mm: factor zone-device page init helpers out of
    __init_zone_device_page
  mm: add a set_page_section_from_pfn() helper
  mm: add a template-based fast path for zone-device page init
  mm: extend the template fast path to zone-device compound tails
  string: introduce memcpy_nt() helpers
  x86/string: extend memcpy_flushcache() fixed-size fastpaths
  mm: use memcpy_nt() in zone-device template copies

 arch/x86/include/asm/string_64.h |  78 ++++++++++++-
 include/linux/mm.h               |  15 ++-
 include/linux/string.h           |  23 ++++
 mm/mm_init.c                     | 186 ++++++++++++++++++++++++++-----
 4 files changed, 273 insertions(+), 29 deletions(-)

---
v5: https://lore.kernel.org/all/20260701090553.62691-1-lizhe.67@bytedance.com/
v4: https://lore.kernel.org/all/20260603080152.64728-1-lizhe.67@bytedance.com/
v3: https://lore.kernel.org/all/20260527033636.28231-1-lizhe.67@bytedance.com/
v2: https://lore.kernel.org/all/20260521040124.10608-1-lizhe.67@bytedance.com/
v1: https://lore.kernel.org/all/20260515082045.63029-1-lizhe.67@bytedance.com/

Changelogs:

v5->v6:
- Rework patch 6 so the x86 memcpy_nt_drain() helper uses wmb()
  instead of an open-coded sfence, while architectures that do not
  override memcpy_nt() keep the generic no-op drain fallback.
  Suggested by Borislav Petkov.
- Rework patch 6 to use the usual self-macro override pattern for
  memcpy_nt() and memcpy_nt_drain() instead of a dedicated
  __HAVE_ARCH_MEMCPY_NT feature macro. Suggested by David
  Hildenbrand.
- Drop the default: case from pagemap_resets_refcount() and drop the
  WARN_ONCE() after the switch so newly added enum memory_type values
  remain visible to compiler switch checking. Suggested by David
  Hildenbrand.
- Drop the unnecessary empty set_page_section() stub from patch 3, keep
  set_page_section_from_pfn() as the only !SECTION_IN_PAGE_FLAGS no-op
  helper and fix its indentation. Suggested by David Hildenbrand.
- Rework patch 4 so the reusable head-page template is seeded from the
  first loop iteration, keeping the main loop intact while ensuring the
  first page is initialized only once; also drop the extra template
  memcpy helper, as suggested by Alistair Popple.
- Rework patch 5 similarly for compound tails: seed the reusable tail
  template from the first tail-page iteration so the first tail page is
  initialized only once, while keeping the main tail loop intact, as
  suggested by Alistair Popple.
- Rework patch 7 to rename the internal fixed-size MOVNTI helpers to
  movnti_*(), keep memcpy_flushcache() as the externally visible
  wrapper, fold the new 32/48/64/80/96-byte copies into the main
  fixed-size switch, clarify in the code and changelog that the
  ZONE_DEVICE template-copy path relies on the 64/80/96-byte struct
  page copies while 32/48-byte copies stay inline as well, and drop the
  separate alignment check based on the physical-host microbenchmark
  results. Suggested by Borislav Petkov.
- Run v6 VM spot checks for the fsdax/devdax map=dev paths. The
  results stayed in the same ballpark as v5, so keep the previously
  posted v5 full-run numbers in this version instead of replacing them
  with another partial data set.

For changelogs of earlier revisions, please refer to the v5 cover letter.

-- 
2.20.1

[PATCH v6 1/8] mm: fix stale ZONE_DEVICE refcount comment

[Li Zhe]

The comment in __init_zone_device_page() still uses the old
MEMORY_TYPE_* names and implies that FS_DAX pages regain a
refcount of 1 in the free path. That no longer matches the code.

Update the comment to describe the current policy correctly:
MEMORY_DEVICE_GENERIC pages regain a refcount of 1 in the free path,
while the remaining ZONE_DEVICE types start from 0 here and raise the
count again when the allocator or driver hands the page out.

No functional change intended.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Alistair Popple <apopple@nvidia.com>
---
 mm/mm_init.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/mm/mm_init.c b/mm/mm_init.c
index 0f64909e8d20..95808ab5cfdb 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -1030,13 +1030,9 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn,
 	page->zone_device_data = NULL;
 
 	/*
-	 * ZONE_DEVICE pages other than MEMORY_TYPE_GENERIC are released
-	 * directly to the driver page allocator which will set the page count
-	 * to 1 when allocating the page.
-	 *
-	 * MEMORY_TYPE_GENERIC and MEMORY_TYPE_FS_DAX pages automatically have
-	 * their refcount reset to one whenever they are freed (ie. after
-	 * their refcount drops to 0).
+	 * MEMORY_DEVICE_GENERIC pages regain a refcount of 1 in the free
+	 * path. The remaining ZONE_DEVICE types start from 0 here and raise
+	 * the count again when the allocator or driver hands the page out.
 	 */
 	switch (pgmap->type) {
 	case MEMORY_DEVICE_FS_DAX:
-- 
2.20.1

[PATCH v6 2/8] mm: factor zone-device page init helpers out of __init_zone_device_page

[Li Zhe]

memmap_init_zone_device() currently mixes refcount policy and core
ZONE_DEVICE page setup in a single helper.

Factor the refcount-reset predicate into pagemap_resets_refcount(), move
the common page initialization into __zone_device_page_init(), and wrap
the existing slow path in zone_device_page_init_slow().

This keeps the slow-path behaviour unchanged and gives later patches
reusable helper boundaries.

No functional change intended.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
---
 mm/mm_init.c | 56 ++++++++++++++++++++++++++++++++++------------------
 1 file changed, 37 insertions(+), 19 deletions(-)

diff --git a/mm/mm_init.c b/mm/mm_init.c
index 95808ab5cfdb..5fccfbacf855 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -1005,11 +1005,37 @@ static void __init memmap_init(void)
 }
 
 #ifdef CONFIG_ZONE_DEVICE
-static void __ref __init_zone_device_page(struct page *page, unsigned long pfn,
+/*
+ * Return true when the free path for this pagemap type restores the page
+ * refcount to 1, so memmap_init_zone_device() can keep the count set by
+ * __init_single_page(). Otherwise initialize the refcount to 0 and leave
+ * it to the allocator or pgmap callbacks to raise it when the page is
+ * handed out again.
+ */
+static inline bool pagemap_resets_refcount(const struct dev_pagemap *pgmap)
+{
+	/*
+	 * MEMORY_DEVICE_GENERIC pages regain a refcount of 1 in the free
+	 * path. The remaining ZONE_DEVICE types start from 0 here and raise
+	 * the count again when the allocator or driver hands the page out.
+	 */
+	switch (pgmap->type) {
+	case MEMORY_DEVICE_FS_DAX:
+	case MEMORY_DEVICE_PRIVATE:
+	case MEMORY_DEVICE_COHERENT:
+	case MEMORY_DEVICE_PCI_P2PDMA:
+		return false;
+	case MEMORY_DEVICE_GENERIC:
+		return true;
+	}
+
+	return true;
+}
+
+static void __ref __zone_device_page_init(struct page *page, unsigned long pfn,
 					  unsigned long zone_idx, int nid,
 					  struct dev_pagemap *pgmap)
 {
-
 	__init_single_page(page, pfn, zone_idx, nid);
 
 	/*
@@ -1028,23 +1054,15 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn,
 	 */
 	page_folio(page)->pgmap = pgmap;
 	page->zone_device_data = NULL;
+}
 
-	/*
-	 * MEMORY_DEVICE_GENERIC pages regain a refcount of 1 in the free
-	 * path. The remaining ZONE_DEVICE types start from 0 here and raise
-	 * the count again when the allocator or driver hands the page out.
-	 */
-	switch (pgmap->type) {
-	case MEMORY_DEVICE_FS_DAX:
-	case MEMORY_DEVICE_PRIVATE:
-	case MEMORY_DEVICE_COHERENT:
-	case MEMORY_DEVICE_PCI_P2PDMA:
+static void __ref zone_device_page_init_slow(struct page *page,
+		unsigned long pfn, unsigned long zone_idx, int nid,
+		struct dev_pagemap *pgmap)
+{
+	__zone_device_page_init(page, pfn, zone_idx, nid, pgmap);
+	if (!pagemap_resets_refcount(pgmap))
 		set_page_count(page, 0);
-		break;
-
-	case MEMORY_DEVICE_GENERIC:
-		break;
-	}
 }
 
 /*
@@ -1090,7 +1108,7 @@ static void __ref memmap_init_compound(struct page *head,
 	for (pfn = head_pfn + 1; pfn < end_pfn; pfn++) {
 		struct page *page = pfn_to_page(pfn);
 
-		__init_zone_device_page(page, pfn, zone_idx, nid, pgmap);
+		zone_device_page_init_slow(page, pfn, zone_idx, nid, pgmap);
 		prep_compound_tail(page, head, order);
 		set_page_count(page, 0);
 	}
@@ -1126,7 +1144,7 @@ void __ref memmap_init_zone_device(struct zone *zone,
 	for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) {
 		struct page *page = pfn_to_page(pfn);
 
-		__init_zone_device_page(page, pfn, zone_idx, nid, pgmap);
+		zone_device_page_init_slow(page, pfn, zone_idx, nid, pgmap);
 
 		if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
 			cond_resched();
-- 
2.20.1

[PATCH v6 3/8] mm: add a set_page_section_from_pfn() helper

[Li Zhe]

Callers that want to update section bits from a PFN currently need to
open-code:

	set_page_section(page, pfn_to_section_nr(pfn));

and guard that sequence with #ifdef SECTION_IN_PAGE_FLAGS.

Add set_page_section_from_pfn() to wrap that update in one place. When
section bits are stored in page flags, the helper derives the section
number from the PFN and updates the page flags. Otherwise keep it as a
no-op so callers can use one helper without open-coding
SECTION_IN_PAGE_FLAGS.

Convert set_page_links() to use the new helper so later ZONE_DEVICE
fast-path patches can also update section bits without open-coding
SECTION_IN_PAGE_FLAGS at each callsite.

This keeps the PFN-to-section translation local to the configurations
that actually store section bits in struct page flags, and avoids
exposing that detail to generic callers.

No functional change intended.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
---
 include/linux/mm.h | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 485df9c2dbdd..43343bfce493 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2541,11 +2541,22 @@ static inline void set_page_section(struct page *page, unsigned long section)
 	page->flags.f |= (section & SECTIONS_MASK) << SECTIONS_PGSHIFT;
 }
 
+static inline void set_page_section_from_pfn(struct page *page,
+		unsigned long pfn)
+{
+	set_page_section(page, pfn_to_section_nr(pfn));
+}
+
 static inline unsigned long memdesc_section(memdesc_flags_t mdf)
 {
 	return (mdf.f >> SECTIONS_PGSHIFT) & SECTIONS_MASK;
 }
 #else /* !SECTION_IN_PAGE_FLAGS */
+static inline void set_page_section_from_pfn(struct page *page,
+		unsigned long pfn)
+{
+}
+
 static inline unsigned long memdesc_section(memdesc_flags_t mdf)
 {
 	return 0;
@@ -2768,9 +2779,7 @@ static inline void set_page_links(struct page *page, enum zone_type zone,
 {
 	set_page_zone(page, zone);
 	set_page_node(page, node);
-#ifdef SECTION_IN_PAGE_FLAGS
-	set_page_section(page, pfn_to_section_nr(pfn));
-#endif
+	set_page_section_from_pfn(page, pfn);
 }
 
 /**
-- 
2.20.1

[PATCH v6 4/8] mm: add a template-based fast path for zone-device page init

[Li Zhe]

memmap_init_zone_device() repeats nearly identical head-page
initialization for each PFN. Prepare one reusable ZONE_DEVICE head-page
template through the existing slow path, refresh the PFN-dependent
fields in that template before each copy, and memcpy it into each
destination page.

The optimized path assigns _refcount through the copied template, so
keep it disabled when the page_ref_set tracepoint is enabled.

This patch accelerates head-page initialization. The pfns_per_compound
== 1 case gets the full benefit here, compound tails are handled in the
next patch.

Tested in a VM with a 100 GB fsdax namespace device configured with
map=dev on Intel Ice Lake server. This test exercises the nd_pmem rebind
path (pfns_per_compound == 1).

Test procedure:
Rebind the nd_pmem driver 30 times and collect the memmap initialization
time from the pr_debug() output of memmap_init_zone_device().

Base(v7.2-rc1):
  First binding: 1456 ms
  Average of subsequent rebinds: 244.28 ms

With this patch and its prerequisites applied:
  First binding: 1440 ms
  Average of subsequent rebinds: 217.19 ms

This reduces the average rebind time from 244.28 ms to 217.19 ms, or
about 11%.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 mm/mm_init.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 58 insertions(+), 1 deletion(-)

diff --git a/mm/mm_init.c b/mm/mm_init.c
index 5fccfbacf855..170021e182e0 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -1065,6 +1065,44 @@ static void __ref zone_device_page_init_slow(struct page *page,
 		set_page_count(page, 0);
 }
 
+static inline bool zone_device_page_init_optimization_enabled(void)
+{
+	/*
+	 * The template fast path copies a preinitialized struct page image.
+	 * Skip it when the page_ref_set tracepoint is enabled.
+	 */
+	return !page_ref_tracepoint_active(page_ref_set);
+}
+
+/*
+ * 'template' is a reusable page prototype rather than a strictly immutable
+ * object. Most ZONE_DEVICE fields stay constant across the pages covered by
+ * the current template, but section bits and page->virtual may still depend
+ * on the PFN. Refresh those PFN-dependent fields in the template before
+ * copying it into @page.
+ */
+static inline void zone_device_page_update_template(struct page *template,
+		unsigned long pfn)
+{
+	set_page_section_from_pfn(template, pfn);
+#ifdef WANT_PAGE_VIRTUAL
+	if (!is_highmem_idx(ZONE_DEVICE))
+		set_page_address(template, __va(pfn << PAGE_SHIFT));
+#endif
+}
+
+static void zone_device_page_init_from_template(struct page *page,
+		unsigned long pfn, struct page *template)
+{
+	/*
+	 * 'template' carries the invariant portion of a ZONE_DEVICE struct
+	 * page. Update the PFN-dependent fields in place before copying it
+	 * to the destination page.
+	 */
+	zone_device_page_update_template(template, pfn);
+	memcpy(page, template, sizeof(*page));
+}
+
 /*
  * With compound page geometry and when struct pages are stored in ram most
  * tail pages are reused. Consequently, the amount of unique struct pages to
@@ -1120,6 +1158,7 @@ void __ref memmap_init_zone_device(struct zone *zone,
 				   unsigned long nr_pages,
 				   struct dev_pagemap *pgmap)
 {
+	bool use_template = zone_device_page_init_optimization_enabled();
 	unsigned long pfn, end_pfn = start_pfn + nr_pages;
 	struct pglist_data *pgdat = zone->zone_pgdat;
 	struct vmem_altmap *altmap = pgmap_altmap(pgmap);
@@ -1127,6 +1166,7 @@ void __ref memmap_init_zone_device(struct zone *zone,
 	unsigned long zone_idx = zone_idx(zone);
 	unsigned long start = jiffies;
 	int nid = pgdat->node_id;
+	struct page template;
 
 	if (WARN_ON_ONCE(!pgmap || zone_idx != ZONE_DEVICE))
 		return;
@@ -1144,7 +1184,24 @@ void __ref memmap_init_zone_device(struct zone *zone,
 	for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) {
 		struct page *page = pfn_to_page(pfn);
 
-		zone_device_page_init_slow(page, pfn, zone_idx, nid, pgmap);
+		if (!use_template) {
+			zone_device_page_init_slow(page, pfn, zone_idx,
+						   nid, pgmap);
+		} else if (pfn == start_pfn) {
+			/*
+			 * Seed the reusable head-page template from the
+			 * first real struct page, because the existing
+			 * page-init and pageblock helpers expect a real
+			 * memmap entry rather than a stack object.
+			 */
+			zone_device_page_init_slow(page, pfn, zone_idx,
+						   nid, pgmap);
+			/* init template page */
+			memcpy(&template, page, sizeof(*page));
+		} else {
+			zone_device_page_init_from_template(page, pfn,
+							    &template);
+		}
 
 		if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
 			cond_resched();
-- 
2.20.1

[PATCH v6 5/8] mm: extend the template fast path to zone-device compound tails

[Li Zhe]

The template fast path from the previous patch only accelerates head
pages. Compound tails in memmap_init_compound() still go through the
slow path one by one.

Build separate head and tail templates and reuse one prepared tail
template across the tail pages in a compound range. Head pages preserve
the existing refcount policy, while compound tails always start with a
refcount of 0 after prep_compound_tail().

This extends the template-copy fast path to pfns_per_compound > 1
without changing the existing slow path. Tail-page PFN-dependent fields
are refreshed in the reusable tail template before each copy.

Tested in a VM with a 100 GB devdax namespace (align=2097152) on Intel
Ice Lake server. This test exercises the dax_pmem rebind path and
measures memmap initialization latency.

Test procedure:
Unbind and rebind the dax_pmem driver 30 times, collect memmap
initialization time from the pr_debug() output of memmap_init_zone_device().

Base(v7.2-rc1):
  First binding: 1462 ms
  Average of subsequent rebinds: 273.31 ms

With this patch and its prerequisites applied:
  First binding: 1403 ms
  Average of subsequent rebinds: 244.37 ms

This reduces the average rebind time from 273.31 ms to 244.37 ms, or
about 10.6%.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 mm/mm_init.c | 40 +++++++++++++++++++++++++++++++++++-----
 1 file changed, 35 insertions(+), 5 deletions(-)

diff --git a/mm/mm_init.c b/mm/mm_init.c
index 170021e182e0..fb855bb0437a 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -1074,6 +1074,16 @@ static inline bool zone_device_page_init_optimization_enabled(void)
 	return !page_ref_tracepoint_active(page_ref_set);
 }
 
+static inline void zone_device_tail_page_init(struct page *page,
+		unsigned long pfn, unsigned long zone_idx, int nid,
+		struct dev_pagemap *pgmap, const struct page *head,
+		unsigned int order)
+{
+	zone_device_page_init_slow(page, pfn, zone_idx, nid, pgmap);
+	prep_compound_tail(page, head, order);
+	set_page_count(page, 0);
+}
+
 /*
  * 'template' is a reusable page prototype rather than a strictly immutable
  * object. Most ZONE_DEVICE fields stay constant across the pages covered by
@@ -1131,10 +1141,12 @@ static void __ref memmap_init_compound(struct page *head,
 				       unsigned long head_pfn,
 				       unsigned long zone_idx, int nid,
 				       struct dev_pagemap *pgmap,
-				       unsigned long nr_pages)
+				       unsigned long nr_pages,
+				       bool use_template)
 {
 	unsigned long pfn, end_pfn = head_pfn + nr_pages;
 	unsigned int order = pgmap->vmemmap_shift;
+	struct page template;
 
 	/*
 	 * We have to initialize the pages, including setting up page links.
@@ -1143,12 +1155,29 @@ static void __ref memmap_init_compound(struct page *head,
 	 * the pages in the same go.
 	 */
 	__SetPageHead(head);
+
 	for (pfn = head_pfn + 1; pfn < end_pfn; pfn++) {
 		struct page *page = pfn_to_page(pfn);
 
-		zone_device_page_init_slow(page, pfn, zone_idx, nid, pgmap);
-		prep_compound_tail(page, head, order);
-		set_page_count(page, 0);
+		if (!use_template) {
+			zone_device_tail_page_init(page, pfn, zone_idx, nid,
+						   pgmap, head, order);
+		} else if (pfn == head_pfn + 1) {
+			/*
+			 * All tails of the same compound page share the
+			 * state established by prep_compound_tail(). Reuse
+			 * one tail template for the whole range and
+			 * refresh only the PFN-dependent fields in that
+			 * template before each copy.
+			 */
+			zone_device_tail_page_init(page, pfn, zone_idx, nid,
+						   pgmap, head, order);
+			/* init template page */
+			memcpy(&template, page, sizeof(*page));
+		} else {
+			zone_device_page_init_from_template(page, pfn,
+							    &template);
+		}
 	}
 	prep_compound_head(head, order);
 }
@@ -1210,7 +1239,8 @@ void __ref memmap_init_zone_device(struct zone *zone,
 			continue;
 
 		memmap_init_compound(page, pfn, zone_idx, nid, pgmap,
-				     compound_nr_pages(pfn, altmap, pgmap));
+				     compound_nr_pages(pfn, altmap, pgmap),
+				     use_template);
 	}
 
 	pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE);
-- 
2.20.1

[PATCH v6 6/8] string: introduce memcpy_nt() helpers

[Li Zhe]

Introduce memcpy_nt() and memcpy_nt_drain() for write-once copy sites
that want a named non-temporal copy primitive plus an explicit drain
step.

On x86_64, override both helpers in arch/x86/include/asm/string_64.h
using the usual self-macro pattern, next to the existing
memcpy_flushcache() backend that memcpy_nt() wraps. The x86_64
implementation maps memcpy_nt() to memcpy_flushcache() and uses wmb()
for memcpy_nt_drain(), because that backend issues MOVNTI stores and
callers need an ordering point before later normal stores that depend
on those writes becoming visible.

include/linux/string.h provides the generic fallback under
memcpy_nt() as plain memcpy() and leaves memcpy_nt_drain() empty, so
architectures that do not override memcpy_nt() do not pay an
unconditional barrier. Architectures that later grow a specialized
memcpy_nt() backend can override memcpy_nt_drain() with whatever
drain primitive their memory-ordering rules require.

The immediate user is the ZONE_DEVICE template-copy path. It populates
struct page descriptors in a write-once pattern, so a regular cached
memcpy() can incur avoidable write-allocate traffic and cache
pollution for data with little near-term reuse.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 arch/x86/include/asm/string_64.h | 22 ++++++++++++++++++++++
 include/linux/string.h           | 23 +++++++++++++++++++++++
 2 files changed, 45 insertions(+)

diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h
index 4635616863f5..6cb9e0ac7fa0 100644
--- a/arch/x86/include/asm/string_64.h
+++ b/arch/x86/include/asm/string_64.h
@@ -4,6 +4,7 @@
 
 #ifdef __KERNEL__
 #include <linux/jump_label.h>
+#include <asm/barrier.h>
 
 /* Written 2002 by Andi Kleen */
 
@@ -100,6 +101,27 @@ static __always_inline void memcpy_flushcache(void *dst, const void *src, size_t
 	}
 	__memcpy_flushcache(dst, src, cnt);
 }
+
+#define memcpy_nt memcpy_nt
+/*
+ * Reuse the existing x86 flushcache backend as the nt copy primitive.
+ * Callers pair it with memcpy_nt_drain() when later stores must be
+ * ordered after the copy.
+ */
+static __always_inline void memcpy_nt(void *dst, const void *src, size_t cnt)
+{
+	memcpy_flushcache(dst, src, cnt);
+}
+
+#define memcpy_nt_drain memcpy_nt_drain
+static __always_inline void memcpy_nt_drain(void)
+{
+	/*
+	 * Order the prior MOVNTI stores issued by memcpy_flushcache()
+	 * before later normal stores.
+	 */
+	wmb();
+}
 #endif
 
 #endif /* __KERNEL__ */
diff --git a/include/linux/string.h b/include/linux/string.h
index 5702daca4326..a109b2f86ca6 100644
--- a/include/linux/string.h
+++ b/include/linux/string.h
@@ -278,6 +278,29 @@ static inline void memcpy_flushcache(void *dst, const void *src, size_t cnt)
 }
 #endif
 
+#ifndef memcpy_nt
+/*
+ * memcpy_nt() requests a non-temporal copy when the architecture has a
+ * suitable backend. Architectures that do not override it fall back to
+ * memcpy().
+ */
+static inline void memcpy_nt(void *dst, const void *src, size_t cnt)
+{
+	memcpy(dst, src, cnt);
+}
+#endif
+
+#ifndef memcpy_nt_drain
+/*
+ * Callers use memcpy_nt_drain() before later normal stores that need to
+ * be ordered after memcpy_nt(). Architectures without a specialized
+ * backend can leave it empty.
+ */
+static inline void memcpy_nt_drain(void)
+{
+}
+#endif
+
 void *memchr_inv(const void *s, int c, size_t n);
 char *strreplace(char *str, char old, char new);
 
-- 
2.20.1

[PATCH v6 7/8] x86/string: extend memcpy_flushcache() fixed-size fastpaths

[Li Zhe]

The new x86 memcpy_nt() helper in this series maps to
memcpy_flushcache(), and the ZONE_DEVICE fast path uses that primitive
for constant-sized struct page template copies.

The immediately relevant x86_64 sizeof(struct page) values here are 64,
80, and 96 bytes. memcpy_flushcache() currently inlines only the 4, 8,
and 16-byte cases, so even those constant-sized struct page copies fall
through to __memcpy_flushcache().

Add 32, 48, 64, 80, and 96-byte MOVNTI sequences to the main fixed-size
switch so those constant-sized copies can stay on the inline path.
Factor the larger sequences into movnti_8()/16()/32()/64() helpers so
the switch can reuse them without duplicating the inline assembly.
While the ZONE_DEVICE template-copy path only needs 64/80/96-byte
copies, keep 32-byte and 48-byte copies inline as well rather than
sending those nearby fixed-size cases back to __memcpy_flushcache().

A microbenchmark compared two memcpy_flushcache()-style helpers that
shared the same generic body and differed only in whether
32/48/64/80/96-byte copies stayed in the fixed-size switch or were
redirected to __memcpy_flushcache() when the destination was not 8-byte
aligned. The timed interval covered the copy loop only; wmb() was used
only to separate rounds.

In pseudo-code, the two variants were:

variant A:
	switch (len) {
		case 32:
		case 48:
		case 64:
		case 80:
		case 96:
			do fixed-size MOVNTI stores;
			return;
		default:
			generic __memcpy_flushcache()-style body;
	}

variant B:
	switch (len) {
		case 32:
		case 48:
		case 64:
		case 80:
		case 96:
			if (!IS_ALIGNED(dst, 8))
				goto generic_path;
			do fixed-size MOVNTI stores;
			return;
		default:
	generic_path:
			generic __memcpy_flushcache()-style body;
	}

For offsets 1..7 averaged, keeping those sizes in the fixed-size
switch took about 0.410/0.411/0.426/0.426/0.428 us per
32/48/64/80/96-byte copy, while redirecting the same cases to
__memcpy_flushcache() took about 0.962/0.956/0.923/0.998/1.001 us.

Based on that measurement, keep these new cases in the main
memcpy_flushcache() fixed-size switch without a separate
alignment check.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 arch/x86/include/asm/string_64.h | 56 +++++++++++++++++++++++++++++++-
 1 file changed, 55 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h
index 6cb9e0ac7fa0..1e634264a31b 100644
--- a/arch/x86/include/asm/string_64.h
+++ b/arch/x86/include/asm/string_64.h
@@ -83,7 +83,35 @@ int strcmp(const char *cs, const char *ct);
 #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE
 #define __HAVE_ARCH_MEMCPY_FLUSHCACHE 1
 void __memcpy_flushcache(void *dst, const void *src, size_t cnt);
-static __always_inline void memcpy_flushcache(void *dst, const void *src, size_t cnt)
+
+static __always_inline void movnti_8(void *dst, const void *src)
+{
+	asm volatile("movntiq %1, %0"
+		     : "=m"(*(u64 *)dst)
+		     : "r"(*(const u64 *)src)
+		     : "memory");
+}
+
+static __always_inline void movnti_16(void *dst, const void *src)
+{
+	movnti_8(dst, src);
+	movnti_8(dst + 8, src + 8);
+}
+
+static __always_inline void movnti_32(void *dst, const void *src)
+{
+	movnti_16(dst, src);
+	movnti_16(dst + 16, src + 16);
+}
+
+static __always_inline void movnti_64(void *dst, const void *src)
+{
+	movnti_32(dst, src);
+	movnti_32(dst + 32, src + 32);
+}
+
+static __always_inline void memcpy_flushcache(void *dst, const void *src,
+					      size_t cnt)
 {
 	if (__builtin_constant_p(cnt)) {
 		switch (cnt) {
@@ -97,8 +125,34 @@ static __always_inline void memcpy_flushcache(void *dst, const void *src, size_t
 				asm ("movntiq %1, %0" : "=m"(*(u64 *)dst) : "r"(*(u64 *)src));
 				asm ("movntiq %1, %0" : "=m"(*(u64 *)(dst + 8)) : "r"(*(u64 *)(src + 8)));
 				return;
+			/*
+			 * The relevant fixed-size copies here are the
+			 * x86_64 struct page sizes: 64, 80, and 96 bytes.
+			 * Keep 32-byte and 48-byte copies inline as well
+			 * instead of sending those nearby fixed-size
+			 * cases back to __memcpy_flushcache().
+			 */
+			case 32:
+				movnti_32(dst, src);
+				return;
+			case 48:
+				movnti_32(dst, src);
+				movnti_16(dst + 32, src + 32);
+				return;
+			case 64:
+				movnti_64(dst, src);
+				return;
+			case 80:
+				movnti_64(dst, src);
+				movnti_16(dst + 64, src + 64);
+				return;
+			case 96:
+				movnti_64(dst, src);
+				movnti_32(dst + 64, src + 64);
+				return;
 		}
 	}
+
 	__memcpy_flushcache(dst, src, cnt);
 }
 
-- 
2.20.1

[PATCH v6 8/8] mm: use memcpy_nt() in zone-device template copies

[Li Zhe]

The template fast path currently uses memcpy() for the actual struct
page copy. Switch zone_device_page_init_from_template() to memcpy_nt()
and add memcpy_nt_drain() before memmap_init_compound(), before
prep_compound_head() updates overlapping tail metadata, and before
returning from memmap_init_zone_device().

ZONE_DEVICE memmap initialization is largely write-once: each struct
page is populated once, and most destination cachelines are not expected
to be reused immediately afterwards. On x86, a regular cached memcpy()
can therefore incur write-allocate traffic by pulling destination
cachelines into the cache before writeback, and can populate the cache
with data that has little near-term reuse. Using memcpy_nt() lets this
path request non-temporal stores for that copy pattern, which can reduce
cache pollution and avoid part of the associated write-allocate
overhead, while architectures without a specialized backend still fall
back to memcpy().

When memcpy_nt() maps to non-temporal stores, order those stores before
memmap_init_compound(), before prep_compound_head() updates overlapping
compound metadata, and before returning from memmap_init_zone_device().

Keep sanitized builds on the slow path so KASAN/KMSAN retain their
instrumented stores.

Tested in a VM with a 100 GB fsdax namespace device configured with
map=dev and a 100 GB devdax namespace (align=2097152) on Intel Ice Lake
server.

Test procedure:
Rebind the nd_pmem and dax_pmem driver 30 times and collect the memmap
initialization time from the pr_debug() output of
memmap_init_zone_device().

Base(v7.2-rc1):
  First binding for nd_pmem driver: 1456 ms
  Average of subsequent rebinds: 244.28 ms

  First binding for dax_pmem driver: 1462 ms
  Average of subsequent rebinds: 273.31 ms

With this series:
  First binding for nd_pmem driver: 1272 ms
  Average of subsequent rebinds: 96.79 ms

  First binding for dax_pmem driver: 1354 ms
  Average of subsequent rebinds: 119.04 ms

This reduces the average rebind time by about 60.4% for nd_pmem and
56.4% for dax_pmem.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 mm/mm_init.c | 39 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 37 insertions(+), 2 deletions(-)

diff --git a/mm/mm_init.c b/mm/mm_init.c
index fb855bb0437a..addb4969587e 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -1067,11 +1067,21 @@ static void __ref zone_device_page_init_slow(struct page *page,
 
 static inline bool zone_device_page_init_optimization_enabled(void)
 {
+	/*
+	 * Keep sanitized builds on the slow path so their stores stay
+	 * instrumented.
+	 */
+	if (IS_ENABLED(CONFIG_KASAN) || IS_ENABLED(CONFIG_KMSAN))
+		return false;
+
 	/*
 	 * The template fast path copies a preinitialized struct page image.
 	 * Skip it when the page_ref_set tracepoint is enabled.
 	 */
-	return !page_ref_tracepoint_active(page_ref_set);
+	if (page_ref_tracepoint_active(page_ref_set))
+		return false;
+
+	return true;
 }
 
 static inline void zone_device_tail_page_init(struct page *page,
@@ -1110,7 +1120,7 @@ static void zone_device_page_init_from_template(struct page *page,
 	 * to the destination page.
 	 */
 	zone_device_page_update_template(template, pfn);
-	memcpy(page, template, sizeof(*page));
+	memcpy_nt(page, template, sizeof(*page));
 }
 
 /*
@@ -1179,6 +1189,15 @@ static void __ref memmap_init_compound(struct page *head,
 							    &template);
 		}
 	}
+
+	/*
+	 * When the template path is enabled, order the preceding tail-page copies
+	 * before prep_compound_head() updates the overlapping compound metadata
+	 * in the first tail-page descriptors. If memcpy_nt() fell back to
+	 * regular cached stores, memcpy_nt_drain() may be a no-op.
+	 */
+	if (use_template)
+		memcpy_nt_drain();
 	prep_compound_head(head, order);
 }
 
@@ -1238,10 +1257,26 @@ void __ref memmap_init_zone_device(struct zone *zone,
 		if (pfns_per_compound == 1)
 			continue;
 
+		/*
+		 * When the template path is enabled, order the preceding head-page copy
+		 * before memmap_init_compound(), which immediately updates compound-head
+		 * metadata. If memcpy_nt() fell back to regular cached stores,
+		 * memcpy_nt_drain() may be a no-op.
+		 */
+		if (use_template)
+			memcpy_nt_drain();
+
 		memmap_init_compound(page, pfn, zone_idx, nid, pgmap,
 				     compound_nr_pages(pfn, altmap, pgmap),
 				     use_template);
 	}
+	/*
+	 * Ensure any prior template copies are ordered before returning.
+	 * On architectures where memcpy_nt() used regular cached stores,
+	 * memcpy_nt_drain() may be a no-op.
+	 */
+	if (use_template)
+		memcpy_nt_drain();
 
 	pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE);
 
-- 
2.20.1

Re: [PATCH v6 0/8] mm: optimize zone-device memmap initialization

[Balbir Singh]

On Thu, Jul 09, 2026 at 07:25:12PM +0800, Li Zhe wrote:
> memmap_init_zone_device() can take a noticeable amount of time when large
> pmem namespaces are bound or rebound, because it initializes nearly
> identical struct page descriptors one PFN at a time. This series reduces
> that ZONE_DEVICE memmap initialization overhead by reusing prepared
> struct page templates and, on x86, using memcpy_nt() for the template
> copy path.
>

Is the slowness specific to an architecture? Do you have any test
results from any other arch?

> The main target is large fsdax/devdax pmem configurations, where the
> cost of initializing the memmap shows up directly in nd_pmem/dax_pmem
> bind and rebind latency.
> 
> Patches 1-3 are preparatory cleanups and helper extraction. Patches 4-5
> add the template-copy fast path for head pages and compound tails.
> Patches 6-8 introduce memcpy_nt()/memcpy_nt_drain(), extend the x86
> fixed-size memcpy_flushcache() inline cases used by that helper, and
> switch the template-copy path over to memcpy_nt().
> 
> The fast path remains disabled when the page_ref_set tracepoint is
> active, and sanitized builds stay on the slow path so their instrumented
> stores are preserved. Architectures without a specialized memcpy_nt()
> backend continue to fall back to memcpy().
> 
> Tested in a VM with a 100 GB fsdax namespace device configured with
> map=dev and a 100 GB devdax namespace (align=2097152) on Intel Ice Lake
> server.
> 
> Test procedure:
> Rebind the nd_pmem and dax_pmem driver 30 times and collect the memmap
> initialization time from the pr_debug() output of
> memmap_init_zone_device().
> 
> Base(v7.2-rc1):
>   First binding for nd_pmem driver: 1456 ms
>   Average of subsequent rebinds: 244.28 ms
> 
>   First binding for dax_pmem driver: 1462 ms
>   Average of subsequent rebinds: 273.31 ms
> 
> With this series applied:
>   First binding for nd_pmem driver: 1272 ms
>   Average of subsequent rebinds: 96.79 ms
> 
>   First binding for dax_pmem driver: 1354 ms
>   Average of subsequent rebinds: 119.04 ms
> 
> This reduces the average rebind time by about 60.4% for nd_pmem and
> 56.4% for dax_pmem.
> 
> As an additional data point, I also ran a smaller set of measurements on
> the same physical x86_64 host with a 100 GB PMEM region created via the
> memmap= kernel command line, configured as fsdax and devdax namespaces
> with map=dev and 2 MiB alignment.
> 
> For brevity, the individual patches keep only the VM results rather than
> including a second set of physical-host measurements throughout the
> series. The physical-host numbers below are included only as
> supplemental evidence that the same optimization also provides a similar
> benefit on a non-virtualized system.
> 
> Test procedure:
> Reconfigure the namespace mode, rebind the nd_pmem or dax_pmem driver
> once, and collect the memmap initialization time from the pr_debug()
> output of memmap_init_zone_device().
> 
> Base (v7.2-rc1):
>   nd_pmem / fsdax: 179 ms
>   dax_pmem / devdax: 264 ms
> 
> With this series applied:
>   nd_pmem / fsdax: 82 ms
>   dax_pmem / devdax: 113 ms
> 
> This reduces the measured rebind time by about 54.2% for nd_pmem and
> 57.2% for dax_pmem on that setup, which is broadly consistent with the
> VM results above.
> 
> As another supplemental data point, I also measured the test_hmm.ko
> module on the same physical x86_64 host, using the test_hmm.ko setup
> from the previous discussion that times ten 64 GB
> memremap_pages()/memunmap_pages() iterations during module insertion[1].
> By default, module insertion initializes two DEVICE_PRIVATE dmirror
> devices, so two avg memremap values are reported; each value is the
> average for one 64 GB chunk.
> 
> This is not the primary target workload of the series, but it exercises
> the same large ZONE_DEVICE memmap initialization path and shows the same
> direction of improvement.
> 
> Base (v7.2-rc1):
>   avg memremap reported during module insertion: 116689362 ns, 116539263 ns
> 
> With this series applied:
>   avg memremap reported during module insertion: 54607108 ns, 54458236 ns
> 
> This corresponds to about a 53.2% reduction based on the mean of the
> reported values, which is again consistent with the pmem bind/rebind
> results above.
>

That is a good result, thanks for sharing

> [1] https://lore.kernel.org/all/aiEoByaQdRR3xtM5@nvdebian.thelocal/
> 
> Li Zhe (8):
>   mm: fix stale ZONE_DEVICE refcount comment
>   mm: factor zone-device page init helpers out of
>     __init_zone_device_page
>   mm: add a set_page_section_from_pfn() helper
>   mm: add a template-based fast path for zone-device page init
>   mm: extend the template fast path to zone-device compound tails
>   string: introduce memcpy_nt() helpers
>   x86/string: extend memcpy_flushcache() fixed-size fastpaths
>   mm: use memcpy_nt() in zone-device template copies
> 
>  arch/x86/include/asm/string_64.h |  78 ++++++++++++-
>  include/linux/mm.h               |  15 ++-
>  include/linux/string.h           |  23 ++++
>  mm/mm_init.c                     | 186 ++++++++++++++++++++++++++-----
>  4 files changed, 273 insertions(+), 29 deletions(-)
> 
> ---
> v5: https://lore.kernel.org/all/20260701090553.62691-1-lizhe.67@bytedance.com/
> v4: https://lore.kernel.org/all/20260603080152.64728-1-lizhe.67@bytedance.com/
> v3: https://lore.kernel.org/all/20260527033636.28231-1-lizhe.67@bytedance.com/
> v2: https://lore.kernel.org/all/20260521040124.10608-1-lizhe.67@bytedance.com/
> v1: https://lore.kernel.org/all/20260515082045.63029-1-lizhe.67@bytedance.com/
> 
> Changelogs:
> 
> v5->v6:
> - Rework patch 6 so the x86 memcpy_nt_drain() helper uses wmb()
>   instead of an open-coded sfence, while architectures that do not
>   override memcpy_nt() keep the generic no-op drain fallback.
>   Suggested by Borislav Petkov.
> - Rework patch 6 to use the usual self-macro override pattern for
>   memcpy_nt() and memcpy_nt_drain() instead of a dedicated
>   __HAVE_ARCH_MEMCPY_NT feature macro. Suggested by David
>   Hildenbrand.
> - Drop the default: case from pagemap_resets_refcount() and drop the
>   WARN_ONCE() after the switch so newly added enum memory_type values
>   remain visible to compiler switch checking. Suggested by David
>   Hildenbrand.
> - Drop the unnecessary empty set_page_section() stub from patch 3, keep
>   set_page_section_from_pfn() as the only !SECTION_IN_PAGE_FLAGS no-op
>   helper and fix its indentation. Suggested by David Hildenbrand.
> - Rework patch 4 so the reusable head-page template is seeded from the
>   first loop iteration, keeping the main loop intact while ensuring the
>   first page is initialized only once; also drop the extra template
>   memcpy helper, as suggested by Alistair Popple.
> - Rework patch 5 similarly for compound tails: seed the reusable tail
>   template from the first tail-page iteration so the first tail page is
>   initialized only once, while keeping the main tail loop intact, as
>   suggested by Alistair Popple.
> - Rework patch 7 to rename the internal fixed-size MOVNTI helpers to
>   movnti_*(), keep memcpy_flushcache() as the externally visible
>   wrapper, fold the new 32/48/64/80/96-byte copies into the main
>   fixed-size switch, clarify in the code and changelog that the
>   ZONE_DEVICE template-copy path relies on the 64/80/96-byte struct
>   page copies while 32/48-byte copies stay inline as well, and drop the
>   separate alignment check based on the physical-host microbenchmark
>   results. Suggested by Borislav Petkov.
> - Run v6 VM spot checks for the fsdax/devdax map=dev paths. The
>   results stayed in the same ballpark as v5, so keep the previously
>   posted v5 full-run numbers in this version instead of replacing them
>   with another partial data set.
> 
> For changelogs of earlier revisions, please refer to the v5 cover letter.
> 
> -- 
> 2.20.1

Balbir

Re: [PATCH v6 0/8] mm: optimize zone-device memmap initialization

[ByteDance]

On 7/13/26 9:44 AM, Balbir Singh wrote:
> On Thu, Jul 09, 2026 at 07:25:12PM +0800, Li Zhe wrote:
>> memmap_init_zone_device() can take a noticeable amount of time when large
>> pmem namespaces are bound or rebound, because it initializes nearly
>> identical struct page descriptors one PFN at a time. This series reduces
>> that ZONE_DEVICE memmap initialization overhead by reusing prepared
>> struct page templates and, on x86, using memcpy_nt() for the template
>> copy path.
>>
> Is the slowness specific to an architecture? Do you have any test
> results from any other arch?


The slowness I am targeting is not specific to x86.

The generic part is that memmap_init_zone_device() initializes nearly
identical struct page state one PFN at a time. Patches 4-5 address that
part by reusing prepared head/tail templates. Patches 6-8 then add an
x86-specific optimization for the copy step via memcpy_nt().

I only have x86_64 measurements at the moment, so I do not want to
claim results for another architecture without data. On the x86_64
setup, patches 4-5 already reduced the average rebind time by about
11.0% for the fsdax case and 10.6% for the devdax case before the
memcpy_nt() changes were added.

Thanks,
Zhe

>> The main target is large fsdax/devdax pmem configurations, where the
>> cost of initializing the memmap shows up directly in nd_pmem/dax_pmem
>> bind and rebind latency.
>>
>> Patches 1-3 are preparatory cleanups and helper extraction. Patches 4-5
>> add the template-copy fast path for head pages and compound tails.
>> Patches 6-8 introduce memcpy_nt()/memcpy_nt_drain(), extend the x86
>> fixed-size memcpy_flushcache() inline cases used by that helper, and
>> switch the template-copy path over to memcpy_nt().
>>
>> The fast path remains disabled when the page_ref_set tracepoint is
>> active, and sanitized builds stay on the slow path so their instrumented
>> stores are preserved. Architectures without a specialized memcpy_nt()
>> backend continue to fall back to memcpy().
>>
>> Tested in a VM with a 100 GB fsdax namespace device configured with
>> map=dev and a 100 GB devdax namespace (align=2097152) on Intel Ice Lake
>> server.
>>
>> Test procedure:
>> Rebind the nd_pmem and dax_pmem driver 30 times and collect the memmap
>> initialization time from the pr_debug() output of
>> memmap_init_zone_device().
>>
>> Base(v7.2-rc1):
>>    First binding for nd_pmem driver: 1456 ms
>>    Average of subsequent rebinds: 244.28 ms
>>
>>    First binding for dax_pmem driver: 1462 ms
>>    Average of subsequent rebinds: 273.31 ms
>>
>> With this series applied:
>>    First binding for nd_pmem driver: 1272 ms
>>    Average of subsequent rebinds: 96.79 ms
>>
>>    First binding for dax_pmem driver: 1354 ms
>>    Average of subsequent rebinds: 119.04 ms
>>
>> This reduces the average rebind time by about 60.4% for nd_pmem and
>> 56.4% for dax_pmem.
>>
>> As an additional data point, I also ran a smaller set of measurements on
>> the same physical x86_64 host with a 100 GB PMEM region created via the
>> memmap= kernel command line, configured as fsdax and devdax namespaces
>> with map=dev and 2 MiB alignment.
>>
>> For brevity, the individual patches keep only the VM results rather than
>> including a second set of physical-host measurements throughout the
>> series. The physical-host numbers below are included only as
>> supplemental evidence that the same optimization also provides a similar
>> benefit on a non-virtualized system.
>>
>> Test procedure:
>> Reconfigure the namespace mode, rebind the nd_pmem or dax_pmem driver
>> once, and collect the memmap initialization time from the pr_debug()
>> output of memmap_init_zone_device().
>>
>> Base (v7.2-rc1):
>>    nd_pmem / fsdax: 179 ms
>>    dax_pmem / devdax: 264 ms
>>
>> With this series applied:
>>    nd_pmem / fsdax: 82 ms
>>    dax_pmem / devdax: 113 ms
>>
>> This reduces the measured rebind time by about 54.2% for nd_pmem and
>> 57.2% for dax_pmem on that setup, which is broadly consistent with the
>> VM results above.
>>
>> As another supplemental data point, I also measured the test_hmm.ko
>> module on the same physical x86_64 host, using the test_hmm.ko setup
>> from the previous discussion that times ten 64 GB
>> memremap_pages()/memunmap_pages() iterations during module insertion[1].
>> By default, module insertion initializes two DEVICE_PRIVATE dmirror
>> devices, so two avg memremap values are reported; each value is the
>> average for one 64 GB chunk.
>>
>> This is not the primary target workload of the series, but it exercises
>> the same large ZONE_DEVICE memmap initialization path and shows the same
>> direction of improvement.
>>
>> Base (v7.2-rc1):
>>    avg memremap reported during module insertion: 116689362 ns, 116539263 ns
>>
>> With this series applied:
>>    avg memremap reported during module insertion: 54607108 ns, 54458236 ns
>>
>> This corresponds to about a 53.2% reduction based on the mean of the
>> reported values, which is again consistent with the pmem bind/rebind
>> results above.
>>
> That is a good result, thanks for sharing
>
>> [1] https://lore.kernel.org/all/aiEoByaQdRR3xtM5@nvdebian.thelocal/
>>
>> Li Zhe (8):
>>    mm: fix stale ZONE_DEVICE refcount comment
>>    mm: factor zone-device page init helpers out of
>>      __init_zone_device_page
>>    mm: add a set_page_section_from_pfn() helper
>>    mm: add a template-based fast path for zone-device page init
>>    mm: extend the template fast path to zone-device compound tails
>>    string: introduce memcpy_nt() helpers
>>    x86/string: extend memcpy_flushcache() fixed-size fastpaths
>>    mm: use memcpy_nt() in zone-device template copies
>>
>>   arch/x86/include/asm/string_64.h |  78 ++++++++++++-
>>   include/linux/mm.h               |  15 ++-
>>   include/linux/string.h           |  23 ++++
>>   mm/mm_init.c                     | 186 ++++++++++++++++++++++++++-----
>>   4 files changed, 273 insertions(+), 29 deletions(-)
>>
>> ---
>> v5: https://lore.kernel.org/all/20260701090553.62691-1-lizhe.67@bytedance.com/
>> v4: https://lore.kernel.org/all/20260603080152.64728-1-lizhe.67@bytedance.com/
>> v3: https://lore.kernel.org/all/20260527033636.28231-1-lizhe.67@bytedance.com/
>> v2: https://lore.kernel.org/all/20260521040124.10608-1-lizhe.67@bytedance.com/
>> v1: https://lore.kernel.org/all/20260515082045.63029-1-lizhe.67@bytedance.com/
>>
>> Changelogs:
>>
>> v5->v6:
>> - Rework patch 6 so the x86 memcpy_nt_drain() helper uses wmb()
>>    instead of an open-coded sfence, while architectures that do not
>>    override memcpy_nt() keep the generic no-op drain fallback.
>>    Suggested by Borislav Petkov.
>> - Rework patch 6 to use the usual self-macro override pattern for
>>    memcpy_nt() and memcpy_nt_drain() instead of a dedicated
>>    __HAVE_ARCH_MEMCPY_NT feature macro. Suggested by David
>>    Hildenbrand.
>> - Drop the default: case from pagemap_resets_refcount() and drop the
>>    WARN_ONCE() after the switch so newly added enum memory_type values
>>    remain visible to compiler switch checking. Suggested by David
>>    Hildenbrand.
>> - Drop the unnecessary empty set_page_section() stub from patch 3, keep
>>    set_page_section_from_pfn() as the only !SECTION_IN_PAGE_FLAGS no-op
>>    helper and fix its indentation. Suggested by David Hildenbrand.
>> - Rework patch 4 so the reusable head-page template is seeded from the
>>    first loop iteration, keeping the main loop intact while ensuring the
>>    first page is initialized only once; also drop the extra template
>>    memcpy helper, as suggested by Alistair Popple.
>> - Rework patch 5 similarly for compound tails: seed the reusable tail
>>    template from the first tail-page iteration so the first tail page is
>>    initialized only once, while keeping the main tail loop intact, as
>>    suggested by Alistair Popple.
>> - Rework patch 7 to rename the internal fixed-size MOVNTI helpers to
>>    movnti_*(), keep memcpy_flushcache() as the externally visible
>>    wrapper, fold the new 32/48/64/80/96-byte copies into the main
>>    fixed-size switch, clarify in the code and changelog that the
>>    ZONE_DEVICE template-copy path relies on the 64/80/96-byte struct
>>    page copies while 32/48-byte copies stay inline as well, and drop the
>>    separate alignment check based on the physical-host microbenchmark
>>    results. Suggested by Borislav Petkov.
>> - Run v6 VM spot checks for the fsdax/devdax map=dev paths. The
>>    results stayed in the same ballpark as v5, so keep the previously
>>    posted v5 full-run numbers in this version instead of replacing them
>>    with another partial data set.
>>
>> For changelogs of earlier revisions, please refer to the v5 cover letter.
>>
>> -- 
>> 2.20.1
> Balbir

Re: [PATCH v6 0/8] mm: optimize zone-device memmap initialization

[Muchun Song]

> On Jul 9, 2026, at 19:25, Li Zhe <lizhe.67@bytedance.com> wrote:
> 
> memmap_init_zone_device() can take a noticeable amount of time when large
> pmem namespaces are bound or rebound, because it initializes nearly
> identical struct page descriptors one PFN at a time. This series reduces

Hi, I have a detailed question regarding the timing of the bound or rebound
process.

1. How long does the entire process take?
2. Specifically, how long does the struct page initialization take?

I'm trying to understand the time distribution and what percentage of the
total process is occupied by the struct page initialization.

> that ZONE_DEVICE memmap initialization overhead by reusing prepared
> struct page templates and, on x86, using memcpy_nt() for the template
> copy path.
> 
> The main target is large fsdax/devdax pmem configurations, where the
> cost of initializing the memmap shows up directly in nd_pmem/dax_pmem
> bind and rebind latency.
> 
> Patches 1-3 are preparatory cleanups and helper extraction. Patches 4-5
> add the template-copy fast path for head pages and compound tails.
> Patches 6-8 introduce memcpy_nt()/memcpy_nt_drain(), extend the x86
> fixed-size memcpy_flushcache() inline cases used by that helper, and
> switch the template-copy path over to memcpy_nt().
> 
> The fast path remains disabled when the page_ref_set tracepoint is
> active, and sanitized builds stay on the slow path so their instrumented
> stores are preserved. Architectures without a specialized memcpy_nt()
> backend continue to fall back to memcpy().
> 
> Tested in a VM with a 100 GB fsdax namespace device configured with
> map=dev and a 100 GB devdax namespace (align=2097152) on Intel Ice Lake
> server.
> 
> Test procedure:
> Rebind the nd_pmem and dax_pmem driver 30 times and collect the memmap
> initialization time from the pr_debug() output of
> memmap_init_zone_device().
> 
> Base(v7.2-rc1):
>  First binding for nd_pmem driver: 1456 ms
>  Average of subsequent rebinds: 244.28 ms
> 
>  First binding for dax_pmem driver: 1462 ms
>  Average of subsequent rebinds: 273.31 ms
> 
> With this series applied:
>  First binding for nd_pmem driver: 1272 ms
>  Average of subsequent rebinds: 96.79 ms
> 
>  First binding for dax_pmem driver: 1354 ms
>  Average of subsequent rebinds: 119.04 ms
> 
> This reduces the average rebind time by about 60.4% for nd_pmem and
> 56.4% for dax_pmem.

Regarding the 60% reduction in time: does this refer to the entire process,
or specifically to the initialization of struct pages in memmap_init_zone_device()?

Thanks.

> 
> As an additional data point, I also ran a smaller set of measurements on
> the same physical x86_64 host with a 100 GB PMEM region created via the
> memmap= kernel command line, configured as fsdax and devdax namespaces
> with map=dev and 2 MiB alignment.
> 
> For brevity, the individual patches keep only the VM results rather than
> including a second set of physical-host measurements throughout the
> series. The physical-host numbers below are included only as
> supplemental evidence that the same optimization also provides a similar
> benefit on a non-virtualized system.
> 
> Test procedure:
> Reconfigure the namespace mode, rebind the nd_pmem or dax_pmem driver
> once, and collect the memmap initialization time from the pr_debug()
> output of memmap_init_zone_device().
> 
> Base (v7.2-rc1):
>  nd_pmem / fsdax: 179 ms
>  dax_pmem / devdax: 264 ms
> 
> With this series applied:
>  nd_pmem / fsdax: 82 ms
>  dax_pmem / devdax: 113 ms
> 
> This reduces the measured rebind time by about 54.2% for nd_pmem and
> 57.2% for dax_pmem on that setup, which is broadly consistent with the
> VM results above.
> 
> As another supplemental data point, I also measured the test_hmm.ko
> module on the same physical x86_64 host, using the test_hmm.ko setup
> from the previous discussion that times ten 64 GB
> memremap_pages()/memunmap_pages() iterations during module insertion[1].
> By default, module insertion initializes two DEVICE_PRIVATE dmirror
> devices, so two avg memremap values are reported; each value is the
> average for one 64 GB chunk.
> 
> This is not the primary target workload of the series, but it exercises
> the same large ZONE_DEVICE memmap initialization path and shows the same
> direction of improvement.
> 
> Base (v7.2-rc1):
>  avg memremap reported during module insertion: 116689362 ns, 116539263 ns
> 
> With this series applied:
>  avg memremap reported during module insertion: 54607108 ns, 54458236 ns
> 
> This corresponds to about a 53.2% reduction based on the mean of the
> reported values, which is again consistent with the pmem bind/rebind
> results above.
> 
> [1] https://lore.kernel.org/all/aiEoByaQdRR3xtM5@nvdebian.thelocal/
> 
> Li Zhe (8):
>  mm: fix stale ZONE_DEVICE refcount comment
>  mm: factor zone-device page init helpers out of
>    __init_zone_device_page
>  mm: add a set_page_section_from_pfn() helper
>  mm: add a template-based fast path for zone-device page init
>  mm: extend the template fast path to zone-device compound tails
>  string: introduce memcpy_nt() helpers
>  x86/string: extend memcpy_flushcache() fixed-size fastpaths
>  mm: use memcpy_nt() in zone-device template copies
> 
> arch/x86/include/asm/string_64.h |  78 ++++++++++++-
> include/linux/mm.h               |  15 ++-
> include/linux/string.h           |  23 ++++
> mm/mm_init.c                     | 186 ++++++++++++++++++++++++++-----
> 4 files changed, 273 insertions(+), 29 deletions(-)
> 
> ---
> v5: https://lore.kernel.org/all/20260701090553.62691-1-lizhe.67@bytedance.com/
> v4: https://lore.kernel.org/all/20260603080152.64728-1-lizhe.67@bytedance.com/
> v3: https://lore.kernel.org/all/20260527033636.28231-1-lizhe.67@bytedance.com/
> v2: https://lore.kernel.org/all/20260521040124.10608-1-lizhe.67@bytedance.com/
> v1: https://lore.kernel.org/all/20260515082045.63029-1-lizhe.67@bytedance.com/
> 
> Changelogs:
> 
> v5->v6:
> - Rework patch 6 so the x86 memcpy_nt_drain() helper uses wmb()
>  instead of an open-coded sfence, while architectures that do not
>  override memcpy_nt() keep the generic no-op drain fallback.
>  Suggested by Borislav Petkov.
> - Rework patch 6 to use the usual self-macro override pattern for
>  memcpy_nt() and memcpy_nt_drain() instead of a dedicated
>  __HAVE_ARCH_MEMCPY_NT feature macro. Suggested by David
>  Hildenbrand.
> - Drop the default: case from pagemap_resets_refcount() and drop the
>  WARN_ONCE() after the switch so newly added enum memory_type values
>  remain visible to compiler switch checking. Suggested by David
>  Hildenbrand.
> - Drop the unnecessary empty set_page_section() stub from patch 3, keep
>  set_page_section_from_pfn() as the only !SECTION_IN_PAGE_FLAGS no-op
>  helper and fix its indentation. Suggested by David Hildenbrand.
> - Rework patch 4 so the reusable head-page template is seeded from the
>  first loop iteration, keeping the main loop intact while ensuring the
>  first page is initialized only once; also drop the extra template
>  memcpy helper, as suggested by Alistair Popple.
> - Rework patch 5 similarly for compound tails: seed the reusable tail
>  template from the first tail-page iteration so the first tail page is
>  initialized only once, while keeping the main tail loop intact, as
>  suggested by Alistair Popple.
> - Rework patch 7 to rename the internal fixed-size MOVNTI helpers to
>  movnti_*(), keep memcpy_flushcache() as the externally visible
>  wrapper, fold the new 32/48/64/80/96-byte copies into the main
>  fixed-size switch, clarify in the code and changelog that the
>  ZONE_DEVICE template-copy path relies on the 64/80/96-byte struct
>  page copies while 32/48-byte copies stay inline as well, and drop the
>  separate alignment check based on the physical-host microbenchmark
>  results. Suggested by Borislav Petkov.
> - Run v6 VM spot checks for the fsdax/devdax map=dev paths. The
>  results stayed in the same ballpark as v5, so keep the previously
>  posted v5 full-run numbers in this version instead of replacing them
>  with another partial data set.
> 
> For changelogs of earlier revisions, please refer to the v5 cover letter.
> 
> --
> 2.20.1

Re: [PATCH v6 4/8] mm: add a template-based fast path for zone-device page init

[Muchun Song]

On 2026/7/9 19:25, Li Zhe wrote:
> memmap_init_zone_device() repeats nearly identical head-page
> initialization for each PFN. Prepare one reusable ZONE_DEVICE head-page
> template through the existing slow path, refresh the PFN-dependent
> fields in that template before each copy, and memcpy it into each
> destination page.
>
> The optimized path assigns _refcount through the copied template, so
> keep it disabled when the page_ref_set tracepoint is enabled.
>
> This patch accelerates head-page initialization. The pfns_per_compound
> == 1 case gets the full benefit here, compound tails are handled in the
> next patch.
>
> Tested in a VM with a 100 GB fsdax namespace device configured with
> map=dev on Intel Ice Lake server. This test exercises the nd_pmem rebind
> path (pfns_per_compound == 1).
>
> Test procedure:
> Rebind the nd_pmem driver 30 times and collect the memmap initialization
> time from the pr_debug() output of memmap_init_zone_device().
>
> Base(v7.2-rc1):
>    First binding: 1456 ms
>    Average of subsequent rebinds: 244.28 ms
>
> With this patch and its prerequisites applied:
>    First binding: 1440 ms
>    Average of subsequent rebinds: 217.19 ms
>
> This reduces the average rebind time from 244.28 ms to 217.19 ms, or
> about 11%.

An 11% reduction is definitely a solid achievement. While the
absolute gain is in the range of a few dozen milliseconds, I’d
love to better understand the compounding impact. In which
specific latency-sensitive scenarios or workloads does saving
these few dozen milliseconds translate into a major breakthrough
for the user experience?

Muchun,
Thanks.

>
> Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
> ---
>   mm/mm_init.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 58 insertions(+), 1 deletion(-)
>
> diff --git a/mm/mm_init.c b/mm/mm_init.c
> index 5fccfbacf855..170021e182e0 100644
> --- a/mm/mm_init.c
> +++ b/mm/mm_init.c
> @@ -1065,6 +1065,44 @@ static void __ref zone_device_page_init_slow(struct page *page,
>   		set_page_count(page, 0);
>   }
>   
> +static inline bool zone_device_page_init_optimization_enabled(void)
> +{
> +	/*
> +	 * The template fast path copies a preinitialized struct page image.
> +	 * Skip it when the page_ref_set tracepoint is enabled.
> +	 */
> +	return !page_ref_tracepoint_active(page_ref_set);
> +}
> +
> +/*
> + * 'template' is a reusable page prototype rather than a strictly immutable
> + * object. Most ZONE_DEVICE fields stay constant across the pages covered by
> + * the current template, but section bits and page->virtual may still depend
> + * on the PFN. Refresh those PFN-dependent fields in the template before
> + * copying it into @page.
> + */
> +static inline void zone_device_page_update_template(struct page *template,
> +		unsigned long pfn)
> +{
> +	set_page_section_from_pfn(template, pfn);
> +#ifdef WANT_PAGE_VIRTUAL
> +	if (!is_highmem_idx(ZONE_DEVICE))
> +		set_page_address(template, __va(pfn << PAGE_SHIFT));
> +#endif
> +}
> +
> +static void zone_device_page_init_from_template(struct page *page,
> +		unsigned long pfn, struct page *template)
> +{
> +	/*
> +	 * 'template' carries the invariant portion of a ZONE_DEVICE struct
> +	 * page. Update the PFN-dependent fields in place before copying it
> +	 * to the destination page.
> +	 */
> +	zone_device_page_update_template(template, pfn);
> +	memcpy(page, template, sizeof(*page));
> +}
> +
>   /*
>    * With compound page geometry and when struct pages are stored in ram most
>    * tail pages are reused. Consequently, the amount of unique struct pages to
> @@ -1120,6 +1158,7 @@ void __ref memmap_init_zone_device(struct zone *zone,
>   				   unsigned long nr_pages,
>   				   struct dev_pagemap *pgmap)
>   {
> +	bool use_template = zone_device_page_init_optimization_enabled();
>   	unsigned long pfn, end_pfn = start_pfn + nr_pages;
>   	struct pglist_data *pgdat = zone->zone_pgdat;
>   	struct vmem_altmap *altmap = pgmap_altmap(pgmap);
> @@ -1127,6 +1166,7 @@ void __ref memmap_init_zone_device(struct zone *zone,
>   	unsigned long zone_idx = zone_idx(zone);
>   	unsigned long start = jiffies;
>   	int nid = pgdat->node_id;
> +	struct page template;
>   
>   	if (WARN_ON_ONCE(!pgmap || zone_idx != ZONE_DEVICE))
>   		return;
> @@ -1144,7 +1184,24 @@ void __ref memmap_init_zone_device(struct zone *zone,
>   	for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) {
>   		struct page *page = pfn_to_page(pfn);
>   
> -		zone_device_page_init_slow(page, pfn, zone_idx, nid, pgmap);
> +		if (!use_template) {
> +			zone_device_page_init_slow(page, pfn, zone_idx,
> +						   nid, pgmap);
> +		} else if (pfn == start_pfn) {
> +			/*
> +			 * Seed the reusable head-page template from the
> +			 * first real struct page, because the existing
> +			 * page-init and pageblock helpers expect a real
> +			 * memmap entry rather than a stack object.
> +			 */
> +			zone_device_page_init_slow(page, pfn, zone_idx,
> +						   nid, pgmap);
> +			/* init template page */
> +			memcpy(&template, page, sizeof(*page));
> +		} else {
> +			zone_device_page_init_from_template(page, pfn,
> +							    &template);
> +		}
>
>   		if (IS_ALIGNED(pfn, PAGES_PER_SECTION))
>   			cond_resched();
> --
> 2.20.1