Re: [PATCH v6 0/8] mm: optimize zone-device memmap initialization

[Balbir Singh <balbirs@nvidia.com>]

On Thu, Jul 09, 2026 at 07:25:12PM +0800, Li Zhe wrote:
> memmap_init_zone_device() can take a noticeable amount of time when large
> pmem namespaces are bound or rebound, because it initializes nearly
> identical struct page descriptors one PFN at a time. This series reduces
> that ZONE_DEVICE memmap initialization overhead by reusing prepared
> struct page templates and, on x86, using memcpy_nt() for the template
> copy path.
>

Is the slowness specific to an architecture? Do you have any test
results from any other arch?

> The main target is large fsdax/devdax pmem configurations, where the
> cost of initializing the memmap shows up directly in nd_pmem/dax_pmem
> bind and rebind latency.
> 
> Patches 1-3 are preparatory cleanups and helper extraction. Patches 4-5
> add the template-copy fast path for head pages and compound tails.
> Patches 6-8 introduce memcpy_nt()/memcpy_nt_drain(), extend the x86
> fixed-size memcpy_flushcache() inline cases used by that helper, and
> switch the template-copy path over to memcpy_nt().
> 
> The fast path remains disabled when the page_ref_set tracepoint is
> active, and sanitized builds stay on the slow path so their instrumented
> stores are preserved. Architectures without a specialized memcpy_nt()
> backend continue to fall back to memcpy().
> 
> Tested in a VM with a 100 GB fsdax namespace device configured with
> map=dev and a 100 GB devdax namespace (align=2097152) on Intel Ice Lake
> server.
> 
> Test procedure:
> Rebind the nd_pmem and dax_pmem driver 30 times and collect the memmap
> initialization time from the pr_debug() output of
> memmap_init_zone_device().
> 
> Base(v7.2-rc1):
>   First binding for nd_pmem driver: 1456 ms
>   Average of subsequent rebinds: 244.28 ms
> 
>   First binding for dax_pmem driver: 1462 ms
>   Average of subsequent rebinds: 273.31 ms
> 
> With this series applied:
>   First binding for nd_pmem driver: 1272 ms
>   Average of subsequent rebinds: 96.79 ms
> 
>   First binding for dax_pmem driver: 1354 ms
>   Average of subsequent rebinds: 119.04 ms
> 
> This reduces the average rebind time by about 60.4% for nd_pmem and
> 56.4% for dax_pmem.
> 
> As an additional data point, I also ran a smaller set of measurements on
> the same physical x86_64 host with a 100 GB PMEM region created via the
> memmap= kernel command line, configured as fsdax and devdax namespaces
> with map=dev and 2 MiB alignment.
> 
> For brevity, the individual patches keep only the VM results rather than
> including a second set of physical-host measurements throughout the
> series. The physical-host numbers below are included only as
> supplemental evidence that the same optimization also provides a similar
> benefit on a non-virtualized system.
> 
> Test procedure:
> Reconfigure the namespace mode, rebind the nd_pmem or dax_pmem driver
> once, and collect the memmap initialization time from the pr_debug()
> output of memmap_init_zone_device().
> 
> Base (v7.2-rc1):
>   nd_pmem / fsdax: 179 ms
>   dax_pmem / devdax: 264 ms
> 
> With this series applied:
>   nd_pmem / fsdax: 82 ms
>   dax_pmem / devdax: 113 ms
> 
> This reduces the measured rebind time by about 54.2% for nd_pmem and
> 57.2% for dax_pmem on that setup, which is broadly consistent with the
> VM results above.
> 
> As another supplemental data point, I also measured the test_hmm.ko
> module on the same physical x86_64 host, using the test_hmm.ko setup
> from the previous discussion that times ten 64 GB
> memremap_pages()/memunmap_pages() iterations during module insertion[1].
> By default, module insertion initializes two DEVICE_PRIVATE dmirror
> devices, so two avg memremap values are reported; each value is the
> average for one 64 GB chunk.
> 
> This is not the primary target workload of the series, but it exercises
> the same large ZONE_DEVICE memmap initialization path and shows the same
> direction of improvement.
> 
> Base (v7.2-rc1):
>   avg memremap reported during module insertion: 116689362 ns, 116539263 ns
> 
> With this series applied:
>   avg memremap reported during module insertion: 54607108 ns, 54458236 ns
> 
> This corresponds to about a 53.2% reduction based on the mean of the
> reported values, which is again consistent with the pmem bind/rebind
> results above.
>

That is a good result, thanks for sharing

> [1] https://lore.kernel.org/all/aiEoByaQdRR3xtM5@nvdebian.thelocal/
> 
> Li Zhe (8):
>   mm: fix stale ZONE_DEVICE refcount comment
>   mm: factor zone-device page init helpers out of
>     __init_zone_device_page
>   mm: add a set_page_section_from_pfn() helper
>   mm: add a template-based fast path for zone-device page init
>   mm: extend the template fast path to zone-device compound tails
>   string: introduce memcpy_nt() helpers
>   x86/string: extend memcpy_flushcache() fixed-size fastpaths
>   mm: use memcpy_nt() in zone-device template copies
> 
>  arch/x86/include/asm/string_64.h |  78 ++++++++++++-
>  include/linux/mm.h               |  15 ++-
>  include/linux/string.h           |  23 ++++
>  mm/mm_init.c                     | 186 ++++++++++++++++++++++++++-----
>  4 files changed, 273 insertions(+), 29 deletions(-)
> 
> ---
> v5: https://lore.kernel.org/all/20260701090553.62691-1-lizhe.67@bytedance.com/
> v4: https://lore.kernel.org/all/20260603080152.64728-1-lizhe.67@bytedance.com/
> v3: https://lore.kernel.org/all/20260527033636.28231-1-lizhe.67@bytedance.com/
> v2: https://lore.kernel.org/all/20260521040124.10608-1-lizhe.67@bytedance.com/
> v1: https://lore.kernel.org/all/20260515082045.63029-1-lizhe.67@bytedance.com/
> 
> Changelogs:
> 
> v5->v6:
> - Rework patch 6 so the x86 memcpy_nt_drain() helper uses wmb()
>   instead of an open-coded sfence, while architectures that do not
>   override memcpy_nt() keep the generic no-op drain fallback.
>   Suggested by Borislav Petkov.
> - Rework patch 6 to use the usual self-macro override pattern for
>   memcpy_nt() and memcpy_nt_drain() instead of a dedicated
>   __HAVE_ARCH_MEMCPY_NT feature macro. Suggested by David
>   Hildenbrand.
> - Drop the default: case from pagemap_resets_refcount() and drop the
>   WARN_ONCE() after the switch so newly added enum memory_type values
>   remain visible to compiler switch checking. Suggested by David
>   Hildenbrand.
> - Drop the unnecessary empty set_page_section() stub from patch 3, keep
>   set_page_section_from_pfn() as the only !SECTION_IN_PAGE_FLAGS no-op
>   helper and fix its indentation. Suggested by David Hildenbrand.
> - Rework patch 4 so the reusable head-page template is seeded from the
>   first loop iteration, keeping the main loop intact while ensuring the
>   first page is initialized only once; also drop the extra template
>   memcpy helper, as suggested by Alistair Popple.
> - Rework patch 5 similarly for compound tails: seed the reusable tail
>   template from the first tail-page iteration so the first tail page is
>   initialized only once, while keeping the main tail loop intact, as
>   suggested by Alistair Popple.
> - Rework patch 7 to rename the internal fixed-size MOVNTI helpers to
>   movnti_*(), keep memcpy_flushcache() as the externally visible
>   wrapper, fold the new 32/48/64/80/96-byte copies into the main
>   fixed-size switch, clarify in the code and changelog that the
>   ZONE_DEVICE template-copy path relies on the 64/80/96-byte struct
>   page copies while 32/48-byte copies stay inline as well, and drop the
>   separate alignment check based on the physical-host microbenchmark
>   results. Suggested by Borislav Petkov.
> - Run v6 VM spot checks for the fsdax/devdax map=dev paths. The
>   results stayed in the same ballpark as v5, so keep the previously
>   posted v5 full-run numbers in this version instead of replacing them
>   with another partial data set.
> 
> For changelogs of earlier revisions, please refer to the v5 cover letter.
> 
> -- 
> 2.20.1

Balbir