[PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()

[PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()

[Shivank Garg]

folio_mc_copy() copies a folio one page at a time with copy_mc_highpage(),
so a 2MB folio makes 512 separate copy_mc_to_kernel() calls. Each page
pays the startup cost of the string instruction ("REP; MOVSB" on x86
CPUs with ERMS) plus a cond_resched().

Introduce copy_mc_highpages(), the page range variant of
copy_mc_highpage(). When copy_mc_to_kernel() is available and the pages
are in the linear map, copy the whole range with a single
copy_mc_to_kernel() call, so the CPU can optimize for the larger extent.
Otherwise, fall back to copy_mc_highpage() one page at a time.

On a machine check, queue the poisoned source page for memory_failure(),
as copy_mc_highpage() does, and return -EHWPOISON, as folio_mc_copy()
does today.

copy_mc_to_kernel() cannot reschedule, so only take the single-call path
when CONFIG_PREEMPTION is set and the copy can be preempted. Since commit
7dadeaa6e851 ("sched: Further restrict the preemption modes"), x86 and
powerpc, the only architectures that select ARCH_HAS_COPY_MC, only offer
the full, lazy and RT preemption models, also with PREEMPT_DYNAMIC, so
they always take this path. The per-page fallback calls cond_resched()
before each page, as clear_contig_highpages() does. Other architectures
keep copying one page at a time.

Switch folio_mc_copy() to copy the whole folio with copy_mc_highpages().

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
On a Zen 6 system with CXL memory, move_pages() gets ~1.4x faster for
2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below).
When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.

Changes in v3:
- Copy the whole range with one copy_mc_to_kernel() call only when
  CONFIG_PREEMPTION is set, otherwise copy page by page with
  cond_resched(), following clear_contig_highpages() [1] (Pedro, David).
- Handle the machine check first in copy_mc_highpages() (Pedro).
- Rebase onto mm/core.git for-next.
- Link to v2: https://lore.kernel.org/r/20261019-batpagecopy-v2-1-d75e7d0dfaae@amd.com

[1] https://lore.kernel.org/r/20261109-clear-pages-batch-v3-1-6693efe18dff@amd.com

Changes in v2:
- Drop the folio_copy() changes as there is little to gain here.
  On architectures without copy_mc_to_kernel(), copy_mc_highpages()
  falls back to the per-page copy_mc_highpage(), which ends up in
  copy_highpage(). A batched copy_highpages() could help there too,
  but I haven't tested it on those architectures, so that is left for
  a follow-up.
- Add COPY_MC_PAGES_NON_PREEMPT_BATCH to limit how much is copied
  between cond_resched() calls on non-preemptible kernels.
- Rebase onto mm-new (2887e63d2e8e).
- v1: https://lore.kernel.org/all/20260427142036.111940-2-shivankg@amd.com

Test setup: AMD EPYC Zen 6, single socket, NPS2 (nodes 0 and 1 are
DRAM, node 2 is CXL), with and without this patch.

move_pages() throughput
-----------------------
A single move_pages() call migrates 1GB of anonymous memory, as 512
2MB THPs or as four 1GB hugetlb pages. Mean ± stddev (10 runs each),
in GiB/s:

2MB THP:
  direction         base            patched         speedup
  DRAM0 -> DRAM1    19.57 ± 0.17    27.51 ± 0.78    1.41x
  DRAM1 -> DRAM0    19.44 ± 0.12    27.33 ± 0.27    1.41x
  DRAM0 -> CXL      18.66 ± 0.18    26.90 ± 0.51    1.44x
  CXL   -> DRAM0    15.38 ± 0.59    16.60 ± 0.24    1.08x

1GB hugetlb:
  direction         base            patched         speedup
  DRAM0 -> DRAM1    20.34 ± 0.13    53.88 ± 1.48    2.65x
  DRAM1 -> DRAM0    19.10 ± 2.65    53.78 ± 0.55    2.82x
  DRAM0 -> CXL      19.53 ± 0.08    44.10 ± 0.19    2.26x
  CXL   -> DRAM0    16.41 ± 0.07    20.89 ± 0.06    1.27x

kswapd demotion
---------------
Node 0 is limited to 16GB of free memory and demotion is enabled. A
single thread bound to node 0 writes 96GB of anonymous THP memory, so
~80GB has to be demoted to CXL. Median of 10 runs:

                               base        patched
  kswapd CPU time              4.25 s      3.44 s     (-19%)
  kswapd CPU per GB demoted    63.6 ms     44.1 ms    (-31%)
  demoted by direct reclaim    14.1 GB     0.46 GB
  allocstall                   7427        370

With the patch, kswapd keeps up with the writer, so the writer rarely
falls back to direct reclaim.
---
 include/linux/highmem.h | 51 +++++++++++++++++++++++++++++++++++++++++++++++++
 mm/util.c               | 14 ++------------
 2 files changed, 53 insertions(+), 12 deletions(-)

diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index d7aac9de1c8a..a79e9d68359c 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -484,6 +484,57 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
 }
 #endif
 
+/**
+ * copy_mc_highpages() - copy a page range with machine check handling
+ * @to: first destination page
+ * @from: first source page
+ * @nr_pages: number of pages
+ *
+ * Range variant of copy_mc_highpage(). On a machine check, the poisoned
+ * source page is queued for memory_failure().
+ *
+ * Context: Process context. May sleep.
+ * Return: 0 on success, -EHWPOISON if a machine check was handled.
+ */
+static inline int copy_mc_highpages(struct page *to, struct page *from,
+		unsigned long nr_pages)
+{
+	unsigned long len = nr_pages << PAGE_SHIFT;
+	unsigned long i, ret;
+
+	might_sleep();
+
+	/*
+	 * When copying we want to operate on the largest extent possible to
+	 * allow for architecture specific extent based optimizations.
+	 *
+	 * copy_mc_to_kernel() does not call cond_resched(). That is fine on
+	 * preemptible kernels, where the scheduler can preempt the copy at
+	 * any point, so copy the whole range at once if copy_mc_to_kernel() is
+	 * available and the pages are in the linear map. Otherwise, copy one
+	 * page at a time and reschedule in between.
+	 */
+	if (IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) && IS_ENABLED(CONFIG_PREEMPTION) &&
+	    !IS_ENABLED(CONFIG_HIGHMEM)) {
+		ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
+		if (ret) {
+			memory_failure_queue(page_to_pfn(from) +
+					     ((len - ret) >> PAGE_SHIFT), 0);
+			return -EHWPOISON;
+		}
+		for (i = 0; i < nr_pages; i++)
+			kmsan_copy_page_meta(to + i, from + i);
+		return 0;
+	}
+
+	for (i = 0; i < nr_pages; i++) {
+		cond_resched();
+		if (copy_mc_highpage(to + i, from + i))
+			return -EHWPOISON;
+	}
+	return 0;
+}
+
 static inline void memcpy_page(struct page *dst_page, size_t dst_off,
 			       struct page *src_page, size_t src_off,
 			       size_t len)
diff --git a/mm/util.c b/mm/util.c
index c9c3dc4b490f..1277daa3ccd6 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -776,18 +776,8 @@ EXPORT_SYMBOL(folio_copy);
 
 int folio_mc_copy(struct folio *dst, struct folio *src)
 {
-	long nr = folio_nr_pages(src);
-	long i = 0;
-
-	for (;;) {
-		if (copy_mc_highpage(folio_page(dst, i), folio_page(src, i)))
-			return -EHWPOISON;
-		if (++i == nr)
-			break;
-		cond_resched();
-	}
-
-	return 0;
+	return copy_mc_highpages(folio_page(dst, 0), folio_page(src, 0),
+				 folio_nr_pages(src));
 }
 EXPORT_SYMBOL(folio_mc_copy);
 

---
base-commit: c6e0cad6c9b4180bb229db06e5b062c5ce0723bd
change-id: 20260714-batpagecopy-5a2b45910e5d

Best regards,
-- 
Shivank Garg <shivankg@amd.com>

Re: [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found no new issues
in this patch, but noticed 1 related pre-existing issue it does not
introduce:
- [Critical] mm: missing hwpoison check in copy_mc_user_highpage() and
  copy_mc_highpage() [1]
--

commit fae95970ccf2607d670f06862e40d3ba9313200a
Author: Shivank Garg <shivankg@amd.com>

mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()

This commit introduces copy_mc_highpages() to copy page ranges with a
single copy_mc_to_kernel() call when possible and updates folio_mc_copy()
to use it.

> diff --git a/include/linux/highmem.h b/include/linux/highmem.h
> index d7aac9de1c8a4..a79e9d68359c8 100644
> --- a/include/linux/highmem.h
> +++ b/include/linux/highmem.h
> @@ -484,6 +484,57 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
>  }
>  #endif
>  
> +/**
> + * copy_mc_highpages() - copy a page range with machine check handling
> + * @to: first destination page
> + * @from: first source page
> + * @nr_pages: number of pages
> + *
> + * Range variant of copy_mc_highpage(). On a machine check, the poisoned
> + * source page is queued for memory_failure().
> + *
> + * Context: Process context. May sleep.
> + * Return: 0 on success, -EHWPOISON if a machine check was handled.
> + */
> +static inline int copy_mc_highpages(struct page *to, struct page *from,
> +		unsigned long nr_pages)
> +{
> +	unsigned long len = nr_pages << PAGE_SHIFT;
> +	unsigned long i, ret;
> +
> +	might_sleep();
> +
> +	/*
> +	 * When copying we want to operate on the largest extent possible to
> +	 * allow for architecture specific extent based optimizations.
> +	 *
> +	 * copy_mc_to_kernel() does not call cond_resched(). That is fine on
> +	 * preemptible kernels, where the scheduler can preempt the copy at
> +	 * any point, so copy the whole range at once if copy_mc_to_kernel() is
> +	 * available and the pages are in the linear map. Otherwise, copy one
> +	 * page at a time and reschedule in between.
> +	 */
> +	if (IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) && IS_ENABLED(CONFIG_PREEMPTION) &&
> +	    !IS_ENABLED(CONFIG_HIGHMEM)) {
> +		ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
> +		if (ret) {
> +			memory_failure_queue(page_to_pfn(from) +
> +					     ((len - ret) >> PAGE_SHIFT), 0);
> +			return -EHWPOISON;
> +		}
> +		for (i = 0; i < nr_pages; i++)
> +			kmsan_copy_page_meta(to + i, from + i);
> +		return 0;
> +	}
> +
> +	for (i = 0; i < nr_pages; i++) {
> +		cond_resched();
> +		if (copy_mc_highpage(to + i, from + i))
> +			return -EHWPOISON;
> +	}

[Severity: Critical]
This problem wasn't introduced by this patch, but can copy_mc_highpage()
and copy_mc_user_highpage() (as well as the new copy_mc_highpages() fast
path) return 0 when the source page or its containing folio is already
marked PG_hwpoison [1]?

Neither helper checks PageHWPoison() or folio_contain_hwpoisoned_page()
before copying.  copy_mc_to_kernel() only catches synchronous #MC
exceptions raised during the copy itself (and falls back to plain memcpy()
when unavailable or on x86 without X86_FEATURE_ERMS or
copy_mc_fragile_enabled).

If a source folio was poisoned via software injection (such as
MADV_HWPOISON) or a prior memory_failure() where reading does not trigger
a synchronous #MC, callers such as folio_mc_copy() or wp_page_copy() treat
the copy as successful and copy corrupted data into a fresh destination
folio without PG_hwpoison set, or trigger an unrecoverable kernel #MC when
copy_mc_to_kernel() is not active.

[1] https://sashiko.dev/#/bug/linux-3af4aa7d-d182-430b-b982-583f6aea4594

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261113-batpagecopy-v3-1-297f5b2fd28e@amd.com?part=1