Linux-mm Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
@ 2026-11-13 17:09 Shivank Garg
  2026-11-14 17:12 ` sashiko-bot
  2026-11-22  7:30 ` Andrew Morton
  0 siblings, 2 replies; 6+ messages in thread
From: Shivank Garg @ 2026-11-13 17:09 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
	Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Borislav Petkov, Ankur Arora,
	Dave Hansen, Thomas Gleixner, Bharata B Rao, Hrushikesh Salunke,
	Ingo Molnar, H . Peter Anvin, David Laight, David Rientjes,
	Linus Torvalds, Teja Vojjala, Huan Nguyen, Pedro Falcato,
	Peter Zijlstra
  Cc: linux-mm, linux-kernel, x86, Shivank Garg

folio_mc_copy() copies a folio one page at a time with copy_mc_highpage(),
so a 2MB folio makes 512 separate copy_mc_to_kernel() calls. Each page
pays the startup cost of the string instruction ("REP; MOVSB" on x86
CPUs with ERMS) plus a cond_resched().

Introduce copy_mc_highpages(), the page range variant of
copy_mc_highpage(). When copy_mc_to_kernel() is available and the pages
are in the linear map, copy the whole range with a single
copy_mc_to_kernel() call, so the CPU can optimize for the larger extent.
Otherwise, fall back to copy_mc_highpage() one page at a time.

On a machine check, queue the poisoned source page for memory_failure(),
as copy_mc_highpage() does, and return -EHWPOISON, as folio_mc_copy()
does today.

copy_mc_to_kernel() cannot reschedule, so only take the single-call path
when CONFIG_PREEMPTION is set and the copy can be preempted. Since commit
7dadeaa6e851 ("sched: Further restrict the preemption modes"), x86 and
powerpc, the only architectures that select ARCH_HAS_COPY_MC, only offer
the full, lazy and RT preemption models, also with PREEMPT_DYNAMIC, so
they always take this path. The per-page fallback calls cond_resched()
before each page, as clear_contig_highpages() does. Other architectures
keep copying one page at a time.

Switch folio_mc_copy() to copy the whole folio with copy_mc_highpages().

Signed-off-by: Shivank Garg <shivankg@amd.com>
---
On a Zen 6 system with CXL memory, move_pages() gets ~1.4x faster for
2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below).
When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.

Changes in v3:
- Copy the whole range with one copy_mc_to_kernel() call only when
  CONFIG_PREEMPTION is set, otherwise copy page by page with
  cond_resched(), following clear_contig_highpages() [1] (Pedro, David).
- Handle the machine check first in copy_mc_highpages() (Pedro).
- Rebase onto mm/core.git for-next.
- Link to v2: https://lore.kernel.org/r/20261019-batpagecopy-v2-1-d75e7d0dfaae@amd.com

[1] https://lore.kernel.org/r/20261109-clear-pages-batch-v3-1-6693efe18dff@amd.com

Changes in v2:
- Drop the folio_copy() changes as there is little to gain here.
  On architectures without copy_mc_to_kernel(), copy_mc_highpages()
  falls back to the per-page copy_mc_highpage(), which ends up in
  copy_highpage(). A batched copy_highpages() could help there too,
  but I haven't tested it on those architectures, so that is left for
  a follow-up.
- Add COPY_MC_PAGES_NON_PREEMPT_BATCH to limit how much is copied
  between cond_resched() calls on non-preemptible kernels.
- Rebase onto mm-new (2887e63d2e8e).
- v1: https://lore.kernel.org/all/20260427142036.111940-2-shivankg@amd.com

Test setup: AMD EPYC Zen 6, single socket, NPS2 (nodes 0 and 1 are
DRAM, node 2 is CXL), with and without this patch.

move_pages() throughput
-----------------------
A single move_pages() call migrates 1GB of anonymous memory, as 512
2MB THPs or as four 1GB hugetlb pages. Mean ± stddev (10 runs each),
in GiB/s:

2MB THP:
  direction         base            patched         speedup
  DRAM0 -> DRAM1    19.57 ± 0.17    27.51 ± 0.78    1.41x
  DRAM1 -> DRAM0    19.44 ± 0.12    27.33 ± 0.27    1.41x
  DRAM0 -> CXL      18.66 ± 0.18    26.90 ± 0.51    1.44x
  CXL   -> DRAM0    15.38 ± 0.59    16.60 ± 0.24    1.08x

1GB hugetlb:
  direction         base            patched         speedup
  DRAM0 -> DRAM1    20.34 ± 0.13    53.88 ± 1.48    2.65x
  DRAM1 -> DRAM0    19.10 ± 2.65    53.78 ± 0.55    2.82x
  DRAM0 -> CXL      19.53 ± 0.08    44.10 ± 0.19    2.26x
  CXL   -> DRAM0    16.41 ± 0.07    20.89 ± 0.06    1.27x

kswapd demotion
---------------
Node 0 is limited to 16GB of free memory and demotion is enabled. A
single thread bound to node 0 writes 96GB of anonymous THP memory, so
~80GB has to be demoted to CXL. Median of 10 runs:

                               base        patched
  kswapd CPU time              4.25 s      3.44 s     (-19%)
  kswapd CPU per GB demoted    63.6 ms     44.1 ms    (-31%)
  demoted by direct reclaim    14.1 GB     0.46 GB
  allocstall                   7427        370

With the patch, kswapd keeps up with the writer, so the writer rarely
falls back to direct reclaim.
---
 include/linux/highmem.h | 51 +++++++++++++++++++++++++++++++++++++++++++++++++
 mm/util.c               | 14 ++------------
 2 files changed, 53 insertions(+), 12 deletions(-)

diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index d7aac9de1c8a..a79e9d68359c 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -484,6 +484,57 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
 }
 #endif
 
+/**
+ * copy_mc_highpages() - copy a page range with machine check handling
+ * @to: first destination page
+ * @from: first source page
+ * @nr_pages: number of pages
+ *
+ * Range variant of copy_mc_highpage(). On a machine check, the poisoned
+ * source page is queued for memory_failure().
+ *
+ * Context: Process context. May sleep.
+ * Return: 0 on success, -EHWPOISON if a machine check was handled.
+ */
+static inline int copy_mc_highpages(struct page *to, struct page *from,
+		unsigned long nr_pages)
+{
+	unsigned long len = nr_pages << PAGE_SHIFT;
+	unsigned long i, ret;
+
+	might_sleep();
+
+	/*
+	 * When copying we want to operate on the largest extent possible to
+	 * allow for architecture specific extent based optimizations.
+	 *
+	 * copy_mc_to_kernel() does not call cond_resched(). That is fine on
+	 * preemptible kernels, where the scheduler can preempt the copy at
+	 * any point, so copy the whole range at once if copy_mc_to_kernel() is
+	 * available and the pages are in the linear map. Otherwise, copy one
+	 * page at a time and reschedule in between.
+	 */
+	if (IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) && IS_ENABLED(CONFIG_PREEMPTION) &&
+	    !IS_ENABLED(CONFIG_HIGHMEM)) {
+		ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
+		if (ret) {
+			memory_failure_queue(page_to_pfn(from) +
+					     ((len - ret) >> PAGE_SHIFT), 0);
+			return -EHWPOISON;
+		}
+		for (i = 0; i < nr_pages; i++)
+			kmsan_copy_page_meta(to + i, from + i);
+		return 0;
+	}
+
+	for (i = 0; i < nr_pages; i++) {
+		cond_resched();
+		if (copy_mc_highpage(to + i, from + i))
+			return -EHWPOISON;
+	}
+	return 0;
+}
+
 static inline void memcpy_page(struct page *dst_page, size_t dst_off,
 			       struct page *src_page, size_t src_off,
 			       size_t len)
diff --git a/mm/util.c b/mm/util.c
index c9c3dc4b490f..1277daa3ccd6 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -776,18 +776,8 @@ EXPORT_SYMBOL(folio_copy);
 
 int folio_mc_copy(struct folio *dst, struct folio *src)
 {
-	long nr = folio_nr_pages(src);
-	long i = 0;
-
-	for (;;) {
-		if (copy_mc_highpage(folio_page(dst, i), folio_page(src, i)))
-			return -EHWPOISON;
-		if (++i == nr)
-			break;
-		cond_resched();
-	}
-
-	return 0;
+	return copy_mc_highpages(folio_page(dst, 0), folio_page(src, 0),
+				 folio_nr_pages(src));
 }
 EXPORT_SYMBOL(folio_mc_copy);
 

---
base-commit: c6e0cad6c9b4180bb229db06e5b062c5ce0723bd
change-id: 20260714-batpagecopy-5a2b45910e5d

Best regards,
-- 
Shivank Garg <shivankg@amd.com>



^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
  2026-11-13 17:09 [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy() Shivank Garg
@ 2026-11-14 17:12 ` sashiko-bot
  2026-11-17  5:27   ` Garg, Shivank
  2026-11-22  7:30 ` Andrew Morton
  1 sibling, 1 reply; 6+ messages in thread
From: sashiko-bot @ 2026-11-14 17:12 UTC (permalink / raw)
  To: Shivank Garg, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
	Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Borislav Petkov, Ankur Arora,
	Dave Hansen, Thomas Gleixner, Bharata B Rao, Hrushikesh Salunke,
	Ingo Molnar, H . Peter Anvin, David Laight, David Rientjes,
	Linus Torvalds, Teja Vojjala, Huan Nguyen, Pedro Falcato,
	Peter Zijlstra
  Cc: linux-mm, x86

Thank you for your contribution! Sashiko AI review found no new issues
in this patch, but noticed 1 related pre-existing issue it does not
introduce:
- [Critical] mm: missing hwpoison check in copy_mc_user_highpage() and
  copy_mc_highpage() [1]
--

commit fae95970ccf2607d670f06862e40d3ba9313200a
Author: Shivank Garg <shivankg@amd.com>

mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()

This commit introduces copy_mc_highpages() to copy page ranges with a
single copy_mc_to_kernel() call when possible and updates folio_mc_copy()
to use it.

> diff --git a/include/linux/highmem.h b/include/linux/highmem.h
> index d7aac9de1c8a4..a79e9d68359c8 100644
> --- a/include/linux/highmem.h
> +++ b/include/linux/highmem.h
> @@ -484,6 +484,57 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
>  }
>  #endif
>  
> +/**
> + * copy_mc_highpages() - copy a page range with machine check handling
> + * @to: first destination page
> + * @from: first source page
> + * @nr_pages: number of pages
> + *
> + * Range variant of copy_mc_highpage(). On a machine check, the poisoned
> + * source page is queued for memory_failure().
> + *
> + * Context: Process context. May sleep.
> + * Return: 0 on success, -EHWPOISON if a machine check was handled.
> + */
> +static inline int copy_mc_highpages(struct page *to, struct page *from,
> +		unsigned long nr_pages)
> +{
> +	unsigned long len = nr_pages << PAGE_SHIFT;
> +	unsigned long i, ret;
> +
> +	might_sleep();
> +
> +	/*
> +	 * When copying we want to operate on the largest extent possible to
> +	 * allow for architecture specific extent based optimizations.
> +	 *
> +	 * copy_mc_to_kernel() does not call cond_resched(). That is fine on
> +	 * preemptible kernels, where the scheduler can preempt the copy at
> +	 * any point, so copy the whole range at once if copy_mc_to_kernel() is
> +	 * available and the pages are in the linear map. Otherwise, copy one
> +	 * page at a time and reschedule in between.
> +	 */
> +	if (IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) && IS_ENABLED(CONFIG_PREEMPTION) &&
> +	    !IS_ENABLED(CONFIG_HIGHMEM)) {
> +		ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
> +		if (ret) {
> +			memory_failure_queue(page_to_pfn(from) +
> +					     ((len - ret) >> PAGE_SHIFT), 0);
> +			return -EHWPOISON;
> +		}
> +		for (i = 0; i < nr_pages; i++)
> +			kmsan_copy_page_meta(to + i, from + i);
> +		return 0;
> +	}
> +
> +	for (i = 0; i < nr_pages; i++) {
> +		cond_resched();
> +		if (copy_mc_highpage(to + i, from + i))
> +			return -EHWPOISON;
> +	}

[Severity: Critical]
This problem wasn't introduced by this patch, but can copy_mc_highpage()
and copy_mc_user_highpage() (as well as the new copy_mc_highpages() fast
path) return 0 when the source page or its containing folio is already
marked PG_hwpoison [1]?

Neither helper checks PageHWPoison() or folio_contain_hwpoisoned_page()
before copying.  copy_mc_to_kernel() only catches synchronous #MC
exceptions raised during the copy itself (and falls back to plain memcpy()
when unavailable or on x86 without X86_FEATURE_ERMS or
copy_mc_fragile_enabled).

If a source folio was poisoned via software injection (such as
MADV_HWPOISON) or a prior memory_failure() where reading does not trigger
a synchronous #MC, callers such as folio_mc_copy() or wp_page_copy() treat
the copy as successful and copy corrupted data into a fresh destination
folio without PG_hwpoison set, or trigger an unrecoverable kernel #MC when
copy_mc_to_kernel() is not active.

[1] https://sashiko.dev/#/bug/linux-3af4aa7d-d182-430b-b982-583f6aea4594

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261113-batpagecopy-v3-1-297f5b2fd28e@amd.com?part=1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
  2026-11-14 17:12 ` sashiko-bot
@ 2026-11-17  5:27   ` Garg, Shivank
  2026-11-19 17:27     ` David Hildenbrand (Arm)
  0 siblings, 1 reply; 6+ messages in thread
From: Garg, Shivank @ 2026-11-17  5:27 UTC (permalink / raw)
  To: tejavojjala@google.com, Rao, Bharata Bhasker, rppt@kernel.org,
	david.laight.linux@gmail.com, ankur.a.arora@oracle.com,
	dave.hansen@linux.intel.com, vbabka@kernel.org,
	ying.huang@linux.alibaba.com, david@kernel.org,
	akpm@linux-foundation.org, mingo@redhat.com, surenb@google.com,
	sashiko-reviews@lists.linux.dev, xiangzao@linux.alibaba.com,
	peterz@infradead.org, nhuhuan@google.com, Salunke, Hrushikesh,
	tglx@kernel.org, liam@infradead.org, hpa@zytor.com,
	mhocko@suse.com, bp@alien8.de, ljs@kernel.org,
	rientjes@google.com, pfalcato@suse.de,
	torvalds@linuxfoundation.org, kaitao.cheng@linux.dev
  Cc: linux-mm@kvack.org, x86@kernel.org

On Sat, 2026-11-14 at 17:12 +0000, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found no new issues
> in this patch, but noticed 1 related pre-existing issue it does not
> introduce:
> - [Critical] mm: missing hwpoison check in copy_mc_user_highpage() and
>   copy_mc_highpage() [1]
> --
> 
> commit fae95970ccf2607d670f06862e40d3ba9313200a
> Author: Shivank Garg <shivankg@amd.com>
> 
> mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
> 
> This commit introduces copy_mc_highpages() to copy page ranges with a
> single copy_mc_to_kernel() call when possible and updates folio_mc_copy()
> to use it.
> 
> > diff --git a/include/linux/highmem.h b/include/linux/highmem.h
> > index d7aac9de1c8a4..a79e9d68359c8 100644
> > --- a/include/linux/highmem.h
> > +++ b/include/linux/highmem.h
> > @@ -484,6 +484,57 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
> >  }
> >  #endif
> >  
> > +/**
> > + * copy_mc_highpages() - copy a page range with machine check handling
> > + * @to: first destination page
> > + * @from: first source page
> > + * @nr_pages: number of pages
> > + *
> > + * Range variant of copy_mc_highpage(). On a machine check, the poisoned
> > + * source page is queued for memory_failure().
> > + *
> > + * Context: Process context. May sleep.
> > + * Return: 0 on success, -EHWPOISON if a machine check was handled.
> > + */
> > +static inline int copy_mc_highpages(struct page *to, struct page *from,
> > +		unsigned long nr_pages)
> > +{
> > +	unsigned long len = nr_pages << PAGE_SHIFT;
> > +	unsigned long i, ret;
> > +
> > +	might_sleep();
> > +
> > +	/*
> > +	 * When copying we want to operate on the largest extent possible to
> > +	 * allow for architecture specific extent based optimizations.
> > +	 *
> > +	 * copy_mc_to_kernel() does not call cond_resched(). That is fine on
> > +	 * preemptible kernels, where the scheduler can preempt the copy at
> > +	 * any point, so copy the whole range at once if copy_mc_to_kernel() is
> > +	 * available and the pages are in the linear map. Otherwise, copy one
> > +	 * page at a time and reschedule in between.
> > +	 */
> > +	if (IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) && IS_ENABLED(CONFIG_PREEMPTION) &&
> > +	    !IS_ENABLED(CONFIG_HIGHMEM)) {
> > +		ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
> > +		if (ret) {
> > +			memory_failure_queue(page_to_pfn(from) +
> > +					     ((len - ret) >> PAGE_SHIFT), 0);
> > +			return -EHWPOISON;
> > +		}
> > +		for (i = 0; i < nr_pages; i++)
> > +			kmsan_copy_page_meta(to + i, from + i);
> > +		return 0;
> > +	}
> > +
> > +	for (i = 0; i < nr_pages; i++) {
> > +		cond_resched();
> > +		if (copy_mc_highpage(to + i, from + i))
> > +			return -EHWPOISON;
> > +	}
> 
> [Severity: Critical]
> This problem wasn't introduced by this patch, but can copy_mc_highpage()
> and copy_mc_user_highpage() (as well as the new copy_mc_highpages() fast
> path) return 0 when the source page or its containing folio is already
> marked PG_hwpoison [1]?


Yes, the patch keeps the existing behavior. PG_hwpoison marked pages are copied without errors but Real poison issue returns non-zero and folio_mc_copy() fails! Should we consider this as blocker?

Yuanhe's patch adds the check to copy !copy_mc branch but copy_mc branch can still copy such a page. [1]

Since it's pre-existing issue, I'd prefer to keep it out of this patch.
For solution, should we consider folio_contain_hwpoisoned_page() check in the folio_mc_copy()? It's racy but best-effort, and would avoid adding
cost of checking each page within large folio.

[1] https://lore.kernel.org/linux-mm/sashiko-outbox-208876@kernel.org/#t

> Neither helper checks PageHWPoison() or folio_contain_hwpoisoned_page()
> before copying.  copy_mc_to_kernel() only catches synchronous #MC
> exceptions raised during the copy itself (and falls back to plain memcpy()
> when unavailable or on x86 without X86_FEATURE_ERMS or
> copy_mc_fragile_enabled).
> 
> If a source folio was poisoned via software injection (such as
> MADV_HWPOISON) or a prior memory_failure() where reading does not trigger
> a synchronous #MC, callers such as folio_mc_copy() or wp_page_copy() treat
> the copy as successful and copy corrupted data into a fresh destination
> folio without PG_hwpoison set, or trigger an unrecoverable kernel #MC when
> copy_mc_to_kernel() is not active.
> 
> [1] https://sashiko.dev/#/bug/linux-3af4aa7d-d182-430b-b982-583f6aea4594

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
  2026-11-17  5:27   ` Garg, Shivank
@ 2026-11-19 17:27     ` David Hildenbrand (Arm)
  0 siblings, 0 replies; 6+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-19 17:27 UTC (permalink / raw)
  To: Garg, Shivank, tejavojjala@google.com, Rao, Bharata Bhasker,
	rppt@kernel.org, david.laight.linux@gmail.com,
	ankur.a.arora@oracle.com, dave.hansen@linux.intel.com,
	vbabka@kernel.org, ying.huang@linux.alibaba.com,
	akpm@linux-foundation.org, mingo@redhat.com, surenb@google.com,
	sashiko-reviews@lists.linux.dev, xiangzao@linux.alibaba.com,
	peterz@infradead.org, nhuhuan@google.com, Salunke, Hrushikesh,
	tglx@kernel.org, liam@infradead.org, hpa@zytor.com,
	mhocko@suse.com, bp@alien8.de, ljs@kernel.org,
	rientjes@google.com, pfalcato@suse.de,
	torvalds@linuxfoundation.org, kaitao.cheng@linux.dev
  Cc: linux-mm@kvack.org, x86@kernel.org

On 11/17/26 06:27, Garg, Shivank wrote:
> On Sat, 2026-11-14 at 17:12 +0000, sashiko-bot@kernel.org wrote:
>> Thank you for your contribution! Sashiko AI review found no new issues
>> in this patch, but noticed 1 related pre-existing issue it does not
>> introduce:
>> - [Critical] mm: missing hwpoison check in copy_mc_user_highpage() and
>>   copy_mc_highpage() [1]
>> --
>>
>> commit fae95970ccf2607d670f06862e40d3ba9313200a
>> Author: Shivank Garg <shivankg@amd.com>
>>
>> mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
>>
>> This commit introduces copy_mc_highpages() to copy page ranges with a
>> single copy_mc_to_kernel() call when possible and updates folio_mc_copy()
>> to use it.
>>
>>> diff --git a/include/linux/highmem.h b/include/linux/highmem.h
>>> index d7aac9de1c8a4..a79e9d68359c8 100644
>>> --- a/include/linux/highmem.h
>>> +++ b/include/linux/highmem.h
>>> @@ -484,6 +484,57 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
>>>  }
>>>  #endif
>>>  
>>> +/**
>>> + * copy_mc_highpages() - copy a page range with machine check handling
>>> + * @to: first destination page
>>> + * @from: first source page
>>> + * @nr_pages: number of pages
>>> + *
>>> + * Range variant of copy_mc_highpage(). On a machine check, the poisoned
>>> + * source page is queued for memory_failure().
>>> + *
>>> + * Context: Process context. May sleep.
>>> + * Return: 0 on success, -EHWPOISON if a machine check was handled.
>>> + */
>>> +static inline int copy_mc_highpages(struct page *to, struct page *from,
>>> +		unsigned long nr_pages)
>>> +{
>>> +	unsigned long len = nr_pages << PAGE_SHIFT;
>>> +	unsigned long i, ret;
>>> +
>>> +	might_sleep();
>>> +
>>> +	/*
>>> +	 * When copying we want to operate on the largest extent possible to
>>> +	 * allow for architecture specific extent based optimizations.
>>> +	 *
>>> +	 * copy_mc_to_kernel() does not call cond_resched(). That is fine on
>>> +	 * preemptible kernels, where the scheduler can preempt the copy at
>>> +	 * any point, so copy the whole range at once if copy_mc_to_kernel() is
>>> +	 * available and the pages are in the linear map. Otherwise, copy one
>>> +	 * page at a time and reschedule in between.
>>> +	 */
>>> +	if (IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) && IS_ENABLED(CONFIG_PREEMPTION) &&
>>> +	    !IS_ENABLED(CONFIG_HIGHMEM)) {
>>> +		ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
>>> +		if (ret) {
>>> +			memory_failure_queue(page_to_pfn(from) +
>>> +					     ((len - ret) >> PAGE_SHIFT), 0);
>>> +			return -EHWPOISON;
>>> +		}
>>> +		for (i = 0; i < nr_pages; i++)
>>> +			kmsan_copy_page_meta(to + i, from + i);
>>> +		return 0;
>>> +	}
>>> +
>>> +	for (i = 0; i < nr_pages; i++) {
>>> +		cond_resched();
>>> +		if (copy_mc_highpage(to + i, from + i))
>>> +			return -EHWPOISON;
>>> +	}
>>
>> [Severity: Critical]
>> This problem wasn't introduced by this patch, but can copy_mc_highpage()
>> and copy_mc_user_highpage() (as well as the new copy_mc_highpages() fast
>> path) return 0 when the source page or its containing folio is already
>> marked PG_hwpoison [1]?
> 
> 
> Yes, the patch keeps the existing behavior. PG_hwpoison marked pages are copied without errors but Real poison issue returns non-zero and folio_mc_copy() fails! Should we consider this as blocker?
> 
> Yuanhe's patch adds the check to copy !copy_mc branch but copy_mc branch can still copy such a page. [1]
> 
> Since it's pre-existing issue, I'd prefer to keep it out of this patch.

Ack.

-- 
Cheers,

David


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
  2026-11-13 17:09 [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy() Shivank Garg
  2026-11-14 17:12 ` sashiko-bot
@ 2026-11-22  7:30 ` Andrew Morton
  2026-11-22 17:13   ` Garg, Shivank
  1 sibling, 1 reply; 6+ messages in thread
From: Andrew Morton @ 2026-11-22  7:30 UTC (permalink / raw)
  To: Shivank Garg
  Cc: David Hildenbrand, Lorenzo Stoakes, Liam R. Howlett,
	Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
	Borislav Petkov, Ankur Arora, Dave Hansen, Thomas Gleixner,
	Bharata B Rao, Hrushikesh Salunke, Ingo Molnar, H . Peter Anvin,
	David Laight, David Rientjes, Linus Torvalds, Teja Vojjala,
	Huan Nguyen, Pedro Falcato, Peter Zijlstra, linux-mm,
	linux-kernel, x86

On Fri, 13 Nov 2026 17:09:21 +0000 Shivank Garg <shivankg@amd.com> wrote:

> folio_mc_copy() copies a folio one page at a time with copy_mc_highpage(),
> so a 2MB folio makes 512 separate copy_mc_to_kernel() calls. Each page
> pays the startup cost of the string instruction ("REP; MOVSB" on x86
> CPUs with ERMS) plus a cond_resched().
> 
> Introduce copy_mc_highpages(), the page range variant of
> copy_mc_highpage(). When copy_mc_to_kernel() is available and the pages
> are in the linear map, copy the whole range with a single
> copy_mc_to_kernel() call, so the CPU can optimize for the larger extent.
> Otherwise, fall back to copy_mc_highpage() one page at a time.
> 
> On a machine check, queue the poisoned source page for memory_failure(),
> as copy_mc_highpage() does, and return -EHWPOISON, as folio_mc_copy()
> does today.
> 
> copy_mc_to_kernel() cannot reschedule, so only take the single-call path
> when CONFIG_PREEMPTION is set and the copy can be preempted. Since commit
> 7dadeaa6e851 ("sched: Further restrict the preemption modes"), x86 and
> powerpc, the only architectures that select ARCH_HAS_COPY_MC, only offer
> the full, lazy and RT preemption models, also with PREEMPT_DYNAMIC, so
> they always take this path. The per-page fallback calls cond_resched()
> before each page, as clear_contig_highpages() does. Other architectures
> keep copying one page at a time.
> 
> Switch folio_mc_copy() to copy the whole folio with copy_mc_highpages().

OK.

> ---
> On a Zen 6 system with CXL memory, move_pages() gets ~1.4x faster for
> 2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below).
> When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.

But that's the most important part!  It's the whole point of the patch!
Yet here it is below the ---, which means "throw this away".  It
should be the first paragraph, in all caps.

<edit edit fixed>

OK, thanks, queued,

nit: does it really need cond_resched()?  That's not appropriate for
current mainline kernels.  I guess "yes", because this patch looks like
something which downstream people will eagerly backport into older
kernels, even though we didn't cc:stable.

And we care about downstream people.  Globally: "Roughly 60% are on 6.x
and 30% are on 5.x (with the remaining ~10% split between 7.x and 4.x
or older)".

I'll take a look at that Sashiko thing.



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
  2026-11-22  7:30 ` Andrew Morton
@ 2026-11-22 17:13   ` Garg, Shivank
  0 siblings, 0 replies; 6+ messages in thread
From: Garg, Shivank @ 2026-11-22 17:13 UTC (permalink / raw)
  To: akpm@linux-foundation.org
  Cc: tejavojjala@google.com, Rao, Bharata Bhasker, rppt@kernel.org,
	david.laight.linux@gmail.com, ankur.a.arora@oracle.com,
	vbabka@kernel.org, dave.hansen@linux.intel.com, david@kernel.org,
	peterz@infradead.org, mingo@redhat.com, surenb@google.com,
	linux-kernel@vger.kernel.org, linux-mm@kvack.org,
	nhuhuan@google.com, Salunke, Hrushikesh, tglx@kernel.org,
	liam@infradead.org, hpa@zytor.com, bp@alien8.de, mhocko@suse.com,
	ljs@kernel.org, rientjes@google.com, pfalcato@suse.de,
	torvalds@linuxfoundation.org, x86@kernel.org

On Sat, 2026-11-21 at 23:30 -0800, Andrew Morton wrote:
> On Fri, 13 Nov 2026 17:09:21 +0000 Shivank Garg <shivankg@amd.com> wrote:
> 
> > folio_mc_copy() copies a folio one page at a time with copy_mc_highpage(),
> > so a 2MB folio makes 512 separate copy_mc_to_kernel() calls. Each page
> > pays the startup cost of the string instruction ("REP; MOVSB" on x86
> > CPUs with ERMS) plus a cond_resched().
> > 
> > Introduce copy_mc_highpages(), the page range variant of
> > copy_mc_highpage(). When copy_mc_to_kernel() is available and the pages
> > are in the linear map, copy the whole range with a single
> > copy_mc_to_kernel() call, so the CPU can optimize for the larger extent.
> > Otherwise, fall back to copy_mc_highpage() one page at a time.
> > 
> > On a machine check, queue the poisoned source page for memory_failure(),
> > as copy_mc_highpage() does, and return -EHWPOISON, as folio_mc_copy()
> > does today.
> > 
> > copy_mc_to_kernel() cannot reschedule, so only take the single-call path
> > when CONFIG_PREEMPTION is set and the copy can be preempted. Since commit
> > 7dadeaa6e851 ("sched: Further restrict the preemption modes"), x86 and
> > powerpc, the only architectures that select ARCH_HAS_COPY_MC, only offer
> > the full, lazy and RT preemption models, also with PREEMPT_DYNAMIC, so
> > they always take this path. The per-page fallback calls cond_resched()
> > before each page, as clear_contig_highpages() does. Other architectures
> > keep copying one page at a time.
> > 
> > Switch folio_mc_copy() to copy the whole folio with copy_mc_highpages().
> 
> OK.
> 
> > ---
> > On a Zen 6 system with CXL memory, move_pages() gets ~1.4x faster for
> > 2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below).
> > When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.
> 
> But that's the most important part!  It's the whole point of the patch!
> Yet here it is below the ---, which means "throw this away".  It
> should be the first paragraph, in all caps.
> 
> <edit edit fixed>

Sure, thanks. I'll keep this in mind.

> 
> OK, thanks, queued,
> 
> nit: does it really need cond_resched()?  That's not appropriate for
> current mainline kernels.  I guess "yes", because this patch looks like
> something which downstream people will eagerly backport into older
> kernels, even though we didn't cc:stable.
> 
> And we care about downstream people.  Globally: "Roughly 60% are on 6.x
> and 30% are on 5.x (with the remaining ~10% split between 7.x and 4.x
> or older)".
> 
> I'll take a look at that Sashiko thing.


Agreed.

For older kernels, where the non-preemptible modes are still
available, the backport should look like v2 in my opinion (calling
cond_resched() every 32MB). Otherwise a 1GB hugetlb copy can run for
a long time without rescheduling. I'll leave this discussion for
backport time :)

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2026-11-22 17:13 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-13 17:09 [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy() Shivank Garg
2026-11-14 17:12 ` sashiko-bot
2026-11-17  5:27   ` Garg, Shivank
2026-11-19 17:27     ` David Hildenbrand (Arm)
2026-11-22  7:30 ` Andrew Morton
2026-11-22 17:13   ` Garg, Shivank

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox