* [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
@ 2026-11-13 17:09 Shivank Garg
2026-11-14 17:12 ` sashiko-bot
2026-11-22 7:30 ` Andrew Morton
0 siblings, 2 replies; 6+ messages in thread
From: Shivank Garg @ 2026-11-13 17:09 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Borislav Petkov, Ankur Arora,
Dave Hansen, Thomas Gleixner, Bharata B Rao, Hrushikesh Salunke,
Ingo Molnar, H . Peter Anvin, David Laight, David Rientjes,
Linus Torvalds, Teja Vojjala, Huan Nguyen, Pedro Falcato,
Peter Zijlstra
Cc: linux-mm, linux-kernel, x86, Shivank Garg
folio_mc_copy() copies a folio one page at a time with copy_mc_highpage(),
so a 2MB folio makes 512 separate copy_mc_to_kernel() calls. Each page
pays the startup cost of the string instruction ("REP; MOVSB" on x86
CPUs with ERMS) plus a cond_resched().
Introduce copy_mc_highpages(), the page range variant of
copy_mc_highpage(). When copy_mc_to_kernel() is available and the pages
are in the linear map, copy the whole range with a single
copy_mc_to_kernel() call, so the CPU can optimize for the larger extent.
Otherwise, fall back to copy_mc_highpage() one page at a time.
On a machine check, queue the poisoned source page for memory_failure(),
as copy_mc_highpage() does, and return -EHWPOISON, as folio_mc_copy()
does today.
copy_mc_to_kernel() cannot reschedule, so only take the single-call path
when CONFIG_PREEMPTION is set and the copy can be preempted. Since commit
7dadeaa6e851 ("sched: Further restrict the preemption modes"), x86 and
powerpc, the only architectures that select ARCH_HAS_COPY_MC, only offer
the full, lazy and RT preemption models, also with PREEMPT_DYNAMIC, so
they always take this path. The per-page fallback calls cond_resched()
before each page, as clear_contig_highpages() does. Other architectures
keep copying one page at a time.
Switch folio_mc_copy() to copy the whole folio with copy_mc_highpages().
Signed-off-by: Shivank Garg <shivankg@amd.com>
---
On a Zen 6 system with CXL memory, move_pages() gets ~1.4x faster for
2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below).
When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.
Changes in v3:
- Copy the whole range with one copy_mc_to_kernel() call only when
CONFIG_PREEMPTION is set, otherwise copy page by page with
cond_resched(), following clear_contig_highpages() [1] (Pedro, David).
- Handle the machine check first in copy_mc_highpages() (Pedro).
- Rebase onto mm/core.git for-next.
- Link to v2: https://lore.kernel.org/r/20261019-batpagecopy-v2-1-d75e7d0dfaae@amd.com
[1] https://lore.kernel.org/r/20261109-clear-pages-batch-v3-1-6693efe18dff@amd.com
Changes in v2:
- Drop the folio_copy() changes as there is little to gain here.
On architectures without copy_mc_to_kernel(), copy_mc_highpages()
falls back to the per-page copy_mc_highpage(), which ends up in
copy_highpage(). A batched copy_highpages() could help there too,
but I haven't tested it on those architectures, so that is left for
a follow-up.
- Add COPY_MC_PAGES_NON_PREEMPT_BATCH to limit how much is copied
between cond_resched() calls on non-preemptible kernels.
- Rebase onto mm-new (2887e63d2e8e).
- v1: https://lore.kernel.org/all/20260427142036.111940-2-shivankg@amd.com
Test setup: AMD EPYC Zen 6, single socket, NPS2 (nodes 0 and 1 are
DRAM, node 2 is CXL), with and without this patch.
move_pages() throughput
-----------------------
A single move_pages() call migrates 1GB of anonymous memory, as 512
2MB THPs or as four 1GB hugetlb pages. Mean ± stddev (10 runs each),
in GiB/s:
2MB THP:
direction base patched speedup
DRAM0 -> DRAM1 19.57 ± 0.17 27.51 ± 0.78 1.41x
DRAM1 -> DRAM0 19.44 ± 0.12 27.33 ± 0.27 1.41x
DRAM0 -> CXL 18.66 ± 0.18 26.90 ± 0.51 1.44x
CXL -> DRAM0 15.38 ± 0.59 16.60 ± 0.24 1.08x
1GB hugetlb:
direction base patched speedup
DRAM0 -> DRAM1 20.34 ± 0.13 53.88 ± 1.48 2.65x
DRAM1 -> DRAM0 19.10 ± 2.65 53.78 ± 0.55 2.82x
DRAM0 -> CXL 19.53 ± 0.08 44.10 ± 0.19 2.26x
CXL -> DRAM0 16.41 ± 0.07 20.89 ± 0.06 1.27x
kswapd demotion
---------------
Node 0 is limited to 16GB of free memory and demotion is enabled. A
single thread bound to node 0 writes 96GB of anonymous THP memory, so
~80GB has to be demoted to CXL. Median of 10 runs:
base patched
kswapd CPU time 4.25 s 3.44 s (-19%)
kswapd CPU per GB demoted 63.6 ms 44.1 ms (-31%)
demoted by direct reclaim 14.1 GB 0.46 GB
allocstall 7427 370
With the patch, kswapd keeps up with the writer, so the writer rarely
falls back to direct reclaim.
---
include/linux/highmem.h | 51 +++++++++++++++++++++++++++++++++++++++++++++++++
mm/util.c | 14 ++------------
2 files changed, 53 insertions(+), 12 deletions(-)
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index d7aac9de1c8a..a79e9d68359c 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -484,6 +484,57 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
}
#endif
+/**
+ * copy_mc_highpages() - copy a page range with machine check handling
+ * @to: first destination page
+ * @from: first source page
+ * @nr_pages: number of pages
+ *
+ * Range variant of copy_mc_highpage(). On a machine check, the poisoned
+ * source page is queued for memory_failure().
+ *
+ * Context: Process context. May sleep.
+ * Return: 0 on success, -EHWPOISON if a machine check was handled.
+ */
+static inline int copy_mc_highpages(struct page *to, struct page *from,
+ unsigned long nr_pages)
+{
+ unsigned long len = nr_pages << PAGE_SHIFT;
+ unsigned long i, ret;
+
+ might_sleep();
+
+ /*
+ * When copying we want to operate on the largest extent possible to
+ * allow for architecture specific extent based optimizations.
+ *
+ * copy_mc_to_kernel() does not call cond_resched(). That is fine on
+ * preemptible kernels, where the scheduler can preempt the copy at
+ * any point, so copy the whole range at once if copy_mc_to_kernel() is
+ * available and the pages are in the linear map. Otherwise, copy one
+ * page at a time and reschedule in between.
+ */
+ if (IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) && IS_ENABLED(CONFIG_PREEMPTION) &&
+ !IS_ENABLED(CONFIG_HIGHMEM)) {
+ ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
+ if (ret) {
+ memory_failure_queue(page_to_pfn(from) +
+ ((len - ret) >> PAGE_SHIFT), 0);
+ return -EHWPOISON;
+ }
+ for (i = 0; i < nr_pages; i++)
+ kmsan_copy_page_meta(to + i, from + i);
+ return 0;
+ }
+
+ for (i = 0; i < nr_pages; i++) {
+ cond_resched();
+ if (copy_mc_highpage(to + i, from + i))
+ return -EHWPOISON;
+ }
+ return 0;
+}
+
static inline void memcpy_page(struct page *dst_page, size_t dst_off,
struct page *src_page, size_t src_off,
size_t len)
diff --git a/mm/util.c b/mm/util.c
index c9c3dc4b490f..1277daa3ccd6 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -776,18 +776,8 @@ EXPORT_SYMBOL(folio_copy);
int folio_mc_copy(struct folio *dst, struct folio *src)
{
- long nr = folio_nr_pages(src);
- long i = 0;
-
- for (;;) {
- if (copy_mc_highpage(folio_page(dst, i), folio_page(src, i)))
- return -EHWPOISON;
- if (++i == nr)
- break;
- cond_resched();
- }
-
- return 0;
+ return copy_mc_highpages(folio_page(dst, 0), folio_page(src, 0),
+ folio_nr_pages(src));
}
EXPORT_SYMBOL(folio_mc_copy);
---
base-commit: c6e0cad6c9b4180bb229db06e5b062c5ce0723bd
change-id: 20260714-batpagecopy-5a2b45910e5d
Best regards,
--
Shivank Garg <shivankg@amd.com>
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
2026-11-13 17:09 [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy() Shivank Garg
@ 2026-11-14 17:12 ` sashiko-bot
2026-11-17 5:27 ` Garg, Shivank
2026-11-22 7:30 ` Andrew Morton
1 sibling, 1 reply; 6+ messages in thread
From: sashiko-bot @ 2026-11-14 17:12 UTC (permalink / raw)
To: Shivank Garg, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Borislav Petkov, Ankur Arora,
Dave Hansen, Thomas Gleixner, Bharata B Rao, Hrushikesh Salunke,
Ingo Molnar, H . Peter Anvin, David Laight, David Rientjes,
Linus Torvalds, Teja Vojjala, Huan Nguyen, Pedro Falcato,
Peter Zijlstra
Cc: linux-mm, x86
Thank you for your contribution! Sashiko AI review found no new issues
in this patch, but noticed 1 related pre-existing issue it does not
introduce:
- [Critical] mm: missing hwpoison check in copy_mc_user_highpage() and
copy_mc_highpage() [1]
--
commit fae95970ccf2607d670f06862e40d3ba9313200a
Author: Shivank Garg <shivankg@amd.com>
mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
This commit introduces copy_mc_highpages() to copy page ranges with a
single copy_mc_to_kernel() call when possible and updates folio_mc_copy()
to use it.
> diff --git a/include/linux/highmem.h b/include/linux/highmem.h
> index d7aac9de1c8a4..a79e9d68359c8 100644
> --- a/include/linux/highmem.h
> +++ b/include/linux/highmem.h
> @@ -484,6 +484,57 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
> }
> #endif
>
> +/**
> + * copy_mc_highpages() - copy a page range with machine check handling
> + * @to: first destination page
> + * @from: first source page
> + * @nr_pages: number of pages
> + *
> + * Range variant of copy_mc_highpage(). On a machine check, the poisoned
> + * source page is queued for memory_failure().
> + *
> + * Context: Process context. May sleep.
> + * Return: 0 on success, -EHWPOISON if a machine check was handled.
> + */
> +static inline int copy_mc_highpages(struct page *to, struct page *from,
> + unsigned long nr_pages)
> +{
> + unsigned long len = nr_pages << PAGE_SHIFT;
> + unsigned long i, ret;
> +
> + might_sleep();
> +
> + /*
> + * When copying we want to operate on the largest extent possible to
> + * allow for architecture specific extent based optimizations.
> + *
> + * copy_mc_to_kernel() does not call cond_resched(). That is fine on
> + * preemptible kernels, where the scheduler can preempt the copy at
> + * any point, so copy the whole range at once if copy_mc_to_kernel() is
> + * available and the pages are in the linear map. Otherwise, copy one
> + * page at a time and reschedule in between.
> + */
> + if (IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) && IS_ENABLED(CONFIG_PREEMPTION) &&
> + !IS_ENABLED(CONFIG_HIGHMEM)) {
> + ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
> + if (ret) {
> + memory_failure_queue(page_to_pfn(from) +
> + ((len - ret) >> PAGE_SHIFT), 0);
> + return -EHWPOISON;
> + }
> + for (i = 0; i < nr_pages; i++)
> + kmsan_copy_page_meta(to + i, from + i);
> + return 0;
> + }
> +
> + for (i = 0; i < nr_pages; i++) {
> + cond_resched();
> + if (copy_mc_highpage(to + i, from + i))
> + return -EHWPOISON;
> + }
[Severity: Critical]
This problem wasn't introduced by this patch, but can copy_mc_highpage()
and copy_mc_user_highpage() (as well as the new copy_mc_highpages() fast
path) return 0 when the source page or its containing folio is already
marked PG_hwpoison [1]?
Neither helper checks PageHWPoison() or folio_contain_hwpoisoned_page()
before copying. copy_mc_to_kernel() only catches synchronous #MC
exceptions raised during the copy itself (and falls back to plain memcpy()
when unavailable or on x86 without X86_FEATURE_ERMS or
copy_mc_fragile_enabled).
If a source folio was poisoned via software injection (such as
MADV_HWPOISON) or a prior memory_failure() where reading does not trigger
a synchronous #MC, callers such as folio_mc_copy() or wp_page_copy() treat
the copy as successful and copy corrupted data into a fresh destination
folio without PG_hwpoison set, or trigger an unrecoverable kernel #MC when
copy_mc_to_kernel() is not active.
[1] https://sashiko.dev/#/bug/linux-3af4aa7d-d182-430b-b982-583f6aea4594
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261113-batpagecopy-v3-1-297f5b2fd28e@amd.com?part=1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
2026-11-14 17:12 ` sashiko-bot
@ 2026-11-17 5:27 ` Garg, Shivank
2026-11-19 17:27 ` David Hildenbrand (Arm)
0 siblings, 1 reply; 6+ messages in thread
From: Garg, Shivank @ 2026-11-17 5:27 UTC (permalink / raw)
To: tejavojjala@google.com, Rao, Bharata Bhasker, rppt@kernel.org,
david.laight.linux@gmail.com, ankur.a.arora@oracle.com,
dave.hansen@linux.intel.com, vbabka@kernel.org,
ying.huang@linux.alibaba.com, david@kernel.org,
akpm@linux-foundation.org, mingo@redhat.com, surenb@google.com,
sashiko-reviews@lists.linux.dev, xiangzao@linux.alibaba.com,
peterz@infradead.org, nhuhuan@google.com, Salunke, Hrushikesh,
tglx@kernel.org, liam@infradead.org, hpa@zytor.com,
mhocko@suse.com, bp@alien8.de, ljs@kernel.org,
rientjes@google.com, pfalcato@suse.de,
torvalds@linuxfoundation.org, kaitao.cheng@linux.dev
Cc: linux-mm@kvack.org, x86@kernel.org
On Sat, 2026-11-14 at 17:12 +0000, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found no new issues
> in this patch, but noticed 1 related pre-existing issue it does not
> introduce:
> - [Critical] mm: missing hwpoison check in copy_mc_user_highpage() and
> copy_mc_highpage() [1]
> --
>
> commit fae95970ccf2607d670f06862e40d3ba9313200a
> Author: Shivank Garg <shivankg@amd.com>
>
> mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
>
> This commit introduces copy_mc_highpages() to copy page ranges with a
> single copy_mc_to_kernel() call when possible and updates folio_mc_copy()
> to use it.
>
> > diff --git a/include/linux/highmem.h b/include/linux/highmem.h
> > index d7aac9de1c8a4..a79e9d68359c8 100644
> > --- a/include/linux/highmem.h
> > +++ b/include/linux/highmem.h
> > @@ -484,6 +484,57 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
> > }
> > #endif
> >
> > +/**
> > + * copy_mc_highpages() - copy a page range with machine check handling
> > + * @to: first destination page
> > + * @from: first source page
> > + * @nr_pages: number of pages
> > + *
> > + * Range variant of copy_mc_highpage(). On a machine check, the poisoned
> > + * source page is queued for memory_failure().
> > + *
> > + * Context: Process context. May sleep.
> > + * Return: 0 on success, -EHWPOISON if a machine check was handled.
> > + */
> > +static inline int copy_mc_highpages(struct page *to, struct page *from,
> > + unsigned long nr_pages)
> > +{
> > + unsigned long len = nr_pages << PAGE_SHIFT;
> > + unsigned long i, ret;
> > +
> > + might_sleep();
> > +
> > + /*
> > + * When copying we want to operate on the largest extent possible to
> > + * allow for architecture specific extent based optimizations.
> > + *
> > + * copy_mc_to_kernel() does not call cond_resched(). That is fine on
> > + * preemptible kernels, where the scheduler can preempt the copy at
> > + * any point, so copy the whole range at once if copy_mc_to_kernel() is
> > + * available and the pages are in the linear map. Otherwise, copy one
> > + * page at a time and reschedule in between.
> > + */
> > + if (IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) && IS_ENABLED(CONFIG_PREEMPTION) &&
> > + !IS_ENABLED(CONFIG_HIGHMEM)) {
> > + ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
> > + if (ret) {
> > + memory_failure_queue(page_to_pfn(from) +
> > + ((len - ret) >> PAGE_SHIFT), 0);
> > + return -EHWPOISON;
> > + }
> > + for (i = 0; i < nr_pages; i++)
> > + kmsan_copy_page_meta(to + i, from + i);
> > + return 0;
> > + }
> > +
> > + for (i = 0; i < nr_pages; i++) {
> > + cond_resched();
> > + if (copy_mc_highpage(to + i, from + i))
> > + return -EHWPOISON;
> > + }
>
> [Severity: Critical]
> This problem wasn't introduced by this patch, but can copy_mc_highpage()
> and copy_mc_user_highpage() (as well as the new copy_mc_highpages() fast
> path) return 0 when the source page or its containing folio is already
> marked PG_hwpoison [1]?
Yes, the patch keeps the existing behavior. PG_hwpoison marked pages are copied without errors but Real poison issue returns non-zero and folio_mc_copy() fails! Should we consider this as blocker?
Yuanhe's patch adds the check to copy !copy_mc branch but copy_mc branch can still copy such a page. [1]
Since it's pre-existing issue, I'd prefer to keep it out of this patch.
For solution, should we consider folio_contain_hwpoisoned_page() check in the folio_mc_copy()? It's racy but best-effort, and would avoid adding
cost of checking each page within large folio.
[1] https://lore.kernel.org/linux-mm/sashiko-outbox-208876@kernel.org/#t
> Neither helper checks PageHWPoison() or folio_contain_hwpoisoned_page()
> before copying. copy_mc_to_kernel() only catches synchronous #MC
> exceptions raised during the copy itself (and falls back to plain memcpy()
> when unavailable or on x86 without X86_FEATURE_ERMS or
> copy_mc_fragile_enabled).
>
> If a source folio was poisoned via software injection (such as
> MADV_HWPOISON) or a prior memory_failure() where reading does not trigger
> a synchronous #MC, callers such as folio_mc_copy() or wp_page_copy() treat
> the copy as successful and copy corrupted data into a fresh destination
> folio without PG_hwpoison set, or trigger an unrecoverable kernel #MC when
> copy_mc_to_kernel() is not active.
>
> [1] https://sashiko.dev/#/bug/linux-3af4aa7d-d182-430b-b982-583f6aea4594
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
2026-11-17 5:27 ` Garg, Shivank
@ 2026-11-19 17:27 ` David Hildenbrand (Arm)
0 siblings, 0 replies; 6+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-19 17:27 UTC (permalink / raw)
To: Garg, Shivank, tejavojjala@google.com, Rao, Bharata Bhasker,
rppt@kernel.org, david.laight.linux@gmail.com,
ankur.a.arora@oracle.com, dave.hansen@linux.intel.com,
vbabka@kernel.org, ying.huang@linux.alibaba.com,
akpm@linux-foundation.org, mingo@redhat.com, surenb@google.com,
sashiko-reviews@lists.linux.dev, xiangzao@linux.alibaba.com,
peterz@infradead.org, nhuhuan@google.com, Salunke, Hrushikesh,
tglx@kernel.org, liam@infradead.org, hpa@zytor.com,
mhocko@suse.com, bp@alien8.de, ljs@kernel.org,
rientjes@google.com, pfalcato@suse.de,
torvalds@linuxfoundation.org, kaitao.cheng@linux.dev
Cc: linux-mm@kvack.org, x86@kernel.org
On 11/17/26 06:27, Garg, Shivank wrote:
> On Sat, 2026-11-14 at 17:12 +0000, sashiko-bot@kernel.org wrote:
>> Thank you for your contribution! Sashiko AI review found no new issues
>> in this patch, but noticed 1 related pre-existing issue it does not
>> introduce:
>> - [Critical] mm: missing hwpoison check in copy_mc_user_highpage() and
>> copy_mc_highpage() [1]
>> --
>>
>> commit fae95970ccf2607d670f06862e40d3ba9313200a
>> Author: Shivank Garg <shivankg@amd.com>
>>
>> mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
>>
>> This commit introduces copy_mc_highpages() to copy page ranges with a
>> single copy_mc_to_kernel() call when possible and updates folio_mc_copy()
>> to use it.
>>
>>> diff --git a/include/linux/highmem.h b/include/linux/highmem.h
>>> index d7aac9de1c8a4..a79e9d68359c8 100644
>>> --- a/include/linux/highmem.h
>>> +++ b/include/linux/highmem.h
>>> @@ -484,6 +484,57 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
>>> }
>>> #endif
>>>
>>> +/**
>>> + * copy_mc_highpages() - copy a page range with machine check handling
>>> + * @to: first destination page
>>> + * @from: first source page
>>> + * @nr_pages: number of pages
>>> + *
>>> + * Range variant of copy_mc_highpage(). On a machine check, the poisoned
>>> + * source page is queued for memory_failure().
>>> + *
>>> + * Context: Process context. May sleep.
>>> + * Return: 0 on success, -EHWPOISON if a machine check was handled.
>>> + */
>>> +static inline int copy_mc_highpages(struct page *to, struct page *from,
>>> + unsigned long nr_pages)
>>> +{
>>> + unsigned long len = nr_pages << PAGE_SHIFT;
>>> + unsigned long i, ret;
>>> +
>>> + might_sleep();
>>> +
>>> + /*
>>> + * When copying we want to operate on the largest extent possible to
>>> + * allow for architecture specific extent based optimizations.
>>> + *
>>> + * copy_mc_to_kernel() does not call cond_resched(). That is fine on
>>> + * preemptible kernels, where the scheduler can preempt the copy at
>>> + * any point, so copy the whole range at once if copy_mc_to_kernel() is
>>> + * available and the pages are in the linear map. Otherwise, copy one
>>> + * page at a time and reschedule in between.
>>> + */
>>> + if (IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) && IS_ENABLED(CONFIG_PREEMPTION) &&
>>> + !IS_ENABLED(CONFIG_HIGHMEM)) {
>>> + ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
>>> + if (ret) {
>>> + memory_failure_queue(page_to_pfn(from) +
>>> + ((len - ret) >> PAGE_SHIFT), 0);
>>> + return -EHWPOISON;
>>> + }
>>> + for (i = 0; i < nr_pages; i++)
>>> + kmsan_copy_page_meta(to + i, from + i);
>>> + return 0;
>>> + }
>>> +
>>> + for (i = 0; i < nr_pages; i++) {
>>> + cond_resched();
>>> + if (copy_mc_highpage(to + i, from + i))
>>> + return -EHWPOISON;
>>> + }
>>
>> [Severity: Critical]
>> This problem wasn't introduced by this patch, but can copy_mc_highpage()
>> and copy_mc_user_highpage() (as well as the new copy_mc_highpages() fast
>> path) return 0 when the source page or its containing folio is already
>> marked PG_hwpoison [1]?
>
>
> Yes, the patch keeps the existing behavior. PG_hwpoison marked pages are copied without errors but Real poison issue returns non-zero and folio_mc_copy() fails! Should we consider this as blocker?
>
> Yuanhe's patch adds the check to copy !copy_mc branch but copy_mc branch can still copy such a page. [1]
>
> Since it's pre-existing issue, I'd prefer to keep it out of this patch.
Ack.
--
Cheers,
David
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
2026-11-13 17:09 [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy() Shivank Garg
2026-11-14 17:12 ` sashiko-bot
@ 2026-11-22 7:30 ` Andrew Morton
2026-11-22 17:13 ` Garg, Shivank
1 sibling, 1 reply; 6+ messages in thread
From: Andrew Morton @ 2026-11-22 7:30 UTC (permalink / raw)
To: Shivank Garg
Cc: David Hildenbrand, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Borislav Petkov, Ankur Arora, Dave Hansen, Thomas Gleixner,
Bharata B Rao, Hrushikesh Salunke, Ingo Molnar, H . Peter Anvin,
David Laight, David Rientjes, Linus Torvalds, Teja Vojjala,
Huan Nguyen, Pedro Falcato, Peter Zijlstra, linux-mm,
linux-kernel, x86
On Fri, 13 Nov 2026 17:09:21 +0000 Shivank Garg <shivankg@amd.com> wrote:
> folio_mc_copy() copies a folio one page at a time with copy_mc_highpage(),
> so a 2MB folio makes 512 separate copy_mc_to_kernel() calls. Each page
> pays the startup cost of the string instruction ("REP; MOVSB" on x86
> CPUs with ERMS) plus a cond_resched().
>
> Introduce copy_mc_highpages(), the page range variant of
> copy_mc_highpage(). When copy_mc_to_kernel() is available and the pages
> are in the linear map, copy the whole range with a single
> copy_mc_to_kernel() call, so the CPU can optimize for the larger extent.
> Otherwise, fall back to copy_mc_highpage() one page at a time.
>
> On a machine check, queue the poisoned source page for memory_failure(),
> as copy_mc_highpage() does, and return -EHWPOISON, as folio_mc_copy()
> does today.
>
> copy_mc_to_kernel() cannot reschedule, so only take the single-call path
> when CONFIG_PREEMPTION is set and the copy can be preempted. Since commit
> 7dadeaa6e851 ("sched: Further restrict the preemption modes"), x86 and
> powerpc, the only architectures that select ARCH_HAS_COPY_MC, only offer
> the full, lazy and RT preemption models, also with PREEMPT_DYNAMIC, so
> they always take this path. The per-page fallback calls cond_resched()
> before each page, as clear_contig_highpages() does. Other architectures
> keep copying one page at a time.
>
> Switch folio_mc_copy() to copy the whole folio with copy_mc_highpages().
OK.
> ---
> On a Zen 6 system with CXL memory, move_pages() gets ~1.4x faster for
> 2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below).
> When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.
But that's the most important part! It's the whole point of the patch!
Yet here it is below the ---, which means "throw this away". It
should be the first paragraph, in all caps.
<edit edit fixed>
OK, thanks, queued,
nit: does it really need cond_resched()? That's not appropriate for
current mainline kernels. I guess "yes", because this patch looks like
something which downstream people will eagerly backport into older
kernels, even though we didn't cc:stable.
And we care about downstream people. Globally: "Roughly 60% are on 6.x
and 30% are on 5.x (with the remaining ~10% split between 7.x and 4.x
or older)".
I'll take a look at that Sashiko thing.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
2026-11-22 7:30 ` Andrew Morton
@ 2026-11-22 17:13 ` Garg, Shivank
0 siblings, 0 replies; 6+ messages in thread
From: Garg, Shivank @ 2026-11-22 17:13 UTC (permalink / raw)
To: akpm@linux-foundation.org
Cc: tejavojjala@google.com, Rao, Bharata Bhasker, rppt@kernel.org,
david.laight.linux@gmail.com, ankur.a.arora@oracle.com,
vbabka@kernel.org, dave.hansen@linux.intel.com, david@kernel.org,
peterz@infradead.org, mingo@redhat.com, surenb@google.com,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
nhuhuan@google.com, Salunke, Hrushikesh, tglx@kernel.org,
liam@infradead.org, hpa@zytor.com, bp@alien8.de, mhocko@suse.com,
ljs@kernel.org, rientjes@google.com, pfalcato@suse.de,
torvalds@linuxfoundation.org, x86@kernel.org
On Sat, 2026-11-21 at 23:30 -0800, Andrew Morton wrote:
> On Fri, 13 Nov 2026 17:09:21 +0000 Shivank Garg <shivankg@amd.com> wrote:
>
> > folio_mc_copy() copies a folio one page at a time with copy_mc_highpage(),
> > so a 2MB folio makes 512 separate copy_mc_to_kernel() calls. Each page
> > pays the startup cost of the string instruction ("REP; MOVSB" on x86
> > CPUs with ERMS) plus a cond_resched().
> >
> > Introduce copy_mc_highpages(), the page range variant of
> > copy_mc_highpage(). When copy_mc_to_kernel() is available and the pages
> > are in the linear map, copy the whole range with a single
> > copy_mc_to_kernel() call, so the CPU can optimize for the larger extent.
> > Otherwise, fall back to copy_mc_highpage() one page at a time.
> >
> > On a machine check, queue the poisoned source page for memory_failure(),
> > as copy_mc_highpage() does, and return -EHWPOISON, as folio_mc_copy()
> > does today.
> >
> > copy_mc_to_kernel() cannot reschedule, so only take the single-call path
> > when CONFIG_PREEMPTION is set and the copy can be preempted. Since commit
> > 7dadeaa6e851 ("sched: Further restrict the preemption modes"), x86 and
> > powerpc, the only architectures that select ARCH_HAS_COPY_MC, only offer
> > the full, lazy and RT preemption models, also with PREEMPT_DYNAMIC, so
> > they always take this path. The per-page fallback calls cond_resched()
> > before each page, as clear_contig_highpages() does. Other architectures
> > keep copying one page at a time.
> >
> > Switch folio_mc_copy() to copy the whole folio with copy_mc_highpages().
>
> OK.
>
> > ---
> > On a Zen 6 system with CXL memory, move_pages() gets ~1.4x faster for
> > 2MB THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below).
> > When demoting to CXL, kswapd uses ~31% less CPU per GB demoted.
>
> But that's the most important part! It's the whole point of the patch!
> Yet here it is below the ---, which means "throw this away". It
> should be the first paragraph, in all caps.
>
> <edit edit fixed>
Sure, thanks. I'll keep this in mind.
>
> OK, thanks, queued,
>
> nit: does it really need cond_resched()? That's not appropriate for
> current mainline kernels. I guess "yes", because this patch looks like
> something which downstream people will eagerly backport into older
> kernels, even though we didn't cc:stable.
>
> And we care about downstream people. Globally: "Roughly 60% are on 6.x
> and 30% are on 5.x (with the remaining ~10% split between 7.x and 4.x
> or older)".
>
> I'll take a look at that Sashiko thing.
Agreed.
For older kernels, where the non-preemptible modes are still
available, the backport should look like v2 in my opinion (calling
cond_resched() every 32MB). Otherwise a 1GB hugetlb copy can run for
a long time without rescheduling. I'll leave this discussion for
backport time :)
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-11-22 17:13 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-13 17:09 [PATCH v3] mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy() Shivank Garg
2026-11-14 17:12 ` sashiko-bot
2026-11-17 5:27 ` Garg, Shivank
2026-11-19 17:27 ` David Hildenbrand (Arm)
2026-11-22 7:30 ` Andrew Morton
2026-11-22 17:13 ` Garg, Shivank
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox