From: Andrew Morton <akpm@linux-foundation.org>
To: mm-commits@vger.kernel.org,vbabka@kernel.org,torvalds@linuxfoundation.org,
tglx@kernel.org,tejavojjala@google.com,surenb@google.com,rppt@kernel.org,
rientjes@google.com,pfalcato@suse.de,peterz@infradead.org,nhuhuan@google.com,
mingo@redhat.com,mhocko@suse.com,ljs@kernel.org,liam@infradead.org,
hsalunke@amd.com,hpa@zytor.com,david.laight.linux@gmail.com,david@kernel.org,
dave.hansen@linux.intel.com,bp@alien8.de,bharata@amd.com,
ankur.a.arora@oracle.com,shivankg@amd.com,akpm@linux-foundation.org
Subject: + mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy.patch added to mm-unstable branch
Date: Sat, 21 Nov 2026 23:30:25 -0800 [thread overview]
Message-ID: <20261122073025.90A441F000C0@smtp.kernel.org> (raw)
In-Reply-To: <20261113-batpagecopy-v3-1-297f5b2fd28e@amd.com>
The patch titled
Subject: mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
has been added to the -mm mm-unstable branch. Its filename is
mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy.patch
This patch will shortly appear at
https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy.patch
This patch will later appear in the mm-unstable branch at
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Before you just go and hit "reply", please:
a) Consider who else should be cc'ed
b) Prefer to cc a suitable mailing list as well
c) Ideally: find the original patch on the mailing list and do a
reply-to-all to that, adding suitable additional cc's
*** Remember to use Documentation/process/submit-checklist.rst when testing your code ***
The -mm tree is included into linux-next via various
branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
and is updated there most days
------------------------------------------------------
From: Shivank Garg <shivankg@amd.com>
Subject: mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
Date: Fri, 13 Nov 2026 17:09:21 +0000
folio_mc_copy() copies a folio one page at a time with copy_mc_highpage(),
so a 2MB folio makes 512 separate copy_mc_to_kernel() calls. Each page
pays the startup cost of the string instruction ("REP; MOVSB" on x86 CPUs
with ERMS) plus a cond_resched().
Introduce copy_mc_highpages(), the page range variant of
copy_mc_highpage(). When copy_mc_to_kernel() is available and the pages
are in the linear map, copy the whole range with a single
copy_mc_to_kernel() call, so the CPU can optimize for the larger extent.
Otherwise, fall back to copy_mc_highpage() one page at a time.
On a machine check, queue the poisoned source page for memory_failure(),
as copy_mc_highpage() does, and return -EHWPOISON, as folio_mc_copy() does
today.
copy_mc_to_kernel() cannot reschedule, so only take the single-call path
when CONFIG_PREEMPTION is set and the copy can be preempted. Since commit
7dadeaa6e851 ("sched: Further restrict the preemption modes"), x86 and
powerpc, the only architectures that select ARCH_HAS_COPY_MC, only offer
the full, lazy and RT preemption models, also with PREEMPT_DYNAMIC, so
they always take this path. The per-page fallback calls cond_resched()
before each page, as clear_contig_highpages() does. Other architectures
keep copying one page at a time.
Switch folio_mc_copy() to copy the whole folio with copy_mc_highpages().
On a Zen 6 system with CXL memory, move_pages() gets ~1.4x faster for 2MB
THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below). When
demoting to CXL, kswapd uses ~31% less CPU per GB demoted.
Link: https://patch.msgid.link/20261113-batpagecopy-v3-1-297f5b2fd28e@amd.com
Signed-off-by: Shivank Garg <shivankg@amd.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Ankur Arora <ankur.a.arora@oracle.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Bharata B Rao <bharata@amd.com>
Cc: Salunke, Hrushikesh <hsalunke@amd.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Linus Torvalds <torvalds@linuxfoundation.org>
Cc: Teja Vojjala <tejavojjala@google.com>
Cc: Huan Nguyen <nhuhuan@google.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
include/linux/highmem.h | 51 ++++++++++++++++++++++++++++++++++++++
mm/util.c | 14 +---------
2 files changed, 53 insertions(+), 12 deletions(-)
--- a/include/linux/highmem.h~mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy
+++ a/include/linux/highmem.h
@@ -484,6 +484,57 @@ static inline int copy_mc_highpage(struc
}
#endif
+/**
+ * copy_mc_highpages() - copy a page range with machine check handling
+ * @to: first destination page
+ * @from: first source page
+ * @nr_pages: number of pages
+ *
+ * Range variant of copy_mc_highpage(). On a machine check, the poisoned
+ * source page is queued for memory_failure().
+ *
+ * Context: Process context. May sleep.
+ * Return: 0 on success, -EHWPOISON if a machine check was handled.
+ */
+static inline int copy_mc_highpages(struct page *to, struct page *from,
+ unsigned long nr_pages)
+{
+ unsigned long len = nr_pages << PAGE_SHIFT;
+ unsigned long i, ret;
+
+ might_sleep();
+
+ /*
+ * When copying we want to operate on the largest extent possible to
+ * allow for architecture specific extent based optimizations.
+ *
+ * copy_mc_to_kernel() does not call cond_resched(). That is fine on
+ * preemptible kernels, where the scheduler can preempt the copy at
+ * any point, so copy the whole range at once if copy_mc_to_kernel() is
+ * available and the pages are in the linear map. Otherwise, copy one
+ * page at a time and reschedule in between.
+ */
+ if (IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) && IS_ENABLED(CONFIG_PREEMPTION) &&
+ !IS_ENABLED(CONFIG_HIGHMEM)) {
+ ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
+ if (ret) {
+ memory_failure_queue(page_to_pfn(from) +
+ ((len - ret) >> PAGE_SHIFT), 0);
+ return -EHWPOISON;
+ }
+ for (i = 0; i < nr_pages; i++)
+ kmsan_copy_page_meta(to + i, from + i);
+ return 0;
+ }
+
+ for (i = 0; i < nr_pages; i++) {
+ cond_resched();
+ if (copy_mc_highpage(to + i, from + i))
+ return -EHWPOISON;
+ }
+ return 0;
+}
+
static inline void memcpy_page(struct page *dst_page, size_t dst_off,
struct page *src_page, size_t src_off,
size_t len)
--- a/mm/util.c~mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy
+++ a/mm/util.c
@@ -776,18 +776,8 @@ EXPORT_SYMBOL(folio_copy);
int folio_mc_copy(struct folio *dst, struct folio *src)
{
- long nr = folio_nr_pages(src);
- long i = 0;
-
- for (;;) {
- if (copy_mc_highpage(folio_page(dst, i), folio_page(src, i)))
- return -EHWPOISON;
- if (++i == nr)
- break;
- cond_resched();
- }
-
- return 0;
+ return copy_mc_highpages(folio_page(dst, 0), folio_page(src, 0),
+ folio_nr_pages(src));
}
EXPORT_SYMBOL(folio_mc_copy);
_
Patches currently in -mm which might be from shivankg@amd.com are
mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy.patch
mm-remove-unused-helpers-from-pagemaph.patch
mm-swapops-remove-unused-make_guard_swp_entry.patch
mm-swap-remove-unused-folio_to_swp_tb.patch
mm-zsmalloc-remove-unused-zpdesc_lock.patch
mm-nommu-remove-unused-heap_stack_gap.patch
next parent reply other threads:[~2026-11-22 7:30 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20261113-batpagecopy-v3-1-297f5b2fd28e@amd.com>
2026-11-22 7:30 ` Andrew Morton [this message]
2026-11-22 16:00 ` + mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy.patch added to mm-unstable branch David Hildenbrand (Arm)
2026-11-22 16:44 ` [merged] mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy.patch removed from -mm tree Andrew Morton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261122073025.90A441F000C0@smtp.kernel.org \
--to=akpm@linux-foundation.org \
--cc=ankur.a.arora@oracle.com \
--cc=bharata@amd.com \
--cc=bp@alien8.de \
--cc=dave.hansen@linux.intel.com \
--cc=david.laight.linux@gmail.com \
--cc=david@kernel.org \
--cc=hpa@zytor.com \
--cc=hsalunke@amd.com \
--cc=liam@infradead.org \
--cc=ljs@kernel.org \
--cc=mhocko@suse.com \
--cc=mingo@redhat.com \
--cc=mm-commits@vger.kernel.org \
--cc=nhuhuan@google.com \
--cc=peterz@infradead.org \
--cc=pfalcato@suse.de \
--cc=rientjes@google.com \
--cc=rppt@kernel.org \
--cc=shivankg@amd.com \
--cc=surenb@google.com \
--cc=tejavojjala@google.com \
--cc=tglx@kernel.org \
--cc=torvalds@linuxfoundation.org \
--cc=vbabka@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox