Linux MM tree latest commits
 help / color / mirror / Atom feed
From: Andrew Morton <akpm@linux-foundation.org>
To: mm-commits@vger.kernel.org,vbabka@kernel.org,torvalds@linuxfoundation.org,
	tglx@kernel.org,tejavojjala@google.com,surenb@google.com,rppt@kernel.org,
	rientjes@google.com,pfalcato@suse.de,peterz@infradead.org,nhuhuan@google.com,
	mingo@redhat.com,mhocko@suse.com,ljs@kernel.org,liam@infradead.org,
	hsalunke@amd.com,hpa@zytor.com,david.laight.linux@gmail.com,david@kernel.org,
	dave.hansen@linux.intel.com,bp@alien8.de,bharata@amd.com,
	ankur.a.arora@oracle.com,shivankg@amd.com,akpm@linux-foundation.org
Subject: + mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy.patch added to mm-unstable branch
Date: Sat, 21 Nov 2026 23:30:25 -0800	[thread overview]
Message-ID: <20261122073025.90A441F000C0@smtp.kernel.org> (raw)
In-Reply-To: <20261113-batpagecopy-v3-1-297f5b2fd28e@amd.com>


The patch titled
     Subject: mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
has been added to the -mm mm-unstable branch.  Its filename is
     mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy.patch

This patch will shortly appear at
     https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy.patch

This patch will later appear in the mm-unstable branch at
    git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Before you just go and hit "reply", please:
   a) Consider who else should be cc'ed
   b) Prefer to cc a suitable mailing list as well
   c) Ideally: find the original patch on the mailing list and do a
      reply-to-all to that, adding suitable additional cc's

*** Remember to use Documentation/process/submit-checklist.rst when testing your code ***

The -mm tree is included into linux-next via various
branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
and is updated there most days

------------------------------------------------------
From: Shivank Garg <shivankg@amd.com>
Subject: mm: introduce copy_mc_highpages() and batch copy in folio_mc_copy()
Date: Fri, 13 Nov 2026 17:09:21 +0000

folio_mc_copy() copies a folio one page at a time with copy_mc_highpage(),
so a 2MB folio makes 512 separate copy_mc_to_kernel() calls.  Each page
pays the startup cost of the string instruction ("REP; MOVSB" on x86 CPUs
with ERMS) plus a cond_resched().

Introduce copy_mc_highpages(), the page range variant of
copy_mc_highpage().  When copy_mc_to_kernel() is available and the pages
are in the linear map, copy the whole range with a single
copy_mc_to_kernel() call, so the CPU can optimize for the larger extent. 
Otherwise, fall back to copy_mc_highpage() one page at a time.

On a machine check, queue the poisoned source page for memory_failure(),
as copy_mc_highpage() does, and return -EHWPOISON, as folio_mc_copy() does
today.

copy_mc_to_kernel() cannot reschedule, so only take the single-call path
when CONFIG_PREEMPTION is set and the copy can be preempted.  Since commit
7dadeaa6e851 ("sched: Further restrict the preemption modes"), x86 and
powerpc, the only architectures that select ARCH_HAS_COPY_MC, only offer
the full, lazy and RT preemption models, also with PREEMPT_DYNAMIC, so
they always take this path.  The per-page fallback calls cond_resched()
before each page, as clear_contig_highpages() does.  Other architectures
keep copying one page at a time.

Switch folio_mc_copy() to copy the whole folio with copy_mc_highpages().

On a Zen 6 system with CXL memory, move_pages() gets ~1.4x faster for 2MB
THPs and up to ~2.8x faster for 1GB hugetlb pages (numbers below).  When
demoting to CXL, kswapd uses ~31% less CPU per GB demoted.

Link: https://patch.msgid.link/20261113-batpagecopy-v3-1-297f5b2fd28e@amd.com
Signed-off-by: Shivank Garg <shivankg@amd.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Ankur Arora <ankur.a.arora@oracle.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: Bharata B Rao <bharata@amd.com>
Cc: Salunke, Hrushikesh <hsalunke@amd.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: David Laight <david.laight.linux@gmail.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Linus Torvalds <torvalds@linuxfoundation.org>
Cc: Teja Vojjala <tejavojjala@google.com>
Cc: Huan Nguyen <nhuhuan@google.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>

 include/linux/highmem.h |   51 ++++++++++++++++++++++++++++++++++++++
 mm/util.c               |   14 +---------
 2 files changed, 53 insertions(+), 12 deletions(-)

--- a/include/linux/highmem.h~mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy
+++ a/include/linux/highmem.h
@@ -484,6 +484,57 @@ static inline int copy_mc_highpage(struc
 }
 #endif
 
+/**
+ * copy_mc_highpages() - copy a page range with machine check handling
+ * @to: first destination page
+ * @from: first source page
+ * @nr_pages: number of pages
+ *
+ * Range variant of copy_mc_highpage(). On a machine check, the poisoned
+ * source page is queued for memory_failure().
+ *
+ * Context: Process context. May sleep.
+ * Return: 0 on success, -EHWPOISON if a machine check was handled.
+ */
+static inline int copy_mc_highpages(struct page *to, struct page *from,
+		unsigned long nr_pages)
+{
+	unsigned long len = nr_pages << PAGE_SHIFT;
+	unsigned long i, ret;
+
+	might_sleep();
+
+	/*
+	 * When copying we want to operate on the largest extent possible to
+	 * allow for architecture specific extent based optimizations.
+	 *
+	 * copy_mc_to_kernel() does not call cond_resched(). That is fine on
+	 * preemptible kernels, where the scheduler can preempt the copy at
+	 * any point, so copy the whole range at once if copy_mc_to_kernel() is
+	 * available and the pages are in the linear map. Otherwise, copy one
+	 * page at a time and reschedule in between.
+	 */
+	if (IS_ENABLED(CONFIG_ARCH_HAS_COPY_MC) && IS_ENABLED(CONFIG_PREEMPTION) &&
+	    !IS_ENABLED(CONFIG_HIGHMEM)) {
+		ret = copy_mc_to_kernel(page_address(to), page_address(from), len);
+		if (ret) {
+			memory_failure_queue(page_to_pfn(from) +
+					     ((len - ret) >> PAGE_SHIFT), 0);
+			return -EHWPOISON;
+		}
+		for (i = 0; i < nr_pages; i++)
+			kmsan_copy_page_meta(to + i, from + i);
+		return 0;
+	}
+
+	for (i = 0; i < nr_pages; i++) {
+		cond_resched();
+		if (copy_mc_highpage(to + i, from + i))
+			return -EHWPOISON;
+	}
+	return 0;
+}
+
 static inline void memcpy_page(struct page *dst_page, size_t dst_off,
 			       struct page *src_page, size_t src_off,
 			       size_t len)
--- a/mm/util.c~mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy
+++ a/mm/util.c
@@ -776,18 +776,8 @@ EXPORT_SYMBOL(folio_copy);
 
 int folio_mc_copy(struct folio *dst, struct folio *src)
 {
-	long nr = folio_nr_pages(src);
-	long i = 0;
-
-	for (;;) {
-		if (copy_mc_highpage(folio_page(dst, i), folio_page(src, i)))
-			return -EHWPOISON;
-		if (++i == nr)
-			break;
-		cond_resched();
-	}
-
-	return 0;
+	return copy_mc_highpages(folio_page(dst, 0), folio_page(src, 0),
+				 folio_nr_pages(src));
 }
 EXPORT_SYMBOL(folio_mc_copy);
 
_

Patches currently in -mm which might be from shivankg@amd.com are

mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy.patch
mm-remove-unused-helpers-from-pagemaph.patch
mm-swapops-remove-unused-make_guard_swp_entry.patch
mm-swap-remove-unused-folio_to_swp_tb.patch
mm-zsmalloc-remove-unused-zpdesc_lock.patch
mm-nommu-remove-unused-heap_stack_gap.patch


       reply	other threads:[~2026-11-22  7:30 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20261113-batpagecopy-v3-1-297f5b2fd28e@amd.com>
2026-11-22  7:30 ` Andrew Morton [this message]
2026-11-22 16:00   ` + mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy.patch added to mm-unstable branch David Hildenbrand (Arm)
2026-11-22 16:44 ` [merged] mm-introduce-copy_mc_highpages-and-batch-copy-in-folio_mc_copy.patch removed from -mm tree Andrew Morton

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261122073025.90A441F000C0@smtp.kernel.org \
    --to=akpm@linux-foundation.org \
    --cc=ankur.a.arora@oracle.com \
    --cc=bharata@amd.com \
    --cc=bp@alien8.de \
    --cc=dave.hansen@linux.intel.com \
    --cc=david.laight.linux@gmail.com \
    --cc=david@kernel.org \
    --cc=hpa@zytor.com \
    --cc=hsalunke@amd.com \
    --cc=liam@infradead.org \
    --cc=ljs@kernel.org \
    --cc=mhocko@suse.com \
    --cc=mingo@redhat.com \
    --cc=mm-commits@vger.kernel.org \
    --cc=nhuhuan@google.com \
    --cc=peterz@infradead.org \
    --cc=pfalcato@suse.de \
    --cc=rientjes@google.com \
    --cc=rppt@kernel.org \
    --cc=shivankg@amd.com \
    --cc=surenb@google.com \
    --cc=tejavojjala@google.com \
    --cc=tglx@kernel.org \
    --cc=torvalds@linuxfoundation.org \
    --cc=vbabka@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox