[PATCH RFC v2 0/3] Demote to lower tier using non-temporal stores

[PATCH RFC v2 0/3] Demote to lower tier using non-temporal stores

[Yiannis Nikolakopoulos]

In most memory tiering scenarios, the memory to be demoted is expected
to be cold and most likely out of the node's last-level cache (as well
as target pages in the target node). Using non-temporal stores instead
of a standard memcpy path can reduce the cache pollution in the local
node and the bandwidth overhead to the target node. Furthermore, for
certain types of CXL devices that support in-line memory compression,
the last-level cache eviction patterns can negatively affect the
bandwidth of the device. Non-temporal stores can mitigate this.

This patch-set introduces a new migrate_mode flag for using non-temporal
stores that is used only in the demotion path. Patch 1 adds some helpers in
x86 and mm to bring non-temporal stores support to a respective folio_copy
function. Patch 2 adds the new flag and necessary changes for compatibility
with the existing behavior. Patch 3 uses the new flag for demotions.

Experimental data: in a CXL system with 1 memory expander, a microbenchmark
that allocates N=64 GB memory in the local node and then triggers demotion
using memory.reclaim, shows a practically complete elimination of read
traffic on the device, i.e. write traffic is N GB with and without the
patch, while read traffic drops from N to almost 0 with the patch.

Opens:
1. There is some "duplication" in the x86 tree and a bit in mm. Can we do
   something better there? As it is now in copy_mc_to_kernel_nt we
duplicate the machine check functionality, which if available will override
the non-temporal. We were not sure how to prioritize these two and what's
the best approach here. Can we completely skip the machine checked for this
path? Huan Nguyen has some ideas here that we will align for the next
version.
2. I am not sure how this should be structured so that it is easily
adopted in other architecture trees (e.g. aarch64). We rely on
memcpy_flushcache for x86_64 but this does not use non temporal stores
in ARM. ARM support is currently out of our scope but any input is
appreciated.

Signed-off-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
---
Changes in v2:
- Remove the KConfig option that was guarding this feature (patch 3) and
  use it always in x86_64 and memcpy_flushcache (non-temporal stores
should be there).
- Link to v1: https://patch.msgid.link/20260526-rfc-nt-demote-v1-0-eb9c9422daef@zptcorp.com

To: Thomas Gleixner <tglx@kernel.org>
To: Ingo Molnar <mingo@redhat.com>
To: Borislav Petkov <bp@alien8.de>
To: Dave Hansen <dave.hansen@linux.intel.com>
To: x86@kernel.org
To: "H. Peter Anvin" <hpa@zytor.com>
To: Andrew Morton <akpm@linux-foundation.org>
To: David Hildenbrand <david@kernel.org>
To: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
To: "Liam R. Howlett" <Liam.Howlett@oracle.com>
To: Vlastimil Babka <vbabka@kernel.org>
To: Mike Rapoport <rppt@kernel.org>
To: Suren Baghdasaryan <surenb@google.com>
To: Michal Hocko <mhocko@suse.com>
To: Trond Myklebust <trondmy@kernel.org>
To: Anna Schumaker <anna@kernel.org>
To: Zi Yan <ziy@nvidia.com>
To: Matthew Brost <matthew.brost@intel.com>
To: Joshua Hahn <joshua.hahnjy@gmail.com>
To: Rakie Kim <rakie.kim@sk.com>
To: Byungchul Park <byungchul@sk.com>
To: Gregory Price <gourry@gourry.net>
To: Ying Huang <ying.huang@linux.alibaba.com>
To: Alistair Popple <apopple@nvidia.com>
To: Steven Rostedt <rostedt@goodmis.org>
To: Masami Hiramatsu <mhiramat@kernel.org>
To: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To: Brendan Jackman <jackmanb@google.com>
To: Johannes Weiner <hannes@cmpxchg.org>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mm@kvack.org
Cc: linux-nfs@vger.kernel.org
Cc: linux-trace-kernel@vger.kernel.org

---
Alirad Malek (3):
      mm, x86: support copying a folio using non-temporal stores
      mm: new migrate_mode flag for async using non-temporal stores
      mm: use non-temporal stores for demotion

 arch/x86/include/asm/uaccess.h |  4 ++++
 arch/x86/lib/copy_mc.c         | 26 ++++++++++++++++++++++++++
 fs/nfs/write.c                 |  2 +-
 include/linux/highmem.h        | 32 ++++++++++++++++++++++++++++++++
 include/linux/migrate_mode.h   |  9 +++++++++
 include/linux/mm.h             |  1 +
 include/trace/events/migrate.h |  1 +
 mm/compaction.c                | 18 +++++++++---------
 mm/migrate.c                   | 22 +++++++++++++++-------
 mm/util.c                      | 17 +++++++++++++++++
 10 files changed, 115 insertions(+), 17 deletions(-)
---
base-commit: 1f318b96cc84d7c2ab792fcc0bfd42a7ca890681
change-id: 20260526-rfc-nt-demote-0fafadfdd006

Best regards,
--  
Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>

[PATCH RFC v2 1/3] mm, x86: support copying a folio using non-temporal stores

[Yiannis Nikolakopoulos]

From: Alirad Malek <alirad.malek@zptcorp.com>

In x86, use memcpy_flushcache (that uses non-temporal store
instructions) to copy a folio. To achieve that, starting from folio_mc_copy
down to copy_mc_to_kernel, create a series of helpers (named with an _nt
suffix) that have similar behavior to the original counterparts.

Signed-off-by: Alirad Malek <alirad.malek@zptcorp.com>
Co-developed-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
Signed-off-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
---
 arch/x86/include/asm/uaccess.h |  4 ++++
 arch/x86/lib/copy_mc.c         | 26 ++++++++++++++++++++++++++
 include/linux/highmem.h        | 32 ++++++++++++++++++++++++++++++++
 include/linux/mm.h             |  1 +
 mm/util.c                      | 17 +++++++++++++++++
 5 files changed, 80 insertions(+)

diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h
index 367297b188c3..2d0938d3e372 100644
--- a/arch/x86/include/asm/uaccess.h
+++ b/arch/x86/include/asm/uaccess.h
@@ -494,6 +494,10 @@ unsigned long __must_check
 copy_mc_to_kernel(void *to, const void *from, unsigned len);
 #define copy_mc_to_kernel copy_mc_to_kernel
 
+unsigned long __must_check
+copy_mc_to_kernel_nt(void *to, const void *from, unsigned len);
+#define copy_mc_to_kernel_nt copy_mc_to_kernel_nt
+
 unsigned long __must_check
 copy_mc_to_user(void __user *to, const void *from, unsigned len);
 #endif
diff --git a/arch/x86/lib/copy_mc.c b/arch/x86/lib/copy_mc.c
index 97e88e58567b..5a2ee5c2211e 100644
--- a/arch/x86/lib/copy_mc.c
+++ b/arch/x86/lib/copy_mc.c
@@ -81,6 +81,32 @@ unsigned long __must_check copy_mc_to_kernel(void *dst, const void *src, unsigne
 }
 EXPORT_SYMBOL_GPL(copy_mc_to_kernel);
 
+/**
+ * copy_mc_to_kernel_nt - memory copy that handles source exceptions
+ * if enabled, otherwise uses non-temporal stores
+ * @dst: destination address
+ * @src: source address
+ * @len: number of bytes to copy
+ *
+ * Return 0 for success, or number of bytes not copied if there was an
+ * exception.
+ */
+unsigned long __must_check copy_mc_to_kernel_nt(void *dst, const void *src, unsigned len)
+{
+	unsigned long ret;
+
+	if (copy_mc_fragile_enabled) {
+		instrument_memcpy_before(dst, src, len);
+		ret = copy_mc_fragile(dst, src, len);
+		instrument_memcpy_after(dst, src, len, ret);
+		return ret;
+	}
+
+	memcpy_flushcache(dst, src, len);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(copy_mc_to_kernel_nt);
+
 unsigned long __must_check copy_mc_to_user(void __user *dst, const void *src, unsigned len)
 {
 	unsigned long ret;
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index af03db851a1d..a5cb435b9ffe 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -468,6 +468,32 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
 
 	return ret;
 }
+
+#ifdef copy_mc_to_kernel_nt
+static inline int copy_mc_highpage_nt(struct page *to, struct page *from)
+{
+	unsigned long ret;
+	char *vfrom, *vto;
+
+	vfrom = kmap_local_page(from);
+	vto = kmap_local_page(to);
+	ret = copy_mc_to_kernel_nt(vto, vfrom, PAGE_SIZE);
+	if (!ret)
+		kmsan_copy_page_meta(to, from);
+	kunmap_local(vto);
+	kunmap_local(vfrom);
+
+	if (ret)
+		memory_failure_queue(page_to_pfn(from), 0);
+
+	return ret;
+}
+#else
+static inline int copy_mc_highpage_nt(struct page *to, struct page *from)
+{
+	return copy_mc_highpage(to, from);
+}
+#endif
 #else
 static inline int copy_mc_user_highpage(struct page *to, struct page *from,
 					unsigned long vaddr, struct vm_area_struct *vma)
@@ -481,6 +507,12 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
 	copy_highpage(to, from);
 	return 0;
 }
+
+static inline int copy_mc_highpage_nt(struct page *to, struct page *from)
+{
+	copy_highpage(to, from);
+	return 0;
+}
 #endif
 
 static inline void memcpy_page(struct page *dst_page, size_t dst_off,
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 5be3d8a8f806..d07ce478582d 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1644,6 +1644,7 @@ void __folio_put(struct folio *folio);
 void split_page(struct page *page, unsigned int order);
 void folio_copy(struct folio *dst, struct folio *src);
 int folio_mc_copy(struct folio *dst, struct folio *src);
+int folio_mc_copy_nt(struct folio *dst, struct folio *src);
 
 unsigned long nr_free_buffer_pages(void);
 
diff --git a/mm/util.c b/mm/util.c
index b05ab6f97e11..e09e9b5f8eee 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -749,6 +749,23 @@ int folio_mc_copy(struct folio *dst, struct folio *src)
 }
 EXPORT_SYMBOL(folio_mc_copy);
 
+int folio_mc_copy_nt(struct folio *dst, struct folio *src)
+{
+	long nr = folio_nr_pages(src);
+	long i = 0;
+
+	for (;;) {
+		if (copy_mc_highpage_nt(folio_page(dst, i), folio_page(src, i)))
+			return -EHWPOISON;
+		if (++i == nr)
+			break;
+		cond_resched();
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(folio_mc_copy_nt);
+
 int sysctl_overcommit_memory __read_mostly = OVERCOMMIT_GUESS;
 static int sysctl_overcommit_ratio __read_mostly = 50;
 static unsigned long sysctl_overcommit_kbytes __read_mostly;

-- 
2.43.0

[PATCH RFC v2 2/3] mm: new migrate_mode flag for async using non-temporal stores

[Yiannis Nikolakopoulos]

From: Alirad Malek <alirad.malek@zptcorp.com>

In preparation for the following patch, add a new migrate_mode which is
still async but will use non-temporal stores. Add a helper function
that checks for both async modes and replace all plain checks of
MIGRATE_ASYNC.

Signed-off-by: Alirad Malek <alirad.malek@zptcorp.com>
Co-developed-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
Signed-off-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
---
 fs/nfs/write.c                 |  2 +-
 include/linux/migrate_mode.h   |  9 +++++++++
 include/trace/events/migrate.h |  1 +
 mm/compaction.c                | 18 +++++++++---------
 mm/migrate.c                   | 12 ++++++------
 5 files changed, 26 insertions(+), 16 deletions(-)

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 1ed4b3590b1a..beae4441e080 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -2119,7 +2119,7 @@ int nfs_migrate_folio(struct address_space *mapping, struct folio *dst,
 	}
 
 	if (folio_test_private_2(src)) { /* [DEPRECATED] */
-		if (mode == MIGRATE_ASYNC)
+		if (migrate_mode_is_async(mode))
 			return -EBUSY;
 		folio_wait_private_2(src);
 	}
diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h
index 265c4328b36a..f7186e705b48 100644
--- a/include/linux/migrate_mode.h
+++ b/include/linux/migrate_mode.h
@@ -3,6 +3,8 @@
 #define MIGRATE_MODE_H_INCLUDED
 /*
  * MIGRATE_ASYNC means never block
+ * MIGRATE_ASYNC_NON_TEMPORAL_STORES means never block and use non-temporal
+ * stores if supported by the architecture
  * MIGRATE_SYNC_LIGHT in the current implementation means to allow blocking
  *	on most operations but not ->writepage as the potential stall time
  *	is too significant
@@ -10,10 +12,17 @@
  */
 enum migrate_mode {
 	MIGRATE_ASYNC,
+	MIGRATE_ASYNC_NON_TEMPORAL_STORES,
 	MIGRATE_SYNC_LIGHT,
 	MIGRATE_SYNC,
 };
 
+static inline bool migrate_mode_is_async(enum migrate_mode mode)
+{
+	return mode == MIGRATE_ASYNC ||
+		mode == MIGRATE_ASYNC_NON_TEMPORAL_STORES;
+}
+
 enum migrate_reason {
 	MR_COMPACTION,
 	MR_MEMORY_FAILURE,
diff --git a/include/trace/events/migrate.h b/include/trace/events/migrate.h
index cd01dd7b3640..e493207a3f46 100644
--- a/include/trace/events/migrate.h
+++ b/include/trace/events/migrate.h
@@ -9,6 +9,7 @@
 
 #define MIGRATE_MODE						\
 	EM( MIGRATE_ASYNC,	"MIGRATE_ASYNC")		\
+	EM(MIGRATE_ASYNC_NON_TEMPORAL_STORES,		"MIGRATE_ASYNC_NON_TEMPORAL_STORES")	\
 	EM( MIGRATE_SYNC_LIGHT,	"MIGRATE_SYNC_LIGHT")		\
 	EMe(MIGRATE_SYNC,	"MIGRATE_SYNC")
 
diff --git a/mm/compaction.c b/mm/compaction.c
index 1e8f8eca318c..cd26781b7376 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -444,7 +444,7 @@ static void update_cached_migrate(struct compact_control *cc, unsigned long pfn)
 	/* Update where async and sync compaction should restart */
 	if (pfn > zone->compact_cached_migrate_pfn[0])
 		zone->compact_cached_migrate_pfn[0] = pfn;
-	if (cc->mode != MIGRATE_ASYNC &&
+	if (!migrate_mode_is_async(cc->mode) &&
 	    pfn > zone->compact_cached_migrate_pfn[1])
 		zone->compact_cached_migrate_pfn[1] = pfn;
 }
@@ -507,7 +507,7 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags,
 	__acquires(lock)
 {
 	/* Track if the lock is contended in async mode */
-	if (cc->mode == MIGRATE_ASYNC && !cc->contended) {
+	if (migrate_mode_is_async(cc->mode) && !cc->contended) {
 		if (spin_trylock_irqsave(lock, *flags))
 			return true;
 
@@ -864,7 +864,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 			return -EAGAIN;
 
 		/* async migration should just abort */
-		if (cc->mode == MIGRATE_ASYNC)
+		if (migrate_mode_is_async(cc->mode))
 			return -EAGAIN;
 
 		reclaim_throttle(pgdat, VMSCAN_THROTTLE_ISOLATED);
@@ -875,7 +875,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 
 	cond_resched();
 
-	if (cc->direct_compaction && (cc->mode == MIGRATE_ASYNC)) {
+	if (cc->direct_compaction && migrate_mode_is_async(cc->mode)) {
 		skip_on_failure = true;
 		next_skip_pfn = block_end_pfn(low_pfn, cc->order);
 	}
@@ -1364,7 +1364,7 @@ static bool suitable_migration_source(struct compact_control *cc,
 	if (pageblock_skip_persistent(page))
 		return false;
 
-	if ((cc->mode != MIGRATE_ASYNC) || !cc->direct_compaction)
+	if (!migrate_mode_is_async(cc->mode) || !cc->direct_compaction)
 		return true;
 
 	block_mt = get_pageblock_migratetype(page);
@@ -1465,7 +1465,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn)
 		return;
 
 	/* Minimise scanning during async compaction */
-	if (cc->direct_compaction && cc->mode == MIGRATE_ASYNC)
+	if (cc->direct_compaction && migrate_mode_is_async(cc->mode))
 		return;
 
 	/* Pageblock boundaries */
@@ -1705,7 +1705,7 @@ static void isolate_freepages(struct compact_control *cc)
 	block_end_pfn = min(block_start_pfn + pageblock_nr_pages,
 						zone_end_pfn(zone));
 	low_pfn = pageblock_end_pfn(cc->migrate_pfn);
-	stride = cc->mode == MIGRATE_ASYNC ? COMPACT_CLUSTER_MAX : 1;
+	stride = migrate_mode_is_async(cc->mode) ? COMPACT_CLUSTER_MAX : 1;
 
 	/*
 	 * Isolate free pages until enough are available to migrate the
@@ -2514,7 +2514,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
 	unsigned long start_pfn = cc->zone->zone_start_pfn;
 	unsigned long end_pfn = zone_end_pfn(cc->zone);
 	unsigned long last_migrated_pfn;
-	const bool sync = cc->mode != MIGRATE_ASYNC;
+	const bool sync = !migrate_mode_is_async(cc->mode);
 	bool update_cached;
 	unsigned int nr_succeeded = 0, nr_migratepages;
 	int order;
@@ -2537,7 +2537,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
 		ret = compaction_suit_allocation_order(cc->zone, cc->order,
 						       cc->highest_zoneidx,
 						       cc->alloc_flags,
-						       cc->mode == MIGRATE_ASYNC,
+						       migrate_mode_is_async(cc->mode),
 						       !cc->direct_compaction);
 		if (ret != COMPACT_CONTINUE)
 			return ret;
diff --git a/mm/migrate.c b/mm/migrate.c
index 2c3d489ecf51..ff6cf50e7b0b 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -907,7 +907,7 @@ static bool buffer_migrate_lock_buffers(struct buffer_head *head,
 
 	do {
 		if (!trylock_buffer(bh)) {
-			if (mode == MIGRATE_ASYNC)
+			if (migrate_mode_is_async(mode))
 				goto unlock;
 			if (mode == MIGRATE_SYNC_LIGHT && !buffer_uptodate(bh))
 				goto unlock;
@@ -1220,7 +1220,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 	dst->private = NULL;
 
 	if (!folio_trylock(src)) {
-		if (mode == MIGRATE_ASYNC)
+		if (migrate_mode_is_async(mode))
 			goto out;
 
 		/*
@@ -1325,7 +1325,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 		/* Establish migration ptes */
 		VM_BUG_ON_FOLIO(folio_test_anon(src) &&
 			       !folio_test_ksm(src) && !anon_vma, src);
-		try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0);
+		try_to_migrate(src, migrate_mode_is_async(mode) ? TTU_BATCH_FLUSH : 0);
 		old_page_state |= PAGE_WAS_MAPPED;
 	}
 
@@ -1565,7 +1565,7 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f
 {
 	int rc;
 
-	if (mode == MIGRATE_ASYNC) {
+	if (migrate_mode_is_async(mode)) {
 		if (!folio_trylock(folio))
 			return -EAGAIN;
 	} else {
@@ -1799,7 +1799,7 @@ static int migrate_pages_batch(struct list_head *from,
 	LIST_HEAD(dst_folios);
 	bool nosplit = (reason == MR_NUMA_MISPLACED);
 
-	VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
+	VM_WARN_ON_ONCE(!migrate_mode_is_async(mode) &&
 			!list_empty(from) && !list_is_singular(from));
 
 	for (pass = 0; pass < nr_pass && retry; pass++) {
@@ -2107,7 +2107,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 		list_cut_before(&folios, from, &folio2->lru);
 	else
 		list_splice_init(from, &folios);
-	if (mode == MIGRATE_ASYNC)
+	if (migrate_mode_is_async(mode))
 		rc = migrate_pages_batch(&folios, get_new_folio, put_new_folio,
 				private, mode, reason, &ret_folios,
 				&split_folios, &stats,

-- 
2.43.0

[PATCH RFC v2 3/3] mm: use non-temporal stores for demotion

[Yiannis Nikolakopoulos]

From: Alirad Malek <alirad.malek@zptcorp.com>

Memory demoted to a lower tier is assumed to be cold and most likely out of
the CPU's last level cache. Additionally, in certain demotion targets (e.g.
CXL devices with compressed memory) the bandwidth can be negatively
impacted by the eviction patterns of the last level cache when standard
memcpy is used. In supported architectures (currently X86_64), use the
MIGRATE_ASYNC_NON_TEMPORAL_STORES flag in demotions to trigger the folio
copy path using non-temporal stores.

Signed-off-by: Alirad Malek <alirad.malek@zptcorp.com>
Co-developed-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
Signed-off-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
---
 mm/migrate.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index ff6cf50e7b0b..c7c62700d541 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -862,7 +862,10 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
 	if (folio_ref_count(src) != expected_count)
 		return -EAGAIN;
 
-	rc = folio_mc_copy(dst, src);
+	if (mode == MIGRATE_ASYNC_NON_TEMPORAL_STORES)
+		rc = folio_mc_copy_nt(dst, src);
+	else
+		rc = folio_mc_copy(dst, src);
 	if (unlikely(rc))
 		return rc;
 
@@ -2081,6 +2084,11 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	LIST_HEAD(split_folios);
 	struct migrate_pages_stats stats;
 
+	if (IS_ENABLED(CONFIG_X86_64) &&
+		IS_ENABLED(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) &&
+		reason == MR_DEMOTION && mode == MIGRATE_ASYNC)
+		mode = MIGRATE_ASYNC_NON_TEMPORAL_STORES;
+
 	trace_mm_migrate_pages_start(mode, reason);
 
 	memset(&stats, 0, sizeof(stats));

-- 
2.43.0

Re: [PATCH RFC v2 0/3] Demote to lower tier using non-temporal stores

[Gregory Price]

On Thu, Jul 30, 2026 at 05:02:56PM +0200, Yiannis Nikolakopoulos wrote:
> In most memory tiering scenarios, the memory to be demoted is expected
> to be cold and most likely out of the node's last-level cache (as well
> as target pages in the target node). Using non-temporal stores instead
> of a standard memcpy path can reduce the cache pollution in the local
> node and the bandwidth overhead to the target node. Furthermore, for
> certain types of CXL devices that support in-line memory compression,
> the last-level cache eviction patterns can negatively affect the
> bandwidth of the device. Non-temporal stores can mitigate this.
> 
> This patch-set introduces a new migrate_mode flag for using non-temporal
> stores that is used only in the demotion path. Patch 1 adds some helpers in
> x86 and mm to bring non-temporal stores support to a respective folio_copy
> function. Patch 2 adds the new flag and necessary changes for compatibility
> with the existing behavior. Patch 3 uses the new flag for demotions.
> 
> Experimental data: in a CXL system with 1 memory expander, a microbenchmark
> that allocates N=64 GB memory in the local node and then triggers demotion
> using memory.reclaim, shows a practically complete elimination of read
> traffic on the device, i.e. write traffic is N GB with and without the
> patch, while read traffic drops from N to almost 0 with the patch.
> 
> Opens:
> 1. There is some "duplication" in the x86 tree and a bit in mm. Can we do
>    something better there? As it is now in copy_mc_to_kernel_nt we
> duplicate the machine check functionality, which if available will override
> the non-temporal. We were not sure how to prioritize these two and what's
> the best approach here. Can we completely skip the machine checked for this
> path? Huan Nguyen has some ideas here that we will align for the next
> version.
> 2. I am not sure how this should be structured so that it is easily
> adopted in other architecture trees (e.g. aarch64). We rely on
> memcpy_flushcache for x86_64 but this does not use non temporal stores
> in ARM. ARM support is currently out of our scope but any input is
> appreciated.
> 

I'm still a bit confused why using NT Stores needs to be an explicit
option - rather than the default behavior if NT Store is available.

Lets assume we did this for all ASYNC requests, is there a negative
effect? A positive effect? Why the new ASYNC type? Is there a
correctness issue?

~Gregory

Re: [PATCH RFC v2 0/3] Demote to lower tier using non-temporal stores

[Frank van der Linden]

On Thu, Jul 30, 2026 at 10:30 AM Gregory Price <gourry@gourry.net> wrote:
>
> On Thu, Jul 30, 2026 at 05:02:56PM +0200, Yiannis Nikolakopoulos wrote:
> > In most memory tiering scenarios, the memory to be demoted is expected
> > to be cold and most likely out of the node's last-level cache (as well
> > as target pages in the target node). Using non-temporal stores instead
> > of a standard memcpy path can reduce the cache pollution in the local
> > node and the bandwidth overhead to the target node. Furthermore, for
> > certain types of CXL devices that support in-line memory compression,
> > the last-level cache eviction patterns can negatively affect the
> > bandwidth of the device. Non-temporal stores can mitigate this.
> >
> > This patch-set introduces a new migrate_mode flag for using non-temporal
> > stores that is used only in the demotion path. Patch 1 adds some helpers in
> > x86 and mm to bring non-temporal stores support to a respective folio_copy
> > function. Patch 2 adds the new flag and necessary changes for compatibility
> > with the existing behavior. Patch 3 uses the new flag for demotions.
> >
> > Experimental data: in a CXL system with 1 memory expander, a microbenchmark
> > that allocates N=64 GB memory in the local node and then triggers demotion
> > using memory.reclaim, shows a practically complete elimination of read
> > traffic on the device, i.e. write traffic is N GB with and without the
> > patch, while read traffic drops from N to almost 0 with the patch.
> >
> > Opens:
> > 1. There is some "duplication" in the x86 tree and a bit in mm. Can we do
> >    something better there? As it is now in copy_mc_to_kernel_nt we
> > duplicate the machine check functionality, which if available will override
> > the non-temporal. We were not sure how to prioritize these two and what's
> > the best approach here. Can we completely skip the machine checked for this
> > path? Huan Nguyen has some ideas here that we will align for the next
> > version.
> > 2. I am not sure how this should be structured so that it is easily
> > adopted in other architecture trees (e.g. aarch64). We rely on
> > memcpy_flushcache for x86_64 but this does not use non temporal stores
> > in ARM. ARM support is currently out of our scope but any input is
> > appreciated.
> >
>
> I'm still a bit confused why using NT Stores needs to be an explicit
> option - rather than the default behavior if NT Store is available.
>
> Lets assume we did this for all ASYNC requests, is there a negative
> effect? A positive effect? Why the new ASYNC type? Is there a
> correctness issue?
>
> ~Gregory

Yes, this is a good discussion to have. I believe that the initial
reason for using a separate mode here was to be non-invasive, e.g.
"don't break anything else".

But why not use non-temporal stores for everything? I don't know. I
suppose one argument might be that if you know that the destination
will be used immediately, NT stores might be a slight performance hit.

arm64 already seems to use stores with NT hints by default.

In general, though, I agree that for ASYNC requests, just always using
NT seems fine.

Maybe the mode and reason should be folded in to one variable, so
that, further down the stack, a decision can be made as to what type
of copy to use? E.g. if the mode is !MIGRATE_ASYNC and the reason is
MR_DEMOTION, then non-temporal is still a good choice. If the mode is
MIGRATE_ASYNC, NT is still a good idea. If you wanted to get fancy,
any mode with folio_test_waiters(folio) == true should not use NT,
since a task is waiting to use the data, so caching it is better.
Maybe that's overthinking it.

- Frank

Re: [PATCH RFC v2 0/3] Demote to lower tier using non-temporal stores

[Johannes Weiner]

On Thu, Jul 30, 2026 at 10:40:32AM -0700, Frank van der Linden wrote:
> On Thu, Jul 30, 2026 at 10:30 AM Gregory Price <gourry@gourry.net> wrote:
> >
> > On Thu, Jul 30, 2026 at 05:02:56PM +0200, Yiannis Nikolakopoulos wrote:
> > > In most memory tiering scenarios, the memory to be demoted is expected
> > > to be cold and most likely out of the node's last-level cache (as well
> > > as target pages in the target node). Using non-temporal stores instead
> > > of a standard memcpy path can reduce the cache pollution in the local
> > > node and the bandwidth overhead to the target node. Furthermore, for
> > > certain types of CXL devices that support in-line memory compression,
> > > the last-level cache eviction patterns can negatively affect the
> > > bandwidth of the device. Non-temporal stores can mitigate this.
> > >
> > > This patch-set introduces a new migrate_mode flag for using non-temporal
> > > stores that is used only in the demotion path. Patch 1 adds some helpers in
> > > x86 and mm to bring non-temporal stores support to a respective folio_copy
> > > function. Patch 2 adds the new flag and necessary changes for compatibility
> > > with the existing behavior. Patch 3 uses the new flag for demotions.
> > >
> > > Experimental data: in a CXL system with 1 memory expander, a microbenchmark
> > > that allocates N=64 GB memory in the local node and then triggers demotion
> > > using memory.reclaim, shows a practically complete elimination of read
> > > traffic on the device, i.e. write traffic is N GB with and without the
> > > patch, while read traffic drops from N to almost 0 with the patch.
> > >
> > > Opens:
> > > 1. There is some "duplication" in the x86 tree and a bit in mm. Can we do
> > >    something better there? As it is now in copy_mc_to_kernel_nt we
> > > duplicate the machine check functionality, which if available will override
> > > the non-temporal. We were not sure how to prioritize these two and what's
> > > the best approach here. Can we completely skip the machine checked for this
> > > path? Huan Nguyen has some ideas here that we will align for the next
> > > version.
> > > 2. I am not sure how this should be structured so that it is easily
> > > adopted in other architecture trees (e.g. aarch64). We rely on
> > > memcpy_flushcache for x86_64 but this does not use non temporal stores
> > > in ARM. ARM support is currently out of our scope but any input is
> > > appreciated.
> > >
> >
> > I'm still a bit confused why using NT Stores needs to be an explicit
> > option - rather than the default behavior if NT Store is available.
> >
> > Lets assume we did this for all ASYNC requests, is there a negative
> > effect? A positive effect? Why the new ASYNC type? Is there a
> > correctness issue?
> >
> > ~Gregory
> 
> Yes, this is a good discussion to have. I believe that the initial
> reason for using a separate mode here was to be non-invasive, e.g.
> "don't break anything else".
> 
> But why not use non-temporal stores for everything? I don't know. I
> suppose one argument might be that if you know that the destination
> will be used immediately, NT stores might be a slight performance hit.

do_numa_page() -> migrate_misplaced_folio() is such a case. It happens
literally in the access path to that data. MIGRATE_ASYNC just means
don't block, fail fast, because waiting for an IO-bound lock to avoid
a remote NUMA access would be a bad idea.

Compaction on the other hand does use MIGRATE_SYNC* and it's touching
data in PFN order, completely out of execution sequence. That could be
a good candidate for NT stores.

> arm64 already seems to use stores with NT hints by default.
> 
> In general, though, I agree that for ASYNC requests, just always using
> NT seems fine.
> 
> Maybe the mode and reason should be folded in to one variable, so
> that, further down the stack, a decision can be made as to what type
> of copy to use? E.g. if the mode is !MIGRATE_ASYNC and the reason is
> MR_DEMOTION, then non-temporal is still a good choice. If the mode is
> MIGRATE_ASYNC, NT is still a good idea. If you wanted to get fancy,
> any mode with folio_test_waiters(folio) == true should not use NT,
> since a task is waiting to use the data, so caching it is better.
> Maybe that's overthinking it.

Agree. `reason' seems like a much stronger signal.

Re: [PATCH RFC v2 0/3] Demote to lower tier using non-temporal stores

[Gregory Price]

On Thu, Jul 30, 2026 at 02:08:56PM -0400, Johannes Weiner wrote:
> On Thu, Jul 30, 2026 at 10:40:32AM -0700, Frank van der Linden wrote:
> > On Thu, Jul 30, 2026 at 10:30 AM Gregory Price <gourry@gourry.net> wrote:
> 
> > arm64 already seems to use stores with NT hints by default.
> > 
> > In general, though, I agree that for ASYNC requests, just always using
> > NT seems fine.
> > 
> > Maybe the mode and reason should be folded in to one variable, so
> > that, further down the stack, a decision can be made as to what type
> > of copy to use? E.g. if the mode is !MIGRATE_ASYNC and the reason is
> > MR_DEMOTION, then non-temporal is still a good choice. If the mode is
> > MIGRATE_ASYNC, NT is still a good idea. If you wanted to get fancy,
> > any mode with folio_test_waiters(folio) == true should not use NT,
> > since a task is waiting to use the data, so caching it is better.
> > Maybe that's overthinking it.
> 
> Agree. `reason' seems like a much stronger signal.

I think this ends up cleaner as well, it would be a nice improvement.

~Gregory

Re: [PATCH RFC v2 1/3] mm, x86: support copying a folio using non-temporal stores

[Huan Nguyen]

Hi Yiannis,

Although copy_mc_to_kernel_nt() asserts both "mc" (machine check exception)
and "nt" (non-temporal store), the implementation fails to deliver both:
1. copy_mc_fragile() is MCE-safe, but it does not use movnti.
2. memcpy_flushcache() uses movnti, but it is not MCE-safe.

Additionally, according to the Intel manual, since movnti uses the WC protocol
(a weakly-ordered memory model), a fencing operation is required. Because
memcpy_flushcache() lacks an sfence, it can result in reading stale or corrupted
data after folio_unlock().

Furthermore, copy_mc_highpage() is unoptimized for page migration operations
because it reuses copy_mc_to_kernel(), built to handle
variable-length, unaligned
memory. As copy_mc_highpage_nt() follows copy_mc_highpage(), it inherits this
flaw. In contrast, copy_highpage() is optimized for page migration by leveraging
copy_page(), dedicated to 4KiB-aligned memory.

Therefore, copy_mc_highpage_nt() should call a dedicated page-copy assembly
routine that provides MCE safety, movnti stores, and an sfence, rather than the
generic copy_mc_to_kernel_nt(). We can name it copy_mc_page_nt(), following
the naming convention of copy_page().

What are your thoughts on this direction? Any feedback or suggestions would be
greatly appreciated.

Huan Nguyen

On Thu, Jul 30, 2026 at 8:03 AM Yiannis Nikolakopoulos
<yiannis.nikolakop@gmail.com> wrote:
>
> From: Alirad Malek <alirad.malek@zptcorp.com>
>
> In x86, use memcpy_flushcache (that uses non-temporal store
> instructions) to copy a folio. To achieve that, starting from folio_mc_copy
> down to copy_mc_to_kernel, create a series of helpers (named with an _nt
> suffix) that have similar behavior to the original counterparts.
>
> Signed-off-by: Alirad Malek <alirad.malek@zptcorp.com>
> Co-developed-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
> Signed-off-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
> ---
>  arch/x86/include/asm/uaccess.h |  4 ++++
>  arch/x86/lib/copy_mc.c         | 26 ++++++++++++++++++++++++++
>  include/linux/highmem.h        | 32 ++++++++++++++++++++++++++++++++
>  include/linux/mm.h             |  1 +
>  mm/util.c                      | 17 +++++++++++++++++
>  5 files changed, 80 insertions(+)
>
> diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h
> index 367297b188c3..2d0938d3e372 100644
> --- a/arch/x86/include/asm/uaccess.h
> +++ b/arch/x86/include/asm/uaccess.h
> @@ -494,6 +494,10 @@ unsigned long __must_check
>  copy_mc_to_kernel(void *to, const void *from, unsigned len);
>  #define copy_mc_to_kernel copy_mc_to_kernel
>
> +unsigned long __must_check
> +copy_mc_to_kernel_nt(void *to, const void *from, unsigned len);
> +#define copy_mc_to_kernel_nt copy_mc_to_kernel_nt
> +
>  unsigned long __must_check
>  copy_mc_to_user(void __user *to, const void *from, unsigned len);
>  #endif
> diff --git a/arch/x86/lib/copy_mc.c b/arch/x86/lib/copy_mc.c
> index 97e88e58567b..5a2ee5c2211e 100644
> --- a/arch/x86/lib/copy_mc.c
> +++ b/arch/x86/lib/copy_mc.c
> @@ -81,6 +81,32 @@ unsigned long __must_check copy_mc_to_kernel(void *dst, const void *src, unsigne
>  }
>  EXPORT_SYMBOL_GPL(copy_mc_to_kernel);
>
> +/**
> + * copy_mc_to_kernel_nt - memory copy that handles source exceptions
> + * if enabled, otherwise uses non-temporal stores
> + * @dst: destination address
> + * @src: source address
> + * @len: number of bytes to copy
> + *
> + * Return 0 for success, or number of bytes not copied if there was an
> + * exception.
> + */
> +unsigned long __must_check copy_mc_to_kernel_nt(void *dst, const void *src, unsigned len)
> +{
> +       unsigned long ret;
> +
> +       if (copy_mc_fragile_enabled) {
> +               instrument_memcpy_before(dst, src, len);
> +               ret = copy_mc_fragile(dst, src, len);
> +               instrument_memcpy_after(dst, src, len, ret);
> +               return ret;
> +       }
> +
> +       memcpy_flushcache(dst, src, len);
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(copy_mc_to_kernel_nt);
> +
>  unsigned long __must_check copy_mc_to_user(void __user *dst, const void *src, unsigned len)
>  {
>         unsigned long ret;
> diff --git a/include/linux/highmem.h b/include/linux/highmem.h
> index af03db851a1d..a5cb435b9ffe 100644
> --- a/include/linux/highmem.h
> +++ b/include/linux/highmem.h
> @@ -468,6 +468,32 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
>
>         return ret;
>  }
> +
> +#ifdef copy_mc_to_kernel_nt
> +static inline int copy_mc_highpage_nt(struct page *to, struct page *from)
> +{
> +       unsigned long ret;
> +       char *vfrom, *vto;
> +
> +       vfrom = kmap_local_page(from);
> +       vto = kmap_local_page(to);
> +       ret = copy_mc_to_kernel_nt(vto, vfrom, PAGE_SIZE);
> +       if (!ret)
> +               kmsan_copy_page_meta(to, from);
> +       kunmap_local(vto);
> +       kunmap_local(vfrom);
> +
> +       if (ret)
> +               memory_failure_queue(page_to_pfn(from), 0);
> +
> +       return ret;
> +}
> +#else
> +static inline int copy_mc_highpage_nt(struct page *to, struct page *from)
> +{
> +       return copy_mc_highpage(to, from);
> +}
> +#endif
>  #else
>  static inline int copy_mc_user_highpage(struct page *to, struct page *from,
>                                         unsigned long vaddr, struct vm_area_struct *vma)
> @@ -481,6 +507,12 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
>         copy_highpage(to, from);
>         return 0;
>  }
> +
> +static inline int copy_mc_highpage_nt(struct page *to, struct page *from)
> +{
> +       copy_highpage(to, from);
> +       return 0;
> +}
>  #endif
>
>  static inline void memcpy_page(struct page *dst_page, size_t dst_off,
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 5be3d8a8f806..d07ce478582d 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -1644,6 +1644,7 @@ void __folio_put(struct folio *folio);
>  void split_page(struct page *page, unsigned int order);
>  void folio_copy(struct folio *dst, struct folio *src);
>  int folio_mc_copy(struct folio *dst, struct folio *src);
> +int folio_mc_copy_nt(struct folio *dst, struct folio *src);
>
>  unsigned long nr_free_buffer_pages(void);
>
> diff --git a/mm/util.c b/mm/util.c
> index b05ab6f97e11..e09e9b5f8eee 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -749,6 +749,23 @@ int folio_mc_copy(struct folio *dst, struct folio *src)
>  }
>  EXPORT_SYMBOL(folio_mc_copy);
>
> +int folio_mc_copy_nt(struct folio *dst, struct folio *src)
> +{
> +       long nr = folio_nr_pages(src);
> +       long i = 0;
> +
> +       for (;;) {
> +               if (copy_mc_highpage_nt(folio_page(dst, i), folio_page(src, i)))
> +                       return -EHWPOISON;
> +               if (++i == nr)
> +                       break;
> +               cond_resched();
> +       }
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL(folio_mc_copy_nt);
> +
>  int sysctl_overcommit_memory __read_mostly = OVERCOMMIT_GUESS;
>  static int sysctl_overcommit_ratio __read_mostly = 50;
>  static unsigned long sysctl_overcommit_kbytes __read_mostly;
>
> --
> 2.43.0
>

Re: [PATCH RFC v2 2/3] mm: new migrate_mode flag for async using non-temporal stores

[Huang, Ying]

Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com> writes:

[snip]

> diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h
> index 265c4328b36a..f7186e705b48 100644
> --- a/include/linux/migrate_mode.h
> +++ b/include/linux/migrate_mode.h
> @@ -3,6 +3,8 @@
>  #define MIGRATE_MODE_H_INCLUDED
>  /*
>   * MIGRATE_ASYNC means never block
> + * MIGRATE_ASYNC_NON_TEMPORAL_STORES means never block and use non-temporal
> + * stores if supported by the architecture
>   * MIGRATE_SYNC_LIGHT in the current implementation means to allow blocking
>   *	on most operations but not ->writepage as the potential stall time
>   *	is too significant
> @@ -10,10 +12,17 @@
>   */
>  enum migrate_mode {
>  	MIGRATE_ASYNC,
> +	MIGRATE_ASYNC_NON_TEMPORAL_STORES,
>  	MIGRATE_SYNC_LIGHT,
>  	MIGRATE_SYNC,
>  };

Is it better to turn migrate_mode parameter into migrate_flags?

---
Best Regards,
Huang, Ying

Re: [PATCH RFC v2 1/3] mm, x86: support copying a folio using non-temporal stores

[Garg, Shivank]

On Fri, 2026-07-31 at 14:53 -0700, Huan Nguyen wrote:
> Hi Yiannis,
> 
> Although copy_mc_to_kernel_nt() asserts both "mc" (machine check exception)
> and "nt" (non-temporal store), the implementation fails to deliver both:
> 1. copy_mc_fragile() is MCE-safe, but it does not use movnti.
> 2. memcpy_flushcache() uses movnti, but it is not MCE-safe.
> 
> Additionally, according to the Intel manual, since movnti uses the WC protocol
> (a weakly-ordered memory model), a fencing operation is required. Because
> memcpy_flushcache() lacks an sfence, it can result in reading stale or corrupted
> data after folio_unlock().
> 
> Furthermore, copy_mc_highpage() is unoptimized for page migration operations
> because it reuses copy_mc_to_kernel(), built to handle
> variable-length, unaligned
> memory. As copy_mc_highpage_nt() follows copy_mc_highpage(), it inherits this
> flaw. In contrast, copy_highpage() is optimized for page migration by leveraging
> copy_page(), dedicated to 4KiB-aligned memory.
> 

I don't think copy_mc_highpage() is unoptimized for page migration copies.

On ERMS systems, copy_mc_to_kernel() uses copy_mc_enhanced_fast_string(),
which is rep movsb. copy_page() uses rep movsq on REP_GOOD systems.

(The copy_mc_fragile_enabled path looks like a special case and
enable_copy_mc_fragile() is reachable either when mce=recovery or through
one of 4 intel CPUs in x86/kernel/quirks.c)

In my testing on AMD EPYC 9655 (ZEN 5), both folio_mc_copy() and
folio_copy() showed similar performance.

https://lore.kernel.org/all/20260427142036.111940-2-shivankg@amd.com

And there is actually scope of upto ~2X speedup for
folio_copy/folio_mc_copy for large pages if we batch 4KiB copies. 

Thanks,
Shivank

[RFC PATCH v3 0/3] mm: use non-temporal stores for demotion

[Huan Nguyen]

This series continues the non-temporal demotion work originally posted
by Yiannis Nikolakopoulos [1].

Motivation
==========
Memory tier demotion moves cold folios from local DRAM to slower memory
tiers, such as CXL or remote NUMA nodes. Standard store instructions
force the CPU to allocate cache lines for cold data that is unlikely to
be referenced in the near term.

Using non-temporal stores and prefetching on x86 bypasses CPU cache
allocation, protecting active workload working sets and improving
demotion throughput.

Changes in v3
=============
- Unified MCE-safe NT copy (addressing v2 Open #1):
  Implement copy_mc_page_nt() in x86 assembly to provide machine-check
  safe copying using non-temporal stores (movnti) and prefetchnta.

- Architecture abstraction (addressing v2 Open #2):
  v2 relied on memcpy_flushcache(), which lacks non-temporal stores on
  architectures like arm64. RFC v3 replaces this with a dedicated
  __HAVE_ARCH_COPY_PAGE_NT interface and generic fallback, enabling
  architectures to plug in native non-temporal routines independently.

- Memory ordering:
  Add a wmb() barrier at the end of folio_mc_copy_nt() to ensure weakly
  ordered non-temporal stores are drained before page tables are updated.

Discussions
===========
1. Migration mode vs. migration control refactoring:
   Currently, this series introduces MIGRATE_ASYNC_NONTEMPORAL into
   enum migrate_mode. In parallel, Shivank Garg's migration refactoring
   series [2] consolidates migration arguments into migrate_control.
   Once that lands, demotion can simply rely on the migration reason
   instead of adding a dedicated migration mode and helper.

2. Large folio copy optimization:
   Multi-page folios (such as 2MiB THP) are supported by iterating over
   constituent 4KiB pages via copy_mc_page_nt(). Once the batch page
   copy series [3] lands, copy_mc_page_nt() can be extended to copy an
   entire multi-page folio in a single invocation, eliminating per-4KiB
   loop overhead and keeping the prefetchnta stream uninterrupted.

Thanks to Yiannis Nikolakopoulos, Alirad Malek, and Dimitrios
Palyvos-Giannas for their original work, feedback, and collaboration.

References
==========
[1] https://lore.kernel.org/linux-mm/20260730-rfc-nt-demote-v2-0-452dbe3b5073@zptcorp.com/
[2] https://lore.kernel.org/linux-mm/20260902-migrate-refactor-shivank-v1-9-9dcca87669c4@amd.com/
[3] https://lore.kernel.org/linux-mm/20261019-batpagecopy-v2-1-d75e7d0dfaae@amd.com/

Signed-off-by: Huan Nguyen <nhuhuan@google.com>

Huan Nguyen (3):
  mm, x86: support copying a folio using non-temporal stores
  mm/migrate: introduce async non-temporal migration mode
  mm/vmscan: demote folios using non-temporal stores

 arch/x86/include/asm/page_64.h |  2 ++
 arch/x86/lib/copy_page_64.S    | 65 ++++++++++++++++++++++++++++++++++
 fs/nfs/write.c                 |  2 +-
 include/linux/highmem.h        | 21 +++++++++++
 include/linux/migrate.h        |  5 +++
 include/linux/migrate_mode.h   |  1 +
 include/linux/mm.h             |  1 +
 include/trace/events/migrate.h |  1 +
 mm/compaction.c                | 18 +++++-----
 mm/migrate.c                   | 29 +++++++++------
 mm/util.c                      | 29 +++++++++++++++
 mm/vmscan.c                    |  2 +-
 12 files changed, 154 insertions(+), 22 deletions(-)

base-commit: 05441d697f7f9a927264f4058cbbc6a03dfcac9d
-- 
2.56.0.398.g882cbad275-goog

[RFC PATCH v3 1/3] mm, x86: support copying a folio using non-temporal stores

[Huan Nguyen]

On x86_64 architecture, copying cold memory folios using standard store
instructions pollutes CPU caches with cold data, increasing memory
latency and bandwidth overhead.

Support machine-check safe non-temporal copying of folios:
- Add copy_mc_page_nt() to copy a 4KiB page using non-temporal stores,
  prefetching, and MCE exception handling.
- Add copy_mc_highpage_nt() to map and copy a 4KiB page.
- Add folio_mc_copy_nt() to copy a folio using copy_mc_highpage_nt().

Because non-temporal stores are weakly ordered, a wmb() is issued at the
end of folio_mc_copy_nt() before page tables are updated to ensure the
new folio contents are globally visible. In the event of task preemption
and CPU migration mid-folio, the scheduler's locked instructions on the
departing CPU serialize and drain earlier stores, while the final wmb()
covers the remainder on the new CPU.

Suggested-by: Alirad Malek <alirad.malek@zptcorp.com>
Suggested-by: Dimitrios Palyvos-Giannas <dimitrios.palyvos@zptcorp.com>
Signed-off-by: Huan Nguyen <nhuhuan@google.com>
---
 arch/x86/include/asm/page_64.h |  2 ++
 arch/x86/lib/copy_page_64.S    | 65 ++++++++++++++++++++++++++++++++++
 include/linux/highmem.h        | 21 +++++++++++
 include/linux/mm.h             |  1 +
 mm/util.c                      | 29 +++++++++++++++
 5 files changed, 118 insertions(+)

diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h
index 1895c207f629..f37126b56239 100644
--- a/arch/x86/include/asm/page_64.h
+++ b/arch/x86/include/asm/page_64.h
@@ -114,7 +114,9 @@ static inline void clear_page(void *addr)
 	clear_pages(addr, 1);
 }
 
+#define __HAVE_ARCH_COPY_PAGE_NT 1
 void copy_page(void *to, void *from);
+unsigned long copy_mc_page_nt(void *to, const void *from);
 KCFI_REFERENCE(copy_page);
 
 /*
diff --git a/arch/x86/lib/copy_page_64.S b/arch/x86/lib/copy_page_64.S
index d8e87fedc20d..185fa3bc34c0 100644
--- a/arch/x86/lib/copy_page_64.S
+++ b/arch/x86/lib/copy_page_64.S
@@ -6,6 +6,7 @@
 #include <linux/cfi_types.h>
 #include <asm/cpufeatures.h>
 #include <asm/alternative.h>
+#include <asm/asm.h>
 
 /*
  * Some CPUs run faster using the string copy instructions (sane microcode).
@@ -88,3 +89,67 @@ SYM_FUNC_START_LOCAL(copy_page_regs)
 	addq	$2*8, %rsp
 	RET
 SYM_FUNC_END(copy_page_regs)
+
+/*
+ * Copy an entire 4KB page using non-temporal stores (movnti).
+ * Returns 0 on success, or number of bytes not copied on error.
+ */
+SYM_TYPED_FUNC_START(copy_mc_page_nt)
+	subq	$2*8, %rsp
+	movq	%rbx, (%rsp)
+	movq	%r12, 1*8(%rsp)
+
+	xorl	%eax, %eax
+	.p2align 4,,10
+	.p2align 3
+
+.L_nt0:	movq	0x8*0(%rsi,%rax), %rdx
+.L_nt1:	movq	0x8*1(%rsi,%rax), %rcx
+.L_nt2:	movq	0x8*2(%rsi,%rax), %rbx
+.L_nt3:	movq	0x8*3(%rsi,%rax), %r8
+.L_nt4:	movq	0x8*4(%rsi,%rax), %r9
+.L_nt5:	movq	0x8*5(%rsi,%rax), %r10
+.L_nt6:	movq	0x8*6(%rsi,%rax), %r11
+.L_nt7:	movq	0x8*7(%rsi,%rax), %r12
+
+	prefetchnta 5*64(%rsi,%rax)
+
+	movnti	%rdx, 0x8*0(%rdi,%rax)
+	movnti	%rcx, 0x8*1(%rdi,%rax)
+	movnti	%rbx, 0x8*2(%rdi,%rax)
+	movnti	%r8,  0x8*3(%rdi,%rax)
+	movnti	%r9,  0x8*4(%rdi,%rax)
+	movnti	%r10, 0x8*5(%rdi,%rax)
+	movnti	%r11, 0x8*6(%rdi,%rax)
+	movnti	%r12, 0x8*7(%rdi,%rax)
+
+	addq	$64, %rax
+	cmpq	$4096, %rax
+	jne	.L_nt0
+
+	movq	(%rsp), %rbx
+	movq	1*8(%rsp), %r12
+	addq	$2*8, %rsp
+	xorl	%eax, %eax
+	RET
+
+.E_nt:
+	negl	%eax
+	addl	$4096, %eax
+	sfence
+	movq	(%rsp), %rbx
+	movq	1*8(%rsp), %r12
+	addq	$2*8, %rsp
+	RET
+
+	_ASM_EXTABLE_TYPE(.L_nt0, .E_nt, EX_TYPE_DEFAULT_MCE_SAFE)
+	_ASM_EXTABLE_TYPE(.L_nt1, .E_nt, EX_TYPE_DEFAULT_MCE_SAFE)
+	_ASM_EXTABLE_TYPE(.L_nt2, .E_nt, EX_TYPE_DEFAULT_MCE_SAFE)
+	_ASM_EXTABLE_TYPE(.L_nt3, .E_nt, EX_TYPE_DEFAULT_MCE_SAFE)
+	_ASM_EXTABLE_TYPE(.L_nt4, .E_nt, EX_TYPE_DEFAULT_MCE_SAFE)
+	_ASM_EXTABLE_TYPE(.L_nt5, .E_nt, EX_TYPE_DEFAULT_MCE_SAFE)
+	_ASM_EXTABLE_TYPE(.L_nt6, .E_nt, EX_TYPE_DEFAULT_MCE_SAFE)
+	_ASM_EXTABLE_TYPE(.L_nt7, .E_nt, EX_TYPE_DEFAULT_MCE_SAFE)
+
+SYM_FUNC_END(copy_mc_page_nt)
+EXPORT_SYMBOL_GPL(copy_mc_page_nt)
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index d7aac9de1c8a..264b3f109151 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -484,6 +484,27 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
 }
 #endif
 
+#ifdef __HAVE_ARCH_COPY_PAGE_NT
+static inline int copy_mc_highpage_nt(struct page *to, struct page *from)
+{
+	unsigned long ret;
+	char *vfrom, *vto;
+
+	vfrom = kmap_local_page(from);
+	vto = kmap_local_page(to);
+	ret = copy_mc_page_nt(vto, vfrom);
+	if (!ret)
+		kmsan_copy_page_meta(to, from);
+	kunmap_local(vto);
+	kunmap_local(vfrom);
+
+	if (ret)
+		memory_failure_queue(page_to_pfn(from), 0);
+
+	return ret;
+}
+#endif
+
 static inline void memcpy_page(struct page *dst_page, size_t dst_off,
 			       struct page *src_page, size_t src_off,
 			       size_t len)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 9f167950926a..549bb21027e4 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2127,6 +2127,7 @@ void __folio_put(struct folio *folio);
 void split_page(struct page *page, unsigned int order);
 void folio_copy(struct folio *dst, struct folio *src);
 int folio_mc_copy(struct folio *dst, struct folio *src);
+int folio_mc_copy_nt(struct folio *dst, struct folio *src);
 
 unsigned long nr_free_buffer_pages(void);
 
diff --git a/mm/util.c b/mm/util.c
index 8acfe2148ba5..0418529264fd 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -778,6 +778,35 @@ int folio_mc_copy(struct folio *dst, struct folio *src)
 }
 EXPORT_SYMBOL(folio_mc_copy);
 
+int folio_mc_copy_nt(struct folio *dst, struct folio *src)
+{
+	long nr = folio_nr_pages(src);
+	long i = 0;
+
+	for (;;) {
+#ifdef __HAVE_ARCH_COPY_PAGE_NT
+		if (copy_mc_highpage_nt(folio_page(dst, i), folio_page(src, i)))
+			return -EHWPOISON;
+#else
+		if (copy_mc_highpage(folio_page(dst, i), folio_page(src, i)))
+			return -EHWPOISON;
+#endif
+		if (++i == nr)
+			break;
+		cond_resched();
+	}
+
+#ifdef __HAVE_ARCH_COPY_PAGE_NT
+	/*
+	 * Drain weakly ordered non-temporal stores before the folio
+	 * is published.
+	 */
+	wmb();
+#endif
+	return 0;
+}
+EXPORT_SYMBOL(folio_mc_copy_nt);
+
 int sysctl_overcommit_memory __read_mostly = OVERCOMMIT_GUESS;
 static int sysctl_overcommit_ratio __read_mostly = 50;
 static unsigned long sysctl_overcommit_kbytes __read_mostly;
-- 
2.56.0.398.g882cbad275-goog

[RFC PATCH v3 2/3] mm/migrate: introduce async non-temporal migration mode

[Huan Nguyen]

Introduce MIGRATE_ASYNC_NONTEMPORAL to migrate cold folios using
non-temporal stores, avoiding CPU cache pollution.

Existing callers check "mode == MIGRATE_ASYNC" directly, which would
cause new asynchronous modes to fall back to synchronous handling.
Introduce migrate_is_async() and update callers across compaction,
NFS, and migration.

In __migrate_folio(), invoke folio_mc_copy_nt() when the migration
mode is MIGRATE_ASYNC_NONTEMPORAL.

Suggested-by: Alirad Malek <alirad.malek@zptcorp.com>
Suggested-by: Dimitrios Palyvos-Giannas <dimitrios.palyvos@zptcorp.com>
Signed-off-by: Huan Nguyen <nhuhuan@google.com>
---
 fs/nfs/write.c                 |  2 +-
 include/linux/migrate.h        |  5 +++++
 include/linux/migrate_mode.h   |  1 +
 include/trace/events/migrate.h |  1 +
 mm/compaction.c                | 18 +++++++++---------
 mm/migrate.c                   | 29 ++++++++++++++++++-----------
 6 files changed, 35 insertions(+), 21 deletions(-)

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 0b42d31f7a75..ad28ff48e541 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -2128,7 +2128,7 @@ int nfs_migrate_folio(struct address_space *mapping, struct folio *dst,
 	}
 
 	if (folio_test_private_2(src)) { /* [DEPRECATED] */
-		if (mode == MIGRATE_ASYNC)
+		if (migrate_is_async(mode))
 			return -EBUSY;
 		folio_wait_private_2(src);
 	}
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index 78424b3824c2..188f9d891a0a 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -51,6 +51,11 @@ struct movable_operations {
 /* Defined in mm/debug.c: */
 extern const char *migrate_reason_names[MR_TYPES];
 
+static inline bool migrate_is_async(enum migrate_mode mode)
+{
+	return mode == MIGRATE_ASYNC || mode == MIGRATE_ASYNC_NONTEMPORAL;
+}
+
 #ifdef CONFIG_MIGRATION
 
 void putback_movable_pages(struct list_head *l);
diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h
index 05102d4d2490..fd752fddc972 100644
--- a/include/linux/migrate_mode.h
+++ b/include/linux/migrate_mode.h
@@ -12,6 +12,7 @@ enum migrate_mode {
 	MIGRATE_ASYNC,
 	MIGRATE_SYNC_LIGHT,
 	MIGRATE_SYNC,
+	MIGRATE_ASYNC_NONTEMPORAL,
 };
 
 enum migrate_reason {
diff --git a/include/trace/events/migrate.h b/include/trace/events/migrate.h
index 15ee2ef201b5..0bb2444211af 100644
--- a/include/trace/events/migrate.h
+++ b/include/trace/events/migrate.h
@@ -9,6 +9,7 @@
 
 #define MIGRATE_MODE						\
 	EM( MIGRATE_ASYNC,	"MIGRATE_ASYNC")		\
+	EM( MIGRATE_ASYNC_NONTEMPORAL, "MIGRATE_ASYNC_NONTEMPORAL") \
 	EM( MIGRATE_SYNC_LIGHT,	"MIGRATE_SYNC_LIGHT")		\
 	EMe(MIGRATE_SYNC,	"MIGRATE_SYNC")
 
diff --git a/mm/compaction.c b/mm/compaction.c
index f1b2060eb201..c11739e6891f 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -445,7 +445,7 @@ static void update_cached_migrate(struct compact_control *cc, unsigned long pfn)
 	/* Update where async and sync compaction should restart */
 	if (pfn > zone->compact_cached_migrate_pfn[0])
 		zone->compact_cached_migrate_pfn[0] = pfn;
-	if (cc->mode != MIGRATE_ASYNC &&
+	if (!migrate_is_async(cc->mode) &&
 	    pfn > zone->compact_cached_migrate_pfn[1])
 		zone->compact_cached_migrate_pfn[1] = pfn;
 }
@@ -508,7 +508,7 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags,
 	__acquires(lock)
 {
 	/* Track if the lock is contended in async mode */
-	if (cc->mode == MIGRATE_ASYNC && !cc->contended) {
+	if (migrate_is_async(cc->mode) && !cc->contended) {
 		if (spin_trylock_irqsave(lock, *flags))
 			return true;
 
@@ -882,7 +882,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 			return -EAGAIN;
 
 		/* async migration should just abort */
-		if (cc->mode == MIGRATE_ASYNC)
+		if (migrate_is_async(cc->mode))
 			return -EAGAIN;
 
 		reclaim_throttle(pgdat, VMSCAN_THROTTLE_ISOLATED);
@@ -893,7 +893,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 
 	cond_resched();
 
-	if (cc->direct_compaction && (cc->mode == MIGRATE_ASYNC)) {
+	if (cc->direct_compaction && migrate_is_async(cc->mode)) {
 		skip_on_failure = true;
 		next_skip_pfn = block_end_pfn(low_pfn, cc->order);
 	}
@@ -1409,7 +1409,7 @@ static bool suitable_migration_source(struct compact_control *cc,
 	 * allocator fallbacks. For non-movable requests, this helps a
 	 * lot, as they can finally scan movable blocks.
 	 */
-	if (cc->mode != MIGRATE_ASYNC)
+	if (!migrate_is_async(cc->mode))
 		return true;
 
 	/*
@@ -1514,7 +1514,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn)
 		return;
 
 	/* Minimise scanning during async compaction */
-	if (cc->direct_compaction && cc->mode == MIGRATE_ASYNC)
+	if (cc->direct_compaction && migrate_is_async(cc->mode))
 		return;
 
 	/* Pageblock boundaries */
@@ -1753,7 +1753,7 @@ static void isolate_freepages(struct compact_control *cc)
 	block_end_pfn = min(block_start_pfn + pageblock_nr_pages,
 						zone_end_pfn(zone));
 	low_pfn = pageblock_end_pfn(cc->migrate_pfn);
-	stride = cc->mode == MIGRATE_ASYNC ? COMPACT_CLUSTER_MAX : 1;
+	stride = migrate_is_async(cc->mode) ? COMPACT_CLUSTER_MAX : 1;
 
 	/*
 	 * Isolate free pages until enough are available to migrate the
@@ -2565,7 +2565,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
 	unsigned long start_pfn = cc->zone->zone_start_pfn;
 	unsigned long end_pfn = zone_end_pfn(cc->zone);
 	unsigned long last_migrated_pfn;
-	const bool sync = cc->mode != MIGRATE_ASYNC;
+	const bool sync = !migrate_is_async(cc->mode);
 	bool update_cached;
 	unsigned int nr_succeeded = 0, nr_migratepages;
 	int order;
@@ -2588,7 +2588,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
 		ret = compaction_suit_allocation_order(cc->zone, cc->order,
 						       cc->highest_zoneidx,
 						       cc->alloc_flags,
-						       cc->mode == MIGRATE_ASYNC,
+						       migrate_is_async(cc->mode),
 						       !cc->direct_compaction);
 		if (ret != COMPACT_CONTINUE)
 			return ret;
diff --git a/mm/migrate.c b/mm/migrate.c
index 7bdcdb57652f..18470de10857 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -876,9 +876,15 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
 	if (folio_ref_count(src) != expected_count)
 		return -EAGAIN;
 
-	rc = folio_mc_copy(dst, src);
-	if (unlikely(rc))
-		return rc;
+	if (mode == MIGRATE_ASYNC_NONTEMPORAL) {
+		rc = folio_mc_copy_nt(dst, src);
+		if (unlikely(rc))
+			return rc;
+	} else {
+		rc = folio_mc_copy(dst, src);
+		if (unlikely(rc))
+			return rc;
+	}
 
 	rc = __folio_migrate_mapping(mapping, dst, src, expected_count);
 	if (rc)
@@ -921,7 +927,7 @@ static bool buffer_migrate_lock_buffers(struct buffer_head *head,
 
 	do {
 		if (!trylock_buffer(bh)) {
-			if (mode == MIGRATE_ASYNC)
+			if (migrate_is_async(mode))
 				goto unlock;
 			if (mode == MIGRATE_SYNC_LIGHT && !buffer_uptodate(bh))
 				goto unlock;
@@ -1229,7 +1235,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 	dst->migrate_info = 0;
 
 	if (!folio_trylock(src)) {
-		if (mode == MIGRATE_ASYNC)
+		if (migrate_is_async(mode))
 			goto out;
 
 		/*
@@ -1334,7 +1340,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 		/* Establish migration ptes */
 		VM_BUG_ON_FOLIO(folio_test_anon(src) &&
 			       !folio_test_ksm(src) && !anon_vma, src);
-		try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0);
+		try_to_migrate(src, migrate_is_async(mode) ? TTU_BATCH_FLUSH : 0);
 		old_folio_state |= FOLIO_WAS_MAPPED;
 	}
 
@@ -1592,7 +1598,7 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f
 {
 	int rc;
 
-	if (mode == MIGRATE_ASYNC) {
+	if (migrate_is_async(mode)) {
 		if (!folio_trylock(folio))
 			return -EAGAIN;
 	} else {
@@ -1826,7 +1832,7 @@ static int migrate_pages_batch(struct list_head *from,
 	LIST_HEAD(dst_folios);
 	bool nosplit = (reason == MR_NUMA_MISPLACED);
 
-	VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
+	VM_WARN_ON_ONCE(!migrate_is_async(mode) &&
 			!list_empty(from) && !list_is_singular(from));
 
 	for (pass = 0; pass < nr_pass && retry; pass++) {
@@ -2134,7 +2140,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 		list_cut_before(&folios, from, &folio2->lru);
 	else
 		list_splice_init(from, &folios);
-	if (mode == MIGRATE_ASYNC)
+	if (migrate_is_async(mode))
 		rc = migrate_pages_batch(&folios, get_new_folio, put_new_folio,
 				private, mode, reason, &ret_folios,
 				&split_folios, &stats,
@@ -2156,8 +2162,9 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 		 * with minimal effort, force MIGRATE_ASYNC mode and retry once.
 		 */
 		migrate_pages_batch(&split_folios, get_new_folio,
-				put_new_folio, private, MIGRATE_ASYNC, reason,
-				&ret_folios, NULL, &stats, 1);
+				put_new_folio, private,
+				migrate_is_async(mode) ? mode : MIGRATE_ASYNC,
+				reason, &ret_folios, NULL, &stats, 1);
 		list_splice_tail_init(&split_folios, &ret_folios);
 	}
 	rc_gather += rc;
-- 
2.56.0.398.g882cbad275-goog

[RFC PATCH v3 3/3] mm/vmscan: demote folios using non-temporal stores

[Huan Nguyen]

Demoting cold folios to slower memory tiers using standard stores
pollutes CPU caches and incurs memory latency and bandwidth overhead.

Update demote_folio_list() to use MIGRATE_ASYNC_NONTEMPORAL, avoiding
CPU cache allocation and improving demotion throughput.

Suggested-by: Alirad Malek <alirad.malek@zptcorp.com>
Suggested-by: Dimitrios Palyvos-Giannas <dimitrios.palyvos@zptcorp.com>
Signed-off-by: Huan Nguyen <nhuhuan@google.com>
---
 mm/vmscan.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9959ca158ce8..b44b7cdf8a03 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1134,7 +1134,7 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,
 
 	/* Demotion ignores all cpuset and mempolicy settings */
 	migrate_pages(demote_folios, alloc_demote_folio, NULL,
-		      (unsigned long)&mtc, MIGRATE_ASYNC, MR_DEMOTION,
+		      (unsigned long)&mtc, MIGRATE_ASYNC_NONTEMPORAL, MR_DEMOTION,
 		      &nr_succeeded);
 
 	return nr_succeeded;
-- 
2.56.0.398.g882cbad275-goog

Re: [RFC PATCH v3 0/3] mm: use non-temporal stores for demotion

[Zi Yan]

On 3 Nov 2026, at 17:30, Huan Nguyen wrote:

> This series continues the non-temporal demotion work originally posted
> by Yiannis Nikolakopoulos [1].
>
> Motivation
> ==========
> Memory tier demotion moves cold folios from local DRAM to slower memory
> tiers, such as CXL or remote NUMA nodes. Standard store instructions
> force the CPU to allocate cache lines for cold data that is unlikely to
> be referenced in the near term.
>
> Using non-temporal stores and prefetching on x86 bypasses CPU cache
> allocation, protecting active workload working sets and improving
> demotion throughput.
>
> Changes in v3
> =============
> - Unified MCE-safe NT copy (addressing v2 Open #1):
>   Implement copy_mc_page_nt() in x86 assembly to provide machine-check
>   safe copying using non-temporal stores (movnti) and prefetchnta.
>
> - Architecture abstraction (addressing v2 Open #2):
>   v2 relied on memcpy_flushcache(), which lacks non-temporal stores on
>   architectures like arm64. RFC v3 replaces this with a dedicated
>   __HAVE_ARCH_COPY_PAGE_NT interface and generic fallback, enabling
>   architectures to plug in native non-temporal routines independently.
>
> - Memory ordering:
>   Add a wmb() barrier at the end of folio_mc_copy_nt() to ensure weakly
>   ordered non-temporal stores are drained before page tables are updated.
>
> Discussions
> ===========
> 1. Migration mode vs. migration control refactoring:
>    Currently, this series introduces MIGRATE_ASYNC_NONTEMPORAL into
>    enum migrate_mode. In parallel, Shivank Garg's migration refactoring
>    series [2] consolidates migration arguments into migrate_control.
>    Once that lands, demotion can simply rely on the migration reason
>    instead of adding a dedicated migration mode and helper.

If that is the ultimate goal, why not do it in this series to avoid
code churn? You probably can coordinate with Shivank on how to move
the migration control refactoring code to this series.

>
> 2. Large folio copy optimization:
>    Multi-page folios (such as 2MiB THP) are supported by iterating over
>    constituent 4KiB pages via copy_mc_page_nt(). Once the batch page
>    copy series [3] lands, copy_mc_page_nt() can be extended to copy an
>    entire multi-page folio in a single invocation, eliminating per-4KiB
>    loop overhead and keeping the prefetchnta stream uninterrupted.
>
> Thanks to Yiannis Nikolakopoulos, Alirad Malek, and Dimitrios
> Palyvos-Giannas for their original work, feedback, and collaboration.
>
> References
> ==========
> [1] https://lore.kernel.org/linux-mm/20260730-rfc-nt-demote-v2-0-452dbe3b5073@zptcorp.com/
> [2] https://lore.kernel.org/linux-mm/20260902-migrate-refactor-shivank-v1-9-9dcca87669c4@amd.com/
> [3] https://lore.kernel.org/linux-mm/20261019-batpagecopy-v2-1-d75e7d0dfaae@amd.com/
>
> Signed-off-by: Huan Nguyen <nhuhuan@google.com>
>
> Huan Nguyen (3):
>   mm, x86: support copying a folio using non-temporal stores
>   mm/migrate: introduce async non-temporal migration mode
>   mm/vmscan: demote folios using non-temporal stores
>
>  arch/x86/include/asm/page_64.h |  2 ++
>  arch/x86/lib/copy_page_64.S    | 65 ++++++++++++++++++++++++++++++++++
>  fs/nfs/write.c                 |  2 +-
>  include/linux/highmem.h        | 21 +++++++++++
>  include/linux/migrate.h        |  5 +++
>  include/linux/migrate_mode.h   |  1 +
>  include/linux/mm.h             |  1 +
>  include/trace/events/migrate.h |  1 +
>  mm/compaction.c                | 18 +++++-----
>  mm/migrate.c                   | 29 +++++++++------
>  mm/util.c                      | 29 +++++++++++++++
>  mm/vmscan.c                    |  2 +-
>  12 files changed, 154 insertions(+), 22 deletions(-)
>
> base-commit: 05441d697f7f9a927264f4058cbbc6a03dfcac9d
> -- 
> 2.56.0.398.g882cbad275-goog


Best Regards,
Yan, Zi

Re: [RFC PATCH v3 0/3] mm: use non-temporal stores for demotion

[Lorenzo Stoakes (ARM)]

Hi,

Please always send new revisions of series individually, not in-reply-to
any other email.

The easiest way of accomplishing this is to use b4 [0] (the recommended
way of sending patches to mm), otherwise format patches like this:

For a patch series, e.g. at v2:

$ git format-patch -v2 --cover-letter --thread ...

For an individual patch, e.g. at v2:

$ git format-patch -v2 HEAD~1

[0]: https://b4.docs.kernel.org/en/latest/contributor/send.html

-- 
Cheers, Lorenzo