[PATCH RFC v2 0/3] Demote to lower tier using non-temporal stores

[PATCH RFC v2 0/3] Demote to lower tier using non-temporal stores

[Yiannis Nikolakopoulos]

In most memory tiering scenarios, the memory to be demoted is expected
to be cold and most likely out of the node's last-level cache (as well
as target pages in the target node). Using non-temporal stores instead
of a standard memcpy path can reduce the cache pollution in the local
node and the bandwidth overhead to the target node. Furthermore, for
certain types of CXL devices that support in-line memory compression,
the last-level cache eviction patterns can negatively affect the
bandwidth of the device. Non-temporal stores can mitigate this.

This patch-set introduces a new migrate_mode flag for using non-temporal
stores that is used only in the demotion path. Patch 1 adds some helpers in
x86 and mm to bring non-temporal stores support to a respective folio_copy
function. Patch 2 adds the new flag and necessary changes for compatibility
with the existing behavior. Patch 3 uses the new flag for demotions.

Experimental data: in a CXL system with 1 memory expander, a microbenchmark
that allocates N=64 GB memory in the local node and then triggers demotion
using memory.reclaim, shows a practically complete elimination of read
traffic on the device, i.e. write traffic is N GB with and without the
patch, while read traffic drops from N to almost 0 with the patch.

Opens:
1. There is some "duplication" in the x86 tree and a bit in mm. Can we do
   something better there? As it is now in copy_mc_to_kernel_nt we
duplicate the machine check functionality, which if available will override
the non-temporal. We were not sure how to prioritize these two and what's
the best approach here. Can we completely skip the machine checked for this
path? Huan Nguyen has some ideas here that we will align for the next
version.
2. I am not sure how this should be structured so that it is easily
adopted in other architecture trees (e.g. aarch64). We rely on
memcpy_flushcache for x86_64 but this does not use non temporal stores
in ARM. ARM support is currently out of our scope but any input is
appreciated.

Signed-off-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
---
Changes in v2:
- Remove the KConfig option that was guarding this feature (patch 3) and
  use it always in x86_64 and memcpy_flushcache (non-temporal stores
should be there).
- Link to v1: https://patch.msgid.link/20260526-rfc-nt-demote-v1-0-eb9c9422daef@zptcorp.com

To: Thomas Gleixner <tglx@kernel.org>
To: Ingo Molnar <mingo@redhat.com>
To: Borislav Petkov <bp@alien8.de>
To: Dave Hansen <dave.hansen@linux.intel.com>
To: x86@kernel.org
To: "H. Peter Anvin" <hpa@zytor.com>
To: Andrew Morton <akpm@linux-foundation.org>
To: David Hildenbrand <david@kernel.org>
To: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
To: "Liam R. Howlett" <Liam.Howlett@oracle.com>
To: Vlastimil Babka <vbabka@kernel.org>
To: Mike Rapoport <rppt@kernel.org>
To: Suren Baghdasaryan <surenb@google.com>
To: Michal Hocko <mhocko@suse.com>
To: Trond Myklebust <trondmy@kernel.org>
To: Anna Schumaker <anna@kernel.org>
To: Zi Yan <ziy@nvidia.com>
To: Matthew Brost <matthew.brost@intel.com>
To: Joshua Hahn <joshua.hahnjy@gmail.com>
To: Rakie Kim <rakie.kim@sk.com>
To: Byungchul Park <byungchul@sk.com>
To: Gregory Price <gourry@gourry.net>
To: Ying Huang <ying.huang@linux.alibaba.com>
To: Alistair Popple <apopple@nvidia.com>
To: Steven Rostedt <rostedt@goodmis.org>
To: Masami Hiramatsu <mhiramat@kernel.org>
To: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To: Brendan Jackman <jackmanb@google.com>
To: Johannes Weiner <hannes@cmpxchg.org>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mm@kvack.org
Cc: linux-nfs@vger.kernel.org
Cc: linux-trace-kernel@vger.kernel.org

---
Alirad Malek (3):
      mm, x86: support copying a folio using non-temporal stores
      mm: new migrate_mode flag for async using non-temporal stores
      mm: use non-temporal stores for demotion

 arch/x86/include/asm/uaccess.h |  4 ++++
 arch/x86/lib/copy_mc.c         | 26 ++++++++++++++++++++++++++
 fs/nfs/write.c                 |  2 +-
 include/linux/highmem.h        | 32 ++++++++++++++++++++++++++++++++
 include/linux/migrate_mode.h   |  9 +++++++++
 include/linux/mm.h             |  1 +
 include/trace/events/migrate.h |  1 +
 mm/compaction.c                | 18 +++++++++---------
 mm/migrate.c                   | 22 +++++++++++++++-------
 mm/util.c                      | 17 +++++++++++++++++
 10 files changed, 115 insertions(+), 17 deletions(-)
---
base-commit: 1f318b96cc84d7c2ab792fcc0bfd42a7ca890681
change-id: 20260526-rfc-nt-demote-0fafadfdd006

Best regards,
--  
Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>

[PATCH RFC v2 1/3] mm, x86: support copying a folio using non-temporal stores

[Yiannis Nikolakopoulos]

From: Alirad Malek <alirad.malek@zptcorp.com>

In x86, use memcpy_flushcache (that uses non-temporal store
instructions) to copy a folio. To achieve that, starting from folio_mc_copy
down to copy_mc_to_kernel, create a series of helpers (named with an _nt
suffix) that have similar behavior to the original counterparts.

Signed-off-by: Alirad Malek <alirad.malek@zptcorp.com>
Co-developed-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
Signed-off-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
---
 arch/x86/include/asm/uaccess.h |  4 ++++
 arch/x86/lib/copy_mc.c         | 26 ++++++++++++++++++++++++++
 include/linux/highmem.h        | 32 ++++++++++++++++++++++++++++++++
 include/linux/mm.h             |  1 +
 mm/util.c                      | 17 +++++++++++++++++
 5 files changed, 80 insertions(+)

diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h
index 367297b188c3..2d0938d3e372 100644
--- a/arch/x86/include/asm/uaccess.h
+++ b/arch/x86/include/asm/uaccess.h
@@ -494,6 +494,10 @@ unsigned long __must_check
 copy_mc_to_kernel(void *to, const void *from, unsigned len);
 #define copy_mc_to_kernel copy_mc_to_kernel
 
+unsigned long __must_check
+copy_mc_to_kernel_nt(void *to, const void *from, unsigned len);
+#define copy_mc_to_kernel_nt copy_mc_to_kernel_nt
+
 unsigned long __must_check
 copy_mc_to_user(void __user *to, const void *from, unsigned len);
 #endif
diff --git a/arch/x86/lib/copy_mc.c b/arch/x86/lib/copy_mc.c
index 97e88e58567b..5a2ee5c2211e 100644
--- a/arch/x86/lib/copy_mc.c
+++ b/arch/x86/lib/copy_mc.c
@@ -81,6 +81,32 @@ unsigned long __must_check copy_mc_to_kernel(void *dst, const void *src, unsigne
 }
 EXPORT_SYMBOL_GPL(copy_mc_to_kernel);
 
+/**
+ * copy_mc_to_kernel_nt - memory copy that handles source exceptions
+ * if enabled, otherwise uses non-temporal stores
+ * @dst: destination address
+ * @src: source address
+ * @len: number of bytes to copy
+ *
+ * Return 0 for success, or number of bytes not copied if there was an
+ * exception.
+ */
+unsigned long __must_check copy_mc_to_kernel_nt(void *dst, const void *src, unsigned len)
+{
+	unsigned long ret;
+
+	if (copy_mc_fragile_enabled) {
+		instrument_memcpy_before(dst, src, len);
+		ret = copy_mc_fragile(dst, src, len);
+		instrument_memcpy_after(dst, src, len, ret);
+		return ret;
+	}
+
+	memcpy_flushcache(dst, src, len);
+	return 0;
+}
+EXPORT_SYMBOL_GPL(copy_mc_to_kernel_nt);
+
 unsigned long __must_check copy_mc_to_user(void __user *dst, const void *src, unsigned len)
 {
 	unsigned long ret;
diff --git a/include/linux/highmem.h b/include/linux/highmem.h
index af03db851a1d..a5cb435b9ffe 100644
--- a/include/linux/highmem.h
+++ b/include/linux/highmem.h
@@ -468,6 +468,32 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
 
 	return ret;
 }
+
+#ifdef copy_mc_to_kernel_nt
+static inline int copy_mc_highpage_nt(struct page *to, struct page *from)
+{
+	unsigned long ret;
+	char *vfrom, *vto;
+
+	vfrom = kmap_local_page(from);
+	vto = kmap_local_page(to);
+	ret = copy_mc_to_kernel_nt(vto, vfrom, PAGE_SIZE);
+	if (!ret)
+		kmsan_copy_page_meta(to, from);
+	kunmap_local(vto);
+	kunmap_local(vfrom);
+
+	if (ret)
+		memory_failure_queue(page_to_pfn(from), 0);
+
+	return ret;
+}
+#else
+static inline int copy_mc_highpage_nt(struct page *to, struct page *from)
+{
+	return copy_mc_highpage(to, from);
+}
+#endif
 #else
 static inline int copy_mc_user_highpage(struct page *to, struct page *from,
 					unsigned long vaddr, struct vm_area_struct *vma)
@@ -481,6 +507,12 @@ static inline int copy_mc_highpage(struct page *to, struct page *from)
 	copy_highpage(to, from);
 	return 0;
 }
+
+static inline int copy_mc_highpage_nt(struct page *to, struct page *from)
+{
+	copy_highpage(to, from);
+	return 0;
+}
 #endif
 
 static inline void memcpy_page(struct page *dst_page, size_t dst_off,
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 5be3d8a8f806..d07ce478582d 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1644,6 +1644,7 @@ void __folio_put(struct folio *folio);
 void split_page(struct page *page, unsigned int order);
 void folio_copy(struct folio *dst, struct folio *src);
 int folio_mc_copy(struct folio *dst, struct folio *src);
+int folio_mc_copy_nt(struct folio *dst, struct folio *src);
 
 unsigned long nr_free_buffer_pages(void);
 
diff --git a/mm/util.c b/mm/util.c
index b05ab6f97e11..e09e9b5f8eee 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -749,6 +749,23 @@ int folio_mc_copy(struct folio *dst, struct folio *src)
 }
 EXPORT_SYMBOL(folio_mc_copy);
 
+int folio_mc_copy_nt(struct folio *dst, struct folio *src)
+{
+	long nr = folio_nr_pages(src);
+	long i = 0;
+
+	for (;;) {
+		if (copy_mc_highpage_nt(folio_page(dst, i), folio_page(src, i)))
+			return -EHWPOISON;
+		if (++i == nr)
+			break;
+		cond_resched();
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL(folio_mc_copy_nt);
+
 int sysctl_overcommit_memory __read_mostly = OVERCOMMIT_GUESS;
 static int sysctl_overcommit_ratio __read_mostly = 50;
 static unsigned long sysctl_overcommit_kbytes __read_mostly;

-- 
2.43.0

[PATCH RFC v2 2/3] mm: new migrate_mode flag for async using non-temporal stores

[Yiannis Nikolakopoulos]

From: Alirad Malek <alirad.malek@zptcorp.com>

In preparation for the following patch, add a new migrate_mode which is
still async but will use non-temporal stores. Add a helper function
that checks for both async modes and replace all plain checks of
MIGRATE_ASYNC.

Signed-off-by: Alirad Malek <alirad.malek@zptcorp.com>
Co-developed-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
Signed-off-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
---
 fs/nfs/write.c                 |  2 +-
 include/linux/migrate_mode.h   |  9 +++++++++
 include/trace/events/migrate.h |  1 +
 mm/compaction.c                | 18 +++++++++---------
 mm/migrate.c                   | 12 ++++++------
 5 files changed, 26 insertions(+), 16 deletions(-)

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 1ed4b3590b1a..beae4441e080 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -2119,7 +2119,7 @@ int nfs_migrate_folio(struct address_space *mapping, struct folio *dst,
 	}
 
 	if (folio_test_private_2(src)) { /* [DEPRECATED] */
-		if (mode == MIGRATE_ASYNC)
+		if (migrate_mode_is_async(mode))
 			return -EBUSY;
 		folio_wait_private_2(src);
 	}
diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h
index 265c4328b36a..f7186e705b48 100644
--- a/include/linux/migrate_mode.h
+++ b/include/linux/migrate_mode.h
@@ -3,6 +3,8 @@
 #define MIGRATE_MODE_H_INCLUDED
 /*
  * MIGRATE_ASYNC means never block
+ * MIGRATE_ASYNC_NON_TEMPORAL_STORES means never block and use non-temporal
+ * stores if supported by the architecture
  * MIGRATE_SYNC_LIGHT in the current implementation means to allow blocking
  *	on most operations but not ->writepage as the potential stall time
  *	is too significant
@@ -10,10 +12,17 @@
  */
 enum migrate_mode {
 	MIGRATE_ASYNC,
+	MIGRATE_ASYNC_NON_TEMPORAL_STORES,
 	MIGRATE_SYNC_LIGHT,
 	MIGRATE_SYNC,
 };
 
+static inline bool migrate_mode_is_async(enum migrate_mode mode)
+{
+	return mode == MIGRATE_ASYNC ||
+		mode == MIGRATE_ASYNC_NON_TEMPORAL_STORES;
+}
+
 enum migrate_reason {
 	MR_COMPACTION,
 	MR_MEMORY_FAILURE,
diff --git a/include/trace/events/migrate.h b/include/trace/events/migrate.h
index cd01dd7b3640..e493207a3f46 100644
--- a/include/trace/events/migrate.h
+++ b/include/trace/events/migrate.h
@@ -9,6 +9,7 @@
 
 #define MIGRATE_MODE						\
 	EM( MIGRATE_ASYNC,	"MIGRATE_ASYNC")		\
+	EM(MIGRATE_ASYNC_NON_TEMPORAL_STORES,		"MIGRATE_ASYNC_NON_TEMPORAL_STORES")	\
 	EM( MIGRATE_SYNC_LIGHT,	"MIGRATE_SYNC_LIGHT")		\
 	EMe(MIGRATE_SYNC,	"MIGRATE_SYNC")
 
diff --git a/mm/compaction.c b/mm/compaction.c
index 1e8f8eca318c..cd26781b7376 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -444,7 +444,7 @@ static void update_cached_migrate(struct compact_control *cc, unsigned long pfn)
 	/* Update where async and sync compaction should restart */
 	if (pfn > zone->compact_cached_migrate_pfn[0])
 		zone->compact_cached_migrate_pfn[0] = pfn;
-	if (cc->mode != MIGRATE_ASYNC &&
+	if (!migrate_mode_is_async(cc->mode) &&
 	    pfn > zone->compact_cached_migrate_pfn[1])
 		zone->compact_cached_migrate_pfn[1] = pfn;
 }
@@ -507,7 +507,7 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags,
 	__acquires(lock)
 {
 	/* Track if the lock is contended in async mode */
-	if (cc->mode == MIGRATE_ASYNC && !cc->contended) {
+	if (migrate_mode_is_async(cc->mode) && !cc->contended) {
 		if (spin_trylock_irqsave(lock, *flags))
 			return true;
 
@@ -864,7 +864,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 			return -EAGAIN;
 
 		/* async migration should just abort */
-		if (cc->mode == MIGRATE_ASYNC)
+		if (migrate_mode_is_async(cc->mode))
 			return -EAGAIN;
 
 		reclaim_throttle(pgdat, VMSCAN_THROTTLE_ISOLATED);
@@ -875,7 +875,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 
 	cond_resched();
 
-	if (cc->direct_compaction && (cc->mode == MIGRATE_ASYNC)) {
+	if (cc->direct_compaction && migrate_mode_is_async(cc->mode)) {
 		skip_on_failure = true;
 		next_skip_pfn = block_end_pfn(low_pfn, cc->order);
 	}
@@ -1364,7 +1364,7 @@ static bool suitable_migration_source(struct compact_control *cc,
 	if (pageblock_skip_persistent(page))
 		return false;
 
-	if ((cc->mode != MIGRATE_ASYNC) || !cc->direct_compaction)
+	if (!migrate_mode_is_async(cc->mode) || !cc->direct_compaction)
 		return true;
 
 	block_mt = get_pageblock_migratetype(page);
@@ -1465,7 +1465,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn)
 		return;
 
 	/* Minimise scanning during async compaction */
-	if (cc->direct_compaction && cc->mode == MIGRATE_ASYNC)
+	if (cc->direct_compaction && migrate_mode_is_async(cc->mode))
 		return;
 
 	/* Pageblock boundaries */
@@ -1705,7 +1705,7 @@ static void isolate_freepages(struct compact_control *cc)
 	block_end_pfn = min(block_start_pfn + pageblock_nr_pages,
 						zone_end_pfn(zone));
 	low_pfn = pageblock_end_pfn(cc->migrate_pfn);
-	stride = cc->mode == MIGRATE_ASYNC ? COMPACT_CLUSTER_MAX : 1;
+	stride = migrate_mode_is_async(cc->mode) ? COMPACT_CLUSTER_MAX : 1;
 
 	/*
 	 * Isolate free pages until enough are available to migrate the
@@ -2514,7 +2514,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
 	unsigned long start_pfn = cc->zone->zone_start_pfn;
 	unsigned long end_pfn = zone_end_pfn(cc->zone);
 	unsigned long last_migrated_pfn;
-	const bool sync = cc->mode != MIGRATE_ASYNC;
+	const bool sync = !migrate_mode_is_async(cc->mode);
 	bool update_cached;
 	unsigned int nr_succeeded = 0, nr_migratepages;
 	int order;
@@ -2537,7 +2537,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc)
 		ret = compaction_suit_allocation_order(cc->zone, cc->order,
 						       cc->highest_zoneidx,
 						       cc->alloc_flags,
-						       cc->mode == MIGRATE_ASYNC,
+						       migrate_mode_is_async(cc->mode),
 						       !cc->direct_compaction);
 		if (ret != COMPACT_CONTINUE)
 			return ret;
diff --git a/mm/migrate.c b/mm/migrate.c
index 2c3d489ecf51..ff6cf50e7b0b 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -907,7 +907,7 @@ static bool buffer_migrate_lock_buffers(struct buffer_head *head,
 
 	do {
 		if (!trylock_buffer(bh)) {
-			if (mode == MIGRATE_ASYNC)
+			if (migrate_mode_is_async(mode))
 				goto unlock;
 			if (mode == MIGRATE_SYNC_LIGHT && !buffer_uptodate(bh))
 				goto unlock;
@@ -1220,7 +1220,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 	dst->private = NULL;
 
 	if (!folio_trylock(src)) {
-		if (mode == MIGRATE_ASYNC)
+		if (migrate_mode_is_async(mode))
 			goto out;
 
 		/*
@@ -1325,7 +1325,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio,
 		/* Establish migration ptes */
 		VM_BUG_ON_FOLIO(folio_test_anon(src) &&
 			       !folio_test_ksm(src) && !anon_vma, src);
-		try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0);
+		try_to_migrate(src, migrate_mode_is_async(mode) ? TTU_BATCH_FLUSH : 0);
 		old_page_state |= PAGE_WAS_MAPPED;
 	}
 
@@ -1565,7 +1565,7 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f
 {
 	int rc;
 
-	if (mode == MIGRATE_ASYNC) {
+	if (migrate_mode_is_async(mode)) {
 		if (!folio_trylock(folio))
 			return -EAGAIN;
 	} else {
@@ -1799,7 +1799,7 @@ static int migrate_pages_batch(struct list_head *from,
 	LIST_HEAD(dst_folios);
 	bool nosplit = (reason == MR_NUMA_MISPLACED);
 
-	VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC &&
+	VM_WARN_ON_ONCE(!migrate_mode_is_async(mode) &&
 			!list_empty(from) && !list_is_singular(from));
 
 	for (pass = 0; pass < nr_pass && retry; pass++) {
@@ -2107,7 +2107,7 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 		list_cut_before(&folios, from, &folio2->lru);
 	else
 		list_splice_init(from, &folios);
-	if (mode == MIGRATE_ASYNC)
+	if (migrate_mode_is_async(mode))
 		rc = migrate_pages_batch(&folios, get_new_folio, put_new_folio,
 				private, mode, reason, &ret_folios,
 				&split_folios, &stats,

-- 
2.43.0

[PATCH RFC v2 3/3] mm: use non-temporal stores for demotion

[Yiannis Nikolakopoulos]

From: Alirad Malek <alirad.malek@zptcorp.com>

Memory demoted to a lower tier is assumed to be cold and most likely out of
the CPU's last level cache. Additionally, in certain demotion targets (e.g.
CXL devices with compressed memory) the bandwidth can be negatively
impacted by the eviction patterns of the last level cache when standard
memcpy is used. In supported architectures (currently X86_64), use the
MIGRATE_ASYNC_NON_TEMPORAL_STORES flag in demotions to trigger the folio
copy path using non-temporal stores.

Signed-off-by: Alirad Malek <alirad.malek@zptcorp.com>
Co-developed-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
Signed-off-by: Yiannis Nikolakopoulos <yiannis.nikolakop@gmail.com>
---
 mm/migrate.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index ff6cf50e7b0b..c7c62700d541 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -862,7 +862,10 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst,
 	if (folio_ref_count(src) != expected_count)
 		return -EAGAIN;
 
-	rc = folio_mc_copy(dst, src);
+	if (mode == MIGRATE_ASYNC_NON_TEMPORAL_STORES)
+		rc = folio_mc_copy_nt(dst, src);
+	else
+		rc = folio_mc_copy(dst, src);
 	if (unlikely(rc))
 		return rc;
 
@@ -2081,6 +2084,11 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	LIST_HEAD(split_folios);
 	struct migrate_pages_stats stats;
 
+	if (IS_ENABLED(CONFIG_X86_64) &&
+		IS_ENABLED(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) &&
+		reason == MR_DEMOTION && mode == MIGRATE_ASYNC)
+		mode = MIGRATE_ASYNC_NON_TEMPORAL_STORES;
+
 	trace_mm_migrate_pages_start(mode, reason);
 
 	memset(&stats, 0, sizeof(stats));

-- 
2.43.0

Re: [PATCH RFC v2 0/3] Demote to lower tier using non-temporal stores

[Gregory Price]

On Thu, Jul 30, 2026 at 05:02:56PM +0200, Yiannis Nikolakopoulos wrote:
> In most memory tiering scenarios, the memory to be demoted is expected
> to be cold and most likely out of the node's last-level cache (as well
> as target pages in the target node). Using non-temporal stores instead
> of a standard memcpy path can reduce the cache pollution in the local
> node and the bandwidth overhead to the target node. Furthermore, for
> certain types of CXL devices that support in-line memory compression,
> the last-level cache eviction patterns can negatively affect the
> bandwidth of the device. Non-temporal stores can mitigate this.
> 
> This patch-set introduces a new migrate_mode flag for using non-temporal
> stores that is used only in the demotion path. Patch 1 adds some helpers in
> x86 and mm to bring non-temporal stores support to a respective folio_copy
> function. Patch 2 adds the new flag and necessary changes for compatibility
> with the existing behavior. Patch 3 uses the new flag for demotions.
> 
> Experimental data: in a CXL system with 1 memory expander, a microbenchmark
> that allocates N=64 GB memory in the local node and then triggers demotion
> using memory.reclaim, shows a practically complete elimination of read
> traffic on the device, i.e. write traffic is N GB with and without the
> patch, while read traffic drops from N to almost 0 with the patch.
> 
> Opens:
> 1. There is some "duplication" in the x86 tree and a bit in mm. Can we do
>    something better there? As it is now in copy_mc_to_kernel_nt we
> duplicate the machine check functionality, which if available will override
> the non-temporal. We were not sure how to prioritize these two and what's
> the best approach here. Can we completely skip the machine checked for this
> path? Huan Nguyen has some ideas here that we will align for the next
> version.
> 2. I am not sure how this should be structured so that it is easily
> adopted in other architecture trees (e.g. aarch64). We rely on
> memcpy_flushcache for x86_64 but this does not use non temporal stores
> in ARM. ARM support is currently out of our scope but any input is
> appreciated.
> 

I'm still a bit confused why using NT Stores needs to be an explicit
option - rather than the default behavior if NT Store is available.

Lets assume we did this for all ASYNC requests, is there a negative
effect? A positive effect? Why the new ASYNC type? Is there a
correctness issue?

~Gregory

Re: [PATCH RFC v2 0/3] Demote to lower tier using non-temporal stores

[Frank van der Linden]

On Thu, Jul 30, 2026 at 10:30 AM Gregory Price <gourry@gourry.net> wrote:
>
> On Thu, Jul 30, 2026 at 05:02:56PM +0200, Yiannis Nikolakopoulos wrote:
> > In most memory tiering scenarios, the memory to be demoted is expected
> > to be cold and most likely out of the node's last-level cache (as well
> > as target pages in the target node). Using non-temporal stores instead
> > of a standard memcpy path can reduce the cache pollution in the local
> > node and the bandwidth overhead to the target node. Furthermore, for
> > certain types of CXL devices that support in-line memory compression,
> > the last-level cache eviction patterns can negatively affect the
> > bandwidth of the device. Non-temporal stores can mitigate this.
> >
> > This patch-set introduces a new migrate_mode flag for using non-temporal
> > stores that is used only in the demotion path. Patch 1 adds some helpers in
> > x86 and mm to bring non-temporal stores support to a respective folio_copy
> > function. Patch 2 adds the new flag and necessary changes for compatibility
> > with the existing behavior. Patch 3 uses the new flag for demotions.
> >
> > Experimental data: in a CXL system with 1 memory expander, a microbenchmark
> > that allocates N=64 GB memory in the local node and then triggers demotion
> > using memory.reclaim, shows a practically complete elimination of read
> > traffic on the device, i.e. write traffic is N GB with and without the
> > patch, while read traffic drops from N to almost 0 with the patch.
> >
> > Opens:
> > 1. There is some "duplication" in the x86 tree and a bit in mm. Can we do
> >    something better there? As it is now in copy_mc_to_kernel_nt we
> > duplicate the machine check functionality, which if available will override
> > the non-temporal. We were not sure how to prioritize these two and what's
> > the best approach here. Can we completely skip the machine checked for this
> > path? Huan Nguyen has some ideas here that we will align for the next
> > version.
> > 2. I am not sure how this should be structured so that it is easily
> > adopted in other architecture trees (e.g. aarch64). We rely on
> > memcpy_flushcache for x86_64 but this does not use non temporal stores
> > in ARM. ARM support is currently out of our scope but any input is
> > appreciated.
> >
>
> I'm still a bit confused why using NT Stores needs to be an explicit
> option - rather than the default behavior if NT Store is available.
>
> Lets assume we did this for all ASYNC requests, is there a negative
> effect? A positive effect? Why the new ASYNC type? Is there a
> correctness issue?
>
> ~Gregory

Yes, this is a good discussion to have. I believe that the initial
reason for using a separate mode here was to be non-invasive, e.g.
"don't break anything else".

But why not use non-temporal stores for everything? I don't know. I
suppose one argument might be that if you know that the destination
will be used immediately, NT stores might be a slight performance hit.

arm64 already seems to use stores with NT hints by default.

In general, though, I agree that for ASYNC requests, just always using
NT seems fine.

Maybe the mode and reason should be folded in to one variable, so
that, further down the stack, a decision can be made as to what type
of copy to use? E.g. if the mode is !MIGRATE_ASYNC and the reason is
MR_DEMOTION, then non-temporal is still a good choice. If the mode is
MIGRATE_ASYNC, NT is still a good idea. If you wanted to get fancy,
any mode with folio_test_waiters(folio) == true should not use NT,
since a task is waiting to use the data, so caching it is better.
Maybe that's overthinking it.

- Frank

Re: [PATCH RFC v2 0/3] Demote to lower tier using non-temporal stores

[Johannes Weiner]

On Thu, Jul 30, 2026 at 10:40:32AM -0700, Frank van der Linden wrote:
> On Thu, Jul 30, 2026 at 10:30 AM Gregory Price <gourry@gourry.net> wrote:
> >
> > On Thu, Jul 30, 2026 at 05:02:56PM +0200, Yiannis Nikolakopoulos wrote:
> > > In most memory tiering scenarios, the memory to be demoted is expected
> > > to be cold and most likely out of the node's last-level cache (as well
> > > as target pages in the target node). Using non-temporal stores instead
> > > of a standard memcpy path can reduce the cache pollution in the local
> > > node and the bandwidth overhead to the target node. Furthermore, for
> > > certain types of CXL devices that support in-line memory compression,
> > > the last-level cache eviction patterns can negatively affect the
> > > bandwidth of the device. Non-temporal stores can mitigate this.
> > >
> > > This patch-set introduces a new migrate_mode flag for using non-temporal
> > > stores that is used only in the demotion path. Patch 1 adds some helpers in
> > > x86 and mm to bring non-temporal stores support to a respective folio_copy
> > > function. Patch 2 adds the new flag and necessary changes for compatibility
> > > with the existing behavior. Patch 3 uses the new flag for demotions.
> > >
> > > Experimental data: in a CXL system with 1 memory expander, a microbenchmark
> > > that allocates N=64 GB memory in the local node and then triggers demotion
> > > using memory.reclaim, shows a practically complete elimination of read
> > > traffic on the device, i.e. write traffic is N GB with and without the
> > > patch, while read traffic drops from N to almost 0 with the patch.
> > >
> > > Opens:
> > > 1. There is some "duplication" in the x86 tree and a bit in mm. Can we do
> > >    something better there? As it is now in copy_mc_to_kernel_nt we
> > > duplicate the machine check functionality, which if available will override
> > > the non-temporal. We were not sure how to prioritize these two and what's
> > > the best approach here. Can we completely skip the machine checked for this
> > > path? Huan Nguyen has some ideas here that we will align for the next
> > > version.
> > > 2. I am not sure how this should be structured so that it is easily
> > > adopted in other architecture trees (e.g. aarch64). We rely on
> > > memcpy_flushcache for x86_64 but this does not use non temporal stores
> > > in ARM. ARM support is currently out of our scope but any input is
> > > appreciated.
> > >
> >
> > I'm still a bit confused why using NT Stores needs to be an explicit
> > option - rather than the default behavior if NT Store is available.
> >
> > Lets assume we did this for all ASYNC requests, is there a negative
> > effect? A positive effect? Why the new ASYNC type? Is there a
> > correctness issue?
> >
> > ~Gregory
> 
> Yes, this is a good discussion to have. I believe that the initial
> reason for using a separate mode here was to be non-invasive, e.g.
> "don't break anything else".
> 
> But why not use non-temporal stores for everything? I don't know. I
> suppose one argument might be that if you know that the destination
> will be used immediately, NT stores might be a slight performance hit.

do_numa_page() -> migrate_misplaced_folio() is such a case. It happens
literally in the access path to that data. MIGRATE_ASYNC just means
don't block, fail fast, because waiting for an IO-bound lock to avoid
a remote NUMA access would be a bad idea.

Compaction on the other hand does use MIGRATE_SYNC* and it's touching
data in PFN order, completely out of execution sequence. That could be
a good candidate for NT stores.

> arm64 already seems to use stores with NT hints by default.
> 
> In general, though, I agree that for ASYNC requests, just always using
> NT seems fine.
> 
> Maybe the mode and reason should be folded in to one variable, so
> that, further down the stack, a decision can be made as to what type
> of copy to use? E.g. if the mode is !MIGRATE_ASYNC and the reason is
> MR_DEMOTION, then non-temporal is still a good choice. If the mode is
> MIGRATE_ASYNC, NT is still a good idea. If you wanted to get fancy,
> any mode with folio_test_waiters(folio) == true should not use NT,
> since a task is waiting to use the data, so caching it is better.
> Maybe that's overthinking it.

Agree. `reason' seems like a much stronger signal.

Re: [PATCH RFC v2 0/3] Demote to lower tier using non-temporal stores

[Gregory Price]

On Thu, Jul 30, 2026 at 02:08:56PM -0400, Johannes Weiner wrote:
> On Thu, Jul 30, 2026 at 10:40:32AM -0700, Frank van der Linden wrote:
> > On Thu, Jul 30, 2026 at 10:30 AM Gregory Price <gourry@gourry.net> wrote:
> 
> > arm64 already seems to use stores with NT hints by default.
> > 
> > In general, though, I agree that for ASYNC requests, just always using
> > NT seems fine.
> > 
> > Maybe the mode and reason should be folded in to one variable, so
> > that, further down the stack, a decision can be made as to what type
> > of copy to use? E.g. if the mode is !MIGRATE_ASYNC and the reason is
> > MR_DEMOTION, then non-temporal is still a good choice. If the mode is
> > MIGRATE_ASYNC, NT is still a good idea. If you wanted to get fancy,
> > any mode with folio_test_waiters(folio) == true should not use NT,
> > since a task is waiting to use the data, so caching it is better.
> > Maybe that's overthinking it.
> 
> Agree. `reason' seems like a much stronger signal.

I think this ends up cleaner as well, it would be a nice improvement.

~Gregory