[PATCH v2 0/3] sched/numa: refine memory tiering promotion heuristics

[PATCH v2 0/3] sched/numa: refine memory tiering promotion heuristics

[Li Zhe]

This version narrows the series to memory tiering.  The generic normal
NUMA balancing cleanups and heuristic changes from v1 are not included.

Patch 1 makes the free-space promotion bypass configurable while keeping
the existing eager-promotion behavior as the default.  Disabling the
bypass forces slow-tier promotions through the normal hotness and
rate-limit checks, which helps workloads that intentionally keep part of
their memory on a slow tier avoid short cold bursts refilling the fast
tier.

Patch 2 keeps memory-tiering-only promotion faults out of normal AutoNUMA
task-placement statistics.  Slow-tier NUMA hint faults use the folio
last-cpupid field as access-time metadata, so those faults should not be
interpreted as normal placement feedback.  The patch marks them explicitly
and skips the placement accounting path.

Patch 3 builds on that separation.  Once tiering faults are no longer fed
into task placement stats, tiering-only mode can use tiering-specific
outcomes to tune the scan period: hot feedback speeds scanning up, while
cold, throttled, failed or empty windows slow it down.  This prevents
tiering-only promotion feedback from being handled by the normal placement
heuristic, where repeated promotions can otherwise push the scan period
towards the maximum.

Li Zhe (3):
  sched/numa: make free-space tiering promotion bypass configurable
  sched/numa: do not feed tiering promotions into task placement stats
  sched/numa: fix scan-period adjustment for tiering-only mode

 include/linux/mmzone.h               |   5 +-
 include/linux/sched.h                |  17 ++++
 include/linux/sched/numa_balancing.h |  10 ++
 kernel/sched/debug.c                 |   2 +
 kernel/sched/fair.c                  | 140 +++++++++++++++++++++++++--
 kernel/sched/sched.h                 |   1 +
 mm/memory.c                          |   6 +-
 7 files changed, 168 insertions(+), 13 deletions(-)

---
v1: https://lore.kernel.org/all/20261014094347.99157-1-lizhe.67@bytedance.com/

Changes since v1:
- Dropped the generic normal NUMA balancing patches.
- Added patch 3 to dynamically adjust the scan period in
  memory-tiering-only mode.

-- 
2.20.1

[PATCH v2 1/3] sched/numa: make free-space tiering promotion bypass configurable

[Li Zhe]

Memory tiering normally promotes any recently accessed slow-tier page if
the target fast-tier node has enough free space. This preserves the
existing policy of filling available fast memory before relying on
hotness and rate-limit control.

That is not always the desired policy. Some deployments may intentionally
keep part of a workload's memory on CXL-attached memory, for example to
preserve a desired DRAM/CXL capacity split or bandwidth distribution. With
the free-space bypass, short cold accesses and bursts of hot accesses can
refill the fast tier and pull the workload away from that distribution
whenever DRAM has free space.

Redis testing exercises this policy issue. On a CXL system with node0 as
DRAM and node3 as slow-tier memory, a Redis test was run with
memory-tiering-only mode enabled. The test used a Redis dataset with 12
million 4KB values. It reclaimed memory until about 85% of Redis
anonymous memory was on node3, used a Redis benchmark configuration with
a 10% hot-key working set, and then issued a short 256MB access burst to
cold keys outside that working set. The cold-key burst mainly touches
memory outside the hot working set, which is expected to remain mostly on
the slow tier.

The upstream default promoted the cold-key burst through the free-space
bypass. The patched default preserved that behavior, while disabling the
bypass kept the same promotions behind the hotness and rate-limit checks:

                              upstream     patched default    bypass disabled
  cold-key burst ops/s          117250         115481             142505
  cold-key burst P99.9 ms        0.854          0.859              0.686
  pgpromote_candidate              0              0               1488328

Compared to upstream default, disabling the bypass improved cold-key burst
throughput by 21.5% and P99.9 latency by 19.7%.

Add a sched debugfs knob:

  /sys/kernel/debug/sched/numa_balancing/promote_free_space_bypass

It defaults to enabled, preserving the existing behavior. When disabled,
enough free space in the target fast-tier node no longer bypasses the
normal hotness and rate-limit controls.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 include/linux/mmzone.h |  5 ++---
 kernel/sched/debug.c   |  2 ++
 kernel/sched/fair.c    | 14 ++++++++------
 kernel/sched/sched.h   |  1 +
 4 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index ecd1ebd49756..bf2d604c04b3 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -278,9 +278,8 @@ enum node_stat_item {
 	 */
 	PGPROMOTE_CANDIDATE,
 	/**
-	 * Not rate-limited (NRL) candidate pages for those can be promoted
-	 * without considering hot threshold because of enough free pages in
-	 * fast-tier node.  These promotions bypass the regular hotness checks
+	 * Not rate-limited (NRL) candidate pages selected for promotion through
+	 * the free-space bypass. These candidates bypass the regular hotness checks
 	 * and do NOT influence the promotion rate-limiter or
 	 * threshold-adjustment logic.
 	 * This is for statistics/monitoring purposes.
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index e6a3b516c703..a28f0f09c91e 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -736,6 +736,8 @@ static __init int sched_init_debug(void)
 	debugfs_create_u32("scan_period_max_ms", 0644, numa, &sysctl_numa_balancing_scan_period_max);
 	debugfs_create_u32("scan_size_mb", 0644, numa, &sysctl_numa_balancing_scan_size);
 	debugfs_create_u32("hot_threshold_ms", 0644, numa, &sysctl_numa_balancing_hot_threshold);
+	debugfs_create_bool("promote_free_space_bypass", 0644, numa,
+			    &numa_balancing_promote_free_space_bypass);
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_SCHED_CACHE
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index cfc982205b98..46a11f5ddafb 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -134,6 +134,8 @@ static unsigned int sysctl_sched_cfs_bandwidth_slice		= 5000UL;
 #ifdef CONFIG_NUMA_BALANCING
 /* Restrict the NUMA promotion throughput (MB/s) for each target node. */
 static unsigned int sysctl_numa_balancing_promote_rate_limit = 65536;
+/* Preserve the existing default of using free fast-tier space eagerly. */
+bool numa_balancing_promote_free_space_bypass = true;
 #endif
 
 #ifdef CONFIG_SYSCTL
@@ -2759,11 +2761,10 @@ static inline bool cpupid_valid(int cpupid)
 }
 
 /*
- * For memory tiering mode, if there are enough free pages (more than
- * enough watermark defined here) in fast memory node, to take full
- * advantage of fast memory capacity, all recently accessed slow
- * memory pages will be migrated to fast memory node without
- * considering hot threshold.
+ * For memory tiering mode, if the target fast memory node has enough free
+ * pages, recently accessed slow memory pages can bypass the normal hotness
+ * check and rate limit so the fast tier can be filled. The bypass can be
+ * disabled by numa_balancing_promote_free_space_bypass.
  */
 static bool pgdat_free_space_enough(struct pglist_data *pgdat)
 {
@@ -2884,7 +2885,8 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 		long nr = folio_nr_pages(folio);
 
 		pgdat = NODE_DATA(dst_nid);
-		if (pgdat_free_space_enough(pgdat)) {
+		if (numa_balancing_promote_free_space_bypass &&
+		    pgdat_free_space_enough(pgdat)) {
 			/* workload changed, reset hot threshold */
 			pgdat->nbp_threshold = 0;
 			mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 87c5335add37..662b11bd401e 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -3218,6 +3218,7 @@ extern unsigned int sysctl_numa_balancing_scan_period_min;
 extern unsigned int sysctl_numa_balancing_scan_period_max;
 extern unsigned int sysctl_numa_balancing_scan_size;
 extern unsigned int sysctl_numa_balancing_hot_threshold;
+extern bool numa_balancing_promote_free_space_bypass;
 
 #ifdef CONFIG_SCHED_HRTICK
 
-- 
2.20.1

[PATCH v2 2/3] sched/numa: do not feed tiering promotions into task placement stats

[Li Zhe]

NUMA balancing has separate mode bits for normal task placement and
memory tiering. In memory-tiering-only mode, NUMA hinting faults on
slow-tier pages are used to detect hot pages and promote them to a top
tier. They are not normal task-placement faults.

For slow-tier folios, memory tiering reuses the folio last-cpupid field
to store access-time metadata. numa_migrate_check() therefore cannot
return a real cpupid for those faults. Before this change, it returned the
same all-ones last_cpupid value that normal AutoNUMA uses for a first or
unknown access.

That can make a memory-tiering-only promotion look like an ordinary first
private AutoNUMA fault. After a successful promotion, do_numa_page()
reports the target top-tier node to task_numa_fault(), so the existing
slow-node skip no longer applies. task_numa_fault() then accounts the
fault in p->numa_faults and p->numa_faults_locality, and
task_numa_placement() can consume that sample to update normal task
placement state even though normal NUMA balancing is disabled. In other
words, hot-page promotion feedback can leak into the CPU placement
heuristic.

Mark faults that carried access-time metadata with TNF_ACCESS_TIME. If
normal NUMA balancing is disabled, keep the migrated-page accounting but
skip normal task-placement accounting for those faults.

When normal NUMA balancing and memory tiering are both enabled, promoted
pages can still feed task placement. In that mode, the user has enabled
both CPU placement feedback and tiering promotion, so a promoted top-tier
fault is treated as an ordinary unset-cpupid AutoNUMA fault.

This does not change whether a page is promoted. It only prevents
memory-tiering-only promotion faults from being consumed by the normal
AutoNUMA placement heuristic.

Using a fault flag avoids reserving an in-band value from the cpupid
namespace, where any masked value may also be produced by a real CPU/PID
pair.

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 include/linux/sched/numa_balancing.h |  1 +
 kernel/sched/fair.c                  | 13 +++++++++++++
 mm/memory.c                          |  6 ++++--
 3 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/include/linux/sched/numa_balancing.h b/include/linux/sched/numa_balancing.h
index 52b22c5c396d..200f50075d5c 100644
--- a/include/linux/sched/numa_balancing.h
+++ b/include/linux/sched/numa_balancing.h
@@ -14,6 +14,7 @@
 #define TNF_SHARED	0x04
 #define TNF_FAULT_LOCAL	0x08
 #define TNF_MIGRATE_FAIL 0x10
+#define TNF_ACCESS_TIME	0x20
 
 enum numa_vmaskip_reason {
 	NUMAB_SKIP_UNSUITABLE,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 46a11f5ddafb..136207b62d87 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4149,6 +4149,19 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
 	if (!p->mm)
 		return;
 
+	/*
+	 * Slow-tier pages use last_cpupid as access-time metadata instead of
+	 * a real cpupid. Do not feed these faults into normal task placement
+	 * when normal NUMA balancing is disabled.
+	 */
+	if (flags & TNF_ACCESS_TIME) {
+		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL)) {
+			if (migrated)
+				p->numa_pages_migrated += pages;
+			return;
+		}
+	}
+
 	/*
 	 * NUMA faults statistics are unnecessary for the slow memory
 	 * node for memory tiering mode.
diff --git a/mm/memory.c b/mm/memory.c
index 0fb676dd2bd6..00b171e2fe23 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -6241,10 +6241,12 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf,
 	 * For memory tiering mode, cpupid of slow memory page is used
 	 * to record page access time.  So use default value.
 	 */
-	if (folio_use_access_time(folio))
+	if (folio_use_access_time(folio)) {
+		*flags |= TNF_ACCESS_TIME;
 		*last_cpupid = (-1 & LAST_CPUPID_MASK);
-	else
+	} else {
 		*last_cpupid = folio_last_cpupid(folio);
+	}
 
 	/* Record the current PID accessing VMA */
 	vma_set_access_pid_bit(vma);
-- 
2.20.1

[PATCH v2 3/3] sched/numa: fix scan-period adjustment for tiering-only mode

[Li Zhe]

When automatic NUMA balancing runs in memory-tiering-only mode, NUMA hint
faults identify slow-memory pages that are worth promoting. They are not
normal task-placement feedback.

Without separate tiering feedback, successful tiering promotions are fed
to update_task_scan_period() as private remote faults. That leaves
local == 0 and shared == 0, so the function hits its early slow-down path
and doubles p->numa_scan_period. Repeated promotion feedback can therefore
drive the scan period towards the maximum.

Track memory-tiering feedback separately from normal task-placement stats.
Successful free-space bypass promotions are not counted as tiering scan
feedback because they did not pass the hotness and rate-limit checks. Cold,
hot, and throttled feedback is generated only by the hotness/rate-limit
path, while migration failures slow scanning like the normal placement
heuristic does.

Use the same 70% threshold style as the normal scan-period heuristic. A
hot-dominated window speeds scanning up, a cold-dominated window slows it
down, and a throttled ratio of at least 30% also slows it down. If a
window has no tiering feedback, for example because promotions used the
free-space bypass, double the scan period, following the same
no-useful-signal slow-down direction as normal scan-period adjustment.

Evaluate the tiering window when a scan sequence completes as well as when
tiering faults are reported. This lets an empty window back off the scan
period after qualifying tiering feedback stops.

On a CXL system with node0 as DRAM and node3 as slow-tier memory, a
STREAM Triad-derived OpenMP workload was used to model a
bandwidth-sensitive working-set shift. It first accessed one 100G range,
then switched to another 100G range. The workload was first-touch placed
on node3 and then run from node0 with memory-tiering-only mode enabled.

With the free-space bypass disabled, so that promotions have to pass the
hotness and rate-limit checks, a kernel with only the bypass control was
used as the no-feedback baseline. Tiering scan-period feedback improved
the transition window after the active range switch:

                              no feedback       tiering feedback
  first 80s after switch       122.4 GB/s             129.3 GB/s  (+5.7%)
  200-240s after switch        139.9 GB/s             140.6 GB/s  (+0.5%)

Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
---
 include/linux/sched.h                |  17 ++++
 include/linux/sched/numa_balancing.h |   9 +++
 kernel/sched/fair.c                  | 113 ++++++++++++++++++++++++++-
 3 files changed, 137 insertions(+), 2 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 1c93da75226e..c4959747374b 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -833,6 +833,16 @@ struct task_ipi_mask {
 struct task_ipi_mask { };
 #endif
 
+#ifdef CONFIG_NUMA_BALANCING
+enum numa_tiering_fault_stats {
+	NUMA_TIERING_FAULT_COLD,
+	NUMA_TIERING_FAULT_HOT,
+	NUMA_TIERING_FAULT_THROTTLED,
+	NUMA_TIERING_FAULT_MIGRATE_FAIL,
+	NR_NUMA_TIERING_FAULT_STATS
+};
+#endif
+
 struct task_struct {
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/*
@@ -1433,6 +1443,13 @@ struct task_struct {
 	 */
 	unsigned long			numa_faults_locality[3];
 
+	/*
+	 * Memory tiering faults are tracked separately from normal NUMA
+	 * placement statistics. In tiering-only mode, slow-memory hint faults
+	 * are used to decide promotion, not task placement.
+	 */
+	unsigned long			numa_tiering_faults[NR_NUMA_TIERING_FAULT_STATS];
+
 	unsigned long			numa_pages_migrated;
 #endif /* CONFIG_NUMA_BALANCING */
 
diff --git a/include/linux/sched/numa_balancing.h b/include/linux/sched/numa_balancing.h
index 200f50075d5c..12f31ed56d66 100644
--- a/include/linux/sched/numa_balancing.h
+++ b/include/linux/sched/numa_balancing.h
@@ -8,6 +8,7 @@
  */
 
 #include <linux/sched.h>
+#include <linux/sched/sysctl.h>
 
 #define TNF_MIGRATED	0x01
 #define TNF_NO_GROUP	0x02
@@ -16,6 +17,14 @@
 #define TNF_MIGRATE_FAIL 0x10
 #define TNF_ACCESS_TIME	0x20
 
+static inline bool numa_balancing_tiering_only(void)
+{
+	int mode = sysctl_numa_balancing_mode;
+
+	return (mode & NUMA_BALANCING_MEMORY_TIERING) &&
+	       !(mode & NUMA_BALANCING_NORMAL);
+}
+
 enum numa_vmaskip_reason {
 	NUMAB_SKIP_UNSUITABLE,
 	NUMAB_SKIP_SHARED_RO,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 136207b62d87..1d9efc2b7427 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -2760,6 +2760,32 @@ static inline bool cpupid_valid(int cpupid)
 	return cpupid_to_cpu(cpupid) < nr_cpu_ids;
 }
 
+static void task_numa_tiering_placement(struct task_struct *p);
+
+static bool task_numa_tiering_update(struct task_struct *p)
+{
+	if (!numa_balancing_tiering_only() || !p->mm)
+		return false;
+
+	task_numa_tiering_placement(p);
+	return true;
+}
+
+static void task_numa_tiering_no_feedback(struct task_struct *p)
+{
+	task_numa_tiering_update(p);
+}
+
+static void task_numa_tiering_fault(struct task_struct *p,
+				    enum numa_tiering_fault_stats stat,
+				    unsigned long pages)
+{
+	if (!task_numa_tiering_update(p))
+		return;
+
+	p->numa_tiering_faults[stat] += pages;
+}
+
 /*
  * For memory tiering mode, if the target fast memory node has enough free
  * pages, recently accessed slow memory pages can bypass the normal hotness
@@ -2887,6 +2913,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 		pgdat = NODE_DATA(dst_nid);
 		if (numa_balancing_promote_free_space_bypass &&
 		    pgdat_free_space_enough(pgdat)) {
+			task_numa_tiering_no_feedback(p);
 			/* workload changed, reset hot threshold */
 			pgdat->nbp_threshold = 0;
 			mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
@@ -2899,10 +2926,18 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 
 		th = pgdat->nbp_threshold ? : def_th;
 		latency = numa_hint_fault_latency(folio);
-		if (latency >= th)
+		if (latency >= th) {
+			task_numa_tiering_fault(p, NUMA_TIERING_FAULT_COLD, nr);
+			return false;
+		}
+
+		if (numa_promotion_rate_limit(pgdat, rate_limit, nr)) {
+			task_numa_tiering_fault(p, NUMA_TIERING_FAULT_THROTTLED, nr);
 			return false;
+		}
 
-		return !numa_promotion_rate_limit(pgdat, rate_limit, nr);
+		task_numa_tiering_fault(p, NUMA_TIERING_FAULT_HOT, nr);
+		return true;
 	}
 
 	this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
@@ -3708,6 +3743,48 @@ static void update_task_scan_period(struct task_struct *p,
 	       sizeof(p->numa_faults_locality));
 }
 
+/*
+ * Memory tiering uses NUMA hint faults to identify recently accessed slow
+ * memory that should be promoted. In tiering-only mode, those faults are not
+ * normal task-placement feedback, so use tiering-specific outcomes to adjust
+ * the scan period.
+ */
+static void update_task_tiering_scan_period(struct task_struct *p)
+{
+	unsigned long cold, hot, throttled, failed;
+	unsigned long total;
+	s64 period, period_slot;
+	s64 diff = 0;
+
+	cold = p->numa_tiering_faults[NUMA_TIERING_FAULT_COLD];
+	hot = p->numa_tiering_faults[NUMA_TIERING_FAULT_HOT];
+	throttled = p->numa_tiering_faults[NUMA_TIERING_FAULT_THROTTLED];
+	failed = p->numa_tiering_faults[NUMA_TIERING_FAULT_MIGRATE_FAIL];
+	total = cold + hot + throttled + failed;
+
+	period = p->numa_scan_period;
+	period_slot = DIV_ROUND_UP_ULL(period, NUMA_PERIOD_SLOTS);
+
+	if (!total) {
+		diff = period;
+	} else if (failed) {
+		diff = period_slot;
+	} else if (throttled * NUMA_PERIOD_SLOTS >=
+		   total * (NUMA_PERIOD_SLOTS - NUMA_PERIOD_THRESHOLD)) {
+		diff = period_slot;
+	} else if (cold * NUMA_PERIOD_SLOTS >=
+		   total * NUMA_PERIOD_THRESHOLD) {
+		diff = period_slot;
+	} else if (hot * NUMA_PERIOD_SLOTS >=
+		   total * NUMA_PERIOD_THRESHOLD) {
+		diff = -period_slot;
+	}
+
+	p->numa_scan_period = clamp_t(s64, period + diff,
+				      task_scan_min(p), task_scan_max(p));
+	memset(p->numa_tiering_faults, 0, sizeof(p->numa_tiering_faults));
+}
+
 /*
  * Get the fraction of time the task has been running since the last
  * NUMA placement cycle. The scheduler keeps similar statistics, but
@@ -3974,6 +4051,22 @@ static void task_numa_placement(struct task_struct *p)
 	update_task_scan_period(p, fault_types[0], fault_types[1]);
 }
 
+static void task_numa_tiering_placement(struct task_struct *p)
+{
+	int seq;
+
+	if (!p->mm)
+		return;
+
+	seq = READ_ONCE(p->mm->numa_scan_seq);
+	if (p->numa_scan_seq == seq)
+		return;
+	p->numa_scan_seq = seq;
+	p->numa_scan_period_max = task_scan_max(p);
+
+	update_task_tiering_scan_period(p);
+}
+
 static inline int get_numa_group(struct numa_group *grp)
 {
 	return refcount_inc_not_zero(&grp->refcount);
@@ -4105,6 +4198,8 @@ void task_numa_free(struct task_struct *p, bool final)
 	unsigned long flags;
 	int i;
 
+	memset(p->numa_tiering_faults, 0, sizeof(p->numa_tiering_faults));
+
 	if (!numa_faults)
 		return;
 
@@ -4158,6 +4253,11 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
 		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL)) {
 			if (migrated)
 				p->numa_pages_migrated += pages;
+			if (numa_balancing_tiering_only() &&
+			    (flags & TNF_MIGRATE_FAIL))
+				task_numa_tiering_fault(p,
+						NUMA_TIERING_FAULT_MIGRATE_FAIL,
+						pages);
 			return;
 		}
 	}
@@ -4529,6 +4629,14 @@ static void task_numa_work(struct callback_head *work)
 		reset_ptenuma_scan(p);
 	mmap_read_unlock(mm);
 
+	/*
+	 * Tiering-only placement feedback is still scan-window based. Consume
+	 * the window when a scan sequence completes, so an empty tiering
+	 * window can back off after qualifying tiering faults stop.
+	 */
+	if (numa_balancing_tiering_only())
+		task_numa_tiering_placement(p);
+
 	/*
 	 * Make sure tasks use at least 32x as much time to run other code
 	 * than they used here, to limit NUMA PTE scanning overhead to 3% max.
@@ -4562,6 +4670,7 @@ void init_numa_balancing(u64 clone_flags, struct task_struct *p)
 	p->numa_faults			= NULL;
 	p->numa_pages_migrated		= 0;
 	p->total_numa_faults		= 0;
+	memset(p->numa_tiering_faults, 0, sizeof(p->numa_tiering_faults));
 	RCU_INIT_POINTER(p->numa_group, NULL);
 	p->last_task_numa_placement	= 0;
 	p->last_sum_exec_runtime	= 0;
-- 
2.20.1

Re: [PATCH v2 0/3] sched/numa: refine memory tiering promotion heuristics

[Peter Zijlstra]

On Wed, Oct 28, 2026 at 12:12:50PM +0800, Li Zhe wrote:
> This version narrows the series to memory tiering.  The generic normal
> NUMA balancing cleanups and heuristic changes from v1 are not included.

Should we not look at Bharata's pghot stuff first and untangle this numa
balancing vs tiering nonsense, before making an even bigger mess of
things?

Re: [PATCH v2 2/3] sched/numa: do not feed tiering promotions into task placement stats

[Joshua Hahn]

On Wed, Oct 28, 2026 at 12:14 AM Li Zhe <lizhe.67@bytedance.com> wrote:
>
> NUMA balancing has separate mode bits for normal task placement and
> memory tiering. In memory-tiering-only mode, NUMA hinting faults on
> slow-tier pages are used to detect hot pages and promote them to a top
> tier. They are not normal task-placement faults.
>
> For slow-tier folios, memory tiering reuses the folio last-cpupid field
> to store access-time metadata. numa_migrate_check() therefore cannot
> return a real cpupid for those faults. Before this change, it returned the
> same all-ones last_cpupid value that normal AutoNUMA uses for a first or
> unknown access.
>
> That can make a memory-tiering-only promotion look like an ordinary first
> private AutoNUMA fault. After a successful promotion, do_numa_page()
> reports the target top-tier node to task_numa_fault(), so the existing
> slow-node skip no longer applies. task_numa_fault() then accounts the
> fault in p->numa_faults and p->numa_faults_locality, and
> task_numa_placement() can consume that sample to update normal task
> placement state even though normal NUMA balancing is disabled. In other
> words, hot-page promotion feedback can leak into the CPU placement
> heuristic.

Hi Zhe,

Thanks for the revision. To me, this looks more like a fix than an
optimization, since there really is incorrect NUMA balancing behavior
happening that could be visible in userspace (unintended
task-placement and related accounting). On the other hand patch 3/3
doesn't seem like a fix to me despite the name, since this one would
mitigate the incorrect behavior.

How about we pull this one out of the series and submit it
independently with an appropriate Fixes: tag and cc:stable, and
discuss 1 and 3 later on? Unfortunately this is also not a great time
to submit new features or behaviors as we are in the middle of a large
merge window, and there are a bunch of fixes that maintainers might be
busy looking into : -(

Regardless, thank you for the fix, Zhe! I don't have a strong
preference on the ordering of this against pghot, but if other folks
are concerned that this could create more churn, I think it would be
reasonable to wait for patches 1 and 3 after we discuss with Bharata.

I hope you have a great day : -)
Joshua