[PATCH v7 linux 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work

[PATCH v7 linux 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work

[Luo Gengkun]

Hi everyone,
The cache-aware scheduling patches have now been merged into the mainline.
The goal of this series is to reduce the overhead of task_cache_work() by
minimizing the number of scanned CPUs, which yields significant
performance gains in multi-instance scenarios like Redis. To facilitate
testing, a debug patch is appended.

Below are the Redis test results conducted on AMD server.

When NUMA balancing is disabled:

valkey-benchmark rps  |      baseline       |         schedcache              |     schedcache_visit
                      |  (no cache aware)   |    (without this series)        |     (with this series)
----------------------+---------------------+---------------------+-----------+-------------------+--------
                      |   p99 latency(ms)   |    p99 latency(ms)  |   DIFF(%) |   p99 latency(ms) |  DIFF(%)
400000                |      0.436          |        0.554        |   -25.68% |         0.441     |   -1.14% 

Additionally, the output of perf top -e cycles:k highlights the overhead
incurred by task_cache_work():

valkey-benchmark rps  |         schedcache                        |     schedcache_visit
                      |    (without this series)                  |     (with this series)
----------------------+-------------------------------------------+---------------------------------------
400000                |  0.81%  [kernel]  [k] task_cache_work     |  0.02%  [kernel]  [k] task_cache_work

The trace data is shown below:

echo NO_SC_VISIT > /sys/kernel/debug/sched/features
    redis-server-24660   [195] .....  2916.005989: sched_cache_scan: comm=redis-server pid=24660 scan=384
 valkey-benchmar-48836   [317] .....  2916.005993: sched_cache_scan: comm=valkey-benchmar pid=48836 scan=384
 valkey-benchmar-48487   [310] .....  2916.006002: sched_cache_scan: comm=valkey-benchmar pid=48487 scan=384

echo SC_VISIT > /sys/kernel/debug/sched/features
    redis-server-29080   [225] .....  2876.257909: sched_cache_scan: comm=redis-server pid=29080 scan=14    
    redis-server-26694   [018] .....  2876.257909: sched_cache_scan: comm=redis-server pid=26694 scan=13    
 valkey-benchmar-48198   [103] .....  2876.257909: sched_cache_scan: comm=valkey-benchmar pid=48198 scan=16 

The above results show that this series significantly reduces the overhead
of task_cache_work() by decreasing the number of CPUs that need to be
scanned, from 384 to 16, when NUMA balancing is disabled.

Even with NUMA balancing enabled, this patch provides better performance
benefits. The following tests were conducted with NUMA balancing enabled:

valkey-benchmark rps  |      baseline       |         schedcache              |     schedcache_visit
                      |  (no cache aware)   |    (without this series)        |     (with this series)
----------------------+---------------------+---------------------+-----------+-------------------+--------
                      |   p99 latency(ms)   |    p99 latency(ms)  |   DIFF(%) |   p99 latency(ms) |  DIFF(%)
400000                |      0.437          |        0.454        |   -3.89%  |         0.442     |   -1.14%

The overhead of task_cache_work() is as follows:

valkey-benchmark rps  |         schedcache                        |     schedcache_visit
                      |    (without this series)                  |     (with this series)
----------------------+-------------------------------------------+---------------------------------------
400000                |  0.13%  [kernel]  [k] task_cache_work     |  0.03%  [kernel]  [k] task_cache_work 


Hackbench testing was also conducted. The results below show that this
series does not affect the accuracy of cache-aware scheduling.

=========================================
Hackbench Comparison: baseline vs schedcache
=========================================
MODE       GROUPS  FDS |     baseline(std)  |  schedcache(std)   |    DIFF(%) | VERDICT
---------- ------ -----+--------------------+--------------------+------------+-----------
threads         1   10 |    113.200 (4.22%) |     67.300 (1.32%) |     40.55% | IMPROVED
threads         1    2 |     16.555 (4.11%) |     11.020 (1.66%) |     33.43% | IMPROVED
threads         1   20 |    250.774 (1.26%) |    265.026 (5.44%) |     -5.68% | REGRESSED
threads         1    4 |     42.117 (1.44%) |     27.758 (1.64%) |     34.09% | IMPROVED
threads         1    6 |     65.140 (4.31%) |     39.182 (1.38%) |     39.85% | IMPROVED
threads         1    8 |     84.286 (1.29%) |     53.721 (1.58%) |     36.26% | IMPROVED
threads         2   10 |    122.592 (0.44%) |    113.365 (4.93%) |      7.53% | IMPROVED
threads         2    2 |     17.702 (4.09%) |     10.473 (0.42%) |     40.84% | IMPROVED
threads         2   20 |    336.457 (1.77%) |    314.108 (1.51%) |      6.64% | IMPROVED
threads         2    4 |     43.989 (0.88%) |     27.067 (3.38%) |     38.47% | IMPROVED
threads         2    6 |     69.322 (0.85%) |     41.707 (4.19%) |     39.84% | IMPROVED
threads         2    8 |    103.767 (1.81%) |     58.518 (3.00%) |     43.61% | IMPROVED
threads         4   10 |    148.882 (3.56%) |    149.449 (1.06%) |     -0.38% | REGRESSED
threads         4    2 |     18.909 (2.96%) |     11.063 (2.08%) |     41.49% | IMPROVED
threads         4   20 |    724.943 (2.14%) |    631.222 (3.92%) |     12.93% | IMPROVED
threads         4    4 |     48.191 (1.91%) |     27.352 (5.35%) |     43.24% | IMPROVED
threads         4    6 |     79.725 (3.84%) |     78.732 (4.10%) |      1.25% | IMPROVED
threads         4    8 |    108.768 (1.36%) |    105.928 (1.65%) |      2.61% | IMPROVED
     
Hackbench Comparison: schedcache vs schedcache_visit
=========================================
MODE       GROUPS  FDS |    schedcache(std) |schedcache_visit(std) | DIFF(%) | VERDICT
---------- ------ -----+--------------------+----------------------+---------+-----------
threads         1   10 |     67.300 (1.32%) |     67.014 (0.96%)   |   0.42% | IMPROVED
threads         1    2 |     11.020 (1.66%) |     10.557 (1.46%)   |   4.20% | IMPROVED
threads         1   20 |    265.026 (5.44%) |   212.366 (16.32%)   |  19.87% | IMPROVED
threads         1    4 |     27.758 (1.64%) |     25.711 (1.32%)   |   7.37% | IMPROVED
threads         1    6 |     39.182 (1.38%) |     38.914 (0.34%)   |   0.68% | IMPROVED
threads         1    8 |     53.721 (1.58%) |     52.889 (0.27%)   |   1.55% | IMPROVED
threads         2   10 |    121.203 (6.99%) |    124.254 (1.38%)   |  -2.52% | REGRESSED
threads         2    2 |     10.473 (0.42%) |     11.206 (5.91%)   |  -7.00% | REGRESSED
threads         2   20 |    314.108 (1.51%) |    301.754 (1.95%)   |   3.93% | IMPROVED
threads         2    4 |     27.067 (3.38%) |     28.028 (2.01%)   |  -3.55% | REGRESSED
threads         2    6 |     41.707 (4.19%) |     42.149 (3.35%)   |  -1.06% | REGRESSED
threads         2    8 |     58.518 (3.00%) |     57.133 (4.39%)   |   2.37% | IMPROVED
threads         4   10 |    149.449 (1.06%) |    141.407 (0.08%)   |   5.38% | IMPROVED
threads         4    2 |     11.063 (2.08%) |     11.360 (5.85%)   |  -2.68% | REGRESSED
threads         4   20 |    631.222 (3.92%) |    622.780 (2.49%)   |   1.34% | IMPROVED
threads         4    4 |     27.352 (5.35%) |     27.947 (5.37%)   |  -2.18% | REGRESSED
threads         4    6 |     78.732 (4.10%) |     73.911 (0.70%)   |   6.12% | IMPROVED
threads         4    8 |    105.928 (1.65%) |    107.535 (3.29%)   |  -1.52% | REGRESSED 

---
Changes history
**v7 Changes:**
1. Add code comments in task_cache_work() to clarify the race condition
between setting and reading visited_cpus.

Link to v6: https://lore.kernel.org/all/20260717041231.3324851-1-luogengkun2@huawei.com/

**v6 Changes:**
1. Switched to cpumask_var_t to dynamically allocate visited_cpus inside
mm_struct, preventing per-process memory bloat when NR_CPUS is configured
large.
2. Intersected the LLC scheduling domain span directly with visited_cpus
to prevent skipping CPU is cross-node LLC topolgies.
3. Renamed epoch_timeout to epoch_last_visit to better reflect its purpose.
4. Moved __update_mm_sched to execute before the timeout check in
fraction_mm_sched()

Link to v5: https://lore.kernel.org/all/20260709130053.2749834-1-luogengkun2@huawei.com/

**v5 Changes:**
1. Restore get_scan_cpumasks() to avoid violating NUMA_BALANCING constraints.
2. Use for_each_cpu_and() to filter CPUs within the LLC domain.

Link to v4: https://lore.kernel.org/all/20260618064300.80072-1-luogengkun2@huawei.com/

**v4 Changes:**
1. Rebase to the master.
2. epoch_timeout is introduced to evict expired CPUs instead of relying on
epoch, because epoch is refreshed periodically due to invocations of
fraction_mm_sched().
3. Move the increasement of nr_running before fraction_mm_sched().
4. Remove the redundant 'work->next' reset at the end of task_cache_work().
5. Add a debug patch to show the number of CPUs scanned to show the
benefit of this optimization.

Link to v3: https://lore.kernel.org/all/20260423085414.1389749-1-luogengkun2@huawei.com/

**v3 Changes:**
1. Remove the static key and enable this feature by default.
2. Reuse llc_epoch_affinity_timeout instead of introducing
llc_epoch_visited_timeout.
3. Move the calculation of rq->cpu_epoch - pcpu_sched->epoch into
fraction_mm_sched() to avoid race between task_cache_work() and
__update_mm_sched(). 
4. Reset work->next at the end of task_cache_work() to prevent concurrent
executions by multiple threads within the same process.

Link to v2: https://lore.kernel.org/all/20260414150745.225416-1-luogengkun2@huawei.com/

**v2 Changes:**
1. Added a pre-check before set/clear visited_cpus to avoid C2C overhead.
2. Optimized llc_epoch_visited_timeout by using a static key to minimize overhead.

Link to v1: https://lore.kernel.org/all/f2488085-4b52-491d-84be-d30d43954381@huawei.com/
---

Luo Gengkun (2):
  sched/cache: Reduce the overhead of task_cache_work by only scan the
    visisted cpus
  -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched
    feature to track scan cost

 include/linux/mm_types.h     |  6 ++++
 include/linux/sched.h        |  2 ++
 include/trace/events/sched.h | 21 ++++++++++++
 kernel/sched/fair.c          | 66 ++++++++++++++++++++++++++++++------
 kernel/sched/features.h      |  1 +
 5 files changed, 85 insertions(+), 11 deletions(-)

-- 
2.34.1

[PATCH v7 linux 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus

[Luo Gengkun]

The overhead of task_cache_work() is high, especially in multi-NUMA systems.
Currently, task_cache_work() tries to find the pref_llc by scanning all CPUs
in the system. However, most of these scans are meaningless, such as those
for CPUs that have never been visited or were accessed a long time ago.

To address this problem, introduce visited_cpus to track the visited CPUs
and evict them once they have not been accessed for a duration exceeding
llc_epoch_affinity_timeout.

Signed-off-by: Luo Gengkun <luogengkun2@huawei.com>
---
 include/linux/mm_types.h |  6 +++++
 include/linux/sched.h    |  2 ++
 kernel/sched/fair.c      | 48 +++++++++++++++++++++++++++++++---------
 3 files changed, 46 insertions(+), 10 deletions(-)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index b18c2b2e7d2c..35559079e4d4 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1620,6 +1620,11 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm)
 	if (!pcpu_sched)
 		return -ENOMEM;
 
+	if (!zalloc_cpumask_var(&mm->sc_stat.visited_cpus, GFP_KERNEL)) {
+		free_percpu(pcpu_sched);
+		return -ENOMEM;
+	}
+
 	mm_init_sched(mm, pcpu_sched);
 	return 0;
 }
@@ -1630,6 +1635,7 @@ static inline void mm_destroy_sched(struct mm_struct *mm)
 {
 	free_percpu(mm->sc_stat.pcpu_sched);
 	mm->sc_stat.pcpu_sched = NULL;
+	free_cpumask_var(mm->sc_stat.visited_cpus);
 }
 #else /* !CONFIG_SCHED_CACHE */
 
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 373bcc0598d1..b461a71a65da 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2388,6 +2388,7 @@ static __always_inline int task_mm_cid(struct task_struct *t)
 struct sched_cache_time {
 	u64 runtime;
 	unsigned long epoch;
+	unsigned long epoch_last_visit;
 };
 
 struct sched_cache_stat {
@@ -2398,6 +2399,7 @@ struct sched_cache_stat {
 	unsigned long next_scan;
 	unsigned long footprint;
 	int cpu;
+	cpumask_var_t visited_cpus;
 } ____cacheline_aligned_in_smp;
 
 #else
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee1..c56b2df66de8 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1585,6 +1585,7 @@ void mm_init_sched(struct mm_struct *mm,
 		pcpu_sched->runtime = 0;
 		/* a slightly stale cpu epoch is acceptible */
 		pcpu_sched->epoch = rq->cpu_epoch;
+		pcpu_sched->epoch_last_visit = rq->cpu_epoch;
 		epoch = rq->cpu_epoch;
 	}
 
@@ -1635,13 +1636,23 @@ static inline void __update_mm_sched(struct rq *rq,
 	}
 }
 
-static unsigned long fraction_mm_sched(struct rq *rq,
-				       struct sched_cache_time *pcpu_sched)
+static unsigned long fraction_mm_sched(int cpu,
+				       struct mm_struct *mm)
 {
+	struct sched_cache_time *pcpu_sched =
+		per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu);
+	struct rq *rq = cpu_rq(cpu);
+
 	guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock);
 
 	__update_mm_sched(rq, pcpu_sched);
 
+	/* Skip the rq that has not been hit for a long time */
+	if ((rq->cpu_epoch - pcpu_sched->epoch_last_visit) > llc_epoch_affinity_timeout) {
+		cpumask_clear_cpu(cpu, mm->sc_stat.visited_cpus);
+		return 0;
+	}
+
 	/*
 	 * Runtime is a geometric series (r=0.5) and as such will sum to twice
 	 * the accumulation period, this means the multiplcation here should
@@ -1711,6 +1722,9 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 		pcpu_sched->runtime += delta_exec;
 		rq->cpu_runtime += delta_exec;
 		epoch = rq->cpu_epoch;
+		pcpu_sched->epoch_last_visit = epoch;
+		if (!cpumask_test_cpu(cpu_of(rq), mm->sc_stat.visited_cpus))
+			cpumask_set_cpu(cpu_of(rq), mm->sc_stat.visited_cpus);
 	}
 
 	/*
@@ -1867,6 +1881,18 @@ static void task_cache_work(struct callback_head *work)
 		guard(rcu)();
 
 		get_scan_cpumasks(cpus, p);
+		/*
+		 * Data race: While evaluating the visited_cpus without
+		 * a lock, a CPU could be concurrently set by
+		 * account_mm_sched(), meaning the scan might skip the newly
+		 * visited CPU if the bit changes during the scan. This is
+		 * a deliberate trade-off between accuracy and efficiency:
+		 * locking would prevent this race but incur extra overhead.
+		 * The missed runtime contribution is negligible because it
+		 * implies this process hasn't run on that CPU for a long
+		 * time, and will be captured in the next cycle.
+		 */
+		cpumask_and(cpus, cpus, mm->sc_stat.visited_cpus);
 
 		for_each_cpu(cpu, cpus) {
 			/* XXX sched_cluster_active */
@@ -1877,19 +1903,21 @@ static void task_cache_work(struct callback_head *work)
 			if (!sd)
 				continue;
 
-			for_each_cpu(i, sched_domain_span(sd)) {
-				occ = fraction_mm_sched(cpu_rq(i),
-							per_cpu_ptr(mm->sc_stat.pcpu_sched, i));
+			for_each_cpu_and(i, sched_domain_span(sd), mm->sc_stat.visited_cpus) {
+				cur = rcu_dereference_all(cpu_rq(i)->curr);
+				if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) &&
+				    cur->mm == mm)
+					nr_running++;
+
+				occ = fraction_mm_sched(i, mm);
+				if (occ == 0)
+					continue;
+
 				a_occ += occ;
 				if (occ > m_occ) {
 					m_occ = occ;
 					m_cpu = i;
 				}
-
-				cur = rcu_dereference_all(cpu_rq(i)->curr);
-				if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) &&
-				    cur->mm == mm)
-					nr_running++;
 			}
 
 			/*
-- 
2.34.1

[PATCH v7 linux 2/2] -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched feature to track scan cost

[Luo Gengkun]

To evaluate the effectiveness of the previous patch, this debug patch
introduces feature SC_VISIT as well as trace events to facilitate testing.

Signed-off-by: Luo Gengkun <luogengkun2@huawei.com>
---
 include/trace/events/sched.h | 21 +++++++++++++++++++++
 kernel/sched/fair.c          | 26 +++++++++++++++++++++-----
 kernel/sched/features.h      |  1 +
 3 files changed, 43 insertions(+), 5 deletions(-)

diff --git a/include/trace/events/sched.h b/include/trace/events/sched.h
index 535860581f15..aced624f198d 100644
--- a/include/trace/events/sched.h
+++ b/include/trace/events/sched.h
@@ -10,6 +10,27 @@
 #include <linux/tracepoint.h>
 #include <linux/binfmts.h>
 
+TRACE_EVENT(sched_cache_scan,
+
+	TP_PROTO(struct task_struct *t, int scan),
+
+	TP_ARGS(t, scan),
+
+	TP_STRUCT__entry(
+		__string(	comm,	t->comm		)
+		__field(	pid_t,	pid		)
+		__field(	int,	scan		)
+	),
+
+	TP_fast_assign(
+		__assign_str(comm);
+		__entry->pid	= t->pid;
+		__entry->scan	= scan;
+	),
+
+	TP_printk("comm=%s pid=%d scan=%d", __get_str(comm), __entry->pid,
+					__entry->scan)
+);
 /*
  * Tracepoint for calling kthread_stop, performed to end a kthread:
  */
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index c56b2df66de8..e5a2dd241f3e 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1648,7 +1648,8 @@ static unsigned long fraction_mm_sched(int cpu,
 	__update_mm_sched(rq, pcpu_sched);
 
 	/* Skip the rq that has not been hit for a long time */
-	if ((rq->cpu_epoch - pcpu_sched->epoch_last_visit) > llc_epoch_affinity_timeout) {
+	if (sched_feat(SC_VISIT) &&
+	    (rq->cpu_epoch - pcpu_sched->epoch_last_visit) > llc_epoch_affinity_timeout) {
 		cpumask_clear_cpu(cpu, mm->sc_stat.visited_cpus);
 		return 0;
 	}
@@ -1723,7 +1724,8 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 		rq->cpu_runtime += delta_exec;
 		epoch = rq->cpu_epoch;
 		pcpu_sched->epoch_last_visit = epoch;
-		if (!cpumask_test_cpu(cpu_of(rq), mm->sc_stat.visited_cpus))
+		if (sched_feat(SC_VISIT) &&
+		    !cpumask_test_cpu(cpu_of(rq), mm->sc_stat.visited_cpus))
 			cpumask_set_cpu(cpu_of(rq), mm->sc_stat.visited_cpus);
 	}
 
@@ -1846,7 +1848,8 @@ static void task_cache_work(struct callback_head *work)
 	unsigned long curr_m_a_occ = 0;
 	struct mm_struct *mm = p->mm;
 	unsigned long m_a_occ = 0;
-	cpumask_var_t cpus;
+	cpumask_var_t cpus, scan_cpus;
+	int scanned = 0;
 
 	WARN_ON_ONCE(work != &p->cache_work);
 
@@ -1877,6 +1880,11 @@ static void task_cache_work(struct callback_head *work)
 	if (!zalloc_cpumask_var(&cpus, GFP_KERNEL))
 		return;
 
+	if (!zalloc_cpumask_var(&scan_cpus, GFP_KERNEL)) {
+		free_cpumask_var(cpus);
+		return;
+	}
+
 	scoped_guard (cpus_read_lock) {
 		guard(rcu)();
 
@@ -1892,7 +1900,8 @@ static void task_cache_work(struct callback_head *work)
 		 * implies this process hasn't run on that CPU for a long
 		 * time, and will be captured in the next cycle.
 		 */
-		cpumask_and(cpus, cpus, mm->sc_stat.visited_cpus);
+		if (sched_feat(SC_VISIT))
+			cpumask_and(cpus, cpus, mm->sc_stat.visited_cpus);
 
 		for_each_cpu(cpu, cpus) {
 			/* XXX sched_cluster_active */
@@ -1903,12 +1912,17 @@ static void task_cache_work(struct callback_head *work)
 			if (!sd)
 				continue;
 
-			for_each_cpu_and(i, sched_domain_span(sd), mm->sc_stat.visited_cpus) {
+			cpumask_copy(scan_cpus, sched_domain_span(sd));
+			if (sched_feat(SC_VISIT))
+				cpumask_and(scan_cpus, scan_cpus, mm->sc_stat.visited_cpus);
+
+			for_each_cpu(i, scan_cpus) {
 				cur = rcu_dereference_all(cpu_rq(i)->curr);
 				if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) &&
 				    cur->mm == mm)
 					nr_running++;
 
+				scanned++;
 				occ = fraction_mm_sched(i, mm);
 				if (occ == 0)
 					continue;
@@ -1963,6 +1977,8 @@ static void task_cache_work(struct callback_head *work)
 
 	update_avg_scale(&mm->sc_stat.nr_running_avg, nr_running);
 	free_cpumask_var(cpus);
+	free_cpumask_var(scan_cpus);
+	trace_sched_cache_scan(p, scanned);
 }
 
 void init_sched_mm(struct task_struct *p)
diff --git a/kernel/sched/features.h b/kernel/sched/features.h
index 8f0dee8fc475..990714ebabfd 100644
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -142,3 +142,4 @@ SCHED_FEAT(LATENCY_WARN, false)
  */
 SCHED_FEAT(NI_RANDOM, true)
 SCHED_FEAT(NI_RATE, true)
+SCHED_FEAT(SC_VISIT, false)
-- 
2.34.1

Re: [PATCH v7 linux 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus

[Tim Chen]

On Mon, 2026-07-20 at 12:22 +0000, Luo Gengkun wrote:
> The overhead of task_cache_work() is high, especially in multi-NUMA systems.
> Currently, task_cache_work() tries to find the pref_llc by scanning all CPUs
> in the system. However, most of these scans are meaningless, such as those
> for CPUs that have never been visited or were accessed a long time ago.
> 
> To address this problem, introduce visited_cpus to track the visited CPUs
> and evict them once they have not been accessed for a duration exceeding
> llc_epoch_affinity_timeout.
> 
> Signed-off-by: Luo Gengkun <luogengkun2@huawei.com>
> ---
>  include/linux/mm_types.h |  6 +++++
>  include/linux/sched.h    |  2 ++
>  kernel/sched/fair.c      | 48 +++++++++++++++++++++++++++++++---------
>  3 files changed, 46 insertions(+), 10 deletions(-)
> 
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index b18c2b2e7d2c..35559079e4d4 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -1620,6 +1620,11 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm)
>  	if (!pcpu_sched)
>  		return -ENOMEM;
>  
> +	if (!zalloc_cpumask_var(&mm->sc_stat.visited_cpus, GFP_KERNEL)) {
> +		free_percpu(pcpu_sched);
> +		return -ENOMEM;
> +	}
> +
>  	mm_init_sched(mm, pcpu_sched);
>  	return 0;
>  }
> @@ -1630,6 +1635,7 @@ static inline void mm_destroy_sched(struct mm_struct *mm)
>  {
>  	free_percpu(mm->sc_stat.pcpu_sched);
>  	mm->sc_stat.pcpu_sched = NULL;
> +	free_cpumask_var(mm->sc_stat.visited_cpus);
>  }
>  #else /* !CONFIG_SCHED_CACHE */
>  
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 373bcc0598d1..b461a71a65da 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -2388,6 +2388,7 @@ static __always_inline int task_mm_cid(struct task_struct *t)
>  struct sched_cache_time {
>  	u64 runtime;
>  	unsigned long epoch;
> +	unsigned long epoch_last_visit;
>  };
>  
>  struct sched_cache_stat {
> @@ -2398,6 +2399,7 @@ struct sched_cache_stat {
>  	unsigned long next_scan;
>  	unsigned long footprint;
>  	int cpu;
> +	cpumask_var_t visited_cpus;
>  } ____cacheline_aligned_in_smp;
>  
>  #else
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index d78467ec6ee1..c56b2df66de8 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -1585,6 +1585,7 @@ void mm_init_sched(struct mm_struct *mm,
>  		pcpu_sched->runtime = 0;
>  		/* a slightly stale cpu epoch is acceptible */
>  		pcpu_sched->epoch = rq->cpu_epoch;
> +		pcpu_sched->epoch_last_visit = rq->cpu_epoch;
>  		epoch = rq->cpu_epoch;
>  	}
>  
> @@ -1635,13 +1636,23 @@ static inline void __update_mm_sched(struct rq *rq,
>  	}
>  }
>  
> -static unsigned long fraction_mm_sched(struct rq *rq,
> -				       struct sched_cache_time *pcpu_sched)
> +static unsigned long fraction_mm_sched(int cpu,
> +				       struct mm_struct *mm)
>  {
> +	struct sched_cache_time *pcpu_sched =
> +		per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu);
> +	struct rq *rq = cpu_rq(cpu);
> +
>  	guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock);
>  
>  	__update_mm_sched(rq, pcpu_sched);
>  
> +	/* Skip the rq that has not been hit for a long time */
> +	if ((rq->cpu_epoch - pcpu_sched->epoch_last_visit) > llc_epoch_affinity_timeout) {
> +		cpumask_clear_cpu(cpu, mm->sc_stat.visited_cpus);
> +		return 0;
> +	}
> +
>  	/*
>  	 * Runtime is a geometric series (r=0.5) and as such will sum to twice
>  	 * the accumulation period, this means the multiplcation here should
> @@ -1711,6 +1722,9 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
>  		pcpu_sched->runtime += delta_exec;
>  		rq->cpu_runtime += delta_exec;
>  		epoch = rq->cpu_epoch;
> +		pcpu_sched->epoch_last_visit = epoch;

We need to make sure that the epoch_last_visit update is seen by the cpu
running task_cache_work(), before it attempts to do the epoch comparison
and clear the cpu, and causing inconsistency in the visited_cpus.

For example

CPU A (e.g. doing LLC/affinity selection, reading remote pcpu_sched)      CPU B (= `cpu`, running account_mm_sched() locally)
--------------------------------------------------------------------      -----------------------------------------------------
read pcpu_sched->epoch_last_visit  (stale, old value)
  -> looks like it timed out
                                                                            pcpu_sched->epoch_last_visit = epoch   (fresh visit!)
                                                                            cpumask_set_cpu(cpu, visited_cpus)     (correctly marks it visited)
cpumask_clear_cpu(cpu, visited_cpus)   <-- wipes out the fresh set!


Perhaps we will need a memory barrier and double check for expiration before we clear
the visited mask.

 if (rq->cpu_epoch - READ_ONCE(pcpu_sched->epoch_last_visit > llc_epoch_affinity_timeout &&
	cpumask_test_cpu(cpu, mm-sc_stat.visited_cpus)) {

	/*
	 * account_mm_sched() may have raced in right here and refreshed
	 * epoch_last_visit + set the bit again after we read it as stale.
	 * Re-check timeout after applying memory barrier.
	 */
	smp_mb();
	if ((rq->cpu_epoch - READ_ONCE(pcpu_sched->epoch_last_visit)) > llc_epoch_affinity_timeout))
		cpumask_clear_cpu(cpu, mm->sc_stat.visited_cpus);	

 }

This fix is ugly and there's probably a better way.

> +		if (!cpumask_test_cpu(cpu_of(rq), mm->sc_stat.visited_cpus))
> +			cpumask_set_cpu(cpu_of(rq), mm->sc_stat.visited_cpus);
>  	}
>  
>  	/*
> @@ -1867,6 +1881,18 @@ static void task_cache_work(struct callback_head *work)
>  		guard(rcu)();
>  
>  		get_scan_cpumasks(cpus, p);


Now that we know exactly which cpus to scan from visited_cpus,
we can get rid of get_scan_cpumasks(), which attempts to limit
the scope of cpu scan.

Thanks.

Tim


> +		/*
> +		 * Data race: While evaluating the visited_cpus without
> +		 * a lock, a CPU could be concurrently set by
> +		 * account_mm_sched(), meaning the scan might skip the newly
> +		 * visited CPU if the bit changes during the scan. This is
> +		 * a deliberate trade-off between accuracy and efficiency:
> +		 * locking would prevent this race but incur extra overhead.
> +		 * The missed runtime contribution is negligible because it
> +		 * implies this process hasn't run on that CPU for a long
> +		 * time, and will be captured in the next cycle.
> +		 */
> +		cpumask_and(cpus, cpus, mm->sc_stat.visited_cpus);
>  
>  		for_each_cpu(cpu, cpus) {
>  			/* XXX sched_cluster_active */
> @@ -1877,19 +1903,21 @@ static void task_cache_work(struct callback_head *work)
>  			if (!sd)
>  				continue;
>  
> -			for_each_cpu(i, sched_domain_span(sd)) {
> -				occ = fraction_mm_sched(cpu_rq(i),
> -							per_cpu_ptr(mm->sc_stat.pcpu_sched, i));
> +			for_each_cpu_and(i, sched_domain_span(sd), mm->sc_stat.visited_cpus) {
> +				cur = rcu_dereference_all(cpu_rq(i)->curr);
> +				if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) &&
> +				    cur->mm == mm)
> +					nr_running++;
> +
> +				occ = fraction_mm_sched(i, mm);
> +				if (occ == 0)
> +					continue;
> +
>  				a_occ += occ;
>  				if (occ > m_occ) {
>  					m_occ = occ;
>  					m_cpu = i;
>  				}
> -
> -				cur = rcu_dereference_all(cpu_rq(i)->curr);
> -				if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) &&
> -				    cur->mm == mm)
> -					nr_running++;
>  			}
>  
>  			/*

Re: [PATCH v7 linux 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus

[Luo Gengkun]

On 2026/7/21 6:09, Tim Chen wrote:
> On Mon, 2026-07-20 at 12:22 +0000, Luo Gengkun wrote:
>> The overhead of task_cache_work() is high, especially in multi-NUMA systems.
>> Currently, task_cache_work() tries to find the pref_llc by scanning all CPUs
>> in the system. However, most of these scans are meaningless, such as those
>> for CPUs that have never been visited or were accessed a long time ago.
>>
>> To address this problem, introduce visited_cpus to track the visited CPUs
>> and evict them once they have not been accessed for a duration exceeding
>> llc_epoch_affinity_timeout.
>>
>> Signed-off-by: Luo Gengkun <luogengkun2@huawei.com>
>> ---
>>   include/linux/mm_types.h |  6 +++++
>>   include/linux/sched.h    |  2 ++
>>   kernel/sched/fair.c      | 48 +++++++++++++++++++++++++++++++---------
>>   3 files changed, 46 insertions(+), 10 deletions(-)
>>
>> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
>> index b18c2b2e7d2c..35559079e4d4 100644
>> --- a/include/linux/mm_types.h
>> +++ b/include/linux/mm_types.h
>> @@ -1620,6 +1620,11 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm)
>>   	if (!pcpu_sched)
>>   		return -ENOMEM;
>>   
>> +	if (!zalloc_cpumask_var(&mm->sc_stat.visited_cpus, GFP_KERNEL)) {
>> +		free_percpu(pcpu_sched);
>> +		return -ENOMEM;
>> +	}
>> +
>>   	mm_init_sched(mm, pcpu_sched);
>>   	return 0;
>>   }
>> @@ -1630,6 +1635,7 @@ static inline void mm_destroy_sched(struct mm_struct *mm)
>>   {
>>   	free_percpu(mm->sc_stat.pcpu_sched);
>>   	mm->sc_stat.pcpu_sched = NULL;
>> +	free_cpumask_var(mm->sc_stat.visited_cpus);
>>   }
>>   #else /* !CONFIG_SCHED_CACHE */
>>   
>> diff --git a/include/linux/sched.h b/include/linux/sched.h
>> index 373bcc0598d1..b461a71a65da 100644
>> --- a/include/linux/sched.h
>> +++ b/include/linux/sched.h
>> @@ -2388,6 +2388,7 @@ static __always_inline int task_mm_cid(struct task_struct *t)
>>   struct sched_cache_time {
>>   	u64 runtime;
>>   	unsigned long epoch;
>> +	unsigned long epoch_last_visit;
>>   };
>>   
>>   struct sched_cache_stat {
>> @@ -2398,6 +2399,7 @@ struct sched_cache_stat {
>>   	unsigned long next_scan;
>>   	unsigned long footprint;
>>   	int cpu;
>> +	cpumask_var_t visited_cpus;
>>   } ____cacheline_aligned_in_smp;
>>   
>>   #else
>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> index d78467ec6ee1..c56b2df66de8 100644
>> --- a/kernel/sched/fair.c
>> +++ b/kernel/sched/fair.c
>> @@ -1585,6 +1585,7 @@ void mm_init_sched(struct mm_struct *mm,
>>   		pcpu_sched->runtime = 0;
>>   		/* a slightly stale cpu epoch is acceptible */
>>   		pcpu_sched->epoch = rq->cpu_epoch;
>> +		pcpu_sched->epoch_last_visit = rq->cpu_epoch;
>>   		epoch = rq->cpu_epoch;
>>   	}
>>   
>> @@ -1635,13 +1636,23 @@ static inline void __update_mm_sched(struct rq *rq,
>>   	}
>>   }
>>   
>> -static unsigned long fraction_mm_sched(struct rq *rq,
>> -				       struct sched_cache_time *pcpu_sched)
>> +static unsigned long fraction_mm_sched(int cpu,
>> +				       struct mm_struct *mm)
>>   {
>> +	struct sched_cache_time *pcpu_sched =
>> +		per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu);
>> +	struct rq *rq = cpu_rq(cpu);
>> +
>>   	guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock);
>>   
>>   	__update_mm_sched(rq, pcpu_sched);
>>   
>> +	/* Skip the rq that has not been hit for a long time */
>> +	if ((rq->cpu_epoch - pcpu_sched->epoch_last_visit) > llc_epoch_affinity_timeout) {
>> +		cpumask_clear_cpu(cpu, mm->sc_stat.visited_cpus);
>> +		return 0;
>> +	}
>> +
>>   	/*
>>   	 * Runtime is a geometric series (r=0.5) and as such will sum to twice
>>   	 * the accumulation period, this means the multiplcation here should
>> @@ -1711,6 +1722,9 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
>>   		pcpu_sched->runtime += delta_exec;
>>   		rq->cpu_runtime += delta_exec;
>>   		epoch = rq->cpu_epoch;
>> +		pcpu_sched->epoch_last_visit = epoch;
> 
> We need to make sure that the epoch_last_visit update is seen by the cpu
> running task_cache_work(), before it attempts to do the epoch comparison
> and clear the cpu, and causing inconsistency in the visited_cpus.
> 
> For example
> 
> CPU A (e.g. doing LLC/affinity selection, reading remote pcpu_sched)      CPU B (= `cpu`, running account_mm_sched() locally)
> --------------------------------------------------------------------      -----------------------------------------------------
> read pcpu_sched->epoch_last_visit  (stale, old value)
>    -> looks like it timed out
>                                                                              pcpu_sched->epoch_last_visit = epoch   (fresh visit!)
>                                                                              cpumask_set_cpu(cpu, visited_cpus)     (correctly marks it visited)
> cpumask_clear_cpu(cpu, visited_cpus)   <-- wipes out the fresh set!
> 
I don't think such a race condition will occur, because both the read and clear
operations in fraction_mm_sched() are protected by rq->cpu_epoch_lock, just like
account_mm_sched(). Please let me know if I'm missing something.

> 
> Perhaps we will need a memory barrier and double check for expiration before we clear
> the visited mask.
> 
>   if (rq->cpu_epoch - READ_ONCE(pcpu_sched->epoch_last_visit > llc_epoch_affinity_timeout &&
> 	cpumask_test_cpu(cpu, mm-sc_stat.visited_cpus)) {
> 
> 	/*
> 	 * account_mm_sched() may have raced in right here and refreshed
> 	 * epoch_last_visit + set the bit again after we read it as stale.
> 	 * Re-check timeout after applying memory barrier.
> 	 */
> 	smp_mb();
> 	if ((rq->cpu_epoch - READ_ONCE(pcpu_sched->epoch_last_visit)) > llc_epoch_affinity_timeout))
> 		cpumask_clear_cpu(cpu, mm->sc_stat.visited_cpus);	
> 
>   }
> 
> This fix is ugly and there's probably a better way.
> 
>> +		if (!cpumask_test_cpu(cpu_of(rq), mm->sc_stat.visited_cpus))
>> +			cpumask_set_cpu(cpu_of(rq), mm->sc_stat.visited_cpus);
>>   	}
>>   
>>   	/*
>> @@ -1867,6 +1881,18 @@ static void task_cache_work(struct callback_head *work)
>>   		guard(rcu)();
>>   
>>   		get_scan_cpumasks(cpus, p);
> 
> 
> Now that we know exactly which cpus to scan from visited_cpus,
> we can get rid of get_scan_cpumasks(), which attempts to limit
> the scope of cpu scan.

Thank you for your suggestion. I agree with removing get_scan_cpumasks()
because visited_cpus also tracks the numa_preferred_nid. In addition,
based on previous experimental analysis, using visited_cpus yields
better performance benefits.

Thanks
Gengkun
> 
> Thanks.
> 
> Tim
> 
> 
>> +		/*
>> +		 * Data race: While evaluating the visited_cpus without
>> +		 * a lock, a CPU could be concurrently set by
>> +		 * account_mm_sched(), meaning the scan might skip the newly
>> +		 * visited CPU if the bit changes during the scan. This is
>> +		 * a deliberate trade-off between accuracy and efficiency:
>> +		 * locking would prevent this race but incur extra overhead.
>> +		 * The missed runtime contribution is negligible because it
>> +		 * implies this process hasn't run on that CPU for a long
>> +		 * time, and will be captured in the next cycle.
>> +		 */
>> +		cpumask_and(cpus, cpus, mm->sc_stat.visited_cpus);
>>   
>>   		for_each_cpu(cpu, cpus) {
>>   			/* XXX sched_cluster_active */
>> @@ -1877,19 +1903,21 @@ static void task_cache_work(struct callback_head *work)
>>   			if (!sd)
>>   				continue;
>>   
>> -			for_each_cpu(i, sched_domain_span(sd)) {
>> -				occ = fraction_mm_sched(cpu_rq(i),
>> -							per_cpu_ptr(mm->sc_stat.pcpu_sched, i));
>> +			for_each_cpu_and(i, sched_domain_span(sd), mm->sc_stat.visited_cpus) {
>> +				cur = rcu_dereference_all(cpu_rq(i)->curr);
>> +				if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) &&
>> +				    cur->mm == mm)
>> +					nr_running++;
>> +
>> +				occ = fraction_mm_sched(i, mm);
>> +				if (occ == 0)
>> +					continue;
>> +
>>   				a_occ += occ;
>>   				if (occ > m_occ) {
>>   					m_occ = occ;
>>   					m_cpu = i;
>>   				}
>> -
>> -				cur = rcu_dereference_all(cpu_rq(i)->curr);
>> -				if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) &&
>> -				    cur->mm == mm)
>> -					nr_running++;
>>   			}
>>   
>>   			/*

Re: [PATCH v7 linux 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus

[Tim Chen]

On Tue, 2026-07-21 at 10:53 +0800, Luo Gengkun wrote:
> 
> 
snip

> > > -static unsigned long fraction_mm_sched(struct rq *rq,
> > > -				       struct sched_cache_time *pcpu_sched)
> > > +static unsigned long fraction_mm_sched(int cpu,
> > > +				       struct mm_struct *mm)
> > >   {
> > > +	struct sched_cache_time *pcpu_sched =
> > > +		per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu);
> > > +	struct rq *rq = cpu_rq(cpu);
> > > +
> > >   	guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock);
> > >   
> > >   	__update_mm_sched(rq, pcpu_sched);
> > >   
> > > +	/* Skip the rq that has not been hit for a long time */
> > > +	if ((rq->cpu_epoch - pcpu_sched->epoch_last_visit) > llc_epoch_affinity_timeout) {
> > > +		cpumask_clear_cpu(cpu, mm->sc_stat.visited_cpus);
> > > +		return 0;
> > > +	}
> > > +
> > >   	/*
> > >   	 * Runtime is a geometric series (r=0.5) and as such will sum to twice
> > >   	 * the accumulation period, this means the multiplcation here should
> > > @@ -1711,6 +1722,9 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
> > >   		pcpu_sched->runtime += delta_exec;
> > >   		rq->cpu_runtime += delta_exec;
> > >   		epoch = rq->cpu_epoch;
> > > +		pcpu_sched->epoch_last_visit = epoch;
> > 
> > We need to make sure that the epoch_last_visit update is seen by the cpu
> > running task_cache_work(), before it attempts to do the epoch comparison
> > and clear the cpu, and causing inconsistency in the visited_cpus.
> > 
> > For example
> > 
> > CPU A (e.g. doing LLC/affinity selection, reading remote pcpu_sched)      CPU B (= `cpu`, running account_mm_sched() locally)
> > --------------------------------------------------------------------      -----------------------------------------------------
> > read pcpu_sched->epoch_last_visit  (stale, old value)
> >    -> looks like it timed out
> >                                                                              pcpu_sched->epoch_last_visit = epoch   (fresh visit!)
> >                                                                              cpumask_set_cpu(cpu, visited_cpus)     (correctly marks it visited)
> > cpumask_clear_cpu(cpu, visited_cpus)   <-- wipes out the fresh set!
> > 
> I don't think such a race condition will occur, because both the read and clear
> operations in fraction_mm_sched() are protected by rq->cpu_epoch_lock, just like
> account_mm_sched(). Please let me know if I'm missing something.
> 

Yes, the epoch lock should prevent the above race from happening.
Missed that on my end.

> 
Thanks.

Tim