* [PATCH v5 linux 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work
@ 2026-07-09 13:00 Luo Gengkun
2026-07-09 13:00 ` [PATCH v5 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus Luo Gengkun
2026-07-09 13:00 ` [PATCH v5 2/2] -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched feature to track scan cost Luo Gengkun
0 siblings, 2 replies; 7+ messages in thread
From: Luo Gengkun @ 2026-07-09 13:00 UTC (permalink / raw)
To: peterz, mingo, juri.lelli, vincent.guittot, tim.c.chen, yu.c.chen
Cc: dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
kprateek.nayak, linux-kernel
Hi everyone,
The cache-aware scheduling patches have now been merged into the mainline.
The goal of this series is to reduce the overhead of task_cache_work() by
minimizing the number of scanned CPUs, which yields significant
performance gains in multi-instance scenarios like Redis. To facilitate
testing, a debug patch is appended.
Below are the Redis test results conducted on AMD server.
valkey-benchmark rps | baseline | schedcache | schedcache_visit
| (no cache aware) | (without this series) | (with this series)
----------------------+---------------------+---------------------+-----------+-------------------+--------
| p99 latency(ms) | p99 latency(ms) | DIFF(%) | p99 latency(ms) | DIFF(%)
200000 | 0.26 | 0.383 | -47.3% | 0.264 | -1.5%
300000 | 0.343 | 0.475 | -38.4% | 0.35 | -2.0%
400000 | 0.445 | 0.567 | -27.4% | 0.453 | -1.7%
Additionally, the output of perf top -e cycles:k highlights the overhead
incurred by task_cache_work:
valkey-benchmark rps | schedcache | schedcache_visit
----------------------+-------------------------------------------+---------------------------------------
200000 | 1.12% [kernel] [k] task_cache_work | 0.02% [kernel] [k] task_cache_work
300000 | 0.92% [kernel] [k] task_cache_work | 0.02% [kernel] [k] task_cache_work
400000 | 0.82% [kernel] [k] task_cache_work | 0.02% [kernel] [k] task_cache_work
The trace data is shown below:
echo NO_SC_VISIT > /sys/kernel/debug/sched/features
redis-server-24660 [195] ..... 2916.005989: sched_cache_scan: comm=redis-server pid=24660 scan=384
valkey-benchmar-48836 [317] ..... 2916.005993: sched_cache_scan: comm=valkey-benchmar pid=48836 scan=384
valkey-benchmar-48487 [310] ..... 2916.006002: sched_cache_scan: comm=valkey-benchmar pid=48487 scan=384
redis-server-24004 [005] ..... 2916.006016: sched_cache_scan: comm=redis-server pid=24004 scan=384
valkey-benchmar-48810 [322] ..... 2916.006017: sched_cache_scan: comm=valkey-benchmar pid=48810 scan=384
redis-server-29634 [287] ..... 2916.006024: sched_cache_scan: comm=redis-server pid=29634 scan=384
redis-server-26480 [081] ..... 2916.006027: sched_cache_scan: comm=redis-server pid=26480 scan=384
redis-server-29582 [086] ..... 2916.006027: sched_cache_scan: comm=redis-server pid=29582 scan=384
echo SC_VISIT > /sys/kernel/debug/sched/features
redis-server-29080 [225] ..... 2876.257909: sched_cache_scan: comm=redis-server pid=29080 scan=14
redis-server-26694 [018] ..... 2876.257909: sched_cache_scan: comm=redis-server pid=26694 scan=13
valkey-benchmar-48198 [103] ..... 2876.257909: sched_cache_scan: comm=valkey-benchmar pid=48198 scan=16
redis-server-26825 [269] ..... 2876.257909: sched_cache_scan: comm=redis-server pid=26825 scan=1
valkey-benchmar-48878 [295] ..... 2876.257910: sched_cache_scan: comm=valkey-benchmar pid=48878 scan=16
valkey-benchmar-47522 [290] ..... 2876.257910: sched_cache_scan: comm=valkey-benchmar pid=47522 scan=16
redis-server-25282 [048] ..... 2876.257910: sched_cache_scan: comm=redis-server pid=25282 scan=1
redis-server-29634 [071] ..... 2876.257910: sched_cache_scan: comm=redis-server pid=29634 scan=2
The above results show that this series significantly reduces the overhead
of task_cache_work() by decreasing the number of CPUs that need to be
scanned, from 384 to 16.
Hackbench testing was also conducted. The results below show that this
series does not affect the accuracy of cache-aware scheduling.
=========================================
Hackbench Comparison: baseline vs schedcache
=========================================
MODE GROUPS FDS | baseline(std) | schedcache(std) | DIFF(%) | VERDICT
---------- ------ -----+--------------------+--------------------+------------+-----------
threads 1 10 | 113.200 (4.22%) | 67.300 (1.32%) | 40.55% | IMPROVED
threads 1 2 | 16.555 (4.11%) | 11.020 (1.66%) | 33.43% | IMPROVED
threads 1 20 | 250.774 (1.26%) | 265.026 (5.44%) | -5.68% | REGRESSED
threads 1 4 | 42.117 (1.44%) | 27.758 (1.64%) | 34.09% | IMPROVED
threads 1 6 | 65.140 (4.31%) | 39.182 (1.38%) | 39.85% | IMPROVED
threads 1 8 | 84.286 (1.29%) | 53.721 (1.58%) | 36.26% | IMPROVED
threads 2 10 | 122.592 (0.44%) | 113.365 (4.93%) | 7.53% | IMPROVED
threads 2 2 | 17.702 (4.09%) | 10.473 (0.42%) | 40.84% | IMPROVED
threads 2 20 | 336.457 (1.77%) | 314.108 (1.51%) | 6.64% | IMPROVED
threads 2 4 | 43.989 (0.88%) | 27.067 (3.38%) | 38.47% | IMPROVED
threads 2 6 | 69.322 (0.85%) | 41.707 (4.19%) | 39.84% | IMPROVED
threads 2 8 | 103.767 (1.81%) | 58.518 (3.00%) | 43.61% | IMPROVED
threads 4 10 | 148.882 (3.56%) | 149.449 (1.06%) | -0.38% | REGRESSED
threads 4 2 | 18.909 (2.96%) | 11.063 (2.08%) | 41.49% | IMPROVED
threads 4 20 | 724.943 (2.14%) | 631.222 (3.92%) | 12.93% | IMPROVED
threads 4 4 | 48.191 (1.91%) | 27.352 (5.35%) | 43.24% | IMPROVED
threads 4 6 | 79.725 (3.84%) | 78.732 (4.10%) | 1.25% | IMPROVED
threads 4 8 | 108.768 (1.36%) | 105.928 (1.65%) | 2.61% | IMPROVED
Hackbench Comparison: schedcache vs schedcache_visit
=========================================
MODE GROUPS FDS | schedcache(std) |schedcache_visit(std) | DIFF(%) | VERDICT
---------- ------ -----+--------------------+----------------------+---------+-----------
threads 1 10 | 67.300 (1.32%) | 67.014 (0.96%) | 0.42% | IMPROVED
threads 1 2 | 11.020 (1.66%) | 10.557 (1.46%) | 4.20% | IMPROVED
threads 1 20 | 265.026 (5.44%) | 212.366 (16.32%) | 19.87% | IMPROVED
threads 1 4 | 27.758 (1.64%) | 25.711 (1.32%) | 7.37% | IMPROVED
threads 1 6 | 39.182 (1.38%) | 38.914 (0.34%) | 0.68% | IMPROVED
threads 1 8 | 53.721 (1.58%) | 52.889 (0.27%) | 1.55% | IMPROVED
threads 2 10 | 121.203 (6.99%) | 124.254 (1.38%) | -2.52% | REGRESSED
threads 2 2 | 10.473 (0.42%) | 11.206 (5.91%) | -7.00% | REGRESSED
threads 2 20 | 314.108 (1.51%) | 301.754 (1.95%) | 3.93% | IMPROVED
threads 2 4 | 27.067 (3.38%) | 28.028 (2.01%) | -3.55% | REGRESSED
threads 2 6 | 41.707 (4.19%) | 42.149 (3.35%) | -1.06% | REGRESSED
threads 2 8 | 58.518 (3.00%) | 57.133 (4.39%) | 2.37% | IMPROVED
threads 4 10 | 149.449 (1.06%) | 141.407 (0.08%) | 5.38% | IMPROVED
threads 4 2 | 11.063 (2.08%) | 11.360 (5.85%) | -2.68% | REGRESSED
threads 4 20 | 631.222 (3.92%) | 622.780 (2.49%) | 1.34% | IMPROVED
threads 4 4 | 27.352 (5.35%) | 27.947 (5.37%) | -2.18% | REGRESSED
threads 4 6 | 78.732 (4.10%) | 73.911 (0.70%) | 6.12% | IMPROVED
threads 4 8 | 105.928 (1.65%) | 107.535 (3.29%) | -1.52% | REGRESSED
---
Changes history
**v5 Changes:**
1. Restore get_scan_cpumasks() to avoid violating NUMA_BALANCING constraints.
2. Use for_each_cpu_and() to filter CPUs within the LLC domain.
Link to v4: https://lore.kernel.org/all/20260618064300.80072-1-luogengkun2@huawei.com/
**v4 Changes:**
1. Rebase to the master.
2. epoch_timeout is introduced to evict expired CPUs instead of relying on
epoch, because epoch is refreshed periodically due to invocations of
fraction_mm_sched().
3. Move the increasement of nr_running before fraction_mm_sched().
4. Remove the redundant 'work->next' reset at the end of task_cache_work().
5. Add a debug patch to show the number of CPUs scanned to show the
benefit of this optimization.
Link to v3: https://lore.kernel.org/all/20260423085414.1389749-1-luogengkun2@huawei.com/
**v3 Changes:**
1. Remove the static key and enable this feature by default.
2. Reuse llc_epoch_affinity_timeout instead of introducing
llc_epoch_visited_timeout.
3. Move the calculation of rq->cpu_epoch - pcpu_sched->epoch into
fraction_mm_sched() to avoid race between task_cache_work() and
__update_mm_sched().
4. Reset work->next at the end of task_cache_work() to prevent concurrent
executions by multiple threads within the same process.
Link to v2: https://lore.kernel.org/all/20260414150745.225416-1-luogengkun2@huawei.com/
**v2 Changes:**
1. Added a pre-check before set/clear visited_cpus to avoid C2C overhead.
2. Optimized llc_epoch_visited_timeout by using a static key to minimize overhead.
Link to v1: https://lore.kernel.org/all/f2488085-4b52-491d-84be-d30d43954381@huawei.com/
---
Luo Gengkun (2):
sched/cache: Reduce the overhead of task_cache_work by only scan the
visisted cpus
-- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched
feature to track scan cost
include/linux/sched.h | 2 ++
include/trace/events/sched.h | 21 +++++++++++++++
kernel/sched/fair.c | 52 +++++++++++++++++++++++++++++-------
kernel/sched/features.h | 1 +
4 files changed, 66 insertions(+), 10 deletions(-)
--
2.34.1
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH v5 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus 2026-07-09 13:00 [PATCH v5 linux 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work Luo Gengkun @ 2026-07-09 13:00 ` Luo Gengkun 2026-07-09 15:01 ` Chen, Yu C 2026-07-09 13:00 ` [PATCH v5 2/2] -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched feature to track scan cost Luo Gengkun 1 sibling, 1 reply; 7+ messages in thread From: Luo Gengkun @ 2026-07-09 13:00 UTC (permalink / raw) To: peterz, mingo, juri.lelli, vincent.guittot, tim.c.chen, yu.c.chen Cc: dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, kprateek.nayak, linux-kernel The overhead of task_cache_work() is high, especially in multi-NUMA systems. Currently, task_cache_work() tries to find the pref_llc by scanning all CPUs in the system. However, most of these scans are meaningless, such as those for CPUs that have never been visited or were accessed a long time ago. To address this problem, introduce visited_cpus to track the visited CPUs and evict them once they have not been accessed for a duration exceeding llc_epoch_affinity_timeout. Signed-off-by: Luo Gengkun <luogengkun2@huawei.com> --- include/linux/sched.h | 2 ++ kernel/sched/fair.c | 37 ++++++++++++++++++++++++++++--------- 2 files changed, 30 insertions(+), 9 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d1..b187ec927146 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2388,6 +2388,7 @@ static __always_inline int task_mm_cid(struct task_struct *t) struct sched_cache_time { u64 runtime; unsigned long epoch; + unsigned long epoch_timeout; }; struct sched_cache_stat { @@ -2398,6 +2399,7 @@ struct sched_cache_stat { unsigned long next_scan; unsigned long footprint; int cpu; + struct cpumask visited_cpus; } ____cacheline_aligned_in_smp; #else diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6ee1..44552168fcc3 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1585,6 +1585,7 @@ void mm_init_sched(struct mm_struct *mm, pcpu_sched->runtime = 0; /* a slightly stale cpu epoch is acceptible */ pcpu_sched->epoch = rq->cpu_epoch; + pcpu_sched->epoch_timeout = rq->cpu_epoch; epoch = rq->cpu_epoch; } @@ -1594,6 +1595,7 @@ void mm_init_sched(struct mm_struct *mm, mm->sc_stat.next_scan = jiffies; mm->sc_stat.nr_running_avg = 0; mm->sc_stat.footprint = 0; + cpumask_clear(&mm->sc_stat.visited_cpus); /* * The update to mm->sc_stat should not be reordered * before initialization to mm's other fields, in case @@ -1635,11 +1637,21 @@ static inline void __update_mm_sched(struct rq *rq, } } -static unsigned long fraction_mm_sched(struct rq *rq, - struct sched_cache_time *pcpu_sched) +static unsigned long fraction_mm_sched(int cpu, + struct mm_struct *mm) { + struct sched_cache_time *pcpu_sched = + per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu); + struct rq *rq = cpu_rq(cpu); + guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock); + /* Skip the rq that has not been hit for a long time */ + if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) { + cpumask_clear_cpu(cpu, &mm->sc_stat.visited_cpus); + return 0; + } + __update_mm_sched(rq, pcpu_sched); /* @@ -1711,6 +1723,9 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) pcpu_sched->runtime += delta_exec; rq->cpu_runtime += delta_exec; epoch = rq->cpu_epoch; + pcpu_sched->epoch_timeout = epoch; + if (!cpumask_test_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus)) + cpumask_set_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus); } /* @@ -1867,6 +1882,7 @@ static void task_cache_work(struct callback_head *work) guard(rcu)(); get_scan_cpumasks(cpus, p); + cpumask_and(cpus, cpus, &mm->sc_stat.visited_cpus); for_each_cpu(cpu, cpus) { /* XXX sched_cluster_active */ @@ -1877,19 +1893,22 @@ static void task_cache_work(struct callback_head *work) if (!sd) continue; - for_each_cpu(i, sched_domain_span(sd)) { - occ = fraction_mm_sched(cpu_rq(i), - per_cpu_ptr(mm->sc_stat.pcpu_sched, i)); + for_each_cpu_and(i, sched_domain_span(sd), cpus) { + cur = rcu_dereference_all(cpu_rq(i)->curr); + if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) && + cur->mm == mm) + nr_running++; + + occ = fraction_mm_sched(i, mm); + if (occ == 0) + continue; + a_occ += occ; if (occ > m_occ) { m_occ = occ; m_cpu = i; } - cur = rcu_dereference_all(cpu_rq(i)->curr); - if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) && - cur->mm == mm) - nr_running++; } /* -- 2.34.1 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH v5 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus 2026-07-09 13:00 ` [PATCH v5 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus Luo Gengkun @ 2026-07-09 15:01 ` Chen, Yu C 2026-07-14 2:22 ` Luo Gengkun 2026-07-14 22:40 ` Tim Chen 0 siblings, 2 replies; 7+ messages in thread From: Chen, Yu C @ 2026-07-09 15:01 UTC (permalink / raw) To: Luo Gengkun Cc: dietmar.eggemann, rostedt, bsegall, mgorman, peterz, vschneid, kprateek.nayak, linux-kernel, juri.lelli, mingo, vincent.guittot, tim.c.chen, chen.yu@linux.dev Hi Gengkun, thanks for the update, On 7/9/2026 9:00 PM, Luo Gengkun wrote: > +static unsigned long fraction_mm_sched(int cpu, > + struct mm_struct *mm) > { > + struct sched_cache_time *pcpu_sched = > + per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu); > + struct rq *rq = cpu_rq(cpu); > + > guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock); > > + /* Skip the rq that has not been hit for a long time */ > + if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) { > + cpumask_clear_cpu(cpu, &mm->sc_stat.visited_cpus); > + return 0; > + } > + I was wondering if there is a race condition in the scenario above: If the rq has just updated its cpu_epoch field, and the CPU subsequently remains idle, then rq->cpu_epoch stays unchanged. That is to say, a race could occur as follows: if CPU1 becomes idle, CPU2's task_cache_work() might compute rq->cpu_epoch - pcpu_sched->epoch_timeout == 0 and consequently fail to clear CPU1 from visit_cpus. CPU_1 CPU_2 ------------------------------------ --------------------------------- account_mm_sched(rq_X) set CPU1 in visited_cpus (idle: fair task runs here, CPU1.cpu_epoch frozen afterwards, pcpu_sched->epoch_timeout = CPU1.epoch) fraction_mm_sched() CPU1->cpu_epoch - pcpu_sched->epoch_timeout = 0 -> bit CPU1 not cleared __update_mm_sched() ->update cpu_epoch, too late I wonder if we should let __update_mm_sched() be invoked before if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout)? BTW, in your test data: sched_cache_scan: comm=redis-server pid=24660 scan=384 Is it because NUMA balancing is disabled on your platform that it has to scan all the CPUs? It would be helpful to know what the result would be with NUMA balancing enabled. I found some reports from sashiko here that are worth taking a look at: https://sashiko.dev/#/patchset/20260709130053.2749834-1-luogengkun2%40huawei.com thanks, Chenyu ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v5 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus 2026-07-09 15:01 ` Chen, Yu C @ 2026-07-14 2:22 ` Luo Gengkun 2026-07-14 22:40 ` Tim Chen 1 sibling, 0 replies; 7+ messages in thread From: Luo Gengkun @ 2026-07-14 2:22 UTC (permalink / raw) To: Chen, Yu C Cc: dietmar.eggemann, rostedt, bsegall, mgorman, peterz, vschneid, kprateek.nayak, linux-kernel, juri.lelli, mingo, vincent.guittot, tim.c.chen, chen.yu@linux.dev On 2026/7/9 23:01, Chen, Yu C wrote: > Hi Gengkun, > > thanks for the update, > > On 7/9/2026 9:00 PM, Luo Gengkun wrote: > >> +static unsigned long fraction_mm_sched(int cpu, >> + struct mm_struct *mm) >> { >> + struct sched_cache_time *pcpu_sched = >> + per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu); >> + struct rq *rq = cpu_rq(cpu); >> + >> guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock); >> + /* Skip the rq that has not been hit for a long time */ >> + if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) { >> + cpumask_clear_cpu(cpu, &mm->sc_stat.visited_cpus); >> + return 0; >> + } >> + > > I was wondering if there is a race condition in the scenario > above: If the rq has just updated its cpu_epoch field, and the > CPU subsequently remains idle, then rq->cpu_epoch stays unchanged. > That is to say, a race could occur as follows: if CPU1 becomes idle, > CPU2's task_cache_work() might compute > rq->cpu_epoch - pcpu_sched->epoch_timeout == 0 and consequently fail > to clear CPU1 from visit_cpus. > > CPU_1 CPU_2 > ------------------------------------ --------------------------------- > account_mm_sched(rq_X) > set CPU1 in visited_cpus > > > (idle: fair task runs here, > CPU1.cpu_epoch frozen afterwards, > pcpu_sched->epoch_timeout = CPU1.epoch) > fraction_mm_sched() > CPU1->cpu_epoch - pcpu_sched->epoch_timeout = 0 > -> bit CPU1 not cleared > __update_mm_sched() ->update cpu_epoch, too late > I wonder if we should let __update_mm_sched() be invoked before > if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout)? Thanks, I will fix it. > > BTW, in your test data: > sched_cache_scan: comm=redis-server pid=24660 scan=384 > Is it because NUMA balancing is disabled on your platform that it has > to scan all the CPUs? It would be helpful to know what the result would > be with NUMA balancing enabled. The following tests were conducted based on the fixed version. It appears that the --mbind parameter in numactl causes NUMA balancing to fail, so the following test only includes the --cpunodebind parameter. When NUMA balancing is disabled: valkey-benchmark rps | baseline | schedcache | schedcache_visit | (no cache aware) | (without this series) | (with this series) ----------------------+---------------------+---------------------+-----------+-------------------+-------- | p99 latency(ms) | p99 latency(ms) | DIFF(%) | p99 latency(ms) | DIFF(%) 400000 | 0.436 | 0.554 | -25.68% | 0.441 | -1.14% The overhead of task_cache_work is as follows: valkey-benchmark rps | schedcache | schedcache_visit ----------------------+-------------------------------------------+--------------------------------------- 400000 | 0.81% [kernel] [k] task_cache_work | 0.02% [kernel] [k] task_cache_work When NUMA balancing is enabled: valkey-benchmark rps | baseline | schedcache | schedcache_visit | (no cache aware) | (without this series) | (with this series) ----------------------+---------------------+---------------------+-----------+-------------------+-------- | p99 latency(ms) | p99 latency(ms) | DIFF(%) | p99 latency(ms) | DIFF(%) 400000 | 0.437 | 0.454 | -3.89% | 0.442 | -1.14% The overhead of task_cache_work is as follows: valkey-benchmark rps | schedcache | schedcache_visit ----------------------+-------------------------------------------+--------------------------------------- 400000 | 0.13% [kernel] [k] task_cache_work | 0.03% [kernel] [k] task_cache_work The scan count for schedcache has been reduced to 96. Additionally, I also conducted a test without restricting CPU affinity (no numactl), and the results are as follows: When NUMA balancing is disabled: valkey-benchmark rps | baseline | schedcache | schedcache_visit | (no cache aware) | (without this series) | (with this series) ----------------------+---------------------+---------------------+-----------+-------------------+-------- | p99 latency(ms) | p99 latency(ms) | DIFF(%) | p99 latency(ms) | DIFF(%) 400000 | 0.741 | 0.82 | -10.66% | 0.705 | +4.86% When NUMA balancing is enabled: valkey-benchmark rps | baseline | schedcache | schedcache_visit | (no cache aware) | (without this series) | (with this series) ----------------------+---------------------+---------------------+-----------+-------------------+-------- | p99 latency(ms) | p99 latency(ms) | DIFF(%) | p99 latency(ms) | DIFF(%) 400000 | 0.75 | 0.744 | +0.80% | 0.736 | +1.87% > I found some reports from sashiko here that are worth taking a look at: > https://sashiko.dev/#/patchset/20260709130053.2749834-1-luogengkun2%40huawei.com > Thank you for the reminder. I will fix these issues in the next version. thanks, Gengkun > thanks, > Chenyu > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v5 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus 2026-07-09 15:01 ` Chen, Yu C 2026-07-14 2:22 ` Luo Gengkun @ 2026-07-14 22:40 ` Tim Chen 2026-07-15 6:41 ` Luo Gengkun 1 sibling, 1 reply; 7+ messages in thread From: Tim Chen @ 2026-07-14 22:40 UTC (permalink / raw) To: Chen, Yu C, Luo Gengkun Cc: dietmar.eggemann, rostedt, bsegall, mgorman, peterz, vschneid, kprateek.nayak, linux-kernel, juri.lelli, mingo, vincent.guittot, chen.yu@linux.dev On Thu, 2026-07-09 at 23:01 +0800, Chen, Yu C wrote: > Hi Gengkun, > > thanks for the update, > > On 7/9/2026 9:00 PM, Luo Gengkun wrote: > > > +static unsigned long fraction_mm_sched(int cpu, > > + struct mm_struct *mm) > > { > > + struct sched_cache_time *pcpu_sched = > > + per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu); > > + struct rq *rq = cpu_rq(cpu); > > + > > guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock); > > > > + /* Skip the rq that has not been hit for a long time */ > > + if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) { > > + cpumask_clear_cpu(cpu, &mm->sc_stat.visited_cpus); > > + return 0; > > + } > > + > > I was wondering if there is a race condition in the scenario > above: If the rq has just updated its cpu_epoch field, and the > CPU subsequently remains idle, then rq->cpu_epoch stays unchanged. > That is to say, a race could occur as follows: if CPU1 becomes idle, > CPU2's task_cache_work() might compute > rq->cpu_epoch - pcpu_sched->epoch_timeout == 0 and consequently fail > to clear CPU1 from visit_cpus. > > CPU_1 CPU_2 > ------------------------------------ --------------------------------- > account_mm_sched(rq_X) > set CPU1 in visited_cpus > > > (idle: fair task runs here, > CPU1.cpu_epoch frozen afterwards, > pcpu_sched->epoch_timeout = CPU1.epoch) > fraction_mm_sched() > CPU1->cpu_epoch - > pcpu_sched->epoch_timeout = 0 > -> bit CPU1 not cleared > __update_mm_sched() > ->update cpu_epoch, too late > I wonder if we should let __update_mm_sched() be invoked before > if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > > llc_epoch_affinity_timeout)? Also, I am not sure why we need to create this new field pcpu_sched->epoch_timeout? Wouldn't rq->cpu_epoch - pcpu_sched->epoch give the elapsed epochs since the last time that cpu was visited for this mm (as in __update_mm_sched())? There are some races possible when cpu_epoch changes before the whole accounting got done. A CPU could get visited after you think that it hasn't got visited. But you will also have race if you use epoch_timeout. Tim > > BTW, in your test data: > sched_cache_scan: comm=redis-server pid=24660 scan=384 > Is it because NUMA balancing is disabled on your platform that it has > to scan all the CPUs? It would be helpful to know what the result would > be with NUMA balancing enabled. > I found some reports from sashiko here that are worth taking a look at: > https://sashiko.dev/#/patchset/20260709130053.2749834-1-luogengkun2%40huawei.com > > thanks, > Chenyu > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v5 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus 2026-07-14 22:40 ` Tim Chen @ 2026-07-15 6:41 ` Luo Gengkun 0 siblings, 0 replies; 7+ messages in thread From: Luo Gengkun @ 2026-07-15 6:41 UTC (permalink / raw) To: Tim Chen, Chen, Yu C Cc: dietmar.eggemann, rostedt, bsegall, mgorman, peterz, vschneid, kprateek.nayak, linux-kernel, juri.lelli, mingo, vincent.guittot, chen.yu@linux.dev On 2026/7/15 6:40, Tim Chen wrote: > On Thu, 2026-07-09 at 23:01 +0800, Chen, Yu C wrote: >> Hi Gengkun, >> >> thanks for the update, >> >> On 7/9/2026 9:00 PM, Luo Gengkun wrote: >> >>> +static unsigned long fraction_mm_sched(int cpu, >>> + struct mm_struct *mm) >>> { >>> + struct sched_cache_time *pcpu_sched = >>> + per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu); >>> + struct rq *rq = cpu_rq(cpu); >>> + >>> guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock); >>> >>> + /* Skip the rq that has not been hit for a long time */ >>> + if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) { >>> + cpumask_clear_cpu(cpu, &mm->sc_stat.visited_cpus); >>> + return 0; >>> + } >>> + >> >> I was wondering if there is a race condition in the scenario >> above: If the rq has just updated its cpu_epoch field, and the >> CPU subsequently remains idle, then rq->cpu_epoch stays unchanged. >> That is to say, a race could occur as follows: if CPU1 becomes idle, >> CPU2's task_cache_work() might compute >> rq->cpu_epoch - pcpu_sched->epoch_timeout == 0 and consequently fail >> to clear CPU1 from visit_cpus. >> >> CPU_1 CPU_2 >> ------------------------------------ --------------------------------- >> account_mm_sched(rq_X) >> set CPU1 in visited_cpus >> >> >> (idle: fair task runs here, >> CPU1.cpu_epoch frozen afterwards, >> pcpu_sched->epoch_timeout = CPU1.epoch) >> fraction_mm_sched() >> CPU1->cpu_epoch - >> pcpu_sched->epoch_timeout = 0 >> -> bit CPU1 not cleared >> __update_mm_sched() >> ->update cpu_epoch, too late >> I wonder if we should let __update_mm_sched() be invoked before >> if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > >> llc_epoch_affinity_timeout)? > > Also, I am not sure why we need to create this new field pcpu_sched->epoch_timeout? > Wouldn't rq->cpu_epoch - pcpu_sched->epoch give the elapsed > epochs since the last time that cpu was visited for this mm > (as in __update_mm_sched())? > Good question. During previous testing, I found that pcpu_sched->epoch was being flushed by calls to fraction_mm_sched(), even though the task was not running on that specific CPU. Because the flush frequency was higher than the eviction frequency, the eviction mechanism failed. The newly introduced epoch_timeout ensures that the epoch_timeout is only flushed when the task is actually running on the corresponding CPU, thereby preventing the aforementioned issue. > There are some races possible > when cpu_epoch changes before the whole accounting got done. > A CPU could get visited after you think that it hasn't got visited. > But you will also have race if you use epoch_timeout If a CPU was just visited, there are two possible conditions: 1. The CPU is not yet set in visited_cpus, meaning account_mm_sched() has not been called yet. In this case, fraction_mm_sched() won't be triggered either. This is acceptable because it implies pcpu_sched->runtime hasn't been updated for a long time too, and this negligible contribution can be safely ignored. 2. The CPU is already set in visited_cpus. In this case, we can simply accept this contribution instead of performing an early return. We can just clear the CPU from the mask and proceed, as our primary goal is simply to limit the number of scanned CPUs. As shown in the code below. What do you think? @@ -1651,7 +1651,6 @@ static unsigned long fraction_mm_sched(int cpu, if (sched_feat(SC_VISIT) && (rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) { cpumask_clear_cpu(cpu, mm->sc_stat.visited_cpus); - return 0; } /* @@ -1913,8 +1912,6 @@ static void task_cache_work(struct callback_head *work) scanned++; occ = fraction_mm_sched(i, mm); - if (occ == 0) - continue; a_occ += occ; thanks, Gengkun > > Tim > > >> >> BTW, in your test data: >> sched_cache_scan: comm=redis-server pid=24660 scan=384 >> Is it because NUMA balancing is disabled on your platform that it has >> to scan all the CPUs? It would be helpful to know what the result would >> be with NUMA balancing enabled. >> I found some reports from sashiko here that are worth taking a look at: >> https://sashiko.dev/#/patchset/20260709130053.2749834-1-luogengkun2%40huawei.com >> >> thanks, >> Chenyu >> ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v5 2/2] -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched feature to track scan cost 2026-07-09 13:00 [PATCH v5 linux 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work Luo Gengkun 2026-07-09 13:00 ` [PATCH v5 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus Luo Gengkun @ 2026-07-09 13:00 ` Luo Gengkun 1 sibling, 0 replies; 7+ messages in thread From: Luo Gengkun @ 2026-07-09 13:00 UTC (permalink / raw) To: peterz, mingo, juri.lelli, vincent.guittot, tim.c.chen, yu.c.chen Cc: dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, kprateek.nayak, linux-kernel To evaluate the effectiveness of the previous patch, this debug patch re-adds get_scan_cpumasks() and introduces two sched features as well as trace events to facilitate testing. Signed-off-by: Luo Gengkun <luogengkun2@huawei.com> --- include/trace/events/sched.h | 21 +++++++++++++++++++++ kernel/sched/fair.c | 23 ++++++++++++++++++----- kernel/sched/features.h | 1 + 3 files changed, 40 insertions(+), 5 deletions(-) diff --git a/include/trace/events/sched.h b/include/trace/events/sched.h index 535860581f15..aced624f198d 100644 --- a/include/trace/events/sched.h +++ b/include/trace/events/sched.h @@ -10,6 +10,27 @@ #include <linux/tracepoint.h> #include <linux/binfmts.h> +TRACE_EVENT(sched_cache_scan, + + TP_PROTO(struct task_struct *t, int scan), + + TP_ARGS(t, scan), + + TP_STRUCT__entry( + __string( comm, t->comm ) + __field( pid_t, pid ) + __field( int, scan ) + ), + + TP_fast_assign( + __assign_str(comm); + __entry->pid = t->pid; + __entry->scan = scan; + ), + + TP_printk("comm=%s pid=%d scan=%d", __get_str(comm), __entry->pid, + __entry->scan) +); /* * Tracepoint for calling kthread_stop, performed to end a kthread: */ diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 44552168fcc3..f1d24a5c5165 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1647,7 +1647,8 @@ static unsigned long fraction_mm_sched(int cpu, guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock); /* Skip the rq that has not been hit for a long time */ - if ((rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) { + if (sched_feat(SC_VISIT) && + (rq->cpu_epoch - pcpu_sched->epoch_timeout) > llc_epoch_affinity_timeout) { cpumask_clear_cpu(cpu, &mm->sc_stat.visited_cpus); return 0; } @@ -1724,7 +1725,8 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) rq->cpu_runtime += delta_exec; epoch = rq->cpu_epoch; pcpu_sched->epoch_timeout = epoch; - if (!cpumask_test_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus)) + if (sched_feat(SC_VISIT) && + !cpumask_test_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus)) cpumask_set_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus); } @@ -1847,7 +1849,8 @@ static void task_cache_work(struct callback_head *work) unsigned long curr_m_a_occ = 0; struct mm_struct *mm = p->mm; unsigned long m_a_occ = 0; - cpumask_var_t cpus; + cpumask_var_t cpus, scan_cpus; + int scanned = 0; WARN_ON_ONCE(work != &p->cache_work); @@ -1878,11 +1881,15 @@ static void task_cache_work(struct callback_head *work) if (!zalloc_cpumask_var(&cpus, GFP_KERNEL)) return; + if (!zalloc_cpumask_var(&scan_cpus, GFP_KERNEL)) + return; + scoped_guard (cpus_read_lock) { guard(rcu)(); get_scan_cpumasks(cpus, p); - cpumask_and(cpus, cpus, &mm->sc_stat.visited_cpus); + if (sched_feat(SC_VISIT)) + cpumask_and(cpus, cpus, &mm->sc_stat.visited_cpus); for_each_cpu(cpu, cpus) { /* XXX sched_cluster_active */ @@ -1893,12 +1900,17 @@ static void task_cache_work(struct callback_head *work) if (!sd) continue; - for_each_cpu_and(i, sched_domain_span(sd), cpus) { + cpumask_copy(scan_cpus, sched_domain_span(sd)); + if (sched_feat(SC_VISIT)) + cpumask_and(scan_cpus, scan_cpus, cpus); + + for_each_cpu(i, scan_cpus) { cur = rcu_dereference_all(cpu_rq(i)->curr); if (cur && !(cur->flags & (PF_EXITING | PF_KTHREAD)) && cur->mm == mm) nr_running++; + scanned++; occ = fraction_mm_sched(i, mm); if (occ == 0) continue; @@ -1954,6 +1966,7 @@ static void task_cache_work(struct callback_head *work) update_avg_scale(&mm->sc_stat.nr_running_avg, nr_running); free_cpumask_var(cpus); + trace_sched_cache_scan(p, scanned); } void init_sched_mm(struct task_struct *p) diff --git a/kernel/sched/features.h b/kernel/sched/features.h index 8f0dee8fc475..990714ebabfd 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -142,3 +142,4 @@ SCHED_FEAT(LATENCY_WARN, false) */ SCHED_FEAT(NI_RANDOM, true) SCHED_FEAT(NI_RATE, true) +SCHED_FEAT(SC_VISIT, false) -- 2.34.1 ^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-07-15 6:42 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-07-09 13:00 [PATCH v5 linux 0/2] Cache aware scheduling: Reduce the overhead of task_cache_work Luo Gengkun 2026-07-09 13:00 ` [PATCH v5 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus Luo Gengkun 2026-07-09 15:01 ` Chen, Yu C 2026-07-14 2:22 ` Luo Gengkun 2026-07-14 22:40 ` Tim Chen 2026-07-15 6:41 ` Luo Gengkun 2026-07-09 13:00 ` [PATCH v5 2/2] -- DO NOT APPLY!!! -- sched/cache/debug: Add trace event and sched feature to track scan cost Luo Gengkun
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.