From: Chen Yu <chen.yu@linux.dev>
To: Tim Chen <tim.c.chen@linux.intel.com>, Jirka Hladky <jhladky@redhat.com>
Cc: Jirka Hladky <jhladky@redhat.com>,
Peter Zijlstra <peterz@infradead.org>,
"Chen, Yu C" <yu.c.chen@intel.com>,
Ingo Molnar <mingo@redhat.com>,
K Prateek Nayak <kprateek.nayak@amd.com>,
"Gautham R . Shenoy" <gautham.shenoy@amd.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Juri Lelli <juri.lelli@redhat.com>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Valentin Schneider <vschneid@redhat.com>,
Madadi Vineeth Reddy <vineethr@linux.ibm.com>,
Hillf Danton <hdanton@sina.com>,
Shrikanth Hegde <sshegde@linux.ibm.com>,
Jianyong Wu <jianyong.wu@outlook.com>,
Yangyu Chen <cyy@cyyself.name>,
Tingyin Duan <tingyin.duan@gmail.com>,
Vern Hao <vernhao@tencent.com>, Vern Hao <haoxing990@gmail.com>,
Len Brown <len.brown@intel.com>, Aubrey Li <aubrey.li@intel.com>,
Zhao Liu <zhao1.liu@intel.com>, Chen Yu <yu.chen.surf@gmail.com>,
Adam Li <adamli@os.amperecomputing.com>,
Aaron Lu <ziqianlu@bytedance.com>,
Tim Chen <tim.c.chen@intel.com>, Josh Don <joshdon@google.com>,
Gavin Guo <gavinguo@igalia.com>,
Qais Yousef <qyousef@layalina.io>,
Libo Chen <libchen@purestorage.com>,
linux-kernel@vger.kernel.org, "Luck, Tony" <tony.luck@intel.com>,
Reinette Chatre <reinette.chatre@intel.com>,
Philip Auld <pauld@redhat.com>, Joe Mario <jmario@redhat.com>
Subject: Re: [Patch v4 17/22] sched/cache: Avoid cache-aware scheduling for memory-heavy processes
Date: Tue, 17 Nov 2026 20:07:52 +0800 [thread overview]
Message-ID: <avxDmPs4rs39NC9O@three-body> (raw)
In-Reply-To: <879c23213859f24b12a23530d6ae849bbe504b6b.camel@linux.intel.com>
Hi Tim, Jirka,
On Mon, Nov 16, 2026 at 03:16:01PM -0800, Tim Chen wrote:
> On Mon, 2026-11-16 at 15:35 +0100, Jirka Hladky wrote:
>
> Hi Jirka,
>
> Thanks for testing Chen Yu's sketch patch so quickly.
>
>
> > Hi Chen Yu, Tim,
> >
> > > Do you mean accumulate the pages into the group in task_numa_fault()?
> > > ...
> > > Would this decay cause a jitter of the footprint? Anyway, I thought
> > > what you suggested is the following for fix2 ...
> >
> > I built and tested the sketch on the Sierra Forest box: a kernel built
> > from the same 7.4.0-0.rc2.19.eln159 source as my baselines, patched
> > with your sketch plus the fixes needed to compile it, and with one
> > change: the decay is applied once per group per sweep via a
> > last_decay_seq stamp, since decaying per task would halve the group
> > footprint once per thread per sweep.
> >
> > The accounting mechanism works exactly as designed:
> >
> > - The footprint accumulates during the sweep and crosses the
> > 27,648-page LLC threshold at t=4.4 s; the aggregation is withdrawn
> > and stays withdrawn (grp->cpu = -1 in 96% of 27,796 samples). To
> > answer the jitter question: with once-per-sweep decay I saw no
> > oscillation in a run whose footprint reached ~3.95M pages.
> > - Removing the exit()/exec() subtraction held up fine as well.
> >
> > But the regression is not fixed. NAS MG class D, 1x64 threads, n=5,
> > median time:
> >
> > stock rc2.19, CAS off 26.45 s
> > stock rc2.19, CAS on 41.78 s (+58%)
> > sketch, CAS off 26.57 s
> > sketch, CAS on 39.19 s (+47.5%)
> >
> > The regression drops from +58% to +47.5%: 2.6 s of the 15.3 s gap,
> > about one sixth, and the reason is placement hysteresis. The 2.1 s
> > window between the aggregation settling
> > and the gate tripping has already shaped the run: the threads re-spread
> > at only ~1 thread per second (4/60 across the two nodes at t=5 s,
> > 20/44 at t=28 s), and the memory first-touched during the window is
> > 59/41 toward the aggregated node, with NUMA balancing then pulling even
> > more memory there as the run continues. Withdrawing grp->cpu stops the
> > steering but does not undo the placement, the fault statistics, or the
> > page layout.
>
> It does seem like the hysteresis is too strong, and takes too long to
> even out on the MG workload.
>
Agreed, it seems that the window before the first full-scan round is still
noticeable for MG.
> >
> > A few observations for the design:
> >
> > 1. Fix 1 (the RSS gate) looks critical for this workload class. MG's
> > RSS crosses the 108 MiB LLC size within ~0.1 s of the allocation
> > phase, so an RSS gate would refuse aggregation almost immediately,
> > before first-touch lays down the skewed layout.
> >
>
> Yeah, perhaps some form of Fix 1 is needed. However
> fix 1 benefits mostly workload where aggregation is unhelpful.
>
> There is a downside for Fix 1 for processes that actually benefit
> from aggregation.
>
> RSS gate is an upper bound on the cache footprint for the default
> case where cache group is a process. It tends to over estimate and
> it will take several scan windows to shed its effect and we could miss
> aggregation in the mean time.
>
> Chen Yu, on your point that a cache group will no longer be one mm
> and that we have no per-group RSS: I do not think we need one.
> I think mm's RSS serves more as a bound of the footprint when we are
> just starting our scan. For a group spanning several processes
> we can use the largest member, which still refuses when any one member
> cannot fit. It is just an initial value that will get decayed away.
>
OK, this makes sense.
> One more thing about the decay, mostly for Chen Yu. The once-per-group
> version halves the whole footprint, including the sweep that just
> closed. The NUMA fault statistics do not: per node they compute
> mem = mem / 2 + membuf, so the new window is added in full and only
> the history is halved. With B pages per sweep the two settle at 2B
> and B respectively depending on your sample time.
>
OK - mem = mem / 2 + membuf can be used to keep it consistent with the current
NUMA balancing fault statistic. Although I'm not sure why the original NUMA balancing
code uses this strategy, normally, for an EWMA, it should be
mem = p * mem + (1 - p) * membuf, where p is in (0, 1).
> A group whose per sweep footprint sits between
> one and two LLC sizes would flip in and out of aggregation at every
> sweep with the current form.
>
> I have a small patch on top of Chen Yu's patch to mitigate this issue.
> It keeps a count in the scan window and a history count, and combine
> the two when sampled to smooth out any jump.
>
> I also generated another test patch on top to test the RSS gate idea.
>
> > 2. Fix 2's trip time is bounded below by the scanner, not by the fault
> > count: the 4.4 s crossing decomposes into the 1 s scan delay plus
> > the 2-3 s the scanner needs to reach the large VMAs of a 26.4 GiB
> > address space. Even the theoretical ~1.8 s would arrive after ~60%
> > of the working set is first-touched.
> >
> > 3. One caveat on the withdrawal path, shared by both fixes: whenever
> > the gate trips after aggregation has been in effect, passive
> > withdrawal (clearing grp->cpu) recovers almost nothing on this
> > workload; the tasks re-spread at ~1 per second. An effective
> > withdrawal would need to actively re-spread the tasks or reset
> > their NUMA preferences. With Fix 1 in place this matters less for
> > the regression class I reported, since aggregation is refused
> > before it settles; it applies to workloads that grow past the LLC
> > mid-run.
>
> This is a good find and I think it deserves its own thread. One
> suspect: while the threads were aggregated, NUMA balancing moved
> their pages to that node, so numa_preferred_nid points there. After
> grp->cpu is cleared the load balancer can move a thread out, but
> numa_migrate_preferred() pulls it back toward its pages. On a system
> where the LLC is the node, the two mechanisms reinforce each other.
>
> Wonder if NUMA balancing is preventing the re-spread?
> If your traces have numa_preferred_nid and task_numa_migrate activity
> after the trip, that would confirm or rule it out.
>
>
This seems to be another issue of NUMA balancing fighting with generic load
balancing, independent of the current footprint issue.
> >
> > Happy to re-test when the next revision is posted. The exact version I
> > tested, on top of your sketch with the changes above, is below. The diff
> > is against v7.4-rc2; the hunks are in code my downstream patchset does
> > not touch, so it should apply to your tree as well.
> >
> > Thanks,
> > Jirka
> >
>
> Two patches on top of Chen Yu's sketch patch below. Thanks.
>
> Tim
>
> --->8----
> From: Tim Chen <tim.c.chen@linux.intel.com>
> Subject: [PATCH 1/2] sched/cache: Keep the footprint flat across sweep
> boundaries
>
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index cf2d9dac959f..f528be926ab0 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -2392,7 +2392,15 @@ struct sched_cache_group {
> unsigned long epoch;
> u64 nr_running_avg;
> unsigned long next_scan;
> + /*
> + * Footprint estimate in pages. footprint is the decayed history of
> + * the completed scan sweeps, written by one task per sweep.
> + * footprint_cur collects the hint faults of the current sweep from
> + * all tasks. The estimate is max(footprint, footprint / 2 +
> + * footprint_cur), see sched_cache_footprint_decay().
> + */
> unsigned long footprint;
> + atomic_long_t footprint_cur;
> unsigned long last_decay_seq;
> int cpu;
> refcount_t refcnt;
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 44b016cd999e..0e0e8f6fd011 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -1424,7 +1424,7 @@ static inline int get_sched_cache_scale(int mul)
> static bool exceed_llc_capacity(struct sched_cache_group *grp, int cpu)
> {
> #ifdef CONFIG_NUMA_BALANCING
> - unsigned long llc, footprint;
> + unsigned long llc, footprint, hist, cur;
> struct sched_domain *sd;
> int scale;
>
> @@ -1440,7 +1440,14 @@ static bool exceed_llc_capacity(struct sched_cache_group *grp, int cpu)
> * excluded.
> */
> llc = sd->llc_bytes;
> - footprint = READ_ONCE(grp->footprint);
> + /*
> + * The history, or what it becomes at the next sweep
> + * boundary if no more faults arrive, whichever is larger.
> + * See sched_cache_footprint_decay().
> + */
> + hist = READ_ONCE(grp->footprint);
> + cur = atomic_long_read(&grp->footprint_cur);
> + footprint = max(hist, hist / 2 + cur);
>
> /*
> * Scale the LLC size by 256*llc_aggr_tolerance
> @@ -1621,6 +1628,7 @@ int mm_init_sched(struct mm_struct *mm,
> grp->next_scan = jiffies;
> grp->nr_running_avg = 0;
> grp->footprint = 0;
> + atomic_long_set(&grp->footprint_cur, 0);
> grp->last_decay_seq = 0;
> refcount_set(&grp->refcnt, 1);
> /*
> @@ -3817,42 +3825,58 @@ static int preferred_group_nid(struct task_struct *p, int nid)
>
> #if defined(CONFIG_SCHED_CACHE) && defined(CONFIG_NUMA_BALANCING)
> /*
> - * Fix 2 (Chen Yu sketch, test build): accumulate the pages reported by
> - * task_numa_fault() into the group footprint as they arrive within the
> - * current sweep, instead of waiting for task_numa_placement() at the sweep
> - * boundary. Batched per task to limit writes to the shared counter.
> + * Accumulate the pages reported by task_numa_fault() into the footprint of
> + * the current sweep as they arrive, instead of waiting for
> + * task_numa_placement() at the sweep boundary. A sweep of a large address
> + * space can take longer than the workload runs.
> + *
> + * The pages are batched per task before they are added to the shared
> + * counter. One THP hint fault reports HPAGE_PMD_NR pages, so the batch is
> + * one THP, or 512 base pages. task_numa_placement() flushes the batch
> + * once per sweep, so pages below the threshold are missing from the
> + * window for at most the sweep in progress.
> */
> -#define SCHED_CACHE_FP_THRESHOLD 10
> -static void sched_cache_footprint_accum(struct task_struct *p, int pages)
> +#define SCHED_CACHE_FP_THRESHOLD 512
> +static void sched_cache_footprint_flush(struct task_struct *p)
> {
> struct sched_cache_group *grp;
>
> - p->total_footprint += pages;
> -
> - if (p->total_footprint < SCHED_CACHE_FP_THRESHOLD)
> + if (!p->total_footprint)
> return;
>
> guard(rcu)();
> grp = rcu_dereference(p->sched_cache_grp);
> - if (!grp)
> - return;
> -
> - /* drain the per-task footprint into the group */
> - WRITE_ONCE(grp->footprint, READ_ONCE(grp->footprint) + p->total_footprint);
> + if (grp)
> + atomic_long_add(p->total_footprint, &grp->footprint_cur);
> p->total_footprint = 0;
> }
>
> +static void sched_cache_footprint_accum(struct task_struct *p, int pages)
> +{
> + p->total_footprint += pages;
> +
> + if (p->total_footprint >= SCHED_CACHE_FP_THRESHOLD)
> + sched_cache_footprint_flush(p);
> +}
> +
> /*
> - * Decay once per group per sweep: the first task whose placement() observes
> - * a new scan sequence halves the accumulated window. (Decaying per task, as
> - * in the original sketch, would halve the group footprint once per thread
> - * per sweep.)
> + * Close the window at the sweep boundary, once per group per sweep: the
> + * first task whose task_numa_placement() observes a new scan sequence
> + * folds the current window into the history.
> + *
> + * Only the history is decayed. The window that just closed is added in
> + * full, like the per node update of the NUMA fault statistics:
> + *
> + * mem = mem / 2 + membuf
> + *
> + * With B pages touched per sweep the history settles at 2B, the value
> + * the fault statistics report.
> */
> static void sched_cache_footprint_decay(struct task_struct *p)
> {
> struct sched_cache_group *grp;
> unsigned long seq = p->numa_scan_seq;
> - unsigned long old;
> + unsigned long old, cur;
>
> guard(rcu)();
> grp = rcu_dereference(p->sched_cache_grp);
> @@ -3862,12 +3886,16 @@ static void sched_cache_footprint_decay(struct task_struct *p)
> old = READ_ONCE(grp->last_decay_seq);
> if (old == seq)
> return;
> - if (cmpxchg(&grp->last_decay_seq, old, seq) == old)
> - WRITE_ONCE(grp->footprint, READ_ONCE(grp->footprint) / 2);
> + if (cmpxchg(&grp->last_decay_seq, old, seq) != old)
> + return;
I'm not sure if we should introduce this per-group last_decay_seq. The grp->last_decay_seq is
assigned the current p->numa_scan_seq, which is derived from p->mm->numa_scan_seq. In the current
implementation, cache-aware scheduling is per-mm, so this is OK. But soon, after we introduce
cross-process/mm cache-aware scheduling via prctl, assigning different mm->numa_scan_seq values to
the shared grp->last_decay_seq might cause last_decay_seq to bounce between different mms' numa_scan_seq,
which could decay the grp->footprint very fast.
That is to say, since we need a "global" sequence across different mm, can we use jiffies in
task_cache_work() to decay the footprint periodically, rather than decay it in task_numa_placement()?
Besides, task_numa_placement() might only be invoked after a full scan, and only if there are NUMA page
faults. If we put the decay in task_cache_work(), only 1 task of the group is allowed to decay the footprint,
and it might be more reliable to invoked? I wonder if the following method would work, similar to what your
patch does for RSS gating(just compiled)?
diff --git a/include/linux/sched.h b/include/linux/sched.h
index e4f6323d7855..8a54335e4fa9 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2394,6 +2394,8 @@ struct sched_cache_group {
u64 nr_running_avg;
unsigned long next_scan;
unsigned long footprint;
+ atomic_long_t footprint_cur;
+ unsigned long footprint_timeout;
int cpu;
refcount_t refcnt;
struct rcu_head rcu;
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 79f6b3a4fd7f..477b8b5b7229 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1393,6 +1393,7 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)
* tunable or so.
*/
#define EPOCH_PERIOD (HZ / 100) /* 10 ms */
+#define SCHED_CACHE_FP_TIMEOUT_PERIOD (4 * HZ)
#define EPOCH_LLC_AFFINITY_TIMEOUT 5 /* 50 ms */
__read_mostly unsigned int llc_aggr_tolerance = 1;
__read_mostly unsigned int llc_epoch_period = EPOCH_PERIOD;
@@ -1424,7 +1425,7 @@ static inline int get_sched_cache_scale(int mul)
static bool exceed_llc_capacity(struct sched_cache_group *grp, int cpu)
{
#ifdef CONFIG_NUMA_BALANCING
- unsigned long llc, footprint;
+ unsigned long llc, footprint, hist;
struct sched_domain *sd;
int scale;
@@ -1440,7 +1441,8 @@ static bool exceed_llc_capacity(struct sched_cache_group *grp, int cpu)
* excluded.
*/
llc = sd->llc_bytes;
- footprint = READ_ONCE(grp->footprint);
+ hist = READ_ONCE(grp->footprint);
+ footprint = max(hist, hist / 2 + atomic_long_read(&grp->footprint_cur));
/*
* Scale the LLC size by 256*llc_aggr_tolerance
@@ -1621,6 +1623,8 @@ int mm_init_sched(struct mm_struct *mm,
grp->next_scan = jiffies;
grp->nr_running_avg = 0;
grp->footprint = 0;
+ atomic_long_set(&grp->footprint_cur, 0);
+ grp->footprint_timeout = jiffies + SCHED_CACHE_FP_TIMEOUT_PERIOD;
refcount_set(&grp->refcnt, 1);
/*
* The update to grp->pcpu_sched should not be reordered
@@ -1730,19 +1734,6 @@ void sched_cache_exit_mm(struct task_struct *p)
{
struct sched_cache_group *grp = sched_cache_replace_grp(p, NULL);
-#ifdef CONFIG_NUMA_BALANCING
- /*
- * Subtract this task's footprint from the group before dropping the
- * reference, so the group footprint converges as its threads exit.
- * Unlocked for performance; clamp to avoid underflow.
- */
- if (grp && p->total_numa_faults) {
- unsigned long fp = READ_ONCE(grp->footprint);
- unsigned long sub = min(fp, p->total_numa_faults);
-
- WRITE_ONCE(grp->footprint, fp - sub);
- }
-#endif
sched_cache_group_put(grp);
}
@@ -2017,6 +2008,15 @@ static void task_cache_work(struct callback_head *work)
READ_ONCE(llc_epoch_period), 1)))
return;
+ if (time_after_eq(now, grp->footprint_timeout)) {
+ unsigned long hist, cur;
+
+ cur = atomic_long_xchg(&grp->footprint_cur, 0);
+ hist = READ_ONCE(grp->footprint) / 2 + cur;
+ WRITE_ONCE(grp->footprint, hist);
+ grp->footprint_timeout = now + SCHED_CACHE_FP_TIMEOUT_PERIOD;
+ }
+
curr_cpu = task_cpu(p);
if (invalid_llc_nr(grp, p, curr_cpu) ||
exceed_llc_capacity(grp, curr_cpu)) {
@@ -3829,14 +3829,12 @@ static int preferred_group_nid(struct task_struct *p, int nid)
static void task_numa_placement(struct task_struct *p)
__context_unsafe(/* conditional locking */)
{
- struct sched_cache_group __maybe_unused *grp;
int seq, nid, max_nid = NUMA_NO_NODE;
unsigned long max_faults = 0;
unsigned long fault_types[2] = { 0, 0 };
unsigned long total_faults;
u64 runtime, period;
spinlock_t *group_lock = NULL;
- long __maybe_unused new_fp;
struct numa_group *ng;
/*
@@ -3911,36 +3909,6 @@ static void task_numa_placement(struct task_struct *p)
ng->total_faults += diff;
group_faults += ng->faults[mem_idx];
}
-#ifdef CONFIG_SCHED_CACHE
- /*
- * Per task p->numa_faults[mem_idx] converges,
- * so the accumulation of each task's faults
- * converges too - Given the number of threads,
- * it cannot overflow an unsigned long.
- * Racy with concurrent updates from other threads
- * sharing this mm. Acceptable since footprint is a
- * heuristic and occasional lost updates are tolerable.
- *
- * If a task exits, its corresponding footprint must
- * be subtracted from p->sched_cache_grp->footprint,
- * otherwise the footprint will not converge: the
- * exiting thread's footprint remains unchanged/undecayed.
- * See exit_mm().
- *
- * Lost updates and unsynchronized subtraction
- * in exit_mm() can cause footprint + diff to
- * go negative. Clamp to zero to prevent the
- * unsigned footprint from wrapping.
- */
- scoped_guard(rcu) {
- grp = rcu_dereference(p->sched_cache_grp);
-
- if (grp) {
- new_fp = (long)READ_ONCE(grp->footprint) + diff;
- WRITE_ONCE(grp->footprint, max(new_fp, 0L));
- }
- }
-#endif
}
if (!ng) {
@@ -4210,6 +4178,15 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
p->numa_faults[task_faults_idx(NUMA_MEMBUF, mem_node, priv)] += pages;
p->numa_faults[task_faults_idx(NUMA_CPUBUF, cpu_node, priv)] += pages;
p->numa_faults_locality[local] += pages;
+
+#ifdef CONFIG_SCHED_CACHE
+ scoped_guard(rcu) {
+ struct sched_cache_group *grp = rcu_dereference(p->sched_cache_grp);
+
+ if (grp)
+ atomic_long_add(pages, &grp->footprint_cur);
+ }
+#endif
}
static void reset_ptenuma_scan(struct task_struct *p)
--
2.43.0
> +
> + cur = atomic_long_xchg(&grp->footprint_cur, 0);
> + WRITE_ONCE(grp->footprint, READ_ONCE(grp->footprint) / 2 + cur);
> }
>
> #else
> static inline void sched_cache_footprint_accum(struct task_struct *p, int pages) { }
> +static inline void sched_cache_footprint_flush(struct task_struct *p) { }
> static inline void sched_cache_footprint_decay(struct task_struct *p) { }
> #endif
>
> @@ -3985,6 +4013,12 @@ static void task_numa_placement(struct task_struct *p)
> }
>
> update_task_scan_period(p, fault_types[0], fault_types[1]);
> +
> + /*
> + * Once per sweep: add this task's pending pages to the window
> + * before the window may be closed below.
> + */
> + sched_cache_footprint_flush(p);
> sched_cache_footprint_decay(p);
> }
>
> --
>
> From: Tim Chen <tim.c.chen@linux.intel.com>
> Date: Mon, 16 Nov 2026 14:05:45 -0800
> Subject: [PATCH 2/2] sched/cache: Use the RSS as an upper bound while there is
> no footprint
>
> The footprint check of exceed_llc_capacity() needs NUMA balancing hint
> faults. Until the first scan sweep of the address space completes there
> is no estimate, and without NUMA balancing there never is one. In both
> cases the check passes and a process of any size can be aggregated.
>
> Use the RSS of the mm as the footprint while there is no estimate. The
> RSS is a poor estimate of the working set, but it is an upper bound: if
> the RSS fits in the LLC, so does the working set. If it does not fit,
> we cannot tell, and refuse to aggregate.
>
> Not-yet-signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
> ---
> include/linux/sched.h | 6 +++
> kernel/sched/fair.c | 95 ++++++++++++++++++++++++++++---------------
> 2 files changed, 68 insertions(+), 33 deletions(-)
>
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index f528be926ab0..9924a781c9dc 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -2402,6 +2402,12 @@ struct sched_cache_group {
> unsigned long footprint;
> atomic_long_t footprint_cur;
> unsigned long last_decay_seq;
> + /*
> + * RSS of the mm of the last task that scanned for the group, in
> + * pages. An upper bound of the working set, used while there is no
> + * footprint estimate yet. See exceed_llc_capacity().
> + */
> + unsigned long rss_bound;
> int cpu;
> refcount_t refcnt;
> struct rcu_head rcu;
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 0e0e8f6fd011..ef2af7a0c082 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -1423,8 +1423,7 @@ static inline int get_sched_cache_scale(int mul)
>
> static bool exceed_llc_capacity(struct sched_cache_group *grp, int cpu)
> {
> -#ifdef CONFIG_NUMA_BALANCING
> - unsigned long llc, footprint, hist, cur;
> + unsigned long llc, footprint = 0;
> struct sched_domain *sd;
> int scale;
>
> @@ -1434,46 +1433,67 @@ static bool exceed_llc_capacity(struct sched_cache_group *grp, int cpu)
> if (!sd)
> return true;
>
> + /*
> + * Scale the LLC size by 256*llc_aggr_tolerance
> + * and compare it to the task's footprint.
> + *
> + * Suppose the L3 size is 32MB. If the
> + * llc_aggr_tolerance is 1:
> + * When the footprint is larger than 32MB, the
> + * process is regarded as exceeding the LLC
> + * capacity. If the llc_aggr_tolerance is 99:
> + * When the footprint is larger than 784GB, the
> + * process is regarded as exceeding the LLC
> + * capacity:
> + * 784GB = (1 + (99 - 1) * 256) * 32MB
> + * If the llc_aggr_tolerance is 100:
> + * ignore the footprint and do the aggregation
> + * anyway.
> + */
> + scale = get_sched_cache_scale(256);
> + if (scale == INT_MAX)
> + return false;
> +
> + /*
> + * TBD: RDT exclusive LLC ways reserved should be
> + * excluded.
> + */
> + llc = sd->llc_bytes;
> +
> +#ifdef CONFIG_NUMA_BALANCING
> if (static_branch_likely(&sched_numa_balancing)) {
> - /*
> - * TBD: RDT exclusive LLC ways reserved should be
> - * excluded.
> - */
> - llc = sd->llc_bytes;
> + unsigned long hist, cur;
> +
> /*
> * The history, or what it becomes at the next sweep
> * boundary if no more faults arrive, whichever is larger.
> * See sched_cache_footprint_decay().
> + *
> + * The history is 0 until the first scan sweep of the
> + * address space completes. Until then the window alone
> + * is not an estimate of the working set, only a lower
> + * bound of it, so fall back to the RSS bound below.
> */
> hist = READ_ONCE(grp->footprint);
> - cur = atomic_long_read(&grp->footprint_cur);
> - footprint = max(hist, hist / 2 + cur);
> -
> - /*
> - * Scale the LLC size by 256*llc_aggr_tolerance
> - * and compare it to the task's footprint.
> - *
> - * Suppose the L3 size is 32MB. If the
> - * llc_aggr_tolerance is 1:
> - * When the footprint is larger than 32MB, the
> - * process is regarded as exceeding the LLC
> - * capacity. If the llc_aggr_tolerance is 99:
> - * When the footprint is larger than 784GB, the
> - * process is regarded as exceeding the LLC
> - * capacity:
> - * 784GB = (1 + (99 - 1) * 256) * 32MB
> - * If the llc_aggr_tolerance is 100:
> - * ignore the footprint and do the aggregation
> - * anyway.
> - */
> - scale = get_sched_cache_scale(256);
> - if (scale == INT_MAX)
> - return false;
> -
> - return ((llc * (u64)scale) < (footprint * PAGE_SIZE));
> + if (hist) {
> + cur = atomic_long_read(&grp->footprint_cur);
> + footprint = max(hist, hist / 2 + cur);
> + }
> }
> #endif
> - return false;
> + /*
> + * No footprint estimate: the first sweep has not completed, or
> + * NUMA balancing is off. Fall back to the RSS of the mm. It is a
> + * poor estimate of the working set but an upper bound of it: if
> + * the RSS fits in the LLC, so does the working set. If it does
> + * not fit, we cannot tell, and refuse to aggregate. A missed
> + * aggregation forfeits a gain. A wrong one on a memory bandwidth
> + * bound process costs far more.
> + */
> + if (!footprint)
> + footprint = READ_ONCE(grp->rss_bound);
> +
> + return ((llc * (u64)scale) < (footprint * PAGE_SIZE));
> }
>
> static bool invalid_llc_nr(struct sched_cache_group *grp, struct task_struct *p,
> @@ -1630,6 +1650,7 @@ int mm_init_sched(struct mm_struct *mm,
> grp->footprint = 0;
> atomic_long_set(&grp->footprint_cur, 0);
> grp->last_decay_seq = 0;
> + grp->rss_bound = 0;
> refcount_set(&grp->refcnt, 1);
> /*
> * The update to grp->pcpu_sched should not be reordered
> @@ -2013,6 +2034,14 @@ static void task_cache_work(struct callback_head *work)
> READ_ONCE(llc_epoch_period), 1)))
> return;
>
> + /*
> + * Sample the RSS of our own mm once per epoch, for the check
> + * below and for the tick and load balancer paths, which must
> + * not touch the mm of a task that is not current.
> + */
> + if (p->mm)
> + WRITE_ONCE(grp->rss_bound, get_mm_rss(p->mm));
MM_FILEPAGES includes some shared library, which was excluded by NUMA balancing, so
we might also want to exclude it from the RSS. Maybe with above footprint timeout decay
applied, we can decay the rss_bound as well, or use update_avg() to smooth it,
otherwise it never shrink if we use max():
@@ -2014,9 +2019,18 @@ static void task_cache_work(struct callback_head *work)
cur = atomic_long_xchg(&grp->footprint_cur, 0);
hist = READ_ONCE(grp->footprint) / 2 + cur;
WRITE_ONCE(grp->footprint, hist);
+ WRITE_ONCE(grp->rss_bound, READ_ONCE(grp->rss_bound) / 2);
grp->footprint_timeout = now + SCHED_CACHE_FP_TIMEOUT_PERIOD;
}
+ if (p->mm) {
+ unsigned long rss;
+
+ rss = get_mm_counter(p->mm, MM_ANONPAGES) +
+ get_mm_counter(p->mm, MM_SHMEMPAGES);
+ WRITE_ONCE(grp->rss_bound, max(READ_ONCE(grp->rss_bound), rss));
+ }
+
After a second thought, if we apply RSS based Fix2, do we still need Fix1?
thanks,
Chenyu
> +
> curr_cpu = task_cpu(p);
> if (invalid_llc_nr(grp, p, curr_cpu) ||
> exceed_llc_capacity(grp, curr_cpu)) {
> --
> 2.53.0
>
>
>
>
next prev parent reply other threads:[~2026-11-17 12:06 UTC|newest]
Thread overview: 80+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-04-01 21:52 [Patch v4 00/22] Cache aware scheduling Tim Chen
2026-04-01 21:52 ` [Patch v4 01/22] sched/cache: Introduce infrastructure for cache-aware load balancing Tim Chen
2026-04-09 12:41 ` Peter Zijlstra
2026-04-09 19:21 ` Tim Chen
2026-04-09 23:00 ` Peter Zijlstra
2026-04-10 6:30 ` Chen, Yu C
2026-04-15 2:06 ` Vern Hao
2026-04-15 3:34 ` Chen, Yu C
2026-09-14 17:50 ` Zenghui Yu
2026-09-14 23:12 ` Tim Chen
2026-09-18 7:14 ` Zenghui Yu
2026-04-01 21:52 ` [Patch v4 02/22] sched/cache: Limit the scan number of CPUs when calculating task occupancy Tim Chen
2026-04-09 13:17 ` Luo Gengkun
2026-04-09 13:41 ` Peter Zijlstra
2026-04-10 10:12 ` Luo Gengkun
2026-04-10 7:29 ` Chen, Yu C
2026-04-10 10:20 ` Luo Gengkun
2026-04-10 17:12 ` Tim Chen
2026-04-10 17:27 ` Chen, Yu C
2026-04-13 7:23 ` [RFC PATCH] sched/fair: dynamically scale the period of cache work Jianyong Wu
2026-04-13 8:38 ` Chen, Yu C
2026-04-13 11:27 ` Jianyong Wu
2026-04-15 3:31 ` Chen, Yu C
2026-04-16 3:39 ` Jianyong Wu
2026-04-15 17:22 ` Tim Chen
2026-04-16 6:50 ` Jianyong Wu
2026-04-14 15:07 ` [PATCH v2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus Luo Gengkun
2026-04-15 3:10 ` Chen, Yu C
2026-04-18 9:01 ` Luo Gengkun
2026-04-20 7:53 ` Chen, Yu C
2026-04-23 8:54 ` [PATCH v3] " Luo Gengkun
2026-04-01 21:52 ` [Patch v4 03/22] sched/cache: Record per LLC utilization to guide cache aware scheduling decisions Tim Chen
2026-04-01 21:52 ` [Patch v4 04/22] sched/cache: Introduce helper functions to enforce LLC migration policy Tim Chen
2026-04-01 21:52 ` [Patch v4 05/22] sched/cache: Make LLC id continuous Tim Chen
2026-04-01 21:52 ` [Patch v4 06/22] sched/cache: Assign preferred LLC ID to processes Tim Chen
2026-04-01 21:52 ` [Patch v4 07/22] sched/cache: Track LLC-preferred tasks per runqueue Tim Chen
2026-04-01 21:52 ` [Patch v4 08/22] sched/cache: Introduce per CPU's tasks LLC preference counter Tim Chen
2026-04-01 21:52 ` [Patch v4 09/22] sched/cache: Calculate the percpu sd task LLC preference Tim Chen
2026-04-01 21:52 ` [Patch v4 10/22] sched/cache: Count tasks prefering destination LLC in a sched group Tim Chen
2026-04-01 21:52 ` [Patch v4 11/22] sched/cache: Check local_group only once in update_sg_lb_stats() Tim Chen
2026-04-01 21:52 ` [Patch v4 12/22] sched/cache: Prioritize tasks preferring destination LLC during balancing Tim Chen
2026-04-01 21:52 ` [Patch v4 13/22] sched/cache: Add migrate_llc_task migration type for cache-aware balancing Tim Chen
2026-04-01 21:52 ` [Patch v4 14/22] sched/cache: Handle moving single tasks to/from their preferred LLC Tim Chen
2026-04-01 21:52 ` [Patch v4 15/22] sched/cache: Respect LLC preference in task migration and detach Tim Chen
2026-04-01 21:52 ` [Patch v4 16/22] sched/cache: Disable cache aware scheduling for processes with high thread counts Tim Chen
2026-04-09 12:43 ` Peter Zijlstra
2026-04-09 19:27 ` Tim Chen
2026-04-01 21:52 ` [Patch v4 17/22] sched/cache: Avoid cache-aware scheduling for memory-heavy processes Tim Chen
2026-04-09 12:46 ` Peter Zijlstra
2026-04-09 12:55 ` Peter Zijlstra
2026-04-10 8:59 ` Chen, Yu C
2026-04-10 9:20 ` Peter Zijlstra
2026-11-13 16:29 ` Jirka Hladky
2026-11-13 23:06 ` Tim Chen
2026-11-14 0:55 ` Jirka Hladky
2026-11-16 5:46 ` Chen Yu
2026-11-16 14:35 ` Jirka Hladky
2026-11-16 23:16 ` Tim Chen
2026-11-17 12:07 ` Chen Yu [this message]
2026-11-17 12:20 ` Chen Yu
2026-04-01 21:52 ` [Patch v4 18/22] sched/cache: Enable cache aware scheduling for multi LLCs NUMA node Tim Chen
2026-04-09 13:37 ` Peter Zijlstra
2026-04-09 19:39 ` Tim Chen
2026-04-01 21:52 ` [Patch v4 19/22] sched/cache: Allow the user space to turn on and off cache aware scheduling Tim Chen
2026-04-01 21:52 ` [Patch v4 20/22] sched/cache: Add user control to adjust the aggressiveness of cache-aware scheduling Tim Chen
2026-04-01 21:52 ` [Patch v4 21/22] -- DO NOT APPLY!!! -- sched/cache/debug: Display the per LLC occupancy for each process via proc fs Tim Chen
2026-04-01 21:52 ` [Patch v4 22/22] -- DO NOT APPLY!!! -- sched/cache/debug: Add ftrace to track the load balance statistics Tim Chen
2026-04-09 13:54 ` [Patch v4 00/22] Cache aware scheduling Peter Zijlstra
2026-04-09 20:02 ` Tim Chen
2026-04-14 3:20 ` Duan Tingyin
2026-04-15 17:35 ` Tim Chen
2026-04-16 0:27 ` Qais Yousef
2026-04-20 9:01 ` Chen, Yu C
2026-04-21 0:34 ` Qais Yousef
2026-04-21 20:57 ` Tim Chen
2026-04-23 15:06 ` Qais Yousef
2026-04-23 16:48 ` Chen, Yu C
2026-04-25 0:05 ` Qais Yousef
2026-04-23 17:17 ` Chen, Yu C
2026-04-25 0:14 ` Qais Yousef
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=avxDmPs4rs39NC9O@three-body \
--to=chen.yu@linux.dev \
--cc=adamli@os.amperecomputing.com \
--cc=aubrey.li@intel.com \
--cc=bsegall@google.com \
--cc=cyy@cyyself.name \
--cc=dietmar.eggemann@arm.com \
--cc=gautham.shenoy@amd.com \
--cc=gavinguo@igalia.com \
--cc=haoxing990@gmail.com \
--cc=hdanton@sina.com \
--cc=jhladky@redhat.com \
--cc=jianyong.wu@outlook.com \
--cc=jmario@redhat.com \
--cc=joshdon@google.com \
--cc=juri.lelli@redhat.com \
--cc=kprateek.nayak@amd.com \
--cc=len.brown@intel.com \
--cc=libchen@purestorage.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=pauld@redhat.com \
--cc=peterz@infradead.org \
--cc=qyousef@layalina.io \
--cc=reinette.chatre@intel.com \
--cc=rostedt@goodmis.org \
--cc=sshegde@linux.ibm.com \
--cc=tim.c.chen@intel.com \
--cc=tim.c.chen@linux.intel.com \
--cc=tingyin.duan@gmail.com \
--cc=tony.luck@intel.com \
--cc=vernhao@tencent.com \
--cc=vincent.guittot@linaro.org \
--cc=vineethr@linux.ibm.com \
--cc=vschneid@redhat.com \
--cc=yu.c.chen@intel.com \
--cc=yu.chen.surf@gmail.com \
--cc=zhao1.liu@intel.com \
--cc=ziqianlu@bytedance.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox