The Linux Kernel Mailing List
 help / color / mirror / Atom feed
* [PATCH v2] sched/cache: Enable LLC aggregation above the first NUMA level
@ 2026-11-12  6:54 Jianyong Wu
  2026-11-19 11:01 ` Chen Yu
  0 siblings, 1 reply; 2+ messages in thread
From: Jianyong Wu @ 2026-11-12  6:54 UTC (permalink / raw)
  To: mingo, peterz, juri.lelli, vincent.guittot, yu.c.chen, tim.c.chen
  Cc: dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, linux-kernel, wujianyong, jianyong.wu, liuyibin,
	zhongyuan

sched_balance_find_src_group() only considers group_llc_balance when
sds.prefer_sibling is set:

	if (sds.prefer_sibling && local->group_type == group_has_spare &&
	    (busiest->group_type == group_llc_balance ||
	     sibling_imbalance(env, &sds, busiest, local) > 1))
		goto force_balance;

sds.prefer_sibling mirrors SD_PREFER_SIBLING of the busiest group's
child sched domain, and sd_init() clears that flag on every SD_NUMA
domain. So on any domain whose child is itself a NUMA domain, i.e. on
systems with more than one NUMA level such as multiple nodes per
socket, groups tagged with group_llc_balance by llc_balance() can
never take the force_balance path.

Such a group falls into the !group_overloaded branch below and in the
common case ends up in out_balanced, so a task that was placed beyond
the first NUMA level is never pulled into its preferred LLC.

Only require sds.prefer_sibling for the sibling_imbalance() case,
which is what it was originally introduced for.

Split the condition into two explicit tests and gate the llc_balance
one on sched_cache_enabled(), so that the non cache aware case never
reads the group type, and the test disappears at compile time when
CONFIG_SCHED_CACHE is off.

Reported-by: Yibin Liu <liuyibin@hygon.cn>
Suggested-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
---
v1 -> v2:
- Split the condition into two tests and gate the llc_balance one on
  sched_cache_enabled() via busiest_llc_balance() (Tim Chen).
- Add schbench/hackbench results with NUMA balancing enabled and
  disabled (Chen Yu).
- Reword the subject and drop the Fixes tag: this enables aggregation
  above the first NUMA level rather than fixing a bug (Chen Yu).

v1: https://lore.kernel.org/all/20261104141313.185433-1-wujianyong@hygon.cn/

[Testing]

The baseline kernel is fb7bd3a2aad7. The baseline and patched kernels
used the same boot command line.

Hygon machine with the following topology:
* 2 sockets
* 4 NUMA nodes per socket
* 4 LLCs per NUMA node
* 4 cores per LLC domain
* 2 SMT threads per core

With the default aggr_tolerance (1), a process is only aggregated when
its average number of running threads is below 3.2 on this machine
(8 CPUs per LLC, SMT-2). aggr_tolerance was set to 2 so that up to 6.4
running threads can be aggregated. schbench with 3 and 4 worker threads
(4 and 5 running threads including the message thread) shows the main
difference. C2 was disabled on all CPUs for this test to reduce the jitter
from C2 exit latency, which makes the difference between the baseline and
patched kernels easier to see.

schbench -m 1 -t $threads -r 30 -i 100, 30 runs per configuration.
p99 wakeup latency, mean of all runs (lower is better, normalized to
baseline):

NUMA balancing enabled:

threads   baseline          patched           improve
========================================================
3         1.000             0.781             21.9%
4         1.000             0.742             25.8%

NUMA balancing disabled:

threads   baseline          patched           improve
========================================================
3         1.000             0.845             15.5%
4         1.000             0.733             26.7%

The fraction of runs in which all threads stayed in one LLC was also
recorded. Thread placement was checked every 3s, and a run counts as
aggregated only if all threads were in one LLC at every check:

NUMA balancing   threads   baseline     patched
================================================
enabled          3         80%          97%
enabled          4         60%          97%
disabled         3         80%          100%
disabled         4         53%          100%

Runs with all threads in one LLC had lower p99 than runs in which the
threads were spread over more than one LLC, so the higher aggregation
rate with this patch accounts for the improvement.

The other tests below were run with aggr_tolerance at its default (1)
and C2 enabled; all other settings were unchanged, and only NUMA
balancing was toggled. They show no obvious regression.

hackbench:
(lower is better, normalized to baseline; the figure in parentheses
is the standard deviation, as a percentage of the mean)
test cmd: hackbench -T -p -f $f -g $g -l 100000

schbench:
p99 wakeup latency (lower is better, normalized to baseline; the figure
in parentheses is the standard deviation, as a percentage of the mean)
test cmd: schbench -m 1 -t $threads -r 30 -w 5 -i 100

Each configuration was run 10 times; configurations with large
run-to-run variation were re-run 20-40 times.

NUMA balancing disabled:

fds    groups  baseline          patched           improve
=============================================================
2      1       1.000 (6.33%)     0.963 (2.48%)     3.69%
2      2       1.000 (46.47%)    0.925 (19.65%)    7.53%
3      1       1.000 (36.07%)    0.860 (25.27%)    13.99%
4      1       1.000 (34.52%)    0.992 (39.02%)    0.79%
4      2       1.000 (28.45%)    0.925 (31.71%)    7.48%
8      1       1.000 (19.00%)    1.010 (12.27%)    -0.99%
8      2       1.000 (8.94%)     0.962 (10.80%)    3.80%
12     1       1.000 (11.83%)    0.991 (13.08%)    0.89%
12     2       1.000 (7.16%)     1.009 (6.25%)     -0.88%
16     1       1.000 (12.56%)    1.030 (11.36%)    -2.96%
16     2       1.000 (9.49%)     0.997 (7.10%)     0.33%
20     1       1.000 (4.76%)     0.999 (4.51%)     0.06%
20     2       1.000 (3.12%)     0.971 (5.26%)     2.87%
24     1       1.000 (8.05%)     1.017 (3.84%)     -1.66%
24     2       1.000 (5.93%)     1.038 (5.13%)     -3.75%
32     1       1.000 (3.77%)     0.986 (3.89%)     1.40%
32     2       1.000 (2.76%)     1.028 (3.48%)     -2.83%
48     1       1.000 (0.56%)     1.012 (2.06%)     -1.21%
48     2       1.000 (2.41%)     0.989 (1.89%)     1.14%

threads   baseline          patched           improve
========================================================
2         1.000 (7.50%)     1.046 (18.54%)    -4.56%
6         1.000 (14.54%)    1.028 (13.17%)    -2.80%
8         1.000 (11.12%)    0.939 (20.46%)    6.10%
12        1.000 (14.66%)    0.996 (13.49%)    0.38%
16        1.000 (14.12%)    1.004 (13.08%)    -0.36%
32        1.000 (14.16%)    1.030 (12.74%)    -2.98%
48        1.000 (23.37%)    0.932 (11.64%)    6.85%
64        1.000 (71.38%)    0.883 (80.19%)    11.72%
96        1.000 (2.32%)     1.025 (3.42%)     -2.46%
128       1.000 (10.06%)    1.009 (10.12%)    -0.87%

NUMA balancing enabled:

fds    groups  baseline          patched           improve
=============================================================
2      1       1.000 (69.72%)    0.781 (39.69%)    21.88%
2      2       1.000 (44.86%)    0.829 (22.25%)    17.08%
3      1       1.000 (34.37%)    0.933 (32.66%)    6.75%
4      1       1.000 (35.28%)    0.816 (39.43%)    18.38%
4      2       1.000 (7.18%)     0.877 (28.44%)    12.35%
8      1       1.000 (7.66%)     0.950 (8.68%)     5.05%
8      2       1.000 (5.47%)     0.909 (12.19%)    9.08%
12     1       1.000 (6.94%)     0.999 (8.91%)     0.15%
12     2       1.000 (11.10%)    1.025 (8.17%)     -2.55%
16     1       1.000 (6.95%)     0.955 (9.78%)     4.49%
16     2       1.000 (4.78%)     1.017 (6.14%)     -1.70%
20     1       1.000 (11.78%)    1.041 (7.17%)     -4.08%
20     2       1.000 (2.93%)     0.957 (5.59%)     4.27%
24     1       1.000 (3.71%)     0.982 (4.90%)     1.77%
24     2       1.000 (3.12%)     1.018 (2.72%)     -1.80%
32     1       1.000 (3.09%)     0.997 (2.54%)     0.33%
32     2       1.000 (1.91%)     1.003 (2.58%)     -0.26%
48     1       1.000 (2.76%)     1.016 (1.23%)     -1.61%
48     2       1.000 (2.57%)     0.965 (2.14%)     3.50%

threads   baseline          patched           improve
========================================================
2         1.000 (24.63%)    0.901 (5.05%)     9.94%
6         1.000 (11.89%)    0.998 (6.94%)     0.19%
8         1.000 (13.61%)    0.979 (14.44%)    2.10%
12        1.000 (13.42%)    0.958 (16.61%)    4.18%
16        1.000 (21.23%)    0.934 (16.92%)    6.60%
32        1.000 (19.45%)    0.985 (12.27%)    1.48%
48        1.000 (8.24%)     1.003 (13.56%)    -0.34%
64        1.000 (71.56%)    1.020 (75.36%)    -1.96%
96        1.000 (4.27%)     0.981 (2.15%)     1.90%
128       1.000 (9.69%)     0.963 (7.86%)     3.70%

 kernel/sched/fair.c | 35 +++++++++++++++++++++++++++--------
 1 file changed, 27 insertions(+), 8 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index cfc982205b98..01b9f7885574 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -12192,6 +12192,16 @@ static bool update_llc_busiest(struct lb_env *env,
 	 */
 	return sgs->nr_pref_dst_llc > busiest->nr_pref_dst_llc;
 }
+
+/*
+ * The busiest group has tasks that prefer the destination LLC and
+ * cache aware scheduling is enabled.
+ */
+static inline bool busiest_llc_balance(struct sg_lb_stats *busiest)
+{
+	return sched_cache_enabled() &&
+	       busiest->group_type == group_llc_balance;
+}
 #else
 static inline void record_sg_llc_stats(struct lb_env *env, struct sg_lb_stats *sgs,
 				       struct sched_group *group)
@@ -12210,6 +12220,11 @@ static bool update_llc_busiest(struct lb_env *env,
 {
 	return false;
 }
+
+static inline bool busiest_llc_balance(struct sg_lb_stats *busiest)
+{
+	return false;
+}
 #endif
 
 /**
@@ -13303,14 +13318,18 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env)
 			goto out_balanced;
 	}
 
-	/*
-	 * Try to move all excess tasks to a sibling domain of the busiest
-	 * group's child domain.
-	 */
-	if (sds.prefer_sibling && local->group_type == group_has_spare &&
-	    (busiest->group_type == group_llc_balance ||
-	    sibling_imbalance(env, &sds, busiest, local) > 1))
-		goto force_balance;
+	if (local->group_type == group_has_spare) {
+		if (busiest_llc_balance(busiest))
+			goto force_balance;
+
+		/*
+		 * Try to move all excess tasks to a sibling domain of the
+		 * busiest group's child domain.
+		 */
+		if (sds.prefer_sibling &&
+		    sibling_imbalance(env, &sds, busiest, local) > 1)
+			goto force_balance;
+	}
 
 	if (busiest->group_type != group_overloaded) {
 		if (!env->idle) {

base-commit: fb7bd3a2aad70a76232a7168a505386d0a1396af
-- 
2.34.1



^ permalink raw reply related	[flat|nested] 2+ messages in thread

* Re: [PATCH v2] sched/cache: Enable LLC aggregation above the first NUMA level
  2026-11-12  6:54 [PATCH v2] sched/cache: Enable LLC aggregation above the first NUMA level Jianyong Wu
@ 2026-11-19 11:01 ` Chen Yu
  0 siblings, 0 replies; 2+ messages in thread
From: Chen Yu @ 2026-11-19 11:01 UTC (permalink / raw)
  To: Jianyong Wu
  Cc: mingo, peterz, juri.lelli, vincent.guittot, tim.c.chen,
	dietmar.eggemann, rostedt, bsegall, mgorman, vschneid,
	kprateek.nayak, linux-kernel, jianyong.wu, liuyibin, zhongyuan,
	chen.yu

Hello Jianyong,

On Thu, Nov 12, 2026 at 02:54:45PM +0800, Jianyong Wu wrote:
> sched_balance_find_src_group() only considers group_llc_balance when
> sds.prefer_sibling is set:
> 
> 	if (sds.prefer_sibling && local->group_type == group_has_spare &&
> 	    (busiest->group_type == group_llc_balance ||
> 	     sibling_imbalance(env, &sds, busiest, local) > 1))
> 		goto force_balance;
> 
> sds.prefer_sibling mirrors SD_PREFER_SIBLING of the busiest group's
> child sched domain, and sd_init() clears that flag on every SD_NUMA
> domain. So on any domain whose child is itself a NUMA domain, i.e. on
> systems with more than one NUMA level such as multiple nodes per
> socket, groups tagged with group_llc_balance by llc_balance() can
> never take the force_balance path.
> 
> Such a group falls into the !group_overloaded branch below and in the
> common case ends up in out_balanced, so a task that was placed beyond
> the first NUMA level is never pulled into its preferred LLC.
> 
> Only require sds.prefer_sibling for the sibling_imbalance() case,
> which is what it was originally introduced for.
> 
> Split the condition into two explicit tests and gate the llc_balance
> one on sched_cache_enabled(), so that the non cache aware case never
> reads the group type, and the test disappears at compile time when
> CONFIG_SCHED_CACHE is off.
> 
> Reported-by: Yibin Liu <liuyibin@hygon.cn>
> Suggested-by: Tim Chen <tim.c.chen@linux.intel.com>
> Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
> ---
> v1 -> v2:
> - Split the condition into two tests and gate the llc_balance one on
>   sched_cache_enabled() via busiest_llc_balance() (Tim Chen).
> - Add schbench/hackbench results with NUMA balancing enabled and
>   disabled (Chen Yu).
> - Reword the subject and drop the Fixes tag: this enables aggregation
>   above the first NUMA level rather than fixing a bug (Chen Yu).
> 
> v1: https://lore.kernel.org/all/20261104141313.185433-1-wujianyong@hygon.cn/
>

[ ... ]
 
>  kernel/sched/fair.c | 35 +++++++++++++++++++++++++++--------
>  1 file changed, 27 insertions(+), 8 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index cfc982205b98..01b9f7885574 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -12192,6 +12192,16 @@ static bool update_llc_busiest(struct lb_env *env,
>  	 */
>  	return sgs->nr_pref_dst_llc > busiest->nr_pref_dst_llc;
>  }
> +
> +/*
> + * The busiest group has tasks that prefer the destination LLC and
> + * cache aware scheduling is enabled.
> + */
> +static inline bool busiest_llc_balance(struct sg_lb_stats *busiest)
> +{
> +	return sched_cache_enabled() &&
> +	       busiest->group_type == group_llc_balance;
> +}
>  #else
>  static inline void record_sg_llc_stats(struct lb_env *env, struct sg_lb_stats *sgs,
>  				       struct sched_group *group)
> @@ -12210,6 +12220,11 @@ static bool update_llc_busiest(struct lb_env *env,
>  {
>  	return false;
>  }
> +
> +static inline bool busiest_llc_balance(struct sg_lb_stats *busiest)
> +{
> +	return false;
> +}
>  #endif
>  
>  /**
> @@ -13303,14 +13318,18 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env)
>  			goto out_balanced;
>  	}
>  
> -	/*
> -	 * Try to move all excess tasks to a sibling domain of the busiest
> -	 * group's child domain.
> -	 */
> -	if (sds.prefer_sibling && local->group_type == group_has_spare &&
> -	    (busiest->group_type == group_llc_balance ||
> -	    sibling_imbalance(env, &sds, busiest, local) > 1))
> -		goto force_balance;
> +	if (local->group_type == group_has_spare) {
> +		if (busiest_llc_balance(busiest))
> +			goto force_balance;
> +
> +		/*
> +		 * Try to move all excess tasks to a sibling domain of the
> +		 * busiest group's child domain.
> +		 */
> +		if (sds.prefer_sibling &&
> +		    sibling_imbalance(env, &sds, busiest, local) > 1)
> +			goto force_balance;
> +	}
>  
>  	if (busiest->group_type != group_overloaded) {
>  		if (!env->idle) {
>

Got one question below. After cache-aware scheduling is enabled at NUMA level 2,
should we adjust src_cpu in llc_balance()?

Suppose there are 4 NUMA nodes, N1, N2, N3, and N4. Under each NUMA node, there are
sub-NUMA nodes, say N1.1, N1.2, and N2.1, N2.2, and so on.

Before this patch, cache-aware scheduling is allowed to aggregate tasks between N1.1
and N1.2, but not between N1 and Nx. With this patch applied, it becomes legal to
aggregate tasks between N1 and Nx.

Consider N1 as the dst_group. Then we try to find the Nx group as the busiest src_group.
Say we choose N2 as the busiest one. Then in llc_balance(), we evaluate the utilization
between the LLC containing env->dst_cpu and the LLC containing cpumask_first(sched_group_span(N2)).
The latter always returns the first CPU in N2.1, which might be wrong, because the busiest could
be N2.2 as well.

So, we might want to use the busiest CPU in the busiest N2 rather than the first CPU in the
busiest N2, how about something like below:

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 79f6b3a4fd7f..f2f15b20e3c7 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -12149,7 +12149,7 @@ static void record_sg_llc_stats(struct lb_env *env,
  * to run on LLC in idle dst_cpu.
  */
 static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs,
-			       struct sched_group *group)
+			       int src_cpu)
 {
 	if (!sched_cache_enabled())
 		return false;
@@ -12183,8 +12183,7 @@ static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs,
 		return false;
 
 	if (sgs->nr_pref_dst_llc &&
-	    can_migrate_llc(cpumask_first(sched_group_span(group)),
-			    env->dst_cpu, 0, true) == mig_llc)
+	    can_migrate_llc(src_cpu, env->dst_cpu, 0, true) == mig_llc)
 		return true;
 
 	return false;
@@ -12235,6 +12234,8 @@ static inline void update_sg_lb_stats(struct lb_env *env,
 {
 	int i, nr_running, local_group, sd_flags = env->sd->flags;
 	bool balancing_at_rd = !env->sd->parent;
+	unsigned int max_pref_nr = 0;
+	int max_pref_cpu = -1;
 
 	memset(sgs, 0, sizeof(*sgs));
 
@@ -12258,13 +12259,20 @@ static inline void update_sg_lb_stats(struct lb_env *env,
 #ifdef CONFIG_SCHED_CACHE
 		if (sched_cache_enabled()) {
 			struct sched_domain *sd_tmp;
+			unsigned int nr_pref;
 			int dst_llc;
 
 			dst_llc = llc_id(env->dst_cpu);
 			if (llc_id(i) != dst_llc) {
 				sd_tmp = rcu_dereference_all(rq->sd);
-				if (sd_tmp && (unsigned int)dst_llc < sd_tmp->llc_max)
-					sgs->nr_pref_dst_llc += sd_tmp->llc_counts[dst_llc];
+				if (sd_tmp && (unsigned int)dst_llc < sd_tmp->llc_max) {
+					nr_pref = sd_tmp->llc_counts[dst_llc];
+					sgs->nr_pref_dst_llc += nr_pref;
+					if (nr_pref > max_pref_nr) {
+						max_pref_nr = nr_pref;
+						max_pref_cpu = i;
+					}
+				}
 			}
 		}
 #endif
@@ -12335,7 +12343,7 @@ static inline void update_sg_lb_stats(struct lb_env *env,
 			sgs->group_smt_balance = 1;
 
 		/* Check for tasks in this group can be moved to their preferred LLC */
-		if (llc_balance(env, sgs, group))
+		if (llc_balance(env, sgs, max_pref_cpu))
 			sgs->group_llc_balance = 1;
 	}
 
-- 
2.43.0



^ permalink raw reply related	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2026-11-19 11:12 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-12  6:54 [PATCH v2] sched/cache: Enable LLC aggregation above the first NUMA level Jianyong Wu
2026-11-19 11:01 ` Chen Yu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox