The Linux Kernel Mailing List
 help / color / mirror / Atom feed
From: Chen Yu <yu.c.chen@intel.com>
To: Jianyong Wu <wujianyong@hygon.cn>
Cc: <mingo@redhat.com>, <peterz@infradead.org>,
	<juri.lelli@redhat.com>, <vincent.guittot@linaro.org>,
	<tim.c.chen@linux.intel.com>, <dietmar.eggemann@arm.com>,
	<rostedt@goodmis.org>, <bsegall@google.com>, <mgorman@suse.de>,
	<vschneid@redhat.com>, <kprateek.nayak@amd.com>,
	<linux-kernel@vger.kernel.org>, <jianyong.wu@outlook.com>,
	<liuyibin@hygon.cn>, <zhongyuan@hygon.cn>, <chen.yu@linux.dev>
Subject: Re: [PATCH v2] sched/cache: Enable LLC aggregation above the first NUMA level
Date: Thu, 19 Nov 2026 19:01:49 +0800	[thread overview]
Message-ID: <av7XHYkg+XLO+Xm4@chenyu-dev> (raw)
In-Reply-To: <20261112065445.592968-1-wujianyong@hygon.cn>

Hello Jianyong,

On Thu, Nov 12, 2026 at 02:54:45PM +0800, Jianyong Wu wrote:
> sched_balance_find_src_group() only considers group_llc_balance when
> sds.prefer_sibling is set:
> 
> 	if (sds.prefer_sibling && local->group_type == group_has_spare &&
> 	    (busiest->group_type == group_llc_balance ||
> 	     sibling_imbalance(env, &sds, busiest, local) > 1))
> 		goto force_balance;
> 
> sds.prefer_sibling mirrors SD_PREFER_SIBLING of the busiest group's
> child sched domain, and sd_init() clears that flag on every SD_NUMA
> domain. So on any domain whose child is itself a NUMA domain, i.e. on
> systems with more than one NUMA level such as multiple nodes per
> socket, groups tagged with group_llc_balance by llc_balance() can
> never take the force_balance path.
> 
> Such a group falls into the !group_overloaded branch below and in the
> common case ends up in out_balanced, so a task that was placed beyond
> the first NUMA level is never pulled into its preferred LLC.
> 
> Only require sds.prefer_sibling for the sibling_imbalance() case,
> which is what it was originally introduced for.
> 
> Split the condition into two explicit tests and gate the llc_balance
> one on sched_cache_enabled(), so that the non cache aware case never
> reads the group type, and the test disappears at compile time when
> CONFIG_SCHED_CACHE is off.
> 
> Reported-by: Yibin Liu <liuyibin@hygon.cn>
> Suggested-by: Tim Chen <tim.c.chen@linux.intel.com>
> Signed-off-by: Jianyong Wu <wujianyong@hygon.cn>
> ---
> v1 -> v2:
> - Split the condition into two tests and gate the llc_balance one on
>   sched_cache_enabled() via busiest_llc_balance() (Tim Chen).
> - Add schbench/hackbench results with NUMA balancing enabled and
>   disabled (Chen Yu).
> - Reword the subject and drop the Fixes tag: this enables aggregation
>   above the first NUMA level rather than fixing a bug (Chen Yu).
> 
> v1: https://lore.kernel.org/all/20261104141313.185433-1-wujianyong@hygon.cn/
>

[ ... ]
 
>  kernel/sched/fair.c | 35 +++++++++++++++++++++++++++--------
>  1 file changed, 27 insertions(+), 8 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index cfc982205b98..01b9f7885574 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -12192,6 +12192,16 @@ static bool update_llc_busiest(struct lb_env *env,
>  	 */
>  	return sgs->nr_pref_dst_llc > busiest->nr_pref_dst_llc;
>  }
> +
> +/*
> + * The busiest group has tasks that prefer the destination LLC and
> + * cache aware scheduling is enabled.
> + */
> +static inline bool busiest_llc_balance(struct sg_lb_stats *busiest)
> +{
> +	return sched_cache_enabled() &&
> +	       busiest->group_type == group_llc_balance;
> +}
>  #else
>  static inline void record_sg_llc_stats(struct lb_env *env, struct sg_lb_stats *sgs,
>  				       struct sched_group *group)
> @@ -12210,6 +12220,11 @@ static bool update_llc_busiest(struct lb_env *env,
>  {
>  	return false;
>  }
> +
> +static inline bool busiest_llc_balance(struct sg_lb_stats *busiest)
> +{
> +	return false;
> +}
>  #endif
>  
>  /**
> @@ -13303,14 +13318,18 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env)
>  			goto out_balanced;
>  	}
>  
> -	/*
> -	 * Try to move all excess tasks to a sibling domain of the busiest
> -	 * group's child domain.
> -	 */
> -	if (sds.prefer_sibling && local->group_type == group_has_spare &&
> -	    (busiest->group_type == group_llc_balance ||
> -	    sibling_imbalance(env, &sds, busiest, local) > 1))
> -		goto force_balance;
> +	if (local->group_type == group_has_spare) {
> +		if (busiest_llc_balance(busiest))
> +			goto force_balance;
> +
> +		/*
> +		 * Try to move all excess tasks to a sibling domain of the
> +		 * busiest group's child domain.
> +		 */
> +		if (sds.prefer_sibling &&
> +		    sibling_imbalance(env, &sds, busiest, local) > 1)
> +			goto force_balance;
> +	}
>  
>  	if (busiest->group_type != group_overloaded) {
>  		if (!env->idle) {
>

Got one question below. After cache-aware scheduling is enabled at NUMA level 2,
should we adjust src_cpu in llc_balance()?

Suppose there are 4 NUMA nodes, N1, N2, N3, and N4. Under each NUMA node, there are
sub-NUMA nodes, say N1.1, N1.2, and N2.1, N2.2, and so on.

Before this patch, cache-aware scheduling is allowed to aggregate tasks between N1.1
and N1.2, but not between N1 and Nx. With this patch applied, it becomes legal to
aggregate tasks between N1 and Nx.

Consider N1 as the dst_group. Then we try to find the Nx group as the busiest src_group.
Say we choose N2 as the busiest one. Then in llc_balance(), we evaluate the utilization
between the LLC containing env->dst_cpu and the LLC containing cpumask_first(sched_group_span(N2)).
The latter always returns the first CPU in N2.1, which might be wrong, because the busiest could
be N2.2 as well.

So, we might want to use the busiest CPU in the busiest N2 rather than the first CPU in the
busiest N2, how about something like below:

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 79f6b3a4fd7f..f2f15b20e3c7 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -12149,7 +12149,7 @@ static void record_sg_llc_stats(struct lb_env *env,
  * to run on LLC in idle dst_cpu.
  */
 static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs,
-			       struct sched_group *group)
+			       int src_cpu)
 {
 	if (!sched_cache_enabled())
 		return false;
@@ -12183,8 +12183,7 @@ static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs,
 		return false;
 
 	if (sgs->nr_pref_dst_llc &&
-	    can_migrate_llc(cpumask_first(sched_group_span(group)),
-			    env->dst_cpu, 0, true) == mig_llc)
+	    can_migrate_llc(src_cpu, env->dst_cpu, 0, true) == mig_llc)
 		return true;
 
 	return false;
@@ -12235,6 +12234,8 @@ static inline void update_sg_lb_stats(struct lb_env *env,
 {
 	int i, nr_running, local_group, sd_flags = env->sd->flags;
 	bool balancing_at_rd = !env->sd->parent;
+	unsigned int max_pref_nr = 0;
+	int max_pref_cpu = -1;
 
 	memset(sgs, 0, sizeof(*sgs));
 
@@ -12258,13 +12259,20 @@ static inline void update_sg_lb_stats(struct lb_env *env,
 #ifdef CONFIG_SCHED_CACHE
 		if (sched_cache_enabled()) {
 			struct sched_domain *sd_tmp;
+			unsigned int nr_pref;
 			int dst_llc;
 
 			dst_llc = llc_id(env->dst_cpu);
 			if (llc_id(i) != dst_llc) {
 				sd_tmp = rcu_dereference_all(rq->sd);
-				if (sd_tmp && (unsigned int)dst_llc < sd_tmp->llc_max)
-					sgs->nr_pref_dst_llc += sd_tmp->llc_counts[dst_llc];
+				if (sd_tmp && (unsigned int)dst_llc < sd_tmp->llc_max) {
+					nr_pref = sd_tmp->llc_counts[dst_llc];
+					sgs->nr_pref_dst_llc += nr_pref;
+					if (nr_pref > max_pref_nr) {
+						max_pref_nr = nr_pref;
+						max_pref_cpu = i;
+					}
+				}
 			}
 		}
 #endif
@@ -12335,7 +12343,7 @@ static inline void update_sg_lb_stats(struct lb_env *env,
 			sgs->group_smt_balance = 1;
 
 		/* Check for tasks in this group can be moved to their preferred LLC */
-		if (llc_balance(env, sgs, group))
+		if (llc_balance(env, sgs, max_pref_cpu))
 			sgs->group_llc_balance = 1;
 	}
 
-- 
2.43.0



      reply	other threads:[~2026-11-19 11:12 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12  6:54 [PATCH v2] sched/cache: Enable LLC aggregation above the first NUMA level Jianyong Wu
2026-11-19 11:01 ` Chen Yu [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av7XHYkg+XLO+Xm4@chenyu-dev \
    --to=yu.c.chen@intel.com \
    --cc=bsegall@google.com \
    --cc=chen.yu@linux.dev \
    --cc=dietmar.eggemann@arm.com \
    --cc=jianyong.wu@outlook.com \
    --cc=juri.lelli@redhat.com \
    --cc=kprateek.nayak@amd.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=liuyibin@hygon.cn \
    --cc=mgorman@suse.de \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=tim.c.chen@linux.intel.com \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    --cc=wujianyong@hygon.cn \
    --cc=zhongyuan@hygon.cn \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox