Re: [PATCH v2] sched/topology: Check average distances to remote packages

Kyle Meyer
2026-02-25T16:42:25Z

On Wed, Feb 25, 2026 at 01:30:52PM +0100, Peter Zijlstra wrote:
> On Tue, Feb 24, 2026 at 07:43:10PM -0600, Kyle Meyer wrote:
>
> > Here's an 8 socket (2 chassis) HPE system with SNC enabled:
> >
> > node 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> > 0: 10 12 16 16 16 16 18 18 40 40 40 40 40 40 40 40
> > 1: 12 10 16 16 16 16 18 18 40 40 40 40 40 40 40 40
> > 2: 16 16 10 12 18 18 16 16 40 40 40 40 40 40 40 40
> > 3: 16 16 12 10 18 18 16 16 40 40 40 40 40 40 40 40
> > 4: 16 16 18 18 10 12 16 16 40 40 40 40 40 40 40 40
> > 5: 16 16 18 18 12 10 16 16 40 40 40 40 40 40 40 40
> > 6: 18 18 16 16 16 16 10 12 40 40 40 40 40 40 40 40
> > 7: 18 18 16 16 16 16 12 10 40 40 40 40 40 40 40 40
> > 8: 40 40 40 40 40 40 40 40 10 12 16 16 16 16 18 18
> > 9: 40 40 40 40 40 40 40 40 12 10 16 16 16 16 18 18
> > 10: 40 40 40 40 40 40 40 40 16 16 10 12 18 18 16 16
> > 11: 40 40 40 40 40 40 40 40 16 16 12 10 18 18 16 16
> > 12: 40 40 40 40 40 40 40 40 16 16 18 18 10 12 16 16
> > 13: 40 40 40 40 40 40 40 40 16 16 18 18 12 10 16 16
> > 14: 40 40 40 40 40 40 40 40 18 18 16 16 16 16 10 12
> > 15: 40 40 40 40 40 40 40 40 18 18 16 16 16 16 12 10
> >
> > 10 = Same chassis and socket
> > 12 = Same chassis and socket (SNC)
> > 16 = Same chassis and adjacent socket
> > 18 = Same chassis and non-adjacent socket
> > 40 = Different chassis
> >
> > Each processor connects to an ASIC (XNC) that acts as a multiplexer, extending
> > the UPI interconnect across the entire system.
> >
> > We don't experience the scheduler domain issue reported by Tim because our SLIT
> > provides symmetric distances to remote NUMA nodes, but we trigger the WARN_ONCE
> > because we exceed 2 packages.
>
> The original case was for SNC-3, the above looks to be SNC-2. Does your
> system also support SNC-3?

We do not currently use SKUs that support SNC-3.

That distance would be set to 12:

node 0 1 2
0: 10 12 12
1: 12 10 12
2: 12 12 10

That might be changed if there's actually a difference in distance.

Distances to adjacent sockets, non-adjacent sockets, and different chassis would
remain the same.

> Anyway, yes your SLIT table looks sane (unlike that SNC-3 monster Tim
> showed earlier).
>
> And it also shows that using REMOTE_DISTANCE (20) was completely random
> and 'wrong'.
>
> So per 4d6dd05d07d0 ("sched/topology: Fix sched domain build error for GNR, CWF in SNC-3 mode")
>
> Tim's original crazy SNC-3 SLIT table was:
>
> node distances:
> node 0 1 2 3 4 5
> 0: 10 15 17 21 28 26
> 1: 15 10 15 23 26 23
> 2: 17 15 10 26 23 21
> 3: 21 28 26 10 15 17
> 4: 23 26 23 15 10 15
> 5: 26 23 21 17 15 10
>
> And per:
>
> https://lore.kernel.org/lkml/20250825075642.GQ3245006@noisy.programming.kicks-ass.net/
>
> My suggestion was to average the off-trace clusters to restore sanity.
>
> So how about we go about implementing that without reference to magical
> numbers, something like so. This obviously needs a little TLC, but it
> might just work.
>
> Hmm?
>
> ---
> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
> index 5cd6950ab672..cba3e4b14250 100644
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -513,33 +513,55 @@ static void __init build_sched_topology(void)
> }
>
> #ifdef CONFIG_NUMA
> -static int sched_avg_remote_distance;
> -static int avg_remote_numa_distance(void)
> +
> +/*
> + * Find the largest symmetric cluster in an attempt to identify the unit size.
> + *
> + * XXX doesn't respect N_CPU node classes and such.
> + */
> +static int slit_cluster_size(void)
> {
> - int i, j;
> - int distance, nr_remote, total_distance;
> + int i, j, n, m = num_possible_nodes();
>
> - if (sched_avg_remote_distance > 0)
> - return sched_avg_remote_distance;
> -
> - nr_remote = 0;
> - total_distance = 0;
> - for_each_node_state(i, N_CPU) {
> - for_each_node_state(j, N_CPU) {
> - distance = node_distance(i, j);
> -
> - if (distance >= REMOTE_DISTANCE) {
> - nr_remote++;
> - total_distance += distance;
> + for (n = 2; n < m; n++) {
> + for (i = 0; i < n; i++) {
> + for (j = i; j < n; j++) {
> + if (node_distance(i, j) != node_distance(j, i))
> + return n - 1;
> }
> }
> }
> - if (nr_remote)
> - sched_avg_remote_distance = total_distance / nr_remote;
> - else
> - sched_avg_remote_distance = REMOTE_DISTANCE;
>
> - return sched_avg_remote_distance;
> + return m;
> +}
> +
> +static int slit_cluster_distance(int i, int j)
> +{
> + static int u = 0;
> + long d = 0;
> + int x, y;
> +
> + if (!u)
> + u = slit_cluster_size();
> +
> + /*
> + * Is this a unit cluster on the trace?
> + */
> + if ((i / u) == (j / u))
> + return node_distance(i, j);
> +
> + /*
> + * Off-trace cluster, return average of the cluster to force symmetry.
> + */
> + x = i - (i % u);
> + y = j - (j % u);
> +
> + for (i = x; i < x + u; i++) {
> + for (j = y; j < y + u; j++)
> + d += node_distance(i, j);
> + }
> +
> + return d / (u*u);
> }
>
> int arch_sched_node_distance(int from, int to)
> @@ -550,8 +572,7 @@ int arch_sched_node_distance(int from, int to)
> case INTEL_GRANITERAPIDS_X:
> case INTEL_ATOM_DARKMONT_X:
>
> - if (!x86_has_numa_in_package || topology_max_packages() == 1 ||
> - d < REMOTE_DISTANCE)
> + if (!x86_has_numa_in_package || topology_max_packages() == 1)
> return d;
>
> /*
> @@ -571,12 +592,7 @@ int arch_sched_node_distance(int from, int to)
> * packages as average distance to different remote packages
> * could be different.
> */
> - WARN_ONCE(topology_max_packages() > 2,
> - "sched: Expect only up to 2 packages for GNR or CWF, "
> - "but saw %d packages when building sched domains.",
> - topology_max_packages());
> -
> - d = avg_remote_numa_distance();
> + return slit_cluster_distance(from, to);
> }
> return d;
> }