[PATCH v2 0/2] sched/topology: Optimize topology_span_sane()

[PATCH v2 0/2] sched/topology: Optimize topology_span_sane()

[Kyle Meyer]

A soft lockup is being detected in build_sched_domains() on 32 socket
Sapphire Rapids systems with 3840 processors.

topology_span_sane(), called by build_sched_domains(), checks that each
processor's non-NUMA scheduling domains are completely equal or
completely disjoint. If a non-NUMA scheduling domain partially overlaps
another, scheduling groups can break.

This series adds for_each_cpu_from() as a generic cpumask macro to
optimize topology_span_sane() by removing duplicate comparisons. The
total number of comparisons is reduced from N * (N - 1) to
N * (N - 1) / 2 on each non-NUMA scheduling domain level, decreasing
the boot time by approximately 20 seconds and preventing the soft lockup
on the mentioned systems.

Changes in v2:
  * 1/2: Change for_each_cpu()'s description.
  * 2/2: Add more information to the commit message.
  * https://lore.kernel.org/linux-kernel/20240409155250.3660517-1-kyle.meyer@hpe.com/T/

Kyle Meyer (2):
  cpumask: Add for_each_cpu_from()
  sched/topology: Optimize topology_span_sane()

 include/linux/cpumask.h | 10 ++++++++++
 kernel/sched/topology.c |  6 ++----
 2 files changed, 12 insertions(+), 4 deletions(-)

-- 
2.44.0

[PATCH v2 0/2] sched/topology: optimize topology_span_sane()

[Yury Norov]

The function may call cpumask_equal with tl->mask(cpu) == tl->mask(i),
even when cpu != i. In such case, cpumask_equal() would always return
true, and we can proceed to the next iteration immediately.

Valentin Schneider shares on it:

  PKG can potentially hit that condition, and so can any
  sched_domain_mask_f that relies on the node masks...
  
  I'm thinking ideally we should have checks in place to
  ensure all node_to_cpumask_map[] masks are disjoint,
  then we could entirely skip the levels that use these
  masks in topology_span_sane(), but there's unfortunately
  no nice way to flag them... Also there would be cases
  where there's no real difference between PKG and NODE
  other than NODE is still based on a per-cpu cpumask and
  PKG isn't, so I don't see a nicer way to go about this.

v1: https://lore.kernel.org/lkml/ZrJk00cmVaUIAr4G@yury-ThinkPad/T/
v2:
 - defer initialization of 'mc' in patch #1 @Chen Yu;
 - more comments from Valentin Schneider.


Yury Norov (2):
  sched/topology: pre-compute topology_span_sane() loop params
  sched/topology: optimize topology_span_sane()

 kernel/sched/topology.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

-- 
2.43.0

[PATCH 1/2] sched/topology: pre-compute topology_span_sane() loop params

[Yury Norov]

tl->mask() is called inside the loop with the same parameters more than
once. We can pre-calculate it.

Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Yury Norov <yury.norov@gmail.com>
---
 kernel/sched/topology.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 76504b776d03..8af3b48da458 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -2353,12 +2353,15 @@ static struct sched_domain *build_sched_domain(struct sched_domain_topology_leve
 static bool topology_span_sane(struct sched_domain_topology_level *tl,
 			      const struct cpumask *cpu_map, int cpu)
 {
+	const struct cpumask *mi, *mc;
 	int i = cpu + 1;
 
 	/* NUMA levels are allowed to overlap */
 	if (tl->flags & SDTL_OVERLAP)
 		return true;
 
+	mc = tl->mask(cpu);
+
 	/*
 	 * Non-NUMA levels cannot partially overlap - they must be either
 	 * completely equal or completely disjoint. Otherwise we can end up
@@ -2366,14 +2369,15 @@ static bool topology_span_sane(struct sched_domain_topology_level *tl,
 	 * breaks the linking done for an earlier span.
 	 */
 	for_each_cpu_from(i, cpu_map) {
+		mi = tl->mask(i);
+
 		/*
 		 * We should 'and' all those masks with 'cpu_map' to exactly
 		 * match the topology we're about to build, but that can only
 		 * remove CPUs, which only lessens our ability to detect
 		 * overlaps
 		 */
-		if (!cpumask_equal(tl->mask(cpu), tl->mask(i)) &&
-		    cpumask_intersects(tl->mask(cpu), tl->mask(i)))
+		if (!cpumask_equal(mc, mi) && cpumask_intersects(mc, mi))
 			return false;
 	}
 
-- 
2.43.0

[PATCH 2/2] sched/topology: optimize topology_span_sane()

[Yury Norov]

The function may call cpumask_equal with tl->mask(cpu) == tl->mask(i),
even though cpu != i. In such case, cpumask_equal() would always return
true, and we can proceed to the next iteration immediately.

Comment is provided by Valentin Schneider.

Reviewed-by: Valentin Schneider <vschneid@redhat.com>
Signed-off-by: Yury Norov <yury.norov@gmail.com>
---
 kernel/sched/topology.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 8af3b48da458..3661d4173d1f 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -2370,6 +2370,18 @@ static bool topology_span_sane(struct sched_domain_topology_level *tl,
 	 */
 	for_each_cpu_from(i, cpu_map) {
 		mi = tl->mask(i);
+		/*
+		 * Some topology levels (e.g. PKG in default_topology[])
+		 * have a sched_domain_mask_f implementation that reuses
+		 * the same mask for several CPUs (in PKG's case, one mask
+		 * for all CPUs in the same NUMA node).
+		 *
+		 * For such topology levels, repeating cpumask_equal()
+		 * checks is wasteful. Instead, we first check that the
+		 * tl->mask(i) pointers aren't the same.
+		 */
+		if (mi == mc)
+			continue;
 
 		/*
 		 * We should 'and' all those masks with 'cpu_map' to exactly
-- 
2.43.0

Re: [PATCH v2 0/2] sched/topology: optimize topology_span_sane()

[Yury Norov]

Ping?

On Wed, Aug 07, 2024 at 12:05:19PM -0700, Yury Norov wrote:
> The function may call cpumask_equal with tl->mask(cpu) == tl->mask(i),
> even when cpu != i. In such case, cpumask_equal() would always return
> true, and we can proceed to the next iteration immediately.
> 
> Valentin Schneider shares on it:
> 
>   PKG can potentially hit that condition, and so can any
>   sched_domain_mask_f that relies on the node masks...
>   
>   I'm thinking ideally we should have checks in place to
>   ensure all node_to_cpumask_map[] masks are disjoint,
>   then we could entirely skip the levels that use these
>   masks in topology_span_sane(), but there's unfortunately
>   no nice way to flag them... Also there would be cases
>   where there's no real difference between PKG and NODE
>   other than NODE is still based on a per-cpu cpumask and
>   PKG isn't, so I don't see a nicer way to go about this.
> 
> v1: https://lore.kernel.org/lkml/ZrJk00cmVaUIAr4G@yury-ThinkPad/T/
> v2:
>  - defer initialization of 'mc' in patch #1 @Chen Yu;
>  - more comments from Valentin Schneider.
> 
> 
> Yury Norov (2):
>   sched/topology: pre-compute topology_span_sane() loop params
>   sched/topology: optimize topology_span_sane()
> 
>  kernel/sched/topology.c | 20 ++++++++++++++++++--
>  1 file changed, 18 insertions(+), 2 deletions(-)
> 
> -- 
> 2.43.0

Re: [PATCH v2 0/2] sched/topology: optimize topology_span_sane()

[Yury Norov]

Ping again?

On Wed, Aug 14, 2024 at 09:35:04AM -0700, Yury Norov wrote:
> Ping?
> 
> On Wed, Aug 07, 2024 at 12:05:19PM -0700, Yury Norov wrote:
> > The function may call cpumask_equal with tl->mask(cpu) == tl->mask(i),
> > even when cpu != i. In such case, cpumask_equal() would always return
> > true, and we can proceed to the next iteration immediately.
> > 
> > Valentin Schneider shares on it:
> > 
> >   PKG can potentially hit that condition, and so can any
> >   sched_domain_mask_f that relies on the node masks...
> >   
> >   I'm thinking ideally we should have checks in place to
> >   ensure all node_to_cpumask_map[] masks are disjoint,
> >   then we could entirely skip the levels that use these
> >   masks in topology_span_sane(), but there's unfortunately
> >   no nice way to flag them... Also there would be cases
> >   where there's no real difference between PKG and NODE
> >   other than NODE is still based on a per-cpu cpumask and
> >   PKG isn't, so I don't see a nicer way to go about this.
> > 
> > v1: https://lore.kernel.org/lkml/ZrJk00cmVaUIAr4G@yury-ThinkPad/T/
> > v2:
> >  - defer initialization of 'mc' in patch #1 @Chen Yu;
> >  - more comments from Valentin Schneider.
> > 
> > 
> > Yury Norov (2):
> >   sched/topology: pre-compute topology_span_sane() loop params
> >   sched/topology: optimize topology_span_sane()
> > 
> >  kernel/sched/topology.c | 20 ++++++++++++++++++--
> >  1 file changed, 18 insertions(+), 2 deletions(-)
> > 
> > -- 
> > 2.43.0

Re: [PATCH v2 0/2] sched/topology: optimize topology_span_sane()

[Christophe JAILLET]

Le 28/08/2024 à 16:09, Yury Norov a écrit :
> Ping again?
> 

Hi,

The covert letter is v2, but the patch below it are without v2.

In patch 1/2, the Suggested-by: tags in v1 have been removed in this new 
version :(.

CJ

> On Wed, Aug 14, 2024 at 09:35:04AM -0700, Yury Norov wrote:
>> Ping?
>>
>> On Wed, Aug 07, 2024 at 12:05:19PM -0700, Yury Norov wrote:
>>> The function may call cpumask_equal with tl->mask(cpu) == tl->mask(i),
>>> even when cpu != i. In such case, cpumask_equal() would always return
>>> true, and we can proceed to the next iteration immediately.
>>>
>>> Valentin Schneider shares on it:
>>>
>>>    PKG can potentially hit that condition, and so can any
>>>    sched_domain_mask_f that relies on the node masks...
>>>    
>>>    I'm thinking ideally we should have checks in place to
>>>    ensure all node_to_cpumask_map[] masks are disjoint,
>>>    then we could entirely skip the levels that use these
>>>    masks in topology_span_sane(), but there's unfortunately
>>>    no nice way to flag them... Also there would be cases
>>>    where there's no real difference between PKG and NODE
>>>    other than NODE is still based on a per-cpu cpumask and
>>>    PKG isn't, so I don't see a nicer way to go about this.
>>>
>>> v1: https://lore.kernel.org/lkml/ZrJk00cmVaUIAr4G@yury-ThinkPad/T/
>>> v2:
>>>   - defer initialization of 'mc' in patch #1 @Chen Yu;
>>>   - more comments from Valentin Schneider.
>>>
>>>
>>> Yury Norov (2):
>>>    sched/topology: pre-compute topology_span_sane() loop params
>>>    sched/topology: optimize topology_span_sane()
>>>
>>>   kernel/sched/topology.c | 20 ++++++++++++++++++--
>>>   1 file changed, 18 insertions(+), 2 deletions(-)
>>>
>>> -- 
>>> 2.43.0
> 
> 

Re: [PATCH v2 0/2] sched/topology: optimize topology_span_sane()

[Yury Norov]

On Thu, Aug 29, 2024 at 08:11:41AM +0200, Christophe JAILLET wrote:
> Le 28/08/2024 à 16:09, Yury Norov a écrit :
> > Ping again?
> > 
> 
> Hi,
> 
> The covert letter is v2, but the patch below it are without v2.
> 
> In patch 1/2, the Suggested-by: tags in v1 have been removed in this new
> version :(.

My bad. I'll send v3.