[PATCH v3 0/3] sched/topology: optimize topology_span_sane()

[PATCH v3 0/3] sched/topology: optimize topology_span_sane()

[Yury Norov]

The function may call cpumask_equal with tl->mask(cpu) == tl->mask(i),
even when cpu != i. In such case, cpumask_equal() would always return
true, and we can proceed to the next iteration immediately.

Valentin Schneider shares on it:

  PKG can potentially hit that condition, and so can any
  sched_domain_mask_f that relies on the node masks...

  I'm thinking ideally we should have checks in place to
  ensure all node_to_cpumask_map[] masks are disjoint,
  then we could entirely skip the levels that use these
  masks in topology_span_sane(), but there's unfortunately
  no nice way to flag them... Also there would be cases
  where there's no real difference between PKG and NODE
  other than NODE is still based on a per-cpu cpumask and
  PKG isn't, so I don't see a nicer way to go about this.

v1: https://lore.kernel.org/lkml/ZrJk00cmVaUIAr4G@yury-ThinkPad/T/
v2: https://lkml.org/lkml/2024/8/7/1299
v3:
 - add topology_cpumask_equal() helper in #3;
 - re-use 'cpu' as an iterator int the for_each_cpu() loop;
 - add proper versioning for all patches.

Yury Norov (3):
  sched/topology: pre-compute topology_span_sane() loop params
  sched/topology: optimize topology_span_sane()
  sched/topology: reorganize topology_span_sane() checking order

 kernel/sched/topology.c | 29 +++++++++++++++++++++++++----
 1 file changed, 25 insertions(+), 4 deletions(-)

-- 
2.43.0

[PATCH v3 1/3] sched/topology: pre-compute topology_span_sane() loop params

[Yury Norov]

tl->mask() is called inside the loop with the same parameters more than
once. We can pre-calculate it. After that, 'cpu' doesn't have to keep its
value while running the loop iterations. We can drop the 'i' iterator,
and re-use 'cpu' in the for_each_cpu_from() loop.

Reviewed-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Yury Norov <yury.norov@gmail.com>
---
 kernel/sched/topology.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 76504b776d03..ffbe3a28d2d4 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -2353,27 +2353,30 @@ static struct sched_domain *build_sched_domain(struct sched_domain_topology_leve
 static bool topology_span_sane(struct sched_domain_topology_level *tl,
 			      const struct cpumask *cpu_map, int cpu)
 {
-	int i = cpu + 1;
+	const struct cpumask *mi, *mc;
 
 	/* NUMA levels are allowed to overlap */
 	if (tl->flags & SDTL_OVERLAP)
 		return true;
 
+	mc = tl->mask(cpu++);
+
 	/*
 	 * Non-NUMA levels cannot partially overlap - they must be either
 	 * completely equal or completely disjoint. Otherwise we can end up
 	 * breaking the sched_group lists - i.e. a later get_group() pass
 	 * breaks the linking done for an earlier span.
 	 */
-	for_each_cpu_from(i, cpu_map) {
+	for_each_cpu_from(cpu, cpu_map) {
+		mi = tl->mask(cpu);
+
 		/*
 		 * We should 'and' all those masks with 'cpu_map' to exactly
 		 * match the topology we're about to build, but that can only
 		 * remove CPUs, which only lessens our ability to detect
 		 * overlaps
 		 */
-		if (!cpumask_equal(tl->mask(cpu), tl->mask(i)) &&
-		    cpumask_intersects(tl->mask(cpu), tl->mask(i)))
+		if (!cpumask_equal(mc, mi) && cpumask_intersects(mc, mi))
 			return false;
 	}
 
-- 
2.43.0

[PATCH v3 2/3] sched/topology: optimize topology_span_sane()

[Yury Norov]

The function may call cpumask_equal with mi == mc, even though the CPUs are
different. In such case, cpumask_equal() would always return true, and we
can proceed to the next iteration immediately.

This happens when topologies re-use the same mask for many CPUs.
The detailed comment is provided by Valentin Schneider.

Reviewed-by: Valentin Schneider <vschneid@redhat.com>
Signed-off-by: Yury Norov <yury.norov@gmail.com>
---
 kernel/sched/topology.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index ffbe3a28d2d4..04a3b3d7b6f4 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -2370,6 +2370,19 @@ static bool topology_span_sane(struct sched_domain_topology_level *tl,
 	for_each_cpu_from(cpu, cpu_map) {
 		mi = tl->mask(cpu);
 
+		/*
+		 * Some topology levels (e.g. PKG in default_topology[])
+		 * have a sched_domain_mask_f implementation that reuses
+		 * the same mask for several CPUs (in PKG's case, one mask
+		 * for all CPUs in the same NUMA node).
+		 *
+		 * For such topology levels, repeating cpumask_equal()
+		 * checks is wasteful. Instead, we first check that the
+		 * tl->mask(i) pointers aren't the same.
+		 */
+		if (mi == mc)
+			continue;
+
 		/*
 		 * We should 'and' all those masks with 'cpu_map' to exactly
 		 * match the topology we're about to build, but that can only
-- 
2.43.0

[PATCH v3 3/3] sched/topology: reorganize topology_span_sane() checking order

[Yury Norov]

The function currently makes 3 checks:

1. mc == mi;
2. cpumask_equal(mc, mi);
3. cpumask_intersects(mc, mi).

Historically, 2 last checks build a single condition for if() statement.

Logically, #1 and #2 should be tested together, because for the topology
sanity checking purposes, they do the same thing. In contrast, #3 tests
for intersection, which is a different logical unit.

This patch creates a helper for #1 and #2 and puts the corresponding
comment on top of the helper; unloading the main topology_span_sane().

Signed-off-by: Yury Norov <yury.norov@gmail.com>
---
 kernel/sched/topology.c | 31 ++++++++++++++++++-------------
 1 file changed, 18 insertions(+), 13 deletions(-)

diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
index 04a3b3d7b6f4..bbbe7955d37c 100644
--- a/kernel/sched/topology.c
+++ b/kernel/sched/topology.c
@@ -2346,6 +2346,22 @@ static struct sched_domain *build_sched_domain(struct sched_domain_topology_leve
 	return sd;
 }
 
+/*
+ * Some topology levels (e.g. PKG in default_topology[]) have a
+ * sched_domain_mask_f implementation that reuses the same mask for
+ * several CPUs (in PKG's case, one mask * for all CPUs in the same
+ * NUMA node).
+ *
+ * For such topology levels, repeating cpumask_equal() checks is
+ * wasteful. Instead, we first check that the tl->mask(i) pointers
+ * aren't the same.
+ */
+static inline bool topology_cpumask_equal(const struct cpumask *m1,
+					  const struct cpumask *m2)
+{
+	return m1 == m2 || cpumask_equal(m1, m2);
+}
+
 /*
  * Ensure topology masks are sane, i.e. there are no conflicts (overlaps) for
  * any two given CPUs at this (non-NUMA) topology level.
@@ -2369,18 +2385,7 @@ static bool topology_span_sane(struct sched_domain_topology_level *tl,
 	 */
 	for_each_cpu_from(cpu, cpu_map) {
 		mi = tl->mask(cpu);
-
-		/*
-		 * Some topology levels (e.g. PKG in default_topology[])
-		 * have a sched_domain_mask_f implementation that reuses
-		 * the same mask for several CPUs (in PKG's case, one mask
-		 * for all CPUs in the same NUMA node).
-		 *
-		 * For such topology levels, repeating cpumask_equal()
-		 * checks is wasteful. Instead, we first check that the
-		 * tl->mask(i) pointers aren't the same.
-		 */
-		if (mi == mc)
+		if (topology_cpumask_equal(mc, mi))
 			continue;
 
 		/*
@@ -2389,7 +2394,7 @@ static bool topology_span_sane(struct sched_domain_topology_level *tl,
 		 * remove CPUs, which only lessens our ability to detect
 		 * overlaps
 		 */
-		if (!cpumask_equal(mc, mi) && cpumask_intersects(mc, mi))
+		if (cpumask_intersects(mc, mi))
 			return false;
 	}
 
-- 
2.43.0

Re: [PATCH v3 0/3] sched/topology: optimize topology_span_sane()

[Yury Norov]

Ping?

On Mon, Sep 02, 2024 at 11:36:04AM -0700, Yury Norov wrote:
> The function may call cpumask_equal with tl->mask(cpu) == tl->mask(i),
> even when cpu != i. In such case, cpumask_equal() would always return
> true, and we can proceed to the next iteration immediately.
> 
> Valentin Schneider shares on it:
> 
>   PKG can potentially hit that condition, and so can any
>   sched_domain_mask_f that relies on the node masks...
> 
>   I'm thinking ideally we should have checks in place to
>   ensure all node_to_cpumask_map[] masks are disjoint,
>   then we could entirely skip the levels that use these
>   masks in topology_span_sane(), but there's unfortunately
>   no nice way to flag them... Also there would be cases
>   where there's no real difference between PKG and NODE
>   other than NODE is still based on a per-cpu cpumask and
>   PKG isn't, so I don't see a nicer way to go about this.
> 
> v1: https://lore.kernel.org/lkml/ZrJk00cmVaUIAr4G@yury-ThinkPad/T/
> v2: https://lkml.org/lkml/2024/8/7/1299
> v3:
>  - add topology_cpumask_equal() helper in #3;
>  - re-use 'cpu' as an iterator int the for_each_cpu() loop;
>  - add proper versioning for all patches.
> 
> Yury Norov (3):
>   sched/topology: pre-compute topology_span_sane() loop params
>   sched/topology: optimize topology_span_sane()
>   sched/topology: reorganize topology_span_sane() checking order
> 
>  kernel/sched/topology.c | 29 +++++++++++++++++++++++++----
>  1 file changed, 25 insertions(+), 4 deletions(-)
> 
> -- 
> 2.43.0

Re: [PATCH v3 0/3] sched/topology: optimize topology_span_sane()

[Yury Norov]

Ping again?

On Sat, Sep 14, 2024 at 09:54:43AM -0700, Yury Norov wrote:
> Ping?
> 
> On Mon, Sep 02, 2024 at 11:36:04AM -0700, Yury Norov wrote:
> > The function may call cpumask_equal with tl->mask(cpu) == tl->mask(i),
> > even when cpu != i. In such case, cpumask_equal() would always return
> > true, and we can proceed to the next iteration immediately.
> > 
> > Valentin Schneider shares on it:
> > 
> >   PKG can potentially hit that condition, and so can any
> >   sched_domain_mask_f that relies on the node masks...
> > 
> >   I'm thinking ideally we should have checks in place to
> >   ensure all node_to_cpumask_map[] masks are disjoint,
> >   then we could entirely skip the levels that use these
> >   masks in topology_span_sane(), but there's unfortunately
> >   no nice way to flag them... Also there would be cases
> >   where there's no real difference between PKG and NODE
> >   other than NODE is still based on a per-cpu cpumask and
> >   PKG isn't, so I don't see a nicer way to go about this.
> > 
> > v1: https://lore.kernel.org/lkml/ZrJk00cmVaUIAr4G@yury-ThinkPad/T/
> > v2: https://lkml.org/lkml/2024/8/7/1299
> > v3:
> >  - add topology_cpumask_equal() helper in #3;
> >  - re-use 'cpu' as an iterator int the for_each_cpu() loop;
> >  - add proper versioning for all patches.
> > 
> > Yury Norov (3):
> >   sched/topology: pre-compute topology_span_sane() loop params
> >   sched/topology: optimize topology_span_sane()
> >   sched/topology: reorganize topology_span_sane() checking order
> > 
> >  kernel/sched/topology.c | 29 +++++++++++++++++++++++++----
> >  1 file changed, 25 insertions(+), 4 deletions(-)
> > 
> > -- 
> > 2.43.0

Re: [PATCH v3 0/3] sched/topology: optimize topology_span_sane()

[Yury Norov]

Last reminder. If you guys don't care, I don't care either.

On Mon, Sep 30, 2024 at 12:15:02PM -0700, Yury Norov wrote:
> Ping again?
> 
> On Sat, Sep 14, 2024 at 09:54:43AM -0700, Yury Norov wrote:
> > Ping?
> > 
> > On Mon, Sep 02, 2024 at 11:36:04AM -0700, Yury Norov wrote:
> > > The function may call cpumask_equal with tl->mask(cpu) == tl->mask(i),
> > > even when cpu != i. In such case, cpumask_equal() would always return
> > > true, and we can proceed to the next iteration immediately.
> > > 
> > > Valentin Schneider shares on it:
> > > 
> > >   PKG can potentially hit that condition, and so can any
> > >   sched_domain_mask_f that relies on the node masks...
> > > 
> > >   I'm thinking ideally we should have checks in place to
> > >   ensure all node_to_cpumask_map[] masks are disjoint,
> > >   then we could entirely skip the levels that use these
> > >   masks in topology_span_sane(), but there's unfortunately
> > >   no nice way to flag them... Also there would be cases
> > >   where there's no real difference between PKG and NODE
> > >   other than NODE is still based on a per-cpu cpumask and
> > >   PKG isn't, so I don't see a nicer way to go about this.
> > > 
> > > v1: https://lore.kernel.org/lkml/ZrJk00cmVaUIAr4G@yury-ThinkPad/T/
> > > v2: https://lkml.org/lkml/2024/8/7/1299
> > > v3:
> > >  - add topology_cpumask_equal() helper in #3;
> > >  - re-use 'cpu' as an iterator int the for_each_cpu() loop;
> > >  - add proper versioning for all patches.
> > > 
> > > Yury Norov (3):
> > >   sched/topology: pre-compute topology_span_sane() loop params
> > >   sched/topology: optimize topology_span_sane()
> > >   sched/topology: reorganize topology_span_sane() checking order
> > > 
> > >  kernel/sched/topology.c | 29 +++++++++++++++++++++++++----
> > >  1 file changed, 25 insertions(+), 4 deletions(-)
> > > 
> > > -- 
> > > 2.43.0

Re: [PATCH v3 0/3] sched/topology: optimize topology_span_sane()

[Christophe JAILLET]

Le 06/11/2024 à 17:28, Yury Norov a écrit :
> Last reminder. If you guys don't care, I don't care either.

Hi Yury,

I'm the only one in the To: field, but I'm not a maintainer.

Maybe, try to move people in the Cc: field into the To: field, to 
increase visibility?

CJ

> 
> On Mon, Sep 30, 2024 at 12:15:02PM -0700, Yury Norov wrote:
>> Ping again?
>>
>> On Sat, Sep 14, 2024 at 09:54:43AM -0700, Yury Norov wrote:
>>> Ping?
>>>
>>> On Mon, Sep 02, 2024 at 11:36:04AM -0700, Yury Norov wrote:
>>>> The function may call cpumask_equal with tl->mask(cpu) == tl->mask(i),
>>>> even when cpu != i. In such case, cpumask_equal() would always return
>>>> true, and we can proceed to the next iteration immediately.
>>>>
>>>> Valentin Schneider shares on it:
>>>>
>>>>    PKG can potentially hit that condition, and so can any
>>>>    sched_domain_mask_f that relies on the node masks...
>>>>
>>>>    I'm thinking ideally we should have checks in place to
>>>>    ensure all node_to_cpumask_map[] masks are disjoint,
>>>>    then we could entirely skip the levels that use these
>>>>    masks in topology_span_sane(), but there's unfortunately
>>>>    no nice way to flag them... Also there would be cases
>>>>    where there's no real difference between PKG and NODE
>>>>    other than NODE is still based on a per-cpu cpumask and
>>>>    PKG isn't, so I don't see a nicer way to go about this.
>>>>
>>>> v1: https://lore.kernel.org/lkml/ZrJk00cmVaUIAr4G@yury-ThinkPad/T/
>>>> v2: https://lkml.org/lkml/2024/8/7/1299
>>>> v3:
>>>>   - add topology_cpumask_equal() helper in #3;
>>>>   - re-use 'cpu' as an iterator int the for_each_cpu() loop;
>>>>   - add proper versioning for all patches.
>>>>
>>>> Yury Norov (3):
>>>>    sched/topology: pre-compute topology_span_sane() loop params
>>>>    sched/topology: optimize topology_span_sane()
>>>>    sched/topology: reorganize topology_span_sane() checking order
>>>>
>>>>   kernel/sched/topology.c | 29 +++++++++++++++++++++++++----
>>>>   1 file changed, 25 insertions(+), 4 deletions(-)
>>>>
>>>> -- 
>>>> 2.43.0
> 
> 

Re: [PATCH v3 0/3] sched/topology: optimize topology_span_sane()

[Yury Norov]

On Wed, Nov 06, 2024 at 06:58:23PM +0100, Christophe JAILLET wrote:
> Le 06/11/2024 à 17:28, Yury Norov a écrit :
> > Last reminder. If you guys don't care, I don't care either.
> 
> Hi Yury,
> 
> I'm the only one in the To: field, but I'm not a maintainer.
> 
> Maybe, try to move people in the Cc: field into the To: field, to increase
> visibility?

Sure, moving everyone into the To.

Re: [PATCH v3 0/3] sched/topology: optimize topology_span_sane()

[Peter Zijlstra]

On Mon, Sep 02, 2024 at 11:36:04AM -0700, Yury Norov wrote:
> The function may call cpumask_equal with tl->mask(cpu) == tl->mask(i),
> even when cpu != i. In such case, cpumask_equal() would always return
> true, and we can proceed to the next iteration immediately.
> 
> Valentin Schneider shares on it:
> 
>   PKG can potentially hit that condition, and so can any
>   sched_domain_mask_f that relies on the node masks...
> 
>   I'm thinking ideally we should have checks in place to
>   ensure all node_to_cpumask_map[] masks are disjoint,
>   then we could entirely skip the levels that use these
>   masks in topology_span_sane(), but there's unfortunately
>   no nice way to flag them... Also there would be cases
>   where there's no real difference between PKG and NODE
>   other than NODE is still based on a per-cpu cpumask and
>   PKG isn't, so I don't see a nicer way to go about this.
> 
> v1: https://lore.kernel.org/lkml/ZrJk00cmVaUIAr4G@yury-ThinkPad/T/
> v2: https://lkml.org/lkml/2024/8/7/1299
> v3:
>  - add topology_cpumask_equal() helper in #3;
>  - re-use 'cpu' as an iterator int the for_each_cpu() loop;
>  - add proper versioning for all patches.
> 
> Yury Norov (3):
>   sched/topology: pre-compute topology_span_sane() loop params
>   sched/topology: optimize topology_span_sane()
>   sched/topology: reorganize topology_span_sane() checking order

Why are we doing this? Subject says optimize, but I see no performance
numbers?

Re: [PATCH v3 0/3] sched/topology: optimize topology_span_sane()

[Yury Norov]

On Wed, Nov 06, 2024 at 07:06:13PM +0100, Peter Zijlstra wrote:
> On Mon, Sep 02, 2024 at 11:36:04AM -0700, Yury Norov wrote:
> > The function may call cpumask_equal with tl->mask(cpu) == tl->mask(i),
> > even when cpu != i. In such case, cpumask_equal() would always return
> > true, and we can proceed to the next iteration immediately.
> > 
> > Valentin Schneider shares on it:
> > 
> >   PKG can potentially hit that condition, and so can any
> >   sched_domain_mask_f that relies on the node masks...
> > 
> >   I'm thinking ideally we should have checks in place to
> >   ensure all node_to_cpumask_map[] masks are disjoint,
> >   then we could entirely skip the levels that use these
> >   masks in topology_span_sane(), but there's unfortunately
> >   no nice way to flag them... Also there would be cases
> >   where there's no real difference between PKG and NODE
> >   other than NODE is still based on a per-cpu cpumask and
> >   PKG isn't, so I don't see a nicer way to go about this.
> > 
> > v1: https://lore.kernel.org/lkml/ZrJk00cmVaUIAr4G@yury-ThinkPad/T/
> > v2: https://lkml.org/lkml/2024/8/7/1299
> > v3:
> >  - add topology_cpumask_equal() helper in #3;
> >  - re-use 'cpu' as an iterator int the for_each_cpu() loop;
> >  - add proper versioning for all patches.
> > 
> > Yury Norov (3):
> >   sched/topology: pre-compute topology_span_sane() loop params
> >   sched/topology: optimize topology_span_sane()
> >   sched/topology: reorganize topology_span_sane() checking order
> 
> Why are we doing this? Subject says optimize, but I see no performance
> numbers?

Well, if we have a chance that cpumask_equal() is passed with
mask1 == mask2, the new vs old approach is O(1) vs O(N), and this
is the case of PKG. This should have performance impact for many-CPUs
setups (much more than BITS_PER_LONG).

I have no such machine, and can't give you perf numbers. I can drop
'optimization' wording, if you find it confusing without numbers.

Thanks,
Yury