Re: [PATCH sched_ext/for-6.13] sched_ext: Do not enable LLC/NUMA optimizations when domains overlap

[Andrea Righi <arighi@nvidia.com>]

On Tue, Nov 05, 2024 at 01:32:05PM -1000, Tejun Heo wrote:
> Hello, Andrea.
> 
> Sorry about the delay.
> 
> On Thu, Oct 31, 2024 at 08:34:01AM +0100, Andrea Righi wrote:
> > @@ -3154,16 +3154,29 @@ static void update_selcpu_topology(void)
> >       rcu_read_lock();
> >       sd = rcu_dereference(per_cpu(sd_llc, cpu));
> >       if (sd) {
> > -             cpus = sched_domain_span(sd);
> > -             if (cpumask_weight(cpus) < num_possible_cpus())
> > +             llc_cpus = sched_domain_span(sd);
> > +             if (cpumask_weight(llc_cpus) < num_possible_cpus())
> 
> Not from this patch but should the weight be compared against
> num_online_cpus()? Sched domains don't include offline CPUs, right?

That's right, sched domain is definitely updated when a CPU goes offine
(I've just verified), so we should check for num_online_cpus() here.
Thanks for noticing it!

> 
> ...
> > +     /*
> > +      * If the NUMA domain perfectly overlaps with the LLC domain, enable
> > +      * LLC optimization only, as checking for an idle CPU in the same
> > +      * domain twice is redundant.
> > +      */
> > +     if (enable_numa && enable_llc && cpumask_equal(numa_cpus, llc_cpus))
> > +             enable_numa = false;
> > +     /*
> > +      * If all the online CPUs are in the same LLC domain, there is no
> > +      * reason to enable LLC optimization.
> > +      */
> > +     if (enable_llc && cpumask_equal(llc_cpus, cpu_online_mask))
> > +             enable_llc = false;
> 
> The second test looks like it should always be correct. I'm not sure the
> first one is. It probably is good enough but would at least be worthwhile to
> explain why that is?

Let's say we have 2 NUMA nodes, each with 2 sockets, and each socket
has its own L3 cache. In this case, numa_cpus will be larger than
llc_cpus, and enabling both NUMA and LLC optimizations would be
beneficial.

On the other hand, if each NUMA node contains only 1 socket, numa_cpus
and llc_cpus will overlap completely, making it unnecessary to enable
both NUMA and LLC optimizations, so we can have just the LLC in this
case.

Would something like this help clarifying the first test?

Thanks,
-Andrea