Re: [PATCH 2/3] sched_ext: Introduce per-NUMA idle cpumasks

[Andrea Righi <arighi@nvidia.com>]

On Tue, Dec 03, 2024 at 04:38:58PM -0800, Yury Norov wrote:
> On Tue, Dec 03, 2024 at 02:04:15PM -1000, Tejun Heo wrote:
...
> > > +static s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags)
> > > +{
> > > +   int start = cpu_to_node(smp_processor_id());
> > > +   int node, cpu;
> > > +
> > > +   for_each_node_state_wrap(node, N_ONLINE, start) {
> > > +           /*
> > > +            * scx_pick_idle_cpu_from_node() can be expensive and redundant
> > > +            * if none of the CPUs in the NUMA node can be used (according
> > > +            * to cpus_allowed).
> > > +            *
> > > +            * Therefore, check if the NUMA node is usable in advance to
> > > +            * save some CPU cycles.
> > > +            */
> > > +           if (!cpumask_intersects(cpumask_of_node(node), cpus_allowed))
> > > +                   continue;
> > > +           cpu = scx_pick_idle_cpu_from_node(node, cpus_allowed, flags);
> > > +           if (cpu >= 0)
> > > +                   return cpu;
> >
> > This is fine for now but it'd be ideal if the iteration is in inter-node
> > distance order so that each CPU radiates from local node to the furthest
> > ones.
> 
> cpumask_local_spread() does exactly that - traverses CPUs in NUMA-aware
> order. Or we can use for_each_numa_hop_mask() iterator, which does the
> same thing more efficiently.

Nice, for_each_numa_hop_mask() seems to be exactly what I need, there's
also a starting node, so with that we don't need to introduce
for_each_online_node_wrap() and the other new *_wrap() helpers.

Thanks,
-Andrea