Re: [PATCH 1/2] sched_ext: Cache per-node NUMA distance order in scx_idle_init_masks()

[Andrea Righi <arighi@nvidia.com>]

On Tue, Mar 17, 2026 at 07:32:54AM -1000, Tejun Heo wrote:
> On Tue, Mar 17, 2026 at 11:04:11PM +0800, Cheng-Yang Chou wrote:
> > Hi Andrea,
> > 
> > On Mon, Mar 16, 2026 at 04:51:34PM +0100, Andrea Righi wrote:
> > > Hi Cheng-Yang,
> > > 
> > > this has been sitting in my TODO list for a while, so thanks for looking
> > > at it. :)
> > > 
> > > Comments below.
> > > 
> > > On Mon, Mar 16, 2026 at 02:10:14AM +0800, Cheng-Yang Chou wrote:
> > > > Add scx_numa_node_order[] and scx_numa_node_order_cnt[], per-node
> > > > arrays that store NUMA nodes sorted by increasing distance from each
> > > > node. They are allocated and populated once during boot in
> > > > scx_idle_init_masks() using the existing for_each_node_numadist()
> > > > O(N^2) traversal, so the cost is paid only at init time.
> 
> Wasn't the conclusion that given the low numa node count, O(N^2) doesn't
> matter for now here although I can see node count becoming high enough with
> AMD fake LLC NUMA enabled on an actual large NUMA machines. But if we decide
> that's an actual problem, shouldn't it be addressed in the topology code
> rather than from sched_ext side?

Yes, that was the conclusion from the previous discussion on this topic.
Honestly I've never seen NUMA system where the O(N^2) could be an issue.
One thing that I like about this change is the removal of
preempt_disable/enable() when we iterate nodes.

However, speaking of performance, the O(N^2) bitmask iteration on a single
nodemask_t is probably more efficient than the O(N) iteration through the
scx_numa_node_order[] array (it should be more cache friendly). Ideally we
should run some tests on those big Intel NUMA machines that were showing
the bad regressions with the single global idle cpumasks, but I don't think
I have access to them anymore...

So, thinking more about this, I'm still conflicted if we should apply this
change or not, especially because we don't have any performance number...

-Andrea