Re: [PATCH] sched/fair: Stabilize idle SMT core selection with asym-capacity

[Andrea Righi <arighi@nvidia.com>]

Hi Julia,

On Fri, Jul 03, 2026 at 07:20:38AM -0400, Julia Lawall wrote:
> On Fri, 3 Jul 2026, Andrea Righi wrote:
> 
> > Hi Prateek,
> >
> > On Fri, Jul 03, 2026 at 11:21:57AM +0530, K Prateek Nayak wrote:
> > > Hello Andrea,
> > >
> > > On 6/30/2026 8:57 PM, Andrea Righi wrote:
> > > > select_idle_capacity() scans all logical CPUs also when it is looking
> > > > for a fully idle SMT core. Two concurrent wakeups can therefore observe
> > > > the same core as idle, encounter different siblings first, and place one
> > > > task on each sibling while another core remains unused.
> > > >
> > > > Make every logical CPU of a selected idle core resolve to the same
> > > > stable CPU representative within the scan's existing affinity and
> > > > scheduling-domain mask. If the first task is enqueued before the next
> > > > scan examines the core, that scan rejects the now-busy core. If both
> > > > scans observe the core as idle, they select the same runqueue even if
> > > > the first enqueue becomes visible before the second scan finishes,
> > > > exposing the imbalance to the load balancer.
> > > >
> > > > The symmetric CPU idle selection path is subject to the same race, but
> > > > normally returns as soon as select_idle_core() finds a fully idle core,
> > > > reducing the conflict window. The per-CPU capacity scan can retain an
> > > > idle-core candidate while evaluating other CPUs, giving concurrent
> > > > wakeups more opportunity to select different siblings of the same SMT
> > > > core. Therefore, limit the normalization to the asym-capacity path,
> > > > where this behavior has a measurable impact.
> > > >
> > > > On NVIDIA Vera Rubin (arm64, 176 CPUs/88 cores per NUMA node), a
> > > > CPU-intensive NVPL SGEMM workload restricted to 88 threads (one per
> > > > core) showed a consistent 23% increase in mean throughput across
> > > > multiple runs.
> > >
> > > Interesting! This reads like active balance across cores is not aggressive
> > > enough for this workload and, as a result, stacking somehow helps.
> > >
> > > I would have expected balance within the core would trigger first and that
> > > would just lead to the same scenario as both sibling sibling busy but I
> > > guess there is a higher order effect of stacking.
> >
> > I think the key here is that temporary runqueue stacking is preferable to
> > consuming both SMT siblings when fully-idle SMT cores are available, more than
> > having benfits from the stacking itself.
> 
> Andrea, did you try changing the clock speed?  With ticks every 4ms and an
> EEVDF time slice that rounds up to 4ms, task_hot makes it almost
> impossible for already-idle CPUs to pull tasks.
> 
> julia

Oh I remember you mentioned this. However, the kernel that I'm using has
CONFIG_HZ_1000=y, so the scheduler tick is 1 ms rather than 4 ms. I tried to
play a bit with different migration_cost_ns settings, but didn't get much
benefit from that.

I think I have a lead, and the observed improvement with this patch may not
be a scheduler/load-balancing effect. In practice, I see different performance
on sibling 0 vs sibling 1, apparently sibling 0 is faster, despite the firmware
advertising identical capacity. So I think my patch is helping mostly due the
fact that I'm using cpumask_first_and(), more than the aggressive SMT avoidance.

I'm trying to get more details from the hw/firmware. Will keep you updated.

Thanks,
-Andrea