Re: [PATCH 1/2] arm64: topology: Prefer PE0 on NVIDIA Olympus SMT cores

[Andrea Righi <arighi@nvidia.com>]

Hi Christian,

On Tue, Sep 01, 2026 at 09:32:56AM +0100, Christian Loehle wrote:
> On 8/31/26 22:43, Andrea Righi wrote:
...
> > I can experiment with this combined priority, but I think removing
> > SD_ASYM_CPUCAPACITY is a separate policy change rather than an alternative
> > implementation of this fix.
> 
> Cool thanks, and sorry for curveballing the approach like this, I wish I had
> the platform to test these ideas myself :/ 

No problem, thanks for looking into this! Access to Vera systems is problematic
also on my side, especially for all the time it takes to run all these tests. :)

> 
> > 
> > A static asym-packing priority does not preserve the capacity-aware semantics
> > used for task fitting, uclamp, misfit handling and migration. The current
> > approach keeps those semantics when selecting a physical core, then applies the
> > PE preference only within that core.
> 
> Right, but arguably most of these semantics become questionable as soon as the
> core enters two-thread mode, since the capacity available to each PE then
> depends on the state of its sibling.

Yes, I agree with that the current capacity model doesn't account the capacity
lost when an SMT sibling becomes active. And Olympus makes that limitation
particularly visible.

That said, I think the static CPU capacity is still meaningful when there's no
SMT contention. SD_ASYM_CPUCAPACITY can compare the task's demand against the
standalone capacity of the candidate physical cores and select an appropriate
one. And looking at the results, this appears to be beneficial. Then the
SMT-local asym-packing priority can select the preferred PE within that core,
which helps keep its sibling idle and preserve the uncontended state whenever
possible.

Once every usable physical core already has an active PE, any additional work
invitably introduces SMT contention. At that point, the effective capacity
becomes sibling-state-dependent and neither SD_ASYM_CPUCAPACITY nor a combined
static asym-packing priority can accurately model it.

> 
> Task fitting:
> We consider two tasks with util=400 to fit on two capacity=1000 PEs, even
> though once both PEs are active neither may have anything close to capacity
> 1000 available. In other words, the capacity used for fitting doesn't account
> for the capacity "stolen" by activating the sibling.

Correct, util_fits_cpu() doesn't reduce capacity merely because the sibling is
busy. The scheduler handles this through the SD_SHARE_CPUCAPACITY topology,
idle-core selection and SMT balancing, which tries to move work from a busy SMT
core to an idle core when possible. It doesn't provide a dynamic numerical
capacity for each PE.

But the same limitation remains with the combined asym-packing. Once both
siblings must be used, neither static priority describes how the core resources
are partitioned.

> 
> Uclamp:
> Isn't uclamp, and particularly its bucket implementation, fundamentally a poor
> fit for these platforms in the first place? Even if we tried to represent these
> small capacity differences through uclamp, we'd need something like
> UCLAMP_BUCKETS_COUNT=512 or 1024 to get useful resolution. We currently limit
> it to 20, and for good reason: the overhead.

Yeah, the uclamp buckets are used to aggregate runnable-task clamps on a
runqueue, they don't encode CPU capacity classes. So it's a different story.

> 
> Misfit handling:
> This seems problematic for essentially the same reason as task fitting. A task
> can be classified as fitting while the core is in one-thread mode, then lose a
> substantial fraction of its effective CPU capacity when the sibling becomes
> active, without the static CPU capacity reflecting that change. Conversely,
> migrating it to an otherwise equivalent core and allowing that core to return
> to one-thread mode changes the effective capacity (and therefore utilization)
> again.

Yes, misfit handling compares a task against the CPU's standalone capacity, it
doesn't dynamically reduce that capacity when an SMT sibling becomes busy. That
is true for regular SMT as well, and sibling contention is handled separately by
the SMT balancing logic. The combined asym-packing priority doesn't change this,
it orders CPUs but doesn't make capacity depend on sibling state.

> 
> I'm assuming the CPU_CYCLES counter advancement isn't affected by the
> one-thread/two-thread mode transition?

My understanding is that the core counter continues to advance according to the
PE clock while the PE is active. The mode transition doesn't change the clock
frequency, so the AMU ratio will not reflect the loss of issue/cache/vector
resources.

So, yes, you are right that the existing capacity model does not fully describe
SMT interference. This is probably a broader dynamic-SMT capacity issue and
neither of the policies discussed here models it explicitly.

That said, this series achieves the intended result for the GEMM benchmark (with
similar results observed also for other CPU-intensive workloads): with SMT off,
running one CPU-intensive task per physical core reaches the same ~10 TFLOP/s as
the patched kernel with SMT enabled and the same number of tasks. So the
scheduler now appears to select the preferred PE consistently and avoid the
unwanted resource-mode transitions.

> 
> > 
> > Also, encoding the combined priority alone would not fix the problem addressed
> > by patch 2: the idle-selection paths currently do not consult asymmetric SMT
> > priority. They can still return an arbitrary idle sibling regardless of how
> > arch_asym_cpu_priority() is defined. Patch 2 adds that missing behavior and
> > scopes it to the shared-capacity SMT domain.
> 
> Sure, patch 2 is a different story altogether.

Ok.

> 
> > 
> >> I had suggested this a while ago, did you have a stab at that by any chance,
> >> too?
> > 
> > I tested your CPPC-based asym-packing series, but not this particular
> > combined-priority variant. IIUC the earlier proposal was replacing
> > capacity-aware scheduling for minor physical-core capacity differences, SMT
> > sibling ordering looks like an orthogonal problem.
> > 
> > And at the time, the combined SMT-aware SD_ASYM_CPUCAPACITY approach also gave
> > the best Vera results of the alternatives I tested, which is another reason I
> > kept physical-core capacity selection separate here.
> > 
> >> Am I missing something altogether?
> > 
> > Combining the priorities is a valid experiment, but I'm not sure if it
> > completely solves the problem by itself, I'll give it a try and share the
> > results.
> 
> Thanks again, i'll have a look and give it some more thoughts myself.

Thanks!
-Andrea