Re: [PATCH v2 00/17] sched/paravirt: Introduce cpu_preferred_mask and steal-driven vCPU backoff

[Shrikanth Hegde <sshegde@linux.ibm.com>]

Hi Hillf.

On 4/9/26 10:45 AM, Hillf Danton wrote:
> On Wed, 8 Apr 2026 19:19:05 +0530 Shrikanth Hegde wrote:
>> On 4/8/26 3:44 PM, Hillf Danton wrote:
>>> On Wed,  8 Apr 2026 00:49:33 +0530 Shrikanth Hegde wrote:
>>>> Core idea is:
>>>> - Maintain set of CPUs which can be used by workload. It is denoted as
>>>>     cpu_preferred_mask
>>>> - Periodically compute the steal time. If steal time is high/low based
>>>>     on the thresholds, either reduce/increase the preferred CPUs.
>>>> - If a CPU is marked as non-preferred, push the task running on it if
>>>>     possible.
>>>> - Use this CPU state in wakeup and load balance to ensure tasks run
>>>>     within preferred CPUs.
>>>>
>>>> For the host kernel, there is no steal time, so no changes to its preferred
>>>> CPUs. So series would affect only the guest kernels.
>>>>
>>> Changes are added to guest in order to detect if pCPU is overloaded, and if
>>> that is true (I mean it is layer violation), why not ask the pCPU governor,
>>> hypervisor, to monitor the loads on pCPU and migrate vCPUs forth and back
>>> if necessary.
>>>
>>
>> AFAIK, there in no information in the host scheduler on what
>> each vCPU is running. It maybe holding a mutex, spinlock with irq disabled
> 
> This is what layer means (particularly in the data center environment).
> 

Host / hypervisor scheduler
- Schedules vCPU threads as opaque entities.
Has no visibility into:
- whether a vCPU is holding a spinlock
- whether IRQs are disabled
- whether a guest mutex is contended
- guest scheduler state
Can only ensure fairness between vCPUs

Guest scheduler
Knows exact task‑level semantics
- lock ownership
- preemption state
- affinity constraints.
But does not control pCPUs directly, unless there is vCPU pinning.

Steal time is precisely the contract boundary between those layers:
So, This is not a layer violation. Guest is acting on its CPUs based on
the hint which host already provides.

Actual layer violation would be:
- host peeking into guest scheduler data
- host deciding which guest vCPUs are “important”
- host understanding guest locks or IRQ state

Or I am not understanding what you mean by layer violation.
If so, please explain to me.

Today, why is steal time is being reported?
So that guest/host can make appropriate decision. right?

When you see high steal values, You have two choices.
Either increase the underlying resource by re-partitioning the host
with more cores or reduce the incoming request from guest such that
host can meet. If the host is already at max cores,
then there is only option.

One could, say with series high steal values may not be seen, how will system
admin re-size the host. Just look at preferred vs online. If they are not same
then there was contention and preferred became subset of online. We might have
update the documentation of steal time section.

>> or maybe in interrupt context. Moving/migrating the vCPUs threads without
>> that knowledge will hurt the guest. And it has to ensure fairness.
>>
> We have to pay the cost for vCPU.
> 
>> This has to work across different archs, some have linux as hypervisor, some
>> has non-linux hypervisor such as powerpc, s390.
>>
> Yeah, in the car cockpit product environment in Shenzhen Linux, Android and
> XYZ guests run on QNX, and your steal time approach looks half baked.
> 

They likely don't have this problem. IIUC, they would prefer deterministic behavior in
automotive hypervisors. Having steal time brings unbounded latency.

If the guests are not linux, then yes. Same logic will have to be there in each guest.
But that problem exists in other direction too. You have to inform the host somehow, which of my
vCPU threads are important. That is going to be way more complex in IMHO.
Even in linux we don't have that interface today. And then repeat the same in other non-linux
guest. One could say that is even worse.

If the guests are all indeed linux, then solution would work just fine.

Just re-iterate:
- For host kernel - No Change as it can't have steal time construct. Minimal overhead.
- Guests don't have steal time - No functional change. Minimal overhead.
- Guest with steal time - NO_STEAL_MONITOR - No functional change. Minimal overhead.
- Guest with steal time - STEAL_MONITOR - Functional changes - Steal driven vCPU backoff.


>> Steal time in guest is common construct in all archs. I don't think such
>> commonality exists in host schedulers.
>>
>> If done in guest, guest actually knows what it is running and whats more important.
>> It can make better decisions IMHO.