Co-stop

Co-stop

[Troels Arvin]

Hello,

According to https://www.linux-kvm.org/page/Lists,_IRC it's OK to use 
this list for KVM user questions, so here it goes:

How important is it to consider "co-stop" with KVM?

I haven't been able to find anything about this for KVM, but there's 
some material about it for VMWare (albeit typically rather old 
material). Based on VMWare material, by "co-stop" I mean the following 
situation:

A VM called "x" has been assigned quite a few vCPUs, e.g. 10, on a 
hypervisor with e.g. 20 physical cores. The hypervisor is hosting many 
other VMs, and the many VMs take turn running on the hardware.
Now, if it's often the case that, e.g., only 7 physical threads are 
available when it's x's turn to run, the hypervisor needs to postpone 
running x till all 10 cores can be allocated at the same time. During 
this waiting time, x is stopped (co-stop).

Is my understanding correct? Or will KVM allow x to run on (e.g.) 7 
cores, even though the VM thinks it has 10 vCPUs available?

It's my understanding that with KVM, the co-stopped situation is 
reflected in the VM's "steal time" metric in a tool like "top".

Does hyperthreading in either the hypervisor and/or the guest impact the 
risk of co-stop?

-- 
Regards,
Troels Arvin

Re: Co-stop

[Sean Christopherson]

On Fri, Sep 05, 2025, Troels Arvin wrote:
> Hello,
> 
> According to https://www.linux-kvm.org/page/Lists,_IRC it's OK to use this
> list for KVM user questions, so here it goes:
> 
> How important is it to consider "co-stop" with KVM?

From a _KVM_ perspective, it's not important at all.  Outside of a few targeted
paravirt yielding scenarios, KVM isn't involved in vCPU scheduling.

> I haven't been able to find anything about this for KVM, but there's some
> material about it for VMWare (albeit typically rather old material). Based
> on VMWare material, by "co-stop" I mean the following situation:
> 
> A VM called "x" has been assigned quite a few vCPUs, e.g. 10, on a
> hypervisor with e.g. 20 physical cores. The hypervisor is hosting many other
> VMs, and the many VMs take turn running on the hardware.
> Now, if it's often the case that, e.g., only 7 physical threads are
> available when it's x's turn to run, the hypervisor needs to postpone
> running x till all 10 cores can be allocated at the same time. During this
> waiting time, x is stopped (co-stop).
> 
> Is my understanding correct?

Maybe?  How vCPUs are scheduled isn't even really a kernel decision, it's much
more of a userspace decision.  E.g. the kernel scheduler will balance competing
tasks, but which vCPUs are allowed to run on which pCPUs, and when (at a macro
level), is entirely userspace controlled.

> Or will KVM allow x to run on (e.g.) 7 cores, even though the VM thinks it
> has 10 vCPUs available?

As above, KVM doesn't care.  KVM does enumerate the maximum number of
recommended vCPUs per VM, e.g. to help userspace avoid doing something truly
stupid, but that's just a recommendation.  Nothing prevents userspace from
creating a 128 vCPU and pinning all vCPUs tasks to a single pCPU.  Performance
will obviously be terrible, but functionally it works.

> It's my understanding that with KVM, the co-stopped situation is reflected
> in the VM's "steal time" metric in a tool like "top".

More or less.  Steal time captures how much time a vCPU wanted to run, but
couldn't because a different task (or tasks) in the host preempted the vCPU.
Note, I specifically say "tasks" and not "vCPUs", because in Linux+KVM, vCPUs
are regular tasks from a scheduling perspective, and so depending on how a host
is configured, a vCPU task may compete against other host tasks.

And to really warp your mind... nothing in KVM requires a 1:1 task:vCPU mapping.
VMMs typically use a dedicated task to run a vCPU, but KVM allows running a vCPU
on different tasks (though obviously not at the same time), and KVM also allows
running multiple vCPUs on a single task (again, obviously not at the same time).

> Does hyperthreading in either the hypervisor and/or the guest impact the
> risk of co-stop?

Not directly, no.  But hyperthreading can factor into the platform owner's
scheduling policies, and that in turn can certain impact contention.  E.g. with
all of the side channels that are suspecitble to cross-SMT attacks, it's common
for cloud providers to do "core scheduling", where vCPUs scheduled on a per-core
basic, not a per-CPU basis.