[Question] Userspace throttling + "sched/fair: Combine detach into dequeue when migrating task" causes guest boot hang

[Question] Userspace throttling + "sched/fair: Combine detach into dequeue when migrating task" causes guest boot hang

[Chen Jinghuang]

Hi, I'm seeing a VM boot hang on mainline, and I'd like to understand the
interaction between userspace throttling and the scheduler patch
"e1f078f50478 sched/fair: Combine detach into dequeue when migrating
task".

Host:
aarch64, 96 CPUs (0-95), 4 NUMA nodes:
	node0: 0-23, node1: 24-47, node2: 48-71, node3: 72-95
Mainline kernel tag: 7.2-rc1.

Guest(libvirt/KVM) - described in words:
An aarch64 (virt-6.2) UEFI VM launched with `virsh create`; key config:

- 128 vCPUs (statically placed, oversubscribed — the host has only 96
  physical CPUs).
- host-passthrough CPU model; GICv3; 64 GiB RAM.
- <cputune> has <global_quota> set to 400000; all <vcpupin> and
  <emulatorpin> entries are commented out, so there is no vCPU pinning.
- Storage: qcow2 on virtio-scsi (cache=none, io=native). HPET disabled.

Userspace throttling:
The VM runs under a CPU-quota cap applied on the host. The actual values
from the cgroup controller are:

	cpu.cfs_period_us = 100000
	cpu.cfs_quota_us  = 400000

I also found that if I set cpu.cfs_quota_us to -1, or enlarge it beyond a
certain point, the guest boots fine.

Symptom:
The guest hangs at some command early in boot and never reaches the login
prompt.

Observations:
Only reverting both of the following together makes it boot (neither one
alone suffices):

1. The kernel patch for userspace throttling.
2. The scheduler patch:
   e1f078f50478 ("sched/fair: Combine detach into dequeue when migrating
   task").

Reverting only one of them still hangs; reverting both together boots fine.

Question:
I don't fully understand how these two interact. My rough guess: e1f078f50478
("sched/fair: Combine detach into dequeue when migrating task") affects the
PELT accounting, and the userspace throttling also has logic that affects PELT
accounting. When both are combined, load balancing and subsequent scheduling
behavior may end up misbehaving, stalling the guest.

This looks like a real regression on mainline in the 128-vCPU oversubscribed
VM on a 96-core/4-NUMA host scenario. Any pointer to the correct mechanism or
a fix direction would be very much appreciated.

Thanks,
Chen Jinghuang

Re: [Question] Userspace throttling + "sched/fair: Combine detach into dequeue when migrating task" causes guest boot hang

[K Prateek Nayak]

Hello Chen,

Thank you for your report.

On 8/25/2026 5:36 PM, Chen Jinghuang wrote:
> Hi, I'm seeing a VM boot hang on mainline, and I'd like to understand the
> interaction between userspace throttling and the scheduler patch
> "e1f078f50478 sched/fair: Combine detach into dequeue when migrating
> task".
> 
> Host:
> aarch64, 96 CPUs (0-95), 4 NUMA nodes:
> 	node0: 0-23, node1: 24-47, node2: 48-71, node3: 72-95
> Mainline kernel tag: 7.2-rc1.
> 
> Guest(libvirt/KVM) - described in words:
> An aarch64 (virt-6.2) UEFI VM launched with `virsh create`; key config:
> 
> - 128 vCPUs (statically placed, oversubscribed — the host has only 96
>   physical CPUs).
> - host-passthrough CPU model; GICv3; 64 GiB RAM.
> - <cputune> has <global_quota> set to 400000; all <vcpupin> and
>   <emulatorpin> entries are commented out, so there is no vCPU pinning.
> - Storage: qcow2 on virtio-scsi (cache=none, io=native). HPET disabled.
> 
> Userspace throttling:
> The VM runs under a CPU-quota cap applied on the host. The actual values
> from the cgroup controller are:
> 
> 	cpu.cfs_period_us = 100000
> 	cpu.cfs_quota_us  = 400000

400ms across 96 CPUs per 100ms seems awfully low. Let me go see if I can
reproduce this.

> 
> I also found that if I set cpu.cfs_quota_us to -1, or enlarge it beyond a
> certain point, the guest boots fine.

Sounds a lot like guest side lock-holder preemption stalling the guest.
If you give it enough time, does the guest progress?

> 
> Symptom:
> The guest hangs at some command early in boot and never reaches the login
> prompt.

What happens if you allow it to boot and then enforce the more
the aggressive limits later? Do you see RCU stalls / lockups?

> 
> Observations:
> Only reverting both of the following together makes it boot (neither one
> alone suffices):
> 
> 1. The kernel patch for userspace throttling.

Are these Aaron's patches too or just the recent rework that I did?
Could you please paste a log of all the reverts.

> 2. The scheduler patch:
>    e1f078f50478 ("sched/fair: Combine detach into dequeue when migrating
>    task").
> 
> Reverting only one of them still hangs; reverting both together boots fine.

Can you check your cgroup stats to see how much time the vCPUs are getting
before and after the revert? Very surprising that e1f078f50478 has some
effect here.

> 
> Question:
> I don't fully understand how these two interact. My rough guess: e1f078f50478
> ("sched/fair: Combine detach into dequeue when migrating task") affects the
> PELT accounting, and the userspace throttling also has logic that affects PELT
> accounting. When both are combined, load balancing and subsequent scheduling
> behavior may end up misbehaving, stalling the guest.
> 
> This looks like a real regression on mainline in the 128-vCPU oversubscribed
> VM on a 96-core/4-NUMA host scenario. Any pointer to the correct mechanism or
> a fix direction would be very much appreciated.

Both, with exit-to-user throttling, and the legacy method, we would have
preempted the vCPU in xfer_to_guest_mode_work():

    if (ti_work & (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY))
        schedule();

    if (ti_work & _TIF_NOTIFY_RESUME)
        resume_user_mode_work(NULL);

Previously, task would have taken the schedule() route out, and now it
is done via resume_user_mode_work() -> schedule() / preempt_schedule()

Since e1f078f50478 only takes effect at migration, does 1:1 pinning
help progress the boot?

Are there any splats in your dmesg?

-- 
Thanks and Regards,
Prateek

[Question] Userspace throttling + "sched/fair: Combine detach into dequeue when migrating task" causes guest boot hang

[Chen Jinghuang]

Hi, I'm seeing a VM boot hang on mainline, and I'd like to understand the
interaction between userspace throttling and the scheduler patch
"e1f078f50478 sched/fair: Combine detach into dequeue when migrating
task".

Host:
aarch64, 96 CPUs (0-95), 4 NUMA nodes:
	node0: 0-23, node1: 24-47, node2: 48-71, node3: 72-95
Mainline kernel tag: 7.2-rc1.

Guest(libvirt/KVM) - described in words:
An aarch64 (virt-6.2) UEFI VM launched with `virsh create`; key config:

- 128 vCPUs (statically placed, oversubscribed — the host has only 96
  physical CPUs).
- host-passthrough CPU model; GICv3; 64 GiB RAM.
- <cputune> has <global_quota> set to 400000; all <vcpupin> and
  <emulatorpin> entries are commented out, so there is no vCPU pinning.
- Storage: qcow2 on virtio-scsi (cache=none, io=native). HPET disabled.

Userspace throttling:
The VM runs under a CPU-quota cap applied on the host. The actual values
from the cgroup controller are:

	cpu.cfs_period_us = 100000
	cpu.cfs_quota_us  = 400000

I also found that if I set cpu.cfs_quota_us to -1, or enlarge it beyond a
certain point, the guest boots fine.

Symptom:
The guest hangs at some command early in boot and never reaches the login
prompt.

Observations:
Only reverting both of the following together makes it boot (neither one
alone suffices):

1. The kernel patch for userspace throttling.
2. The scheduler patch:
   e1f078f50478 ("sched/fair: Combine detach into dequeue when migrating
   task").

Reverting only one of them still hangs; reverting both together boots fine.

Question:
I don't fully understand how these two interact. My rough guess: e1f078f50478
("sched/fair: Combine detach into dequeue when migrating task") affects the
PELT accounting, and the userspace throttling also has logic that affects PELT
accounting. When both are combined, load balancing and subsequent scheduling
behavior may end up misbehaving, stalling the guest.

This looks like a real regression on mainline in the 128-vCPU oversubscribed
VM on a 96-core/4-NUMA host scenario. Any pointer to the correct mechanism or
a fix direction would be very much appreciated.

Thanks,
Chen Jinghuang