The SMP RHEL 5.1 PAE guest can't boot up issue

Yang, Sheng
2008-02-22T08:57:34Z

I believe I have found the root cause of SMP RHEL5.1 PAE guest can't boot up
issue. The problem was caused by
kvm:6685637b211ad67bdce21bfd9f91bc888b3acb4f
"KVM: VMX: Ensure vcpu time stamp counter is monotonous" (It didn't take me
much time to found the solution, but a lot of time to find the proper
explanation... :( )

As we guessed, the problem was the monotonous of TSC. I have traced to
the 2.6.18 PAE guest kernel, and finally found it caused by a overflow in the
loop of function update_wall_timer()(kernel/timer.c), when using TSC as
clocksource by default.

The reason is that the patch "KVM: VMX: Ensure vcpu time stamp counter is
monotonous" bring big gap between different VCPUs (error between
TSC_OFFSETs). Though I have proved that the patch can ensure the monotonous
on each VCPU (which rejected my first thought...), the patch
have 2 problems:

1. It have accumulated the error. Each vcpu's TSC is monotonous, but get
slower and slower, compared to the host. That's because the TSC is very
accuracy and the interval between reading TSC is big. But this is not very
critical.

2. The critical one. In normal condition, VCPU0 migrated much more
frequently than other VCPUs. And the patch add more "delta" (always negative
if host TSC is stable) to TSC_OFFSET each
time migrated. Then after boot for a while, VCPU0 became much
slower than others (In my test, VCPU0 was migrated about two times than the
others, and easily to be more than 100k cycles slower). In the guest kernel,
clocksource TSC is global variable, the variable "cycle_last" may got the
VCPU1's TSC value, then turn to VCPU0. For VCPU0's TSC_OFFSET is
smaller than VCPU1's, so it's possible to got the "cycle_last" (from VCPU1)
bigger than current TSC value (from VCPU0) in next tick. Then "u64 offset =
clocksource_read() - cycle_last" overflowed and caused the "infinite" loop.
And it can also explained why Marcelo's patch don't work - it just reduce the
rate of gap increasing.

The freezing didn't happen when using userspace IOAPIC, just because the qemu
APIC didn't implement real LOWPRI(or round_robin) to choose CPU for delivery.
It choose VCPU0 everytime if possible, so CPU1 in guest won't update
cycle_last. :(

This freezing only occurred on RHEL5/5.1 pae (kernel 2.6.18), because of they
set IO-APIC IRQ0's dest_mask to 0x3 (with 2 vcpus) and dest_mode as
LOWEST_PRIOITY, then other vcpus had chance to modify "cycle_last". In
contrast, RHEL5/5.1 32e set IRQ0's dest_mode as FIXED, to CPU0, then don't
have this problem. So does RHEL4(kernel 2.6.9).

I don't know if the patch was still needed now, since it was posted long ago(I
don't know which issue it solved). I'd like to post a revert patch if
necessary.

--
Thanks
Yang, Sheng