Re: Should QEMU (accel=kvm) kvm-clock/guest_tsc stop counting during downtime blackout?

[Dongli Zhang <dongli.zhang@oracle.com>]

On 9/23/25 9:26 AM, David Woodhouse wrote:
> On Mon, 2025-09-22 at 12:37 -0700, Dongli Zhang wrote:
>> On 9/22/25 11:16 AM, David Woodhouse wrote:

[snip]

>>>
>>>>
>>>> As demonstrated in my test, currently guest_tsc doesn't stop counting during
>>>> blackout because of the lack of "MSR_IA32_TSC put" at
>>>> kvmclock_vm_state_change(). Per my understanding, it is a bug and we may need to
>>>> fix it.
>>>>
>>>> BTW, kvmclock_vm_state_change() already utilizes KVM_SET_CLOCK to re-configure
>>>> kvm-clock before continuing the guest VM.
> 
> Yeah, right now it's probably just introducing errors for a stop/start
> of the VM.

But that help can meet the expectation?

Thanks to KVM_GET_CLOCK and KVM_SET_CLOCK, QEMU saves the clock with
KVM_GET_CLOCK when the VM is stopped, and restores it with KVM_SET_CLOCK when
the VM is continued.

This ensures that the clock value itself does not change between stop and cont.

However, QEMU does not adjust the TSC offset via MSR_IA32_TSC during stop.

As a result, when execution resumes, the guest TSC suddenly jumps forward.

> 
>>>>>
>>>>> KVM already lets you restore the TSC correctly. To restore KVM clock
>>>>> correctly, you want something like KVM_SET_CLOCK_GUEST from
>>>>> https://lore.kernel.org/all/20240522001817.619072-4-dwmw2@infradead.org/
>>>>>
>>>>> For cross machine migration, you *do* need to use a realtime clock
>>>>> reference as that's the best you have (make sure you use TAI not UTC
>>>>> and don't get affected by leap seconds or smearing). Use that to
>>>>> restore the *TSC* as well as you can to make it appear to have kept
>>>>> running consistently. And then KVM_SET_CLOCK_GUEST just as you would on
>>>>> the same host.
>>>>
>>>> Indeed QEMU Live Migration also relies on kvmclock_vm_state_change() to
>>>> temporarily stop/cont the source/target VM.
>>>>
>>>> Would you mean we expect something different for live migration, i.e.,
>>>>
>>>> 1. Live Migrate a source VM to a file.
>>>> 2. Copy the file to another server.
>>>> 3. Wait for 1 hour.
>>>> 4. Migrate from the file to target VM.
>>>>
>>>> Although it is equivalent to a one-hour downtime, we do need to count the
>>>> missing one-hour, correct?
>>>
>>> I don't look at it as counting anything. The clock keeps running even
>>> when I'm not looking at it. If I wake up and look at it again, there is
>>> no 'counting' how long I was asleep...
>>>
>>
>> That means:
>>
>> - stop/cont: clock/tsc stop running
>> - savevm/loadvm: clock/tsc stop running
> 
> What does "stop running" even mean here? You can never stop the clock
> running. The only thing you can do is change its offset so that it
> jumps back to an earlier value, when you resume a VM?
> 

Yes, I meant "change its offset so that it jumps back to an earlier value." From
the VM's perspective, this is equivalent to "the clock was stopped."


Could you help explain why we treat stop/cont differently from live migration?

The live migration/update is same as stop/cont because the blackout phase
involves stopping a guest, and continuing execution in a different/same host.
You technically stop the guest in a way that's not controlled by the guest
(compared to say hibernation or suspend-to-idle). and then you continue. Part of
the reason I think 'stop'/'cont' ought to have same behavior as live migration.

Thank you very much!

Dongli Zhang