Re: Should QEMU (accel=kvm) kvm-clock/guest_tsc stop counting during downtime blackout?

[David Woodhouse <dwmw2@infradead.org>]

[-- Attachment #1: Type: text/plain, Size: 4219 bytes --]

On Tue, 2025-09-23 at 10:25 -0700, Dongli Zhang wrote:
> 
> 
> On 9/23/25 9:26 AM, David Woodhouse wrote:
> > On Mon, 2025-09-22 at 12:37 -0700, Dongli Zhang wrote:
> > > On 9/22/25 11:16 AM, David Woodhouse wrote:
> 
> [snip]
> 
> > > > 
> > > > > 
> > > > > As demonstrated in my test, currently guest_tsc doesn't stop counting during
> > > > > blackout because of the lack of "MSR_IA32_TSC put" at
> > > > > kvmclock_vm_state_change(). Per my understanding, it is a bug and we may need to
> > > > > fix it.
> > > > > 
> > > > > BTW, kvmclock_vm_state_change() already utilizes KVM_SET_CLOCK to re-configure
> > > > > kvm-clock before continuing the guest VM.
> > 
> > Yeah, right now it's probably just introducing errors for a stop/start
> > of the VM.
> 
> But that help can meet the expectation?
> 
> Thanks to KVM_GET_CLOCK and KVM_SET_CLOCK, QEMU saves the clock with
> KVM_GET_CLOCK when the VM is stopped, and restores it with KVM_SET_CLOCK when
> the VM is continued.

It saves the actual *value* of the clock. I would prefer to phrase that
as "it makes the clock jump backwards to the time at which the guest
was paused".

> This ensures that the clock value itself does not change between stop and cont.
> 
> However, QEMU does not adjust the TSC offset via MSR_IA32_TSC during stop.
> 
> As a result, when execution resumes, the guest TSC suddenly jumps forward.

Oh wow, that seems really broken. If we're going to make it experience
a time warp, we should at least be *consistent*.

So a guest which uses the TSC for timekeeping should be mostly
unaffected by this and its wallclock should still be accurate. A guest
which uses the KVM clock will be hosed by it.

I think we should fix this so that the KVM clock is unaffected too.

> > 
> > > > > > 
> > > > > > KVM already lets you restore the TSC correctly. To restore KVM clock
> > > > > > correctly, you want something like KVM_SET_CLOCK_GUEST from
> > > > > > https://lore.kernel.org/all/20240522001817.619072-4-dwmw2@infradead.org/
> > > > > > 
> > > > > > For cross machine migration, you *do* need to use a realtime clock
> > > > > > reference as that's the best you have (make sure you use TAI not UTC
> > > > > > and don't get affected by leap seconds or smearing). Use that to
> > > > > > restore the *TSC* as well as you can to make it appear to have kept
> > > > > > running consistently. And then KVM_SET_CLOCK_GUEST just as you would on
> > > > > > the same host.
> > > > > 
> > > > > Indeed QEMU Live Migration also relies on kvmclock_vm_state_change() to
> > > > > temporarily stop/cont the source/target VM.
> > > > > 
> > > > > Would you mean we expect something different for live migration, i.e.,
> > > > > 
> > > > > 1. Live Migrate a source VM to a file.
> > > > > 2. Copy the file to another server.
> > > > > 3. Wait for 1 hour.
> > > > > 4. Migrate from the file to target VM.
> > > > > 
> > > > > Although it is equivalent to a one-hour downtime, we do need to count the
> > > > > missing one-hour, correct?
> > > > 
> > > > I don't look at it as counting anything. The clock keeps running even
> > > > when I'm not looking at it. If I wake up and look at it again, there is
> > > > no 'counting' how long I was asleep...
> > > > 
> > > 
> > > That means:
> > > 
> > > - stop/cont: clock/tsc stop running
> > > - savevm/loadvm: clock/tsc stop running
> > 
> > What does "stop running" even mean here? You can never stop the clock
> > running. The only thing you can do is change its offset so that it
> > jumps back to an earlier value, when you resume a VM?
> > 
> 
> Yes, I meant "change its offset so that it jumps back to an earlier value." From
> the VM's perspective, this is equivalent to "the clock was stopped."

Yeah... let's stop doing that :)

> Could you help explain why we treat stop/cont differently from live migration?

We shouldn't. Hardware companies spent *years* learning the lesson
about clocks. That they should just keep counting. At the same rate.
Unconditionally. Even if the CPU is running slower. Or stopped. Or
whatever. Just count. Do not stop counting.

Let's not repeat the same mistakes.


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]