[BUG REPORT] KVM: x86: Keep kvmclock_offset monotonic across same-host pause/resume

[BUG REPORT] KVM: x86: Keep kvmclock_offset monotonic across same-host pause/resume

[Chenzhendong (alex)]

Hi,

While testing storage snapshots that suspend and resume a VM on the same
host (vm_stop -> snapshot -> vm_start), we observed that the guest
CLOCK_MONOTONIC jumps *backward* by the freeze duration after every
resume.

The problem
-----------

On a same-host pause/resume cycle, QEMU saves the kvmclock value at the
freeze instant (T1) via KVM_GET_CLOCK in vm_stop, and feeds it back via
KVM_SET_CLOCK in vm_start (T2 = T1 + freeze_duration).

kvm_vm_ioctl_set_clock() recomputes the offset unconditionally:

    kvmclock_offset = data.clock - now_raw_ns

with now_raw_ns sampled at T2. Since

    data.clock  = now_raw_ns(T1) + old_kvmclock_offset

this gives

    new_offset  = old_kvmclock_offset - freeze_duration

i.e. the offset is pulled back by the freeze duration.

The guest computes CLOCK_MONOTONIC from the pvclock page as:

    CLOCK_MONOTONIC = system_time + (rdtsc - tsc_timestamp) * mult >> shift

where system_time = kernel_ns + kvmclock_offset is filled by the host at
each vCPU entry. After resume the host re-samples tsc_timestamp, so the
(rdtsc - tsc_timestamp) term starts growing normally again and never
goes backward. The only term that can shrink is system_time: because
kvmclock_offset was pulled back by freeze_duration, the new system_time
is freeze_duration smaller than the value the guest last observed before
the freeze. The guest's next clock_gettime(CLOCK_MONOTONIC) therefore
reads a value freeze_duration smaller than before the freeze -- a
backward jump, violating the invariant that CLOCK_MONOTONIC never goes
backward.

The fix
-------

Only allow kvmclock_offset to move forward (or stay the same). If the
newly computed offset would be smaller than the current one, keep the
current value, so the guest clock keeps running forward and the freeze
duration is simply skipped from the guest's perspective -- which is the
expected behavior for a same-host pause/resume. This is a pure monotonic
protection with no time threshold; the normal forward path (guest boot,
KVM_CLOCK_REALTIME, cross-host migration toward a later destination host
clock) is unaffected.

    old_off = ka->kvmclock_offset;
    new_off = data.clock - now_raw_ns;
    if (new_off >= old_off)
            ka->kvmclock_offset = new_off;

Test
----

Inside the guest, pin a vCPU and call clock_gettime(CLOCK_MONOTONIC)
twice in a row, flagging any backward step. Over 50 snapshot
suspend/resume cycles the backward-step counter stays at 0 with this
patch, while the unpatched kernel reports a backward jump on every cycle
(measured freeze duration 27-36 ms).

To the best of our knowledge this scenario is not handled upstream. Any
feedback on whether monotonic protection is the right approach (vs. e.g.
teaching QEMU to feed a post-resume clock) would be much appreciated.

Signed-off-by: <Alex Chen> <alex.chen@huawei.com>
---
 arch/x86/kvm/x86.c | 19 ++++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 7a0d0e8e5c1f..000000000000 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -XXXX,XX +XXXX,XX @@ static int kvm_vm_ioctl_set_clock(struct kvm *kvm, void __user *argp)
 	struct kvm_arch *ka = &kvm->arch;
 	struct kvm_clock_data data;
 	u64 now_raw_ns;
+	s64 old_off, new_off;
 	...
 	if (ka->use_master_clock)
 		now_raw_ns = ka->master_kernel_ns;
 	else
 		now_raw_ns = get_kvmclock_base_ns();
-	ka->kvmclock_offset = data.clock - now_raw_ns;
+
+	/*
+	 * Monotonic protection: data.clock is the kvmclock value userspace
+	 * saved at vm_stop (freeze instant); now_raw_ns is sampled at resume.
+	 * The naive kvmclock_offset = data.clock - now_raw_ns pulls the
+	 * offset back to the freeze instant, which makes pvti.system_time =
+	 * kernel_ns + kvmclock_offset jump backward by the freeze duration ->
+	 * guest vDSO CLOCK_MONOTONIC goes backward.
+	 *
+	 * Only allow kvmclock_offset to move forward (or stay). If the new
+	 * offset would shrink, keep the old one so the guest clock keeps
+	 * running forward (the freeze duration is simply skipped from the
+	 * guest's perspective, which is the desired behavior for a same-host
+	 * pause/resume). Pure monotonic protection, no time threshold.
+	 */
+	old_off = ka->kvmclock_offset;
+	new_off = data.clock - now_raw_ns;
+	if (new_off >= old_off)
+		ka->kvmclock_offset = new_off;
 	kvm_end_pvclock_update(kvm);

Re: [BUG REPORT] KVM: x86: Keep kvmclock_offset monotonic across same-host pause/resume

[Dongli Zhang]

+ David.

FYI there were some prior discussions related to this topic.

David has a patch at KVM to enhance the KVM API to resolve the issue. It is
still not in mainline.

[PATCH v7 31/36] KVM: x86: Add KVM_[GS]ET_CLOCK_GUEST for accurate KVM clock
migration
https://lore.kernel.org/all/20260728144954.355376-32-dwmw2@infradead.org

I used to try to enhance from QEMU side to avoid clock drift while accounting
for downtime.

[PATCH 1/1] target/i386/kvm: account blackout downtime for kvm-clock and guest TSC
https://lore.kernel.org/qemu-devel/20251009095831.46297-1-dongli.zhang@oracle.com

Here is my RCA about forward drift and backward drift from another perspective.

https://lore.kernel.org/all/935312be-9a86-49fd-8bb4-2c998a68e2df@oracle.com


On Sun, Nov 1, 2026 8:19:14AM -0800, Chenzhendong (alex) wrote:
> Hi,
> 
> While testing storage snapshots that suspend and resume a VM on the same
> host (vm_stop -> snapshot -> vm_start), we observed that the guest
> CLOCK_MONOTONIC jumps *backward* by the freeze duration after every
> resume.
> 
> The problem
> -----------
> 
> On a same-host pause/resume cycle, QEMU saves the kvmclock value at the
> freeze instant (T1) via KVM_GET_CLOCK in vm_stop, and feeds it back via
> KVM_SET_CLOCK in vm_start (T2 = T1 + freeze_duration).
> 
> kvm_vm_ioctl_set_clock() recomputes the offset unconditionally:
> 
>     kvmclock_offset = data.clock - now_raw_ns
> 
> with now_raw_ns sampled at T2. Since
> 
>     data.clock  = now_raw_ns(T1) + old_kvmclock_offset
> 
> this gives
> 
>     new_offset  = old_kvmclock_offset - freeze_duration
> 
> i.e. the offset is pulled back by the freeze duration.
> 
> The guest computes CLOCK_MONOTONIC from the pvclock page as:
> 
>     CLOCK_MONOTONIC = system_time + (rdtsc - tsc_timestamp) * mult >> shift

CLOCK_MONOTONIC and kvm-clock are different topics.

I would suggest here use kvm-clock or PV clock.

> 
> where system_time = kernel_ns + kvmclock_offset is filled by the host at
> each vCPU entry. After resume the host re-samples tsc_timestamp, so the
> (rdtsc - tsc_timestamp) term starts growing normally again and never
> goes backward. The only term that can shrink is system_time: because
> kvmclock_offset was pulled back by freeze_duration, the new system_time
> is freeze_duration smaller than the value the guest last observed before
> the freeze. The guest's next clock_gettime(CLOCK_MONOTONIC) therefore
> reads a value freeze_duration smaller than before the freeze -- a
> backward jump, violating the invariant that CLOCK_MONOTONIC never goes
> backward.
> 
> The fix
> -------
> 
> Only allow kvmclock_offset to move forward (or stay the same). If the

Ideally, neither forward nor backward drift is appreciated.

Thank you very much!

Dongli Zhang


> newly computed offset would be smaller than the current one, keep the
> current value, so the guest clock keeps running forward and the freeze
> duration is simply skipped from the guest's perspective -- which is the
> expected behavior for a same-host pause/resume. This is a pure monotonic
> protection with no time threshold; the normal forward path (guest boot,
> KVM_CLOCK_REALTIME, cross-host migration toward a later destination host
> clock) is unaffected.
> 
>     old_off = ka->kvmclock_offset;
>     new_off = data.clock - now_raw_ns;
>     if (new_off >= old_off)
>             ka->kvmclock_offset = new_off;
> 
> Test
> ----
> 
> Inside the guest, pin a vCPU and call clock_gettime(CLOCK_MONOTONIC)
> twice in a row, flagging any backward step. Over 50 snapshot
> suspend/resume cycles the backward-step counter stays at 0 with this
> patch, while the unpatched kernel reports a backward jump on every cycle
> (measured freeze duration 27-36 ms).
> 
> To the best of our knowledge this scenario is not handled upstream. Any
> feedback on whether monotonic protection is the right approach (vs. e.g.
> teaching QEMU to feed a post-resume clock) would be much appreciated.
> 
> Signed-off-by: <Alex Chen> <alex.chen@huawei.com>
> ---
>  arch/x86/kvm/x86.c | 19 ++++++++++++++++++++-
>  1 file changed, 18 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 7a0d0e8e5c1f..000000000000 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -XXXX,XX +XXXX,XX @@ static int kvm_vm_ioctl_set_clock(struct kvm *kvm, void __user *argp)
>  	struct kvm_arch *ka = &kvm->arch;
>  	struct kvm_clock_data data;
>  	u64 now_raw_ns;
> +	s64 old_off, new_off;
>  	...
>  	if (ka->use_master_clock)
>  		now_raw_ns = ka->master_kernel_ns;
>  	else
>  		now_raw_ns = get_kvmclock_base_ns();
> -	ka->kvmclock_offset = data.clock - now_raw_ns;
> +
> +	/*
> +	 * Monotonic protection: data.clock is the kvmclock value userspace
> +	 * saved at vm_stop (freeze instant); now_raw_ns is sampled at resume.
> +	 * The naive kvmclock_offset = data.clock - now_raw_ns pulls the
> +	 * offset back to the freeze instant, which makes pvti.system_time =
> +	 * kernel_ns + kvmclock_offset jump backward by the freeze duration ->
> +	 * guest vDSO CLOCK_MONOTONIC goes backward.
> +	 *
> +	 * Only allow kvmclock_offset to move forward (or stay). If the new
> +	 * offset would shrink, keep the old one so the guest clock keeps
> +	 * running forward (the freeze duration is simply skipped from the
> +	 * guest's perspective, which is the desired behavior for a same-host
> +	 * pause/resume). Pure monotonic protection, no time threshold.
> +	 */
> +	old_off = ka->kvmclock_offset;
> +	new_off = data.clock - now_raw_ns;
> +	if (new_off >= old_off)
> +		ka->kvmclock_offset = new_off;
>  	kvm_end_pvclock_update(kvm);
> 
> 

Re: [BUG REPORT] KVM: x86: Keep kvmclock_offset monotonic across same-host pause/resume

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 4521 bytes --]

On Sun, 2026-11-01 at 12:30 -0800, Dongli Zhang wrote:
> + David.

Thanks.

> FYI there were some prior discussions related to this topic.
> 
> David has a patch at KVM to enhance the KVM API to resolve the issue. It is
> still not in mainline.
> 
> [PATCH v7 31/36] KVM: x86: Add KVM_[GS]ET_CLOCK_GUEST for accurate KVM clock
> migration
> https://lore.kernel.org/all/20260728144954.355376-32-dwmw2@infradead.org
> 
> I used to try to enhance from QEMU side to avoid clock drift while accounting
> for downtime.
> 
> [PATCH 1/1] target/i386/kvm: account blackout downtime for kvm-clock and guest TSC
> https://lore.kernel.org/qemu-devel/20251009095831.46297-1-dongli.zhang@oracle.com
> 
> Here is my RCA about forward drift and backward drift from another perspective.
> 
> https://lore.kernel.org/all/935312be-9a86-49fd-8bb4-2c998a68e2df@oracle.com
> 
> 
> On Sun, Nov 1, 2026 8:19:14AM -0800, Chenzhendong (alex) wrote:
> > Hi,
> > 
> > While testing storage snapshots that suspend and resume a VM on the same
> > host (vm_stop -> snapshot -> vm_start), we observed that the guest
> > CLOCK_MONOTONIC jumps *backward* by the freeze duration after every
> > resume.
> > 
> > The problem
> > -----------
> > 
> > On a same-host pause/resume cycle, QEMU saves the kvmclock value at the
> > freeze instant (T1) via KVM_GET_CLOCK in vm_stop, and feeds it back via
> > KVM_SET_CLOCK in vm_start (T2 = T1 + freeze_duration).

The whole of this in KVM upstream today is completely insane.

Note that those are completely different clock domains. Your T1 and T2
are CLOCK_REALTIME, which doesn't even run at the same *rate* as the
host's CLOCK_MONOTONIC_RAW which the kvmclock is theoretically bound
to. And if you do it during a leap second, it's doubly hosed because
it's UTC not TAI.

Ignoring the leap seconds, it's almost tolerable to use KVM_SET_CLOCK
with the KVM_CLOCK_REALTIME flag for live *migration*, but it makes
zero sense for live update on the *same* host.

The source of timekeeping is the TSC. The guest's TSC is just an
*offset* (and optional scaling) from the host's TSC. So if you're
running on the same host, the guest *TSC* can be reconstituted with
*exactly* the same relationship to the host TSC as before. Not a single
cycle of difference.

The kvmclock is then advertised to the guest as a mathematical function
of the guest's TSC. There's not much excuse for even a single
nanosecond of difference in that *either*. In fact, even with
KVM_[GS]ET_CLOCK_GUEST in my patch series, there's still a ±1ns
variance across get/set. Fixing that would require a lot *more*
refactoring than I've already done, so probably isn't worth it.

Guests should never be using kvmclock on modern CPUs where the TSC is
reliable, *anyway*.

> > kvm_vm_ioctl_set_clock() recomputes the offset unconditionally:
> > 
> >     kvmclock_offset = data.clock - now_raw_ns
> > 
> > with now_raw_ns sampled at T2. Since
> > 
> >     data.clock  = now_raw_ns(T1) + old_kvmclock_offset
> > 
> > this gives
> > 
> >     new_offset  = old_kvmclock_offset - freeze_duration
> > 
> > i.e. the offset is pulled back by the freeze duration.

Hm? There's no `freeze_duration` in kvm_vm_ioctl_set_clock(). It looks
like this:

	/*
	 * This pairs with kvm_guest_time_update(): when masterclock
is
	 * in use, we use master_kernel_ns + kvmclock_offset to set
	 * unsigned 'system_time' so if we use get_kvmclock_ns()
(which
	 * is slightly ahead) here we risk going negative on unsigned
	 * 'system_time' when 'data.clock' is very small.
	 */
	if (data.flags & KVM_CLOCK_REALTIME) {
		u64 now_real_ns = ktime_get_real_ns();

		/*
		 * Avoid stepping the kvmclock backwards.
		 */
		if (now_real_ns > data.realtime)
			data.clock += now_real_ns - data.realtime;
	}

Does that not already *have* the monotonicity protection that you are
trying to add?

> > The guest computes CLOCK_MONOTONIC from the pvclock page as:
> > 
> >     CLOCK_MONOTONIC = system_time + (rdtsc - tsc_timestamp) * mult >> shift
> 
> CLOCK_MONOTONIC and kvm-clock are different topics.
> 
> I would suggest here use kvm-clock or PV clock.

Right, don't use CLOCK_MONOTONIC in the guest; use CLOCK_MONOTONIC_RAW.

You can do this with KVM selftests though; perform the save/restore
from the host and then just run some code in the guest which checks for
monotonicity. Didn't my series already add a test which is very similar
to that? 

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 6179 bytes --]