[PATCH v13 00/26] KVM: x86: Cleaning up the KVM clock mess, part 2

[PATCH v13 00/26] KVM: x86: Cleaning up the KVM clock mess, part 2

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

This is the second half of the KVM clock series which has been churning
since about 2024. The first half¹ is upstream as of 7.4-rc1 (yay!).  This is
the remainder — the masterclock overhaul, and new API for fixing migration.

Sashiko is helping by finding lots of things which have been fairly hosed
in kvmclock since its inception, as well as things I've managed to screw
up for myself along the way.

As of v12, a guest can no longer force the KVM clock out of masterclock
mode at all. Even the old SYSTEM_TIME MSR (and that SUSE kernel from 2006)
is satisfied by updating the master clock when the MSR is rewritten.

Tree: https://git.infradead.org/users/dwmw2/linux.git/shortlog/refs/heads/kvmclock13

v13 follows one more on-list sashiko round, a second 0day report⁴, and
eleven rounds of review-and-fixup with sashiko locally. Two
pre-existing bugs the review flushed out are split out as standalone
fixes with Fixes: tags (new patches 7 and 9), so the series is now 26
patches. In detail:

KVM_SET_TSC_KHZ:
 • The {guest TSC, host clock} tuple could be sampled just before a sibling
   opened a newer generation, wrapping the elapsed time — and the first
   fix, clamping it to zero, inflated the matching error from |F1-F2|·Δt
   to F1·Δt, spuriously splitting the generation when two vCPUs raced
   through the ioctl.
 • The 1ms matching window for exact kernel-internal samples was
   unreachable unless userspace had already written a TSC.
 • preempt_disable() did not actually make the fallback {TSC, ns} read
   pair coherent; interrupts could still land between the two reads.
 • A creation-time frequency the hardware cannot scale to left the vCPU
   with virtual_tsc_khz == 0 for its whole life.
 • An internal preserved sample of exactly zero took the userspace
   zero-write path and was snapped onto the generation line.
 • A forced synchronization of an unrun vCPU racing a sibling through
   the ioctl could wrap its extrapolated TSC below zero, which an
   unstable-TSC host then re-anchored the guest TSC to at vcpu_load.
 • The 1ms window computation needed libgcc's __udivdi3 on i386 (kernel
   test robot⁴).
 • The fix for stale always-catchup state is now its own patch with a
   Fixes: tag, since the bug predates this series (new patch 9), as is
   the fix for the Hyper-V TSC page latching stale parameters from a
   superseded snapshot (new patch 7).

The master clock:
 • A scaled protected guest TSC (TDX, SNP Secure TSC) left the master
   clock computing with the host's unscaled frequency.
 • The first protected-TSC synchronization cleared the frequency match
   for the life of the VM, since such vCPUs never reach
   kvm_track_tsc_matching() to re-latch it — and the re-latch then had to
   raise the masterclock update itself, or a refresh racing a second
   vCPU's creation latched the master clock off forever.
 • An aborted vCPU creation could clear matched state which a racing
   host-initiated TSC write on the already-live fd had made correct, and
   for a protected-TSC first vCPU it skipped the snapshot refresh forever.
 • KVM_GET_CLOCK's no-vCPU staleness check compared two reads nothing
   kept coherent, so a reader preempted for a second mid-ioctl served the
   flag-less fallback from a healthy snapshot.
 • Deduplicated masterclock updates removed the accidental heal for a
   Hyper-V TSC page rewritten from a stale snapshot: a vCPU preempted
   between its pvclock read and the page write consumed the HOST_CHANGED
   mark against the old snapshot and set HV_TSC_PAGE_SET, permanently.
   kvm_hv_setup_tsc_page() now revalidates the pvclock seqcount under
   its own lock, and the updater re-marks the page once the new snapshot
   is published.
 • The CLOCK_UPDATE broadcast sat inside the pvclock seqcount write
   section, growing the interrupts-off window with the vCPU count.
 • A CPU hotplug runs the backwards-TSC compensation while vCPU threads
   are live, and the unlocked consume of tsc_offset_adjustment in
   vcpu_load could discard a delta the generation reference kept.
 • kvm_vm_ioctl_set_clock()'s epoch-wrap guard left the anchor off
   master_kernel_ns, wrapping system_time for a near-zero clock instead.
 • Patch 12's subject and opening described an earlier patch's work; it
   is now "Keep the master clock trustworthy while the VM has no vCPUs".

KVM_[GS]ET_CLOCK_GUEST:
 • GET after a host suspend with the VM paused combined the post-resume
   offset with the pre-suspend snapshot — frequencies still matched, so
   the destination accepted garbage. GET now services the vCPU's pending
   masterclock update first, and fails with -ENODATA.

Elsewhere:
 • The TSC migration documentation's step 7b multiplication wraps a
   signed 64-bit value after ~51 minutes of pause at 3GHz; it now carries
   the same 128-bit note as step 2d, and the exemplar selftest does the
   arithmetic in 128 bits.
 • Selftests reported a loaded host as a kernel regression instead of a
   TAP skip, printed no TAP output at all, carried a fourth private copy
   of the pvclock arithmetic, and the divergent-offset correction check
   could not fail against a silently ignored KVM_SET_CLOCK_GUEST.
 • The CPUID timing test's ratio cross-check skip keyed off the very
   ratio it exists to verify, and the first-write test's
   frequency-change arm could pass on a buggy kernel under adverse
   scheduling.
 • The clock library helpers now also convert the pre-existing
   open-coded call sites.
 • Various comments and commit messages overclaimed ("invisible to any
   correct guest", "definitely identical", "the only synchronization a
   protected vCPU ever performs") and have been made honest.

¹ https://lore.kernel.org/all/20260826213303.914988-1-seanjc@google.com
² https://lore.kernel.org/all/20261105154955.2525419-1-dwmw2@infradead.org
³ https://lore.kernel.org/all/20261109012112.3415328-1-dwmw2@infradead.org
⁴ https://lore.kernel.org/all/202611091411.tSxUx2Bj-lkp@intel.com


David Woodhouse (23):
  KVM: x86: Improve synchronization in kvm_synchronize_tsc()
  KVM: x86: Kill last_tsc_{nsec,write,offset} fields
  KVM: x86: Replace nr_vcpus_matched_tsc count with all_vcpus_matched_tsc bool
  KVM: x86: Allow KVM master clock mode when TSCs are offset from each other
  KVM: x86: Factor out kvm_use_master_clock()
  KVM: x86: Avoid gratuitous global clock updates
  KVM: x86: Don't latch the Hyper-V TSC page from a superseded snapshot
  KVM: x86: Avoid redundant masterclock updates from multiple vCPUs
  KVM: x86: Stop TSC catchup when a frequency change no longer needs it
  KVM: x86: Re-synchronize TSC after KVM_SET_TSC_KHZ
  KVM: x86: Cache host vclock_mode for masterclock eligibility checks
  KVM: x86: Remove pvclock_gtod_data and private timekeeping code
  KVM: x86: Keep the master clock trustworthy while the VM has no vCPUs
  KVM: selftests: Use UAPI pvclock-abi.h in xen_shinfo_test
  KVM: selftests: Add library helpers for clock and TSC-offset ioctls
  KVM: x86: Add KVM_VCPU_TSC_SCALE and fix the documentation on TSC migration
  KVM: selftests: Add master clock offset test
  KVM: selftests: Add Xen/generic CPUID timing leaf test
  KVM: selftests: Add Xen runstate migration test
  KVM: selftests: Test that the first user TSC write is honoured exactly
  KVM: x86/xen: Drop open-coded fallback clock read in kvm_xen_start_timer()
  KVM: x86: Use a single time snapshot in kvm_vm_ioctl_set_clock()
  KVM: x86: Keep master clock for guests using legacy MSR_KVM_SYSTEM_TIME

Jack Allister (3):
  UAPI: x86: Move pvclock-abi to UAPI for x86 platforms
  KVM: x86: Add KVM_[GS]ET_CLOCK_GUEST for accurate KVM clock migration
  KVM: selftests: Add KVM/PV clock selftest to prove timer correction

 Documentation/virt/kvm/api.rst                |   64 +
 Documentation/virt/kvm/devices/vcpu.rst       |  144 +-
 Documentation/virt/kvm/locking.rst            |    3 +-
 MAINTAINERS                                   |    4 +-
 arch/x86/include/asm/kvm_host.h               |   21 +-
 arch/x86/include/uapi/asm/kvm.h               |    7 +
 arch/x86/include/{ => uapi}/asm/pvclock-abi.h |   27 +-
 arch/x86/kvm/hyperv.c                         |   20 +-
 arch/x86/kvm/hyperv.h                         |   10 +-
 arch/x86/kvm/msrs.c                           |   30 +-
 arch/x86/kvm/trace.h                          |   12 +-
 arch/x86/kvm/x86.c                            | 1275 ++++++++++++++---
 arch/x86/kvm/x86.h                            |    2 +-
 arch/x86/kvm/xen.c                            |   50 +-
 include/uapi/linux/kvm.h                      |    4 +
 scripts/xen-hypercalls.sh                     |    2 +-
 tools/arch/x86/include/asm/pvclock-abi.h      |   48 -
 tools/arch/x86/include/uapi/asm/kvm.h         |    6 +
 tools/testing/selftests/kvm/Makefile.kvm      |    6 +
 .../selftests/kvm/include/x86/processor.h     |   50 +
 .../kvm/system_counter_offset_test.c          |    3 +-
 .../testing/selftests/kvm/x86/hyperv_clock.c  |    2 +-
 .../selftests/kvm/x86/kvm_clock_test.c        |    6 +-
 .../kvm/x86/masterclock_offset_test.c         |  249 ++++
 .../kvm/x86/pvclock_migration_test.c          |  503 +++++++
 .../testing/selftests/kvm/x86/pvclock_test.c  |  525 +++++++
 .../selftests/kvm/x86/tsc_user_set_test.c     |  193 +++
 .../selftests/kvm/x86/xen_cpuid_timing_test.c |  293 ++++
 .../selftests/kvm/x86/xen_migration_test.c    |  227 +++
 .../selftests/kvm/x86/xen_shinfo_test.c       |   35 +-
 30 files changed, 3454 insertions(+), 367 deletions(-)
 rename arch/x86/include/{ => uapi}/asm/pvclock-abi.h (82%)
 delete mode 100644 tools/arch/x86/include/asm/pvclock-abi.h
 create mode 100644 tools/testing/selftests/kvm/x86/masterclock_offset_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/pvclock_migration_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/pvclock_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/tsc_user_set_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/xen_cpuid_timing_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/xen_migration_test.c

-- 
2.43.0

[PATCH v13 01/26] KVM: x86: Improve synchronization in kvm_synchronize_tsc()

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

When synchronizing to an existing TSC (either by explicitly writing
zero, or the legacy hack where the TSC is written within one second's
worth of the previously written TSC), the last_tsc_write and
last_tsc_nsec values were being misrecorded by __kvm_synchronize_tsc().
The *unsynchronized* value of the TSC (perhaps even zero) was being
recorded, along with the current time at which kvm_synchronize_tsc()
was called. This could cause *subsequent* writes to fail to synchronize
correctly.

Fix that by extrapolating the generation's TSC value to the current
time (data = cur_tsc_write + elapsed cycles) before passing it to
__kvm_synchronize_tsc(), so the recorded {data, ns} pair always lies
on the generation's line and the one-second matching window slides
forward with each synchronized write. When the TSC is stable, reuse
the generation's offset exactly; when it is unstable, compute a new
offset for the extrapolated value.

Furthermore, there were *three* different TSC reads used for calculating
the "current" time, all slightly different from each other. Fix that by
using kvm_get_time_and_clockread() where possible and using the same
host_tsc value in all cases.

Extrapolating from the generation's origin at the nominal frequency
means a guest within tsc_tolerance_ppm of the host (where hardware
scaling is not engaged) drifts out of the one-second window after
about 67 minutes; a write which then fails to match still installs
the exact value, at the cost of a fresh generation. The old per-write
re-anchoring bounded that drift, but recorded points the generation's
line did not pass through — the bug this patch exists to fix.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Reviewed-by: Paul Durrant <paul@xen.org>
---
 arch/x86/kvm/x86.c | 39 ++++++++++++++++++++++++++++++---------
 1 file changed, 30 insertions(+), 9 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 32563a91fba9..7a2b0b99373f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -210,6 +210,10 @@ module_param(pi_inject_timer, bint, 0644);
 static bool __read_mostly mitigate_smt_rsb;
 module_param(mitigate_smt_rsb, bool, 0444);
 
+#ifdef CONFIG_X86_64
+static bool kvm_get_time_and_clockread(s64 *kernel_ns, u64 *tsc_timestamp);
+#endif
+
 #define KVM_SUPPORTED_XCR0     (XFEATURE_MASK_FP | XFEATURE_MASK_SSE \
 				| XFEATURE_MASK_YMM | XFEATURE_MASK_BNDREGS \
 				| XFEATURE_MASK_BNDCSR | XFEATURE_MASK_AVX512 \
@@ -1374,14 +1378,23 @@ void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value)
 {
 	u64 data = user_value ? *user_value : 0;
 	struct kvm *kvm = vcpu->kvm;
-	u64 offset, ns, elapsed;
+	u64 offset, host_tsc, elapsed;
+	s64 ns;
 	unsigned long flags;
 	bool matched = false;
 	bool synchronizing = false;
 
 	raw_spin_lock_irqsave(&kvm->arch.tsc_write_lock, flags);
-	offset = kvm_compute_l1_tsc_offset(vcpu, rdtsc(), data);
-	ns = get_kvmclock_base_ns();
+
+#ifdef CONFIG_X86_64
+	if (!kvm_get_time_and_clockread(&ns, &host_tsc))
+#endif
+	{
+		host_tsc = rdtsc();
+		ns = get_kvmclock_base_ns();
+	}
+
+	offset = kvm_compute_l1_tsc_offset(vcpu, host_tsc, data);
 	elapsed = ns - kvm->arch.last_tsc_nsec;
 
 	if (vcpu->arch.virtual_tsc_khz) {
@@ -1424,13 +1437,21 @@ void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value)
          */
 	if (synchronizing &&
 	    vcpu->arch.virtual_tsc_khz == kvm->arch.last_tsc_khz) {
-		if (!kvm_check_tsc_unstable()) {
+		/*
+		 * If synchronizing, advance the reference point to "now"
+		 * so the matching window slides forward with each vCPU.
+		 * Extrapolate from the generation's own origin: 'elapsed'
+		 * is measured from the most recent write, which lies on
+		 * the generation's line only for the first matched write.
+		 */
+		u64 delta = nsec_to_cycles(vcpu, ns - kvm->arch.cur_tsc_nsec);
+
+		data = kvm->arch.cur_tsc_write + delta;
+
+		if (!kvm_check_tsc_unstable())
 			offset = kvm->arch.cur_tsc_offset;
-		} else {
-			u64 delta = nsec_to_cycles(vcpu, elapsed);
-			data += delta;
-			offset = kvm_compute_l1_tsc_offset(vcpu, rdtsc(), data);
-		}
+		else
+			offset = kvm_compute_l1_tsc_offset(vcpu, host_tsc, data);
 		matched = true;
 	}
 
-- 
2.43.0

[PATCH v13 02/26] KVM: x86: Kill last_tsc_{nsec,write,offset} fields

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

These mostly duplicate the cur_tsc_{nsec,write,offset} values. On the
stable-TSC matching paths the offsets used were identical; the stored
{nsec, write} anchor could differ, because a matched
kvm_arch_tsc_set_attr() recorded a freshly measured point, quietly
re-anchoring the 1-second matching window onto the guest's actual TSC
line. Matching now measures from the generation's origin alone. A
guest frequency within tsc_tolerance_ppm of the host's can therefore
drift out of the window after about an hour of generation age; such a
write opens a new generation and still installs the exact value — a
PVCLOCK_TSC_STABLE_BIT drop, healed as the siblings' own writes
rejoin, never an inaccuracy. On the unstable-TSC paths the two values
could genuinely diverge, so kvm_arch_tsc_set_attr() now matches
against the generation's reference offset — the more principled
comparison anyway, and with an unstable TSC the match only affects
generation accounting.

Rename last_tsc_khz and last_tsc_scaling_ratio to cur_tsc_khz and
cur_tsc_scaling_ratio, since they are properties of the current
generation.

With cur_tsc_offset the single source of truth for the generation's
offset, the backwards-TSC compensation on S4 resume must adjust it by
the same (guest-scaled) delta it applies to every vCPU; later
synchronizations would otherwise match against a stale pre-S4 offset.
And since a synchronization installs an absolute offset which already
includes that delta, it clears the vCPU's pending
tsc_offset_adjustment — applying the deferred compensation on top at
the next vcpu_load would double it. Previously a nonzero write could
not hit this: zeroing the last_tsc values broke the matching window
after a backwards TSC.

Three races around that compensation are handled. The consume in
kvm_arch_vcpu_load() now runs under tsc_write_lock, as an unlocked
read-then-zero racing the hotplug-time fold could discard a delta the
generation keeps. kvm_arch_tsc_set_attr() folds a pending adjustment
into the user-supplied offset under the lock, since userspace computed
it against the pre-jump TSC line. And the lockless vm_list walk in
kvm_arch_enable_virtualization_cpu() remains as it was: unsound
against a concurrent kvm_destroy_vm(), but pre-existing, and not
fixable with kvm_lock without recreating the inversion against
cpus_read_lock() that commit 44d174596260 ("KVM: Use dedicated mutex
to protect kvm_usage_count to avoid deadlock") untangled.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Reviewed-by: Paul Durrant <paul@xen.org>
---
 Documentation/virt/kvm/locking.rst |   3 +-
 arch/x86/include/asm/kvm_host.h    |   7 +-
 arch/x86/kvm/x86.c                 | 132 ++++++++++++++++++++++-------
 3 files changed, 105 insertions(+), 37 deletions(-)

diff --git a/Documentation/virt/kvm/locking.rst b/Documentation/virt/kvm/locking.rst
index 662231e958a0..b3621eb3c1cd 100644
--- a/Documentation/virt/kvm/locking.rst
+++ b/Documentation/virt/kvm/locking.rst
@@ -263,7 +263,8 @@ time it will be set using the Dirty tracking mechanism described above.
 
 :Type:		raw_spinlock_t
 :Arch:		x86
-:Protects:	- kvm_arch::{last_tsc_write,last_tsc_nsec,last_tsc_offset}
+:Protects:	- kvm_arch::{cur_tsc_write,cur_tsc_nsec,cur_tsc_offset,cur_tsc_khz,cur_tsc_scaling_ratio}
+		- kvm_vcpu_arch::tsc_offset_adjustment
 		- tsc offset in vmcb
 :Comment:	'raw' because updating the tsc offsets must not be preempted.
 
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index e2eef7057bd3..ea48b1949316 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1236,11 +1236,8 @@ struct kvm_arch {
 	 * preemption-disabled region, so it must be a raw spinlock.
 	 */
 	raw_spinlock_t tsc_write_lock;
-	u64 last_tsc_nsec;
-	u64 last_tsc_write;
-	u32 last_tsc_khz;
-	u64 last_tsc_offset;
-	u64 last_tsc_scaling_ratio;
+	u32 cur_tsc_khz;
+	u64 cur_tsc_scaling_ratio;
 	u64 cur_tsc_nsec;
 	u64 cur_tsc_write;
 	u64 cur_tsc_offset;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 7a2b0b99373f..12f465040efc 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1334,17 +1334,24 @@ static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc,
 		vcpu->kvm->arch.user_set_tsc = true;
 
 	/*
-	 * We also track th most recent recorded KHZ, write and time to
-	 * allow the matching interval to be extended at each write.
+	 * Track the TSC frequency, scaling ratio, and offset for the current
+	 * generation. These are used to detect matching TSC writes and to
+	 * compute the guest TSC from the host clock.
 	 */
-	kvm->arch.last_tsc_nsec = ns;
-	kvm->arch.last_tsc_write = tsc;
-	kvm->arch.last_tsc_khz = vcpu->arch.virtual_tsc_khz;
-	kvm->arch.last_tsc_offset = offset;
-	kvm->arch.last_tsc_scaling_ratio = vcpu->arch.l1_tsc_scaling_ratio;
+	kvm->arch.cur_tsc_khz = vcpu->arch.virtual_tsc_khz;
+	kvm->arch.cur_tsc_scaling_ratio = vcpu->arch.l1_tsc_scaling_ratio;
 
 	vcpu->arch.last_guest_tsc = tsc;
 
+	/*
+	 * The offset written below is absolute, superseding any backwards-TSC
+	 * compensation still pending from kvm_arch_enable_virtualization_cpu()
+	 * (which has already been folded into cur_tsc_offset, and thus into
+	 * a matched @offset); applying it again at the next vcpu_load would
+	 * double it.
+	 */
+	vcpu->arch.tsc_offset_adjustment = 0;
+
 	kvm_vcpu_write_tsc_offset(vcpu, offset);
 
 	if (!matched) {
@@ -1354,8 +1361,6 @@ static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc,
 		 * nanosecond time, offset, and write, so if TSCs are in
 		 * sync, we can match exact offset, and if not, we can match
 		 * exact software computation in compute_guest_tsc()
-		 *
-		 * These values are tracked in kvm->arch.cur_xxx variables.
 		 */
 		kvm->arch.cur_tsc_generation++;
 		kvm->arch.cur_tsc_nsec = ns;
@@ -1395,7 +1400,7 @@ void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value)
 	}
 
 	offset = kvm_compute_l1_tsc_offset(vcpu, host_tsc, data);
-	elapsed = ns - kvm->arch.last_tsc_nsec;
+	elapsed = ns - kvm->arch.cur_tsc_nsec;
 
 	if (vcpu->arch.virtual_tsc_khz) {
 		if (data == 0) {
@@ -1405,7 +1410,7 @@ void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value)
 			 */
 			synchronizing = true;
 		} else if (kvm->arch.user_set_tsc) {
-			u64 tsc_exp = kvm->arch.last_tsc_write +
+			u64 tsc_exp = kvm->arch.cur_tsc_write +
 						nsec_to_cycles(vcpu, elapsed);
 			u64 tsc_hz = vcpu->arch.virtual_tsc_khz * 1000LL;
 			/*
@@ -1436,15 +1441,14 @@ void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value)
 	 * it's better to try to match offsets from the beginning.
          */
 	if (synchronizing &&
-	    vcpu->arch.virtual_tsc_khz == kvm->arch.last_tsc_khz) {
+	    vcpu->arch.virtual_tsc_khz == kvm->arch.cur_tsc_khz) {
 		/*
-		 * If synchronizing, advance the reference point to "now"
-		 * so the matching window slides forward with each vCPU.
-		 * Extrapolate from the generation's own origin: 'elapsed'
-		 * is measured from the most recent write, which lies on
-		 * the generation's line only for the first matched write.
+		 * If synchronizing, extrapolate the generation's value to
+		 * "now": the matching window slides forward with wall time,
+		 * anchored on the generation's origin alone (a matched
+		 * write no longer updates any stored reference point).
 		 */
-		u64 delta = nsec_to_cycles(vcpu, ns - kvm->arch.cur_tsc_nsec);
+		u64 delta = nsec_to_cycles(vcpu, elapsed);
 
 		data = kvm->arch.cur_tsc_write + delta;
 
@@ -1615,7 +1619,7 @@ static void pvclock_update_vm_gtod_copy(struct kvm *kvm)
 		 * get_kvmclock() to compute kvmclock from the host TSC
 		 * without needing a vCPU reference.
 		 */
-		ka->master_tsc_scaling_ratio = ka->last_tsc_scaling_ratio;
+		ka->master_tsc_scaling_ratio = ka->cur_tsc_scaling_ratio;
 		tsc_hz = (u64)get_cpu_tsc_khz() * HZ_PER_KHZ;
 		if (tsc_hz && kvm_caps.has_tsc_control)
 			tsc_hz = kvm_scale_tsc(tsc_hz,
@@ -2603,8 +2607,24 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 
 	/* Apply any externally detected TSC adjustments (due to suspend) */
 	if (unlikely(vcpu->arch.tsc_offset_adjustment)) {
-		adjust_tsc_offset_host(vcpu, vcpu->arch.tsc_offset_adjustment);
+		unsigned long flags;
+		u64 adj;
+
+		/*
+		 * Consume under tsc_write_lock: the writer in
+		 * kvm_arch_enable_virtualization_cpu() folds the same delta
+		 * into the generation's cur_tsc_offset under the lock, and
+		 * an unlocked read-then-zero here could discard a delta the
+		 * generation keeps, leaving this vCPU's offset diverged
+		 * from a generation it is still counted as matching.
+		 */
+		raw_spin_lock_irqsave(&vcpu->kvm->arch.tsc_write_lock, flags);
+		adj = vcpu->arch.tsc_offset_adjustment;
 		vcpu->arch.tsc_offset_adjustment = 0;
+		raw_spin_unlock_irqrestore(&vcpu->kvm->arch.tsc_write_lock,
+					   flags);
+
+		adjust_tsc_offset_host(vcpu, adj);
 		kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu);
 	}
 
@@ -3343,9 +3363,25 @@ static int kvm_arch_tsc_set_attr(struct kvm_vcpu *vcpu,
 
 		raw_spin_lock_irqsave(&kvm->arch.tsc_write_lock, flags);
 
+		/*
+		 * A backwards-TSC compensation which landed after this
+		 * ioctl's vcpu_load() consumed the pending adjustment was
+		 * folded into cur_tsc_offset but not yet into this vCPU —
+		 * and @offset was computed by userspace against the
+		 * pre-jump TSC line. Fold it into @offset here (scaled as
+		 * adjust_tsc_offset_host() would), both so the matched
+		 * comparison below compares like with like and because
+		 * __kvm_synchronize_tsc() will clear the pending
+		 * adjustment as superseded by the absolute offset it
+		 * installs.
+		 */
+		if (unlikely(vcpu->arch.tsc_offset_adjustment))
+			offset += kvm_scale_tsc(vcpu->arch.tsc_offset_adjustment,
+						vcpu->arch.l1_tsc_scaling_ratio);
+
 		matched = (vcpu->arch.virtual_tsc_khz &&
-			   kvm->arch.last_tsc_khz == vcpu->arch.virtual_tsc_khz &&
-			   kvm->arch.last_tsc_offset == offset);
+			   kvm->arch.cur_tsc_khz == vcpu->arch.virtual_tsc_khz &&
+			   kvm->arch.cur_tsc_offset == offset);
 
 		tsc = kvm_scale_tsc(rdtsc(), vcpu->arch.l1_tsc_scaling_ratio) + offset;
 		ns = get_kvmclock_base_ns();
@@ -9803,7 +9839,7 @@ int kvm_arch_enable_virtualization_cpu(void)
 {
 	struct kvm *kvm;
 	struct kvm_vcpu *vcpu;
-	unsigned long i;
+	unsigned long i, flags;
 	int ret;
 	u64 local_tsc;
 	u64 max_tsc = 0;
@@ -9874,6 +9910,44 @@ int kvm_arch_enable_virtualization_cpu(void)
 	if (backwards_tsc) {
 		u64 delta_cyc = max_tsc - local_tsc;
 		list_for_each_entry(kvm, &vm_list, vm_list) {
+			/*
+			 * Hold tsc_write_lock across the per-vCPU adjustments
+			 * AND the generation reference offset adjustment, so
+			 * that a concurrent TSC synchronization cannot observe
+			 * one without the other (it would compute an offset
+			 * with the delta applied twice, or not at all).
+			 * kvm_arch_vcpu_load() consumes tsc_offset_adjustment
+			 * under the same lock, so it cannot discard a delta
+			 * which has been folded into the generation
+			 * reference.
+			 *
+			 * The generation's reference offset is adjusted by the
+			 * same (guest-scaled, as adjust_tsc_offset_host()
+			 * does for each vCPU) delta, so that future
+			 * KVM_SET_TSC / vCPU creation still matches against
+			 * the adjusted TSC timeline.
+			 *
+			 * The recorded {cur_tsc_nsec, cur_tsc_write} pair is
+			 * left alone, and ends up stale by the sleep period:
+			 * cur_tsc_nsec is CLOCK_BOOTTIME, which has the sleep
+			 * injected on resume, while the guest TSC (kept
+			 * continuous across the host TSC jump) does not
+			 * advance during the sleep. It cannot be re-anchored
+			 * here because, as noted above, timekeeping has not
+			 * yet been resumed. The 1-second matching window
+			 * therefore fails to match across a longer sleep,
+			 * and the next nonzero TSC write opens a fresh,
+			 * coherent generation. A forced synchronization
+			 * (vCPU creation, or a zero write) bypasses the
+			 * window and does extrapolate along the stale line,
+			 * overshooting by the sleep period — but it installs
+			 * the compensated offset, so the vCPU's actual TSC
+			 * is correct; only last_guest_tsc overshoots, which
+			 * matters only if the TSC turns unstable before the
+			 * vCPU first enters the guest, exactly as it did
+			 * before this accounting existed.
+			 */
+			raw_spin_lock_irqsave(&kvm->arch.tsc_write_lock, flags);
 			kvm->arch.backwards_tsc_observed = true;
 			kvm_for_each_vcpu(i, vcpu, kvm) {
 				vcpu->arch.tsc_offset_adjustment += delta_cyc;
@@ -9881,14 +9955,10 @@ int kvm_arch_enable_virtualization_cpu(void)
 				kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu);
 			}
 
-			/*
-			 * We have to disable TSC offset matching.. if you were
-			 * booting a VM while issuing an S4 host suspend....
-			 * you may have some problem.  Solving this issue is
-			 * left as an exercise to the reader.
-			 */
-			kvm->arch.last_tsc_nsec = 0;
-			kvm->arch.last_tsc_write = 0;
+			kvm->arch.cur_tsc_offset +=
+				kvm_scale_tsc(delta_cyc,
+					      kvm->arch.cur_tsc_scaling_ratio);
+			raw_spin_unlock_irqrestore(&kvm->arch.tsc_write_lock, flags);
 		}
 
 	}
-- 
2.43.0

[PATCH v13 03/26] KVM: x86: Replace nr_vcpus_matched_tsc count with all_vcpus_matched_tsc bool

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Comparing nr_vcpus_matched_tsc against kvm->online_vcpus at read time
was always racy — a new vCPU can appear between the increment and its
first synchronization — and it made "all vCPUs matched" an accident of
arithmetic rather than a property with defined semantics. Replace it
with a boolean maintained at synchronization time under tsc_write_lock:
set in kvm_track_tsc_matching(), cleared when a new generation opens,
consumed by pvclock_update_vm_gtod_copy(). The count remains as the
input from which the flag is computed, and for the trace event.

The consistent interface is the point. The flag starts true at VM
creation, so a VM whose vCPUs never synchronize — a protected guest
TSC (TDX, SNP Secure TSC) returns early from __kvm_synchronize_tsc() —
is eligible for PVCLOCK_TSC_STABLE_BIT by design rather than by luck
of the read-time arithmetic, and use_master_clock is true from VM
creation on a TSC-clocksource host, before any vCPU exists; later
patches build on exactly that. The default scaling ratio must then be
initialized too (a zero ratio zeroes tsc_hz and turns the master clock
straight back off), and the khz/ratio bookkeeping recorded even for a
protected guest TSC: it is host-side state, which a scaled protected
guest must not leave at the host's unscaled frequency.

The cached state needs care at vCPU creation: the new vCPU is counted
in online_vcpus before it is synchronized, and userspace taking
vcpu->mutex first can hold that window open across a whole first
KVM_RUN. Clear the matched state up front in postcreate — the old
read-time comparison failed conservatively false throughout — and let
the synchronization (or a racing host-initiated TSC write on the
already-live fd) re-latch it; a postcreate aborted by a fatal signal
just propagates whatever the state then is with a masterclock update.
Only the instants between the online_vcpus increment and the clear
remain stale-true.

The explicit clear at new-generation start is currently overwritten by
the unconditional recompute in kvm_track_tsc_matching(); it is
preparation for later patches which consume the previous value.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/include/asm/kvm_host.h |  6 ++-
 arch/x86/kvm/x86.c              | 76 +++++++++++++++++++++++++++------
 2 files changed, 67 insertions(+), 15 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index ea48b1949316..18d7bb9fd8d2 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1232,8 +1232,9 @@ struct kvm_arch {
 	s64 kvmclock_offset;
 
 	/*
-	 * This also protects nr_vcpus_matched_tsc which is read from a
-	 * preemption-disabled region, so it must be a raw spinlock.
+	 * This also protects nr_vcpus_matched_tsc and
+	 * all_vcpus_matched_tsc, which are read from preemption-disabled
+	 * regions, so it must be a raw spinlock.
 	 */
 	raw_spinlock_t tsc_write_lock;
 	u32 cur_tsc_khz;
@@ -1242,6 +1243,7 @@ struct kvm_arch {
 	u64 cur_tsc_write;
 	u64 cur_tsc_offset;
 	u64 cur_tsc_generation;
+	bool all_vcpus_matched_tsc;
 	int nr_vcpus_matched_tsc;
 
 	u32 default_tsc_khz;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 12f465040efc..1bb0136042a6 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1125,8 +1125,10 @@ static void kvm_track_tsc_matching(struct kvm_vcpu *vcpu, bool new_generation)
 	 * and all vCPUs must have matching TSCs.  Note, the count for matching
 	 * vCPUs doesn't include the reference vCPU, hence "+1".
 	 */
-	bool use_master_clock = (ka->nr_vcpus_matched_tsc + 1 ==
-				 atomic_read(&vcpu->kvm->online_vcpus)) &&
+	ka->all_vcpus_matched_tsc = (ka->nr_vcpus_matched_tsc + 1 ==
+				     atomic_read(&vcpu->kvm->online_vcpus));
+
+	bool use_master_clock = ka->all_vcpus_matched_tsc &&
 				gtod_is_based_on_tsc(gtod->clock.vclock_mode);
 
 	/*
@@ -1327,20 +1329,23 @@ static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc,
 
 	lockdep_assert_held(&kvm->arch.tsc_write_lock);
 
-	if (vcpu->arch.guest_tsc_protected)
-		return;
-
-	if (user_set_tsc)
-		vcpu->kvm->arch.user_set_tsc = true;
-
 	/*
-	 * Track the TSC frequency, scaling ratio, and offset for the current
+	 * Track the TSC frequency and scaling ratio for the current
 	 * generation. These are used to detect matching TSC writes and to
-	 * compute the guest TSC from the host clock.
+	 * compute the guest TSC from the host clock. This is host-side
+	 * bookkeeping, recorded even for a protected guest TSC — a scaled
+	 * protected guest (TDX, SNP Secure TSC) must not leave the master
+	 * clock computing with the host's unscaled frequency.
 	 */
 	kvm->arch.cur_tsc_khz = vcpu->arch.virtual_tsc_khz;
 	kvm->arch.cur_tsc_scaling_ratio = vcpu->arch.l1_tsc_scaling_ratio;
 
+	if (vcpu->arch.guest_tsc_protected)
+		return;
+
+	if (user_set_tsc)
+		vcpu->kvm->arch.user_set_tsc = true;
+
 	vcpu->arch.last_guest_tsc = tsc;
 
 	/*
@@ -1367,6 +1372,7 @@ static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc,
 		kvm->arch.cur_tsc_write = tsc;
 		kvm->arch.cur_tsc_offset = offset;
 		kvm->arch.nr_vcpus_matched_tsc = 0;
+		kvm->arch.all_vcpus_matched_tsc = false;
 	} else if (vcpu->arch.this_tsc_generation != kvm->arch.cur_tsc_generation) {
 		kvm->arch.nr_vcpus_matched_tsc++;
 	}
@@ -1593,8 +1599,7 @@ static void pvclock_update_vm_gtod_copy(struct kvm *kvm)
 	bool host_tsc_clocksource, vcpus_matched;
 
 	lockdep_assert_held(&kvm->arch.tsc_write_lock);
-	vcpus_matched = (ka->nr_vcpus_matched_tsc + 1 ==
-			atomic_read(&kvm->online_vcpus));
+	vcpus_matched = ka->all_vcpus_matched_tsc;
 
 	/*
 	 * If the host uses TSC clock, then passthrough TSC as stable
@@ -9584,8 +9589,38 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
 void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
 {
-	if (mutex_lock_killable(&vcpu->mutex))
+	struct kvm_arch *ka = &vcpu->kvm->arch;
+	unsigned long flags;
+
+	/*
+	 * The new vCPU is already counted in online_vcpus but has not
+	 * been synchronized. Clear the cached matched state before
+	 * trying the mutex: the old read-time comparison failed
+	 * conservatively false throughout this window, and userspace
+	 * can hold it open across a whole first KVM_RUN by taking
+	 * vcpu->mutex before we do. The synchronization below (or any
+	 * racing host-initiated TSC write on the already-live fd)
+	 * re-latches the flags. A protected guest TSC never
+	 * participates in offset matching, so for it there is nothing
+	 * to clear — and no later sync to re-latch.
+	 */
+	if (!vcpu->arch.guest_tsc_protected) {
+		raw_spin_lock_irqsave(&ka->tsc_write_lock, flags);
+		WRITE_ONCE(ka->all_vcpus_matched_tsc, false);
+		raw_spin_unlock_irqrestore(&ka->tsc_write_lock, flags);
+	}
+
+	if (mutex_lock_killable(&vcpu->mutex)) {
+		/*
+		 * Aborted by a fatal signal: the synchronization never
+		 * ran. Propagate whatever the matched state now is (a
+		 * racing TSC write may have re-latched it), so the
+		 * master clock does not stay in its previous mode with
+		 * running siblings keeping PVCLOCK_TSC_STABLE_BIT.
+		 */
+		kvm_update_masterclock(vcpu->kvm);
 		return;
+	}
 	vcpu_load(vcpu);
 	kvm_synchronize_tsc(vcpu, NULL);
 	if (kvm_check_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu))
@@ -10061,6 +10096,21 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	ratelimit_set_flags(&kvm->arch.kvmclock_update_rs, RATELIMIT_MSG_ON_RELEASE);
 	kvm->arch.kvmclock_offset = -get_kvmclock_base_ns();
 
+	/*
+	 * Start with all (zero) vCPUs considered matched, so that a VM whose
+	 * vCPUs never reach kvm_track_tsc_matching() — notably when the guest
+	 * TSC is protected (TDX, SNP Secure TSC) and __kvm_synchronize_tsc()
+	 * returns early — is still eligible for PVCLOCK_TSC_STABLE_BIT. The
+	 * scaling ratio must then also be initialized: with vcpus_matched
+	 * now true here, pvclock_update_vm_gtod_copy() reaches the scaling
+	 * path, and a zero ratio would zero tsc_hz and turn the master
+	 * clock straight back off (permanently so for protected-TSC guests,
+	 * whose syncs return before kvm_track_tsc_matching() ever requests
+	 * the masterclock update that would recompute it).
+	 */
+	kvm->arch.all_vcpus_matched_tsc = true;
+	kvm->arch.cur_tsc_scaling_ratio = kvm_caps.default_tsc_scaling_ratio;
+
 	raw_spin_lock_irqsave(&kvm->arch.tsc_write_lock, flags);
 	pvclock_update_vm_gtod_copy(kvm);
 	raw_spin_unlock_irqrestore(&kvm->arch.tsc_write_lock, flags);
-- 
2.43.0

[PATCH v13 04/26] KVM: x86: Allow KVM master clock mode when TSCs are offset from each other

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Previously, a guest writing different TSC values on different vCPUs could
force KVM out of master clock mode. With this change, only a frequency
mismatch disables master clock. The only ways for non-master-clock mode
to happen now are archaic hardware without a TSC-based clocksource, a
VMM that sets different TSC frequencies across vCPUs, or a guest using
the legacy MSR_KVM_SYSTEM_TIME (addressed later in this series).

Running at a different frequency leads to systemic skew between the
clocks observed by different vCPUs, due to arithmetic precision in the
scaling, so that should indeed force the clock back onto the host's
CLOCK_MONOTONIC_RAW. But divergent TSC *offsets* are not a problem: the
KVM clock is a single VM-wide nanosecond timeline, and only its
expression in TSC cycles is per-vCPU. Each vCPU's pvclock structure is
generated by translating the shared master snapshot into that vCPU's
own TSC domain via kvm_read_l1_tsc(), so when the guest computes
system_time + (rdtsc - tsc_timestamp) * rate, its own offset is present
in both rdtsc and tsc_timestamp and cancels out. VM-scoped consumers
(KVM_GET_CLOCK, the wall clock epoch, the Xen runstate times) operate
purely in the nanosecond domain, where the offset never appears.

Only two consumers genuinely depend on the TSC offsets matching, and
both are already guarded by PVCLOCK_TSC_STABLE_BIT:

 1. A guest vCPU reading a *sibling* vCPU's pvclock structure with its
    own TSC would compute garbage — precisely why commit b48aa97e3820
    ("KVM: x86: require matched TSC offsets for master clock") required
    matched offsets. But when PVCLOCK_TSC_STABLE_BIT is clear, guests
    already fall back to the global last_value monotonicity clamp.

 2. The Hyper-V TSC page is a single VM-wide {scale, offset} pair, so
    it cannot represent divergent offsets. It is self-guarding:
    compute_tsc_page_parameters() refuses to populate the page unless
    PVCLOCK_TSC_STABLE_BIT is set, forcing such guests onto the (safe,
    nanosecond domain) TIME_REF_COUNT MSR.

So: frequency match gates master clock eligibility, and the full TSC
match (including offset) gates only PVCLOCK_TSC_STABLE_BIT, which tells
the guest it may skip cross-vCPU monotonicity enforcement.

Frequency matching is tracked with a dedicated generation counter
(cur_tsc_freq_generation) which bumps only on actual frequency changes;
each vCPU is counted once per generation via this_tsc_freq_generation.
The existing offset tracking is restructured to the same pattern, and
both counters use a consistent >= online_vcpus threshold. A vCPU set
to a different frequency and back opens a new generation each time,
and its siblings are only counted back in as they synchronize — the
migration procedure sets every vCPU, which re-latches the match as the
last one lands; frequency excursions on a subset of vCPUs are not
free, by design.

A protected vCPU's synchronization returns before
kvm_track_tsc_matching(), the only other place the match is re-latched,
so its branch re-latches the frequency match itself — the first vCPU's
own bump against the zero initial cur_tsc_khz would otherwise clear it
for the life of the VM — and raises KVM_REQ_MASTERCLOCK_UPDATE on the
transition back to matched, since no later synchronization ever runs
for a protected guest to repair a refresh taken in the window.

KVM_SET_TSC_KHZ on a vCPU which has already run does not pass through
__kvm_synchronize_tsc(), so at this point in the series it leaves the
frequency-match state stale, exactly as it previously left the offset
match stale; the later patch "KVM: x86: Re-synchronize TSC after
KVM_SET_TSC_KHZ" funnels it through the invalidation.

The divergent-offset behaviour is exercised by pvclock_migration_test:
vCPUs with TSCs offset by ±0.5-1s keep the master clock, lose
PVCLOCK_TSC_STABLE_BIT, and still agree on the KVM clock to within a
nanosecond, before and after a simulated live migration.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/include/asm/kvm_host.h |  11 ++-
 arch/x86/kvm/trace.h            |  12 +--
 arch/x86/kvm/x86.c              | 139 ++++++++++++++++++++++++++------
 3 files changed, 130 insertions(+), 32 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 18d7bb9fd8d2..74306a81e882 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -887,6 +887,7 @@ struct kvm_vcpu_arch {
 	u64 this_tsc_nsec;
 	u64 this_tsc_write;
 	u64 this_tsc_generation;
+	u64 this_tsc_freq_generation;
 	bool tsc_catchup;
 	bool tsc_always_catchup;
 	s8 virtual_tsc_shift;
@@ -1232,9 +1233,10 @@ struct kvm_arch {
 	s64 kvmclock_offset;
 
 	/*
-	 * This also protects nr_vcpus_matched_tsc and
-	 * all_vcpus_matched_tsc, which are read from preemption-disabled
-	 * regions, so it must be a raw spinlock.
+	 * This also protects the matched-state fields
+	 * ({nr_vcpus_matched,all_vcpus_matched}_{tsc,freq} and
+	 * cur_tsc_freq_generation), which are read from
+	 * preemption-disabled regions, so it must be a raw spinlock.
 	 */
 	raw_spinlock_t tsc_write_lock;
 	u32 cur_tsc_khz;
@@ -1244,6 +1246,9 @@ struct kvm_arch {
 	u64 cur_tsc_offset;
 	u64 cur_tsc_generation;
 	bool all_vcpus_matched_tsc;
+	bool all_vcpus_matched_freq;
+	int nr_vcpus_matched_freq;
+	u64 cur_tsc_freq_generation;
 	int nr_vcpus_matched_tsc;
 
 	u32 default_tsc_khz;
diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
index 93de876c318c..704683532ef0 100644
--- a/arch/x86/kvm/trace.h
+++ b/arch/x86/kvm/trace.h
@@ -1003,25 +1003,25 @@ TRACE_EVENT(kvm_write_tsc_offset,
 	{VDSO_CLOCKMODE_TSC,  "tsc"}			\
 
 TRACE_EVENT(kvm_update_master_clock,
-	TP_PROTO(bool use_master_clock, unsigned int host_clock, bool offset_matched),
-	TP_ARGS(use_master_clock, host_clock, offset_matched),
+	TP_PROTO(bool use_master_clock, unsigned int host_clock, bool freq_matched),
+	TP_ARGS(use_master_clock, host_clock, freq_matched),
 
 	TP_STRUCT__entry(
 		__field(		bool,	use_master_clock	)
 		__field(	unsigned int,	host_clock		)
-		__field(		bool,	offset_matched		)
+		__field(		bool,	freq_matched		)
 	),
 
 	TP_fast_assign(
 		__entry->use_master_clock	= use_master_clock;
 		__entry->host_clock		= host_clock;
-		__entry->offset_matched		= offset_matched;
+		__entry->freq_matched		= freq_matched;
 	),
 
-	TP_printk("masterclock %d hostclock %s offsetmatched %u",
+	TP_printk("masterclock %d hostclock %s freqmatched %u",
 		  __entry->use_master_clock,
 		  __print_symbolic(__entry->host_clock, host_clocks),
-		  __entry->offset_matched)
+		  __entry->freq_matched)
 );
 
 TRACE_EVENT(kvm_track_tsc,
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 1bb0136042a6..08cc8ab3d699 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1114,30 +1114,72 @@ static inline bool gtod_is_based_on_tsc(int mode)
 }
 #endif
 
-static void kvm_track_tsc_matching(struct kvm_vcpu *vcpu, bool new_generation)
+static void kvm_track_tsc_matching(struct kvm_vcpu *vcpu, bool update_mclock)
 {
 #ifdef CONFIG_X86_64
 	struct kvm_arch *ka = &vcpu->kvm->arch;
 	struct pvclock_gtod_data *gtod = &pvclock_gtod_data;
+	bool prev_matched_tsc = ka->all_vcpus_matched_tsc;
+	bool prev_matched_freq = ka->all_vcpus_matched_freq;
 
 	/*
-	 * To use the masterclock, the host clocksource must be based on TSC
-	 * and all vCPUs must have matching TSCs.  Note, the count for matching
-	 * vCPUs doesn't include the reference vCPU, hence "+1".
+	 * Track whether all vCPUs have matching TSC offsets (for
+	 * PVCLOCK_TSC_STABLE_BIT) and matching frequencies (for
+	 * master clock eligibility).
+	 */
+
+	/*
+	 * A new vCPU might already have incremented ->online_vcpus
+	 * and cause a temporary false negative here. But will then
+	 * call kvm_synchronize_tsc() from kvm_arch_vcpu_postcreate()
+	 * and finish the job.
+	 */
+	int online = atomic_read(&vcpu->kvm->online_vcpus);
+	/*
+	 * WRITE_ONCE() pairs with the READ_ONCE() in kvm_guest_time_update(),
+	 * which samples the flag locklessly (outside both tsc_write_lock and
+	 * its pvclock seqcount read section); the race is benign (a stale
+	 * value merely delays the PVCLOCK_TSC_STABLE_BIT broadcast to the
+	 * masterclock update which is requested below).
+	 */
+	WRITE_ONCE(ka->all_vcpus_matched_tsc,
+		   ka->nr_vcpus_matched_tsc >= online);
+
+	update_mclock |= (prev_matched_tsc != ka->all_vcpus_matched_tsc);
+
+	/*
+	 * all_vcpus_matched_freq starts true and is cleared when
+	 * __kvm_synchronize_tsc() detects a frequency mismatch.
+	 * Re-enable when all vCPUs have synced with matching frequency.
+	 * If all offsets also match, that implies frequencies match too.
 	 */
-	ka->all_vcpus_matched_tsc = (ka->nr_vcpus_matched_tsc + 1 ==
-				     atomic_read(&vcpu->kvm->online_vcpus));
+	if (ka->all_vcpus_matched_tsc ||
+	    ka->nr_vcpus_matched_freq >= online)
+		ka->all_vcpus_matched_freq = true;
 
-	bool use_master_clock = ka->all_vcpus_matched_tsc &&
+	/*
+	 * A frequency-match transition needs a masterclock update too:
+	 * when the last vCPU reaches the new frequency, the masterclock
+	 * scaling factors still describe the old one, and no toggle of
+	 * use_master_clock occurs to trigger the refresh.
+	 */
+	update_mclock |= (prev_matched_freq != ka->all_vcpus_matched_freq);
+
+	/*
+	 * To use the masterclock, the host clocksource must be based on TSC
+	 * and all vCPUs must have matching TSC *frequency*. Different offsets
+	 * are fine — each vCPU's pvclock has its own tsc_timestamp that
+	 * accounts for its offset.
+	 */
+	bool use_master_clock = ka->all_vcpus_matched_freq &&
 				gtod_is_based_on_tsc(gtod->clock.vclock_mode);
 
 	/*
-	 * Request a masterclock update if the masterclock needs to be toggled
-	 * on/off, or when starting a new generation and the masterclock is
-	 * enabled (compute_guest_tsc() requires the masterclock snapshot to be
-	 * taken _after_ the new generation is created).
+	 * Request a masterclock update if needed: toggling master clock,
+	 * new generation requiring a fresh snapshot, or a change in
+	 * all_vcpus_matched_tsc (to broadcast PVCLOCK_TSC_STABLE_BIT).
 	 */
-	if ((ka->use_master_clock && new_generation) ||
+	if ((ka->use_master_clock && update_mclock) ||
 	    (ka->use_master_clock != use_master_clock))
 		kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu);
 
@@ -1336,12 +1378,50 @@ static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc,
 	 * bookkeeping, recorded even for a protected guest TSC — a scaled
 	 * protected guest (TDX, SNP Secure TSC) must not leave the master
 	 * clock computing with the host's unscaled frequency.
+	 *
+	 * If the frequency changed, master clock mode can no longer be used
+	 * since the kvmclock scaling factors differ between vCPUs.
 	 */
+	if (vcpu->arch.virtual_tsc_khz != kvm->arch.cur_tsc_khz) {
+		kvm->arch.cur_tsc_freq_generation++;
+		kvm->arch.all_vcpus_matched_freq = false;
+		kvm->arch.nr_vcpus_matched_freq = 0;
+	}
+
+	/* Count each vCPU once per freq generation */
+	if (vcpu->arch.this_tsc_freq_generation != kvm->arch.cur_tsc_freq_generation) {
+		vcpu->arch.this_tsc_freq_generation = kvm->arch.cur_tsc_freq_generation;
+		kvm->arch.nr_vcpus_matched_freq++;
+	}
+
 	kvm->arch.cur_tsc_khz = vcpu->arch.virtual_tsc_khz;
 	kvm->arch.cur_tsc_scaling_ratio = vcpu->arch.l1_tsc_scaling_ratio;
 
-	if (vcpu->arch.guest_tsc_protected)
+	if (vcpu->arch.guest_tsc_protected) {
+		/*
+		 * A protected vCPU synchronizes only here (from postcreate,
+		 * host-initiated MSR_IA32_TSC writes and KVM_VCPU_TSC_OFFSET
+		 * writes) and never reaches
+		 * kvm_track_tsc_matching() — the only other place the
+		 * frequency match is re-latched. Without this, the first
+		 * vCPU's bump above (cur_tsc_khz starts at zero) would
+		 * clear the match for the life of the VM, turning the
+		 * master clock off for exactly the guests the init-to-true
+		 * serves. On the transition back to matched, request the
+		 * masterclock update that kvm_track_tsc_matching() would
+		 * have: a refresh in the mismatched window (another vCPU's
+		 * creation racing this sync) has latched use_master_clock
+		 * off, and nothing else ever re-runs it for a protected
+		 * guest.
+		 */
+		if (kvm->arch.nr_vcpus_matched_freq >=
+		    atomic_read(&kvm->online_vcpus) &&
+		    !kvm->arch.all_vcpus_matched_freq) {
+			kvm->arch.all_vcpus_matched_freq = true;
+			kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu);
+		}
 		return;
+	}
 
 	if (user_set_tsc)
 		vcpu->kvm->arch.user_set_tsc = true;
@@ -1357,6 +1437,16 @@ static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc,
 	 */
 	vcpu->arch.tsc_offset_adjustment = 0;
 
+	/*
+	 * If the offset changes, the guest-visible pvclock must be
+	 * rewritten before the vCPU runs again: its tsc_timestamp was
+	 * computed with the old offset, and the masterclock (which now
+	 * survives offset changes) will not request the refresh for a
+	 * matched write.
+	 */
+	if (offset != vcpu->arch.l1_tsc_offset)
+		kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu);
+
 	kvm_vcpu_write_tsc_offset(vcpu, offset);
 
 	if (!matched) {
@@ -1368,17 +1458,18 @@ static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc,
 		 * exact software computation in compute_guest_tsc()
 		 */
 		kvm->arch.cur_tsc_generation++;
+		WRITE_ONCE(kvm->arch.all_vcpus_matched_tsc, false);
+		kvm->arch.nr_vcpus_matched_tsc = 0;
 		kvm->arch.cur_tsc_nsec = ns;
 		kvm->arch.cur_tsc_write = tsc;
 		kvm->arch.cur_tsc_offset = offset;
-		kvm->arch.nr_vcpus_matched_tsc = 0;
-		kvm->arch.all_vcpus_matched_tsc = false;
-	} else if (vcpu->arch.this_tsc_generation != kvm->arch.cur_tsc_generation) {
+	}
+
+	if (vcpu->arch.this_tsc_generation != kvm->arch.cur_tsc_generation) {
+		vcpu->arch.this_tsc_generation = kvm->arch.cur_tsc_generation;
 		kvm->arch.nr_vcpus_matched_tsc++;
 	}
 
-	/* Keep track of which generation this VCPU has synchronized to */
-	vcpu->arch.this_tsc_generation = kvm->arch.cur_tsc_generation;
 	vcpu->arch.this_tsc_nsec = kvm->arch.cur_tsc_nsec;
 	vcpu->arch.this_tsc_write = kvm->arch.cur_tsc_write;
 
@@ -1596,10 +1687,10 @@ static void pvclock_update_vm_gtod_copy(struct kvm *kvm)
 #ifdef CONFIG_X86_64
 	struct kvm_arch *ka = &kvm->arch;
 	int vclock_mode;
-	bool host_tsc_clocksource, vcpus_matched;
+	bool host_tsc_clocksource, freqs_matched;
 
 	lockdep_assert_held(&kvm->arch.tsc_write_lock);
-	vcpus_matched = ka->all_vcpus_matched_tsc;
+	freqs_matched = ka->all_vcpus_matched_freq;
 
 	/*
 	 * If the host uses TSC clock, then passthrough TSC as stable
@@ -1609,7 +1700,7 @@ static void pvclock_update_vm_gtod_copy(struct kvm *kvm)
 					&ka->master_kernel_ns,
 					&ka->master_cycle_now);
 
-	ka->use_master_clock = host_tsc_clocksource && vcpus_matched
+	ka->use_master_clock = host_tsc_clocksource && freqs_matched
 				&& !ka->backwards_tsc_observed
 				&& !ka->boot_vcpu_runs_old_kvmclock;
 
@@ -1639,7 +1730,7 @@ static void pvclock_update_vm_gtod_copy(struct kvm *kvm)
 
 	vclock_mode = pvclock_gtod_data.clock.vclock_mode;
 	trace_kvm_update_master_clock(ka->use_master_clock, vclock_mode,
-					vcpus_matched);
+					freqs_matched);
 #endif
 }
 
@@ -1927,7 +2018,7 @@ int kvm_guest_time_update(struct kvm_vcpu *v)
 
 	/* If the host uses TSC clocksource, then it is stable */
 	hv_clock.flags = 0;
-	if (use_master_clock)
+	if (use_master_clock && READ_ONCE(ka->all_vcpus_matched_tsc))
 		hv_clock.flags |= PVCLOCK_TSC_STABLE_BIT;
 
 	if (vcpu->pv_time.active) {
@@ -9607,6 +9698,7 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
 	if (!vcpu->arch.guest_tsc_protected) {
 		raw_spin_lock_irqsave(&ka->tsc_write_lock, flags);
 		WRITE_ONCE(ka->all_vcpus_matched_tsc, false);
+		ka->all_vcpus_matched_freq = false;
 		raw_spin_unlock_irqrestore(&ka->tsc_write_lock, flags);
 	}
 
@@ -10095,6 +10187,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	ratelimit_state_init(&kvm->arch.kvmclock_update_rs, HZ, 10);
 	ratelimit_set_flags(&kvm->arch.kvmclock_update_rs, RATELIMIT_MSG_ON_RELEASE);
 	kvm->arch.kvmclock_offset = -get_kvmclock_base_ns();
+	kvm->arch.all_vcpus_matched_freq = true;
 
 	/*
 	 * Start with all (zero) vCPUs considered matched, so that a VM whose
-- 
2.43.0

[PATCH v13 05/26] KVM: x86: Factor out kvm_use_master_clock()

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Both kvm_track_tsc_matching() and pvclock_update_vm_gtod_copy() make a
decision about whether the KVM clock should be in master clock mode.
They used *different* criteria for the decision though. This isn't
really a problem; it only has the potential to cause unnecessary
invocations of KVM_REQ_MASTERCLOCK_UPDATE if the masterclock was
disabled due to TSC going backwards, or the guest using the old MSR.
But it isn't pretty.

Factor the decision out to a single function. And document the
historical reason why it's disabled for guests that use the old
MSR_KVM_SYSTEM_TIME.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Reviewed-by: Paul Durrant <paul@xen.org>
---
 arch/x86/kvm/x86.c | 40 ++++++++++++++++++++++++++++++----------
 1 file changed, 30 insertions(+), 10 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 08cc8ab3d699..9ad217e8640a 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1112,11 +1112,30 @@ static inline bool gtod_is_based_on_tsc(int mode)
 {
 	return mode == VDSO_CLOCKMODE_TSC || mode == VDSO_CLOCKMODE_HVCLOCK;
 }
-#endif
+
+static bool kvm_use_master_clock(struct kvm *kvm)
+{
+	struct kvm_arch *ka = &kvm->arch;
+
+	/*
+	 * The 'old kvmclock' check is a workaround (from 2015) for a
+	 * SUSE 2.6.16 kernel that didn't boot if the system_time in
+	 * its kvmclock was too far behind the current time. So the
+	 * mode of just setting the reference point and allowing time
+	 * to proceed linearly from there makes it fail to boot.
+	 * Despite that being kind of the *point* of the way the clock
+	 * is exposed to the guest. By coincidence, the offending
+	 * kernels used the old MSR_KVM_SYSTEM_TIME, which was moved
+	 * only because it resided in the wrong number range. So the
+	 * workaround is activated for *all* guests using the old MSR.
+	 */
+	return ka->all_vcpus_matched_freq &&
+		!ka->backwards_tsc_observed &&
+		!ka->boot_vcpu_runs_old_kvmclock;
+}
 
 static void kvm_track_tsc_matching(struct kvm_vcpu *vcpu, bool update_mclock)
 {
-#ifdef CONFIG_X86_64
 	struct kvm_arch *ka = &vcpu->kvm->arch;
 	struct pvclock_gtod_data *gtod = &pvclock_gtod_data;
 	bool prev_matched_tsc = ka->all_vcpus_matched_tsc;
@@ -1171,7 +1190,7 @@ static void kvm_track_tsc_matching(struct kvm_vcpu *vcpu, bool update_mclock)
 	 * are fine — each vCPU's pvclock has its own tsc_timestamp that
 	 * accounts for its offset.
 	 */
-	bool use_master_clock = ka->all_vcpus_matched_freq &&
+	bool use_master_clock = kvm_use_master_clock(vcpu->kvm) &&
 				gtod_is_based_on_tsc(gtod->clock.vclock_mode);
 
 	/*
@@ -1186,8 +1205,11 @@ static void kvm_track_tsc_matching(struct kvm_vcpu *vcpu, bool update_mclock)
 	trace_kvm_track_tsc(vcpu->vcpu_id, ka->nr_vcpus_matched_tsc,
 			    atomic_read(&vcpu->kvm->online_vcpus),
 		            ka->use_master_clock, gtod->clock.vclock_mode);
-#endif
 }
+#else
+static inline void kvm_track_tsc_matching(struct kvm_vcpu *vcpu,
+					  bool update_mclock) {}
+#endif
 
 /*
  * Multiply tsc by a fixed point number represented by ratio.
@@ -1687,10 +1709,9 @@ static void pvclock_update_vm_gtod_copy(struct kvm *kvm)
 #ifdef CONFIG_X86_64
 	struct kvm_arch *ka = &kvm->arch;
 	int vclock_mode;
-	bool host_tsc_clocksource, freqs_matched;
+	bool host_tsc_clocksource;
 
 	lockdep_assert_held(&kvm->arch.tsc_write_lock);
-	freqs_matched = ka->all_vcpus_matched_freq;
 
 	/*
 	 * If the host uses TSC clock, then passthrough TSC as stable
@@ -1700,9 +1721,8 @@ static void pvclock_update_vm_gtod_copy(struct kvm *kvm)
 					&ka->master_kernel_ns,
 					&ka->master_cycle_now);
 
-	ka->use_master_clock = host_tsc_clocksource && freqs_matched
-				&& !ka->backwards_tsc_observed
-				&& !ka->boot_vcpu_runs_old_kvmclock;
+	ka->use_master_clock = host_tsc_clocksource &&
+				kvm_use_master_clock(kvm);
 
 	if (ka->use_master_clock) {
 		u64 tsc_hz;
@@ -1730,7 +1750,7 @@ static void pvclock_update_vm_gtod_copy(struct kvm *kvm)
 
 	vclock_mode = pvclock_gtod_data.clock.vclock_mode;
 	trace_kvm_update_master_clock(ka->use_master_clock, vclock_mode,
-					freqs_matched);
+					ka->all_vcpus_matched_freq);
 #endif
 }
 
-- 
2.43.0

[PATCH v13 06/26] KVM: x86: Avoid gratuitous global clock updates

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Eliminate two sources of unnecessary KVM_REQ_GLOBAL_CLOCK_UPDATE:

1. kvm_write_system_time(): The global clock update was a workaround for
   ever-drifting clocks based on the host's CLOCK_MONOTONIC subject to
   NTP skew. Now that the KVM clock uses CLOCK_MONOTONIC_RAW, the clock
   does not drift with NTP corrections and there is no need to
   synchronize all vCPUs on boot or resume. Use KVM_REQ_CLOCK_UPDATE on
   the vCPU itself, and only when the clock is being enabled, not
   disabled.

2. kvm_arch_vcpu_load(): In master clock mode, migration between pCPUs
   does not require any clock update since the master clock reference is
   shared. Only request a local KVM_REQ_CLOCK_UPDATE for a load with no
   prior physical CPU (vcpu->cpu == -1: its first ever load, or a TDX
   re-association) to generate the pvclock parameters.
   In non-master-clock mode, keep the global update to synchronize all
   vCPUs.

A host in non-master-clock mode because its TSC is unstable still
re-anchors all vCPUs through the global update in
kvm_arch_vcpu_load(), which fires on every load there. What this
deliberately no longer covers is a stable-TSC host manually switched
to a non-TSC clocksource with pinned vCPUs, where vCPUs anchored far
apart can skew by the TSC calibration error over the elapsed time;
the removed broadcast only masked that when a guest happened to
rewrite MSR_KVM_SYSTEM_TIME, which a running guest does not do.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/kvm/msrs.c | 7 ++++---
 arch/x86/kvm/x86.c  | 4 +++-
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kvm/msrs.c b/arch/x86/kvm/msrs.c
index e91650af44c2..a2dffa1c706f 100644
--- a/arch/x86/kvm/msrs.c
+++ b/arch/x86/kvm/msrs.c
@@ -1285,14 +1285,15 @@ static void kvm_write_system_time(struct kvm_vcpu *vcpu, gpa_t system_time,
 	}
 
 	vcpu->arch.time = system_time;
-	kvm_make_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu);
 
 	/* we verify if the enable bit is set... */
-	if (system_time & 1)
+	if (system_time & 1) {
 		kvm_gpc_activate(&vcpu->arch.pv_time, system_time & ~1ULL,
 				 sizeof(struct pvclock_vcpu_time_info));
-	else
+		kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu);
+	} else {
 		kvm_gpc_deactivate(&vcpu->arch.pv_time);
+	}
 
 	return;
 }
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 9ad217e8640a..05c801973a7f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2765,11 +2765,13 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 		 * On a host with synchronized TSC, there is no need to update
 		 * kvmclock on vcpu->cpu migration
 		 */
-		if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu == -1) {
+		if (!vcpu->kvm->arch.use_master_clock) {
 			if (__ratelimit(&vcpu->kvm->arch.kvmclock_update_rs))
 				kvm_make_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu);
 			else
 				kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu);
+		} else if (vcpu->cpu == -1) {
+			kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu);
 		}
 
 		if (vcpu->cpu != cpu)
-- 
2.43.0

[PATCH v13 07/26] KVM: x86: Don't latch the Hyper-V TSC page from a superseded snapshot

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The Hyper-V TSC page is rewritten from kvm_guest_time_update() when its
HOST_CHANGED state is set, and latches to HV_TSC_PAGE_SET, which blocks
further rewrites. But the state is set before the new pvclock snapshot
is published (setting it takes hv_lock, a mutex, so it cannot happen
inside the tsc_write_lock section), and nothing validates the snapshot
the consumer is holding. Two interleavings latch stale parameters which
nothing then rewrites until some unrelated clock event invalidates the
page again:

 - A vCPU handling an unrelated KVM_REQ_CLOCK_UPDATE completes its
   pvclock read, is preempted (or sleeps in kvm_write_guest(), holding
   hv_lock), a KVM_SET_CLOCK completes meanwhile, and the vCPU then
   writes the page from the old snapshot and sets HV_TSC_PAGE_SET —
   consuming the state intended for the new one.

 - The state is consumed legitimately an instant before the publish,
   writing values which were current at that moment and superseded the
   next; the CLOCK_UPDATE broadcast which follows the publish finds
   HV_TSC_PAGE_SET and rewrites nothing.

For a KVM_SET_CLOCK stepping kvmclock_offset — a live migration
restore — the page is left disagreeing with the restored clock by the
full step.

Close the first at the consumer: kvm_hv_setup_tsc_page() revalidates
the pvclock seqcount under hv_lock — it must be under the lock, or the
consumer could be preempted between a check and the consumption — and
requeues the clock update rather than consume the state against a
superseded snapshot. Close the second by re-marking the page once the
new snapshot is published, with a CLOCK_UPDATE queued on every vCPU
behind the re-mark: the broadcast from kvm_end_pvclock_update() can
have been entirely consumed by then, and no vCPU in particular is
guaranteed to ever run again. The hv_lock mutex which lets a consumer
sleep across the whole update is also what serializes its write
against the re-mark, so the queued consumer always finds either fresh
page contents or the HOST_CHANGED state; guests which never enabled
the page skip the broadcast entirely.

Fixes: cc9cfddb0433 ("KVM: x86: hyper-v: Track Hyper-V TSC page status")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/kvm/hyperv.c | 20 ++++++++++++++++++--
 arch/x86/kvm/hyperv.h | 10 ++++++----
 arch/x86/kvm/x86.c    | 36 +++++++++++++++++++++++++++++++++++-
 3 files changed, 59 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c
index 5d2e3617c952..56da410a7f32 100644
--- a/arch/x86/kvm/hyperv.c
+++ b/arch/x86/kvm/hyperv.c
@@ -1182,9 +1182,11 @@ static inline bool tsc_page_update_unsafe(struct kvm_hv *hv)
 		hv->hv_tsc_emulation_control;
 }
 
-void kvm_hv_setup_tsc_page(struct kvm *kvm,
-			   struct pvclock_vcpu_time_info *hv_clock)
+void kvm_hv_setup_tsc_page(struct kvm_vcpu *v,
+			   struct pvclock_vcpu_time_info *hv_clock,
+			   unsigned int seq)
 {
+	struct kvm *kvm = v->kvm;
 	struct kvm_hv *hv = to_kvm_hv(kvm);
 	u32 tsc_seq;
 	u64 gfn;
@@ -1202,6 +1204,20 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm,
 	if (!(hv->hv_tsc_page & HV_X64_MSR_TSC_REFERENCE_ENABLE))
 		return;
 
+	/*
+	 * @hv_clock derives from the pvclock snapshot current at @seq.
+	 * Consuming the HOST_CHANGED state against a superseded snapshot
+	 * would latch stale parameters as HV_TSC_PAGE_SET, which nothing
+	 * then rewrites; the check must be made here, under hv_lock, or
+	 * this vCPU could be preempted between a check and the
+	 * consumption while an update completes. Leave the state alone
+	 * and requeue; the retry reads the new snapshot.
+	 */
+	if (read_seqcount_retry(&kvm->arch.pvclock_sc, seq)) {
+		kvm_make_request(KVM_REQ_CLOCK_UPDATE, v);
+		return;
+	}
+
 	gfn = hv->hv_tsc_page >> HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT;
 	/*
 	 * Because the TSC parameters only vary when there is a
diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h
index 622a6553e9ac..62e7922440aa 100644
--- a/arch/x86/kvm/hyperv.h
+++ b/arch/x86/kvm/hyperv.h
@@ -284,8 +284,9 @@ static inline bool kvm_hv_invtsc_suppressed(struct kvm_vcpu *vcpu)
 
 void kvm_hv_process_stimers(struct kvm_vcpu *vcpu);
 
-void kvm_hv_setup_tsc_page(struct kvm *kvm,
-			   struct pvclock_vcpu_time_info *hv_clock);
+void kvm_hv_setup_tsc_page(struct kvm_vcpu *v,
+			   struct pvclock_vcpu_time_info *hv_clock,
+			   unsigned int seq);
 void kvm_hv_request_tsc_page_update(struct kvm *kvm);
 
 void kvm_hv_xsaves_xsavec_maybe_warn(struct kvm_vcpu *vcpu);
@@ -377,8 +378,9 @@ static inline void kvm_hv_nested_transtion_tlb_flush(struct kvm_vcpu *vcpu,
 
 int kvm_hv_vcpu_flush_tlb(struct kvm_vcpu *vcpu);
 #else /* CONFIG_KVM_HYPERV */
-static inline void kvm_hv_setup_tsc_page(struct kvm *kvm,
-					 struct pvclock_vcpu_time_info *hv_clock) {}
+static inline void kvm_hv_setup_tsc_page(struct kvm_vcpu *v,
+					 struct pvclock_vcpu_time_info *hv_clock,
+					 unsigned int seq) {}
 static inline void kvm_hv_request_tsc_page_update(struct kvm *kvm) {}
 static inline void kvm_hv_xsaves_xsavec_maybe_warn(struct kvm_vcpu *vcpu) {}
 static inline void kvm_hv_init_vm(struct kvm *kvm) {}
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 05c801973a7f..f7af94d8b3bb 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1789,6 +1789,36 @@ static void kvm_end_pvclock_update(struct kvm *kvm)
 		kvm_clear_request(KVM_REQ_MCLOCK_INPROGRESS, vcpu);
 }
 
+/*
+ * Re-mark the Hyper-V TSC page once a new pvclock snapshot is published:
+ * the mark made before the update may have been legitimately consumed an
+ * instant before the publish, writing values the publish then superseded.
+ * The consumer must be queued after the re-mark: the CLOCK_UPDATE
+ * broadcast from kvm_end_pvclock_update() can have been entirely consumed
+ * by the time the re-mark lands, since a consumer may sleep in
+ * kvm_hv_setup_tsc_page() (hv_lock is a mutex) across the whole update —
+ * the same sleep which serializes it against this re-mark. Queue on
+ * every vCPU: any one suffices (the page is VM-wide), but none in
+ * particular is guaranteed to ever run again.
+ */
+static void kvm_hv_remark_tsc_page(struct kvm *kvm)
+{
+#ifdef CONFIG_KVM_HYPERV
+	kvm_hv_request_tsc_page_update(kvm);
+
+	/*
+	 * Racy, but benign: a transition to HV_TSC_PAGE_SET in the
+	 * instants after the re-mark means a consumer has just written
+	 * the page from the current snapshot, and a guest which never
+	 * enabled the page (UNSET/BROKEN, the common case) needs no
+	 * broadcast at all.
+	 */
+	if (READ_ONCE(to_kvm_hv(kvm)->hv_tsc_page_status) ==
+	    HV_TSC_PAGE_HOST_CHANGED)
+		kvm_make_all_cpus_request(kvm, KVM_REQ_CLOCK_UPDATE);
+#endif
+}
+
 static void kvm_update_masterclock(struct kvm *kvm)
 {
 	kvm_hv_request_tsc_page_update(kvm);
@@ -1961,6 +1991,7 @@ int kvm_guest_time_update(struct kvm_vcpu *v)
 	} while (read_seqcount_retry(&ka->pvclock_sc, seq));
 #else
 	const bool use_master_clock = false;
+	unsigned int seq = read_seqcount_begin(&ka->pvclock_sc);
 #endif
 	/*
 	 * Ensure reading the TSC+frequency pair is done on the same CPU.  When
@@ -2056,7 +2087,7 @@ int kvm_guest_time_update(struct kvm_vcpu *v)
 		hv_clock.flags &= ~PVCLOCK_GUEST_STOPPED;
 	}
 
-	kvm_hv_setup_tsc_page(v->kvm, &hv_clock);
+	kvm_hv_setup_tsc_page(v, &hv_clock, seq);
 
 #ifdef CONFIG_KVM_XEN
 	/*
@@ -4572,6 +4603,9 @@ static int kvm_vm_ioctl_set_clock(struct kvm *kvm, void __user *argp)
 		now_raw_ns = get_kvmclock_base_ns();
 	ka->kvmclock_offset = data.clock - now_raw_ns;
 	kvm_end_pvclock_update(kvm);
+
+	/* As in kvm_update_masterclock(): re-mark after publishing. */
+	kvm_hv_remark_tsc_page(kvm);
 	return 0;
 }
 
-- 
2.43.0

[PATCH v13 08/26] KVM: x86: Avoid redundant masterclock updates from multiple vCPUs

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

When a masterclock update is triggered (e.g. by the clocksource change
notifier), KVM_REQ_MASTERCLOCK_UPDATE is set on all vCPUs, and each of
them independently re-executed the whole pvclock_update_vm_gtod_copy()
sequence, serialized only by tsc_write_lock. Every redundant
re-snapshot moves the reference point.

Let the first vCPU to take the lock do the update and clear the
request on the others, which then bail out; the caller in
vcpu_enter_guest() uses the non-clearing kvm_test_request(), since the
clearing happens under the lock. Concurrent updaters are refcounted:
each broadcasts its KVM_REQ_MCLOCK_INPROGRESS kicks before contending
for the lock, and only the last one out releases them, so a finishing
updater cannot free vCPUs into the guest while another update is in
flight. The seqcount write section opens before the pending requests
are consumed — a vCPU which sees its request gone and heads into
kvm_guest_time_update() spins until the new snapshot is live — and
closes before the KVM_REQ_CLOCK_UPDATE broadcast, so readers do not
spin across the request loops. Those loops run with interrupts off
under tsc_write_lock: bounded by the vCPU count, paid once per
deduplicated update.

Since the update now consumes pending requests, the Hyper-V
reenlightenment notifier must invalidate the Hyper-V TSC page itself;
previously a surviving request refreshed the page later.

Deduplication also strips the paper off an old race on the Hyper-V
TSC page which the preceding commit fixed properly: every vCPU
re-running the broadcast used to re-mark the page by accident, so the
masterclock update paths must now re-mark it on purpose, as
KVM_SET_CLOCK already does, once the new snapshot is published.

Suggested-by: Dongli Zhang <dongli.zhang@oracle.com>
Tested-by: Dongli Zhang <dongli.zhang@oracle.com>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/include/asm/kvm_host.h |   1 +
 arch/x86/kvm/msrs.c             |  11 ++-
 arch/x86/kvm/x86.c              | 122 +++++++++++++++++++++++++++-----
 3 files changed, 115 insertions(+), 19 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 74306a81e882..3f5aa1d3f6bb 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1241,6 +1241,7 @@ struct kvm_arch {
 	raw_spinlock_t tsc_write_lock;
 	u32 cur_tsc_khz;
 	u64 cur_tsc_scaling_ratio;
+	int mclock_updaters; /* protected by tsc_write_lock */
 	u64 cur_tsc_nsec;
 	u64 cur_tsc_write;
 	u64 cur_tsc_offset;
diff --git a/arch/x86/kvm/msrs.c b/arch/x86/kvm/msrs.c
index a2dffa1c706f..26eb30f10c43 100644
--- a/arch/x86/kvm/msrs.c
+++ b/arch/x86/kvm/msrs.c
@@ -1278,10 +1278,15 @@ static void kvm_write_system_time(struct kvm_vcpu *vcpu, gpa_t system_time,
 	struct kvm_arch *ka = &vcpu->kvm->arch;
 
 	if (vcpu->vcpu_id == 0 && !host_initiated) {
-		if (ka->boot_vcpu_runs_old_kvmclock != old_msr)
+		/*
+		 * Store the flag before requesting the masterclock update:
+		 * the update may now be performed by another vCPU (which
+		 * consumes this request), and it must observe the new value.
+		 */
+		if (ka->boot_vcpu_runs_old_kvmclock != old_msr) {
+			ka->boot_vcpu_runs_old_kvmclock = old_msr;
 			kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu);
-
-		ka->boot_vcpu_runs_old_kvmclock = old_msr;
+		}
 	}
 
 	vcpu->arch.time = system_time;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index f7af94d8b3bb..0b3b04a156da 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1756,13 +1756,72 @@ static void pvclock_update_vm_gtod_copy(struct kvm *kvm)
 
 static void kvm_make_mclock_inprogress_request(struct kvm *kvm)
 {
+	unsigned long flags;
+
+	/*
+	 * Count updaters in flight: several can broadcast their
+	 * KVM_REQ_MCLOCK_INPROGRESS kicks and then contend for
+	 * tsc_write_lock, and whoever finishes (or bails out) first must
+	 * not clear the kicks out from under the updates still pending —
+	 * vCPUs would re-enter the guest mid-update and miss the
+	 * un-kicked KVM_REQ_CLOCK_UPDATE made at the end.
+	 */
+	raw_spin_lock_irqsave(&kvm->arch.tsc_write_lock, flags);
+	kvm->arch.mclock_updaters++;
+	raw_spin_unlock_irqrestore(&kvm->arch.tsc_write_lock, flags);
+
 	kvm_make_all_cpus_request(kvm, KVM_REQ_MCLOCK_INPROGRESS);
 }
 
-static void __kvm_start_pvclock_update(struct kvm *kvm)
+/* Drop an updater; the last one out re-allows guest entries. */
+static void kvm_put_mclock_updater(struct kvm *kvm)
+{
+	struct kvm_vcpu *vcpu;
+	unsigned long i;
+
+	lockdep_assert_held(&kvm->arch.tsc_write_lock);
+
+	if (--kvm->arch.mclock_updaters)
+		return;
+
+	kvm_for_each_vcpu(i, vcpu, kvm)
+		kvm_clear_request(KVM_REQ_MCLOCK_INPROGRESS, vcpu);
+}
+
+static bool __kvm_start_pvclock_update(struct kvm *kvm, struct kvm_vcpu *requesting_vcpu)
 {
+	struct kvm_vcpu *vcpu;
+	unsigned long i;
+
 	raw_spin_lock_irq(&kvm->arch.tsc_write_lock);
+
+	/*
+	 * If another vCPU already did the update while we were waiting
+	 * for the lock, our request will have been cleared. Bail out,
+	 * dropping this updater's hold on KVM_REQ_MCLOCK_INPROGRESS.
+	 */
+	if (requesting_vcpu &&
+	    !kvm_test_request(KVM_REQ_MASTERCLOCK_UPDATE, requesting_vcpu)) {
+		kvm_put_mclock_updater(kvm);
+		raw_spin_unlock_irq(&kvm->arch.tsc_write_lock);
+		return false;
+	}
+
+	/*
+	 * Open the seqcount before consuming the other vCPUs' requests:
+	 * a vCPU which sees its KVM_REQ_MASTERCLOCK_UPDATE gone and goes
+	 * straight to kvm_guest_time_update() must then spin on the
+	 * seqcount until the new snapshot is live, rather than read the
+	 * old one (and, via kvm_hv_setup_tsc_page(), consume the Hyper-V
+	 * TSC page's pending state against it).
+	 */
 	write_seqcount_begin(&kvm->arch.pvclock_sc);
+
+	/* The update is VM-wide; prevent other vCPUs from redoing it. */
+	kvm_for_each_vcpu(i, vcpu, kvm)
+		kvm_clear_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu);
+
+	return true;
 }
 
 static void kvm_start_pvclock_update(struct kvm *kvm)
@@ -1770,7 +1829,7 @@ static void kvm_start_pvclock_update(struct kvm *kvm)
 	kvm_make_mclock_inprogress_request(kvm);
 
 	/* no guest entries from this point */
-	__kvm_start_pvclock_update(kvm);
+	__kvm_start_pvclock_update(kvm, NULL);
 }
 
 static void kvm_end_pvclock_update(struct kvm *kvm)
@@ -1779,14 +1838,22 @@ static void kvm_end_pvclock_update(struct kvm *kvm)
 	struct kvm_vcpu *vcpu;
 	unsigned long i;
 
+	/*
+	 * The new snapshot is complete; close the seqcount first so
+	 * readers stop spinning while the requests are broadcast. The
+	 * CLOCK_UPDATE requests must precede the release of the
+	 * INPROGRESS holds, so that a vCPU allowed back into the guest
+	 * is guaranteed to refresh its pvclock on the way in.
+	 */
 	write_seqcount_end(&ka->pvclock_sc);
-	raw_spin_unlock_irq(&ka->tsc_write_lock);
+
 	kvm_for_each_vcpu(i, vcpu, kvm)
 		kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu);
 
-	/* guest entries allowed */
-	kvm_for_each_vcpu(i, vcpu, kvm)
-		kvm_clear_request(KVM_REQ_MCLOCK_INPROGRESS, vcpu);
+	/* guest entries allowed, unless another update is in flight */
+	kvm_put_mclock_updater(kvm);
+
+	raw_spin_unlock_irq(&ka->tsc_write_lock);
 }
 
 /*
@@ -1819,12 +1886,27 @@ static void kvm_hv_remark_tsc_page(struct kvm *kvm)
 #endif
 }
 
-static void kvm_update_masterclock(struct kvm *kvm)
+static void kvm_update_masterclock(struct kvm *kvm, struct kvm_vcpu *vcpu)
 {
+	/*
+	 * The Hyper-V TSC page request is made before the dedupe check
+	 * in __kvm_start_pvclock_update() because it takes a mutex and
+	 * so cannot sit inside the tsc_write_lock critical section where
+	 * the winning updater is known. A losing vCPU may therefore mark
+	 * the page HOST_CHANGED after the winner has already rewritten
+	 * it; the next clock update then rewrites the page again with
+	 * the same contents, which the TSC-page sequence protocol makes
+	 * safe against concurrently running guests.
+	 */
 	kvm_hv_request_tsc_page_update(kvm);
-	kvm_start_pvclock_update(kvm);
-	pvclock_update_vm_gtod_copy(kvm);
-	kvm_end_pvclock_update(kvm);
+	kvm_make_mclock_inprogress_request(kvm);
+
+	if (__kvm_start_pvclock_update(kvm, vcpu)) {
+		pvclock_update_vm_gtod_copy(kvm);
+		kvm_end_pvclock_update(kvm);
+
+		kvm_hv_remark_tsc_page(kvm);
+	}
 }
 
 /*
@@ -4604,7 +4686,6 @@ static int kvm_vm_ioctl_set_clock(struct kvm *kvm, void __user *argp)
 	ka->kvmclock_offset = data.clock - now_raw_ns;
 	kvm_end_pvclock_update(kvm);
 
-	/* As in kvm_update_masterclock(): re-mark after publishing. */
 	kvm_hv_remark_tsc_page(kvm);
 	return 0;
 }
@@ -6961,9 +7042,18 @@ static void kvm_hyperv_tsc_notifier(void)
 	kvm_caps.max_guest_tsc_khz = tsc_khz;
 
 	list_for_each_entry(kvm, &vm_list, vm_list) {
-		__kvm_start_pvclock_update(kvm);
+		/*
+		 * Invalidate the Hyper-V TSC page explicitly: the update
+		 * below consumes any pending KVM_REQ_MASTERCLOCK_UPDATE,
+		 * which would otherwise have refreshed the page from
+		 * kvm_update_masterclock().
+		 */
+		kvm_hv_request_tsc_page_update(kvm);
+		__kvm_start_pvclock_update(kvm, NULL);
 		pvclock_update_vm_gtod_copy(kvm);
 		kvm_end_pvclock_update(kvm);
+
+		kvm_hv_remark_tsc_page(kvm);
 	}
 
 	mutex_unlock(&kvm_lock);
@@ -8367,8 +8457,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 			kvm_mmu_free_obsolete_roots(vcpu);
 		if (kvm_check_request(KVM_REQ_MIGRATE_TIMER, vcpu))
 			__kvm_migrate_timers(vcpu);
-		if (kvm_check_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu))
-			kvm_update_masterclock(vcpu->kvm);
+		if (kvm_test_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu))
+			kvm_update_masterclock(vcpu->kvm, vcpu);
 		if (kvm_check_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu))
 			kvm_gen_kvmclock_update(vcpu);
 		if (kvm_check_request(KVM_REQ_CLOCK_UPDATE, vcpu)) {
@@ -9766,13 +9856,13 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
 		 * master clock does not stay in its previous mode with
 		 * running siblings keeping PVCLOCK_TSC_STABLE_BIT.
 		 */
-		kvm_update_masterclock(vcpu->kvm);
+		kvm_update_masterclock(vcpu->kvm, NULL);
 		return;
 	}
 	vcpu_load(vcpu);
 	kvm_synchronize_tsc(vcpu, NULL);
 	if (kvm_check_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu))
-		kvm_update_masterclock(vcpu->kvm);
+		kvm_update_masterclock(vcpu->kvm, NULL);
 	vcpu_put(vcpu);
 
 	/* poll control enabled by default */
-- 
2.43.0

[PATCH v13 09/26] KVM: x86: Stop TSC catchup when a frequency change no longer needs it

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

On hardware without TSC scaling, KVM_SET_TSC_KHZ with a frequency above
the host's puts the vCPU into always-catchup mode. Nothing ever takes
it back out: a later KVM_SET_TSC_KHZ restoring a frequency within
tsc_tolerance_ppm of the host's, where no scaling or catchup is needed,
leaves the vCPU paying the catchup overhead on every clock update for
the rest of its life — and KVM_HC_CLOCK_PAIRING stays disabled, since
it rejects always-catchup mode.

Clear the pairing whenever set_tsc_khz() succeeds with a frequency
which does not need it. Only that pairing: tsc_catchup alone also
serves the unstable-host-TSC compensation in kvm_arch_vcpu_load(),
which must not be disturbed.

Fixes: cc578287e322 ("KVM: Infrastructure for software and hardware based TSC rate scaling")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/kvm/x86.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 0b3b04a156da..1d4b66257b35 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1031,7 +1031,7 @@ static int set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz, bool scale)
 	/* Guest TSC same frequency as host TSC? */
 	if (!scale) {
 		kvm_vcpu_write_tsc_multiplier(vcpu, kvm_caps.default_tsc_scaling_ratio);
-		return 0;
+		goto clear_catchup;
 	}
 
 	/* TSC scaling supported? */
@@ -1057,6 +1057,19 @@ static int set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz, bool scale)
 	}
 
 	kvm_vcpu_write_tsc_multiplier(vcpu, ratio);
+
+clear_catchup:
+	/*
+	 * A frequency which no longer needs catchup must also stop it: a
+	 * previous above-host-rate setting on scaling-less hardware set
+	 * the always-catchup flags, and nothing else ever clears them.
+	 * Only undo that pairing — tsc_catchup alone also serves the
+	 * unstable-host-TSC compensation in kvm_arch_vcpu_load().
+	 */
+	if (vcpu->arch.tsc_always_catchup) {
+		vcpu->arch.tsc_always_catchup = 0;
+		vcpu->arch.tsc_catchup = 0;
+	}
 	return 0;
 }
 
-- 
2.43.0

[PATCH v13 10/26] KVM: x86: Re-synchronize TSC after KVM_SET_TSC_KHZ

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

KVM_SET_TSC_KHZ changes the vCPU's TSC scaling ratio but not the
VM-wide cur_tsc_scaling_ratio used by get_kvmclock(), which then
computes the KVM clock with a stale ratio and drifts away from what
the guest observes.

Re-synchronize the TSC after a frequency change. That updates the
VM-wide scaling ratio, keeps the guest TSC value continuous across the
change, and establishes the new generation for subsequent vCPUs to
match. Two cases are skipped: vCPU creation, which is not yet counted
in online_vcpus (the postcreate synchronization covers it), and an
unchanged frequency, which is documented as a no-op — tested before
anything which could fail, since creation commits a frequency even
when set_tsc_khz() cannot honour it, and a repeated set of that same
frequency must not turn into -EINVAL.

The preserved value is sampled as a {guest TSC, host clock} tuple and
the new offset is computed against the tuple's own reference point,
not against "now", so delay between sample and installation costs
nothing. Preemption stays disabled from the sample until
tsc_write_lock is held: this task's own preempt notifier is the one
writer which could apply a pending backwards-TSC compensation to the
offset the sample was read through, and one still pending at the lock
is folded in there. A failed ioctl commits nothing, so a retry does
not masquerade as a frequency change (creation commits regardless, as
it always has — its caller ignores the return value).

A kernel-internal call is distinguished from a userspace write by an
explicit user_initiated parameter: the internal call must not set
user_set_tsc and arm the 1-second slop heuristic before userspace has
ever written a TSC. Because the internal sample is exact it gets a 1ms
matching window instead of the legacy API's 1-second slop: a vCPU on
the generation's timeline still matches, but one which genuinely
diverged (a guest TSC_ADJUST write) keeps its exact value. Values
preserved from the old frequency diverge from the new generation's
line at |F1-F2| per unit time — the honest cost of preserving them.

Only a vCPU which has run, or whose TSC userspace explicitly wrote,
has a value worth preserving; track that with a new per-vCPU
user_set_tsc flag, since the VM-wide flag would let one vCPU's
restored TSC cause a sibling's unrestored creation-time value to be
preserved. A never-run vCPU is synchronized as for a zero write,
landing on the generation's timeline at the new frequency.

The arithmetic corners — a tuple sampled just before a sibling opens a
newer generation, the wrap-safe matching window, the zero clamp for an
unrun vCPU forced onto a younger generation — are documented where
they live, in __kvm_synchronize_tsc_at(). The virtual_tsc_mult/shift
pair is still published non-atomically, exactly as it always was. A
KVM_GET_CLOCK issued before the resulting masterclock update is
serviced still returns the correct time: the masterclock snapshot is
self-consistent, and the ns it yields is independent of the guest
frequency it encodes.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 Documentation/virt/kvm/api.rst  |   8 +
 arch/x86/include/asm/kvm_host.h |   1 +
 arch/x86/kvm/msrs.c             |   2 +-
 arch/x86/kvm/x86.c              | 284 ++++++++++++++++++++++++++++----
 arch/x86/kvm/x86.h              |   2 +-
 5 files changed, 260 insertions(+), 37 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index ddc61ea36ac5..9487afc4431f 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -2023,6 +2023,14 @@ If the KVM_CAP_VM_TSC_CONTROL capability is advertised, this can also
 be used as a vm ioctl to set the initial tsc frequency of subsequently
 created vCPUs.  Note, the vm ioctl is only allowed prior to creating vCPUs.
 
+Changing the frequency of a vCPU with the vcpu ioctl re-synchronizes
+its TSC: for a vCPU which has run, or whose TSC userspace has written,
+the current guest TSC value is preserved across the change (the TSC
+offset, as visible through the KVM_VCPU_TSC_OFFSET attribute, is
+rewritten accordingly); otherwise the vCPU is synchronized to the
+VM's existing TSC timeline at the new frequency. Setting the current
+frequency again is a no-op.
+
 For TSC protected Confidential Computing (CoCo) VMs where TSC frequency
 is configured once at VM scope and remains unchanged during VM's
 lifetime, the vm ioctl should be used to configure the TSC frequency
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 3f5aa1d3f6bb..773df58898ea 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -888,6 +888,7 @@ struct kvm_vcpu_arch {
 	u64 this_tsc_write;
 	u64 this_tsc_generation;
 	u64 this_tsc_freq_generation;
+	bool user_set_tsc;
 	bool tsc_catchup;
 	bool tsc_always_catchup;
 	s8 virtual_tsc_shift;
diff --git a/arch/x86/kvm/msrs.c b/arch/x86/kvm/msrs.c
index 26eb30f10c43..d07831b5dfb8 100644
--- a/arch/x86/kvm/msrs.c
+++ b/arch/x86/kvm/msrs.c
@@ -1645,7 +1645,7 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 		break;
 	case MSR_IA32_TSC:
 		if (msr_info->host_initiated) {
-			kvm_synchronize_tsc(vcpu, &data);
+			kvm_synchronize_tsc(vcpu, &data, true);
 		} else if (!vcpu->arch.guest_tsc_protected) {
 			u64 adj = kvm_compute_l1_tsc_offset(vcpu, rdtsc(), data) -
 				  vcpu->arch.l1_tsc_offset;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 1d4b66257b35..cf35363366f1 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1073,10 +1073,20 @@ static int set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz, bool scale)
 	return 0;
 }
 
-static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz)
+static void __kvm_synchronize_tsc_at(struct kvm_vcpu *vcpu, u64 *value,
+				     u64 host_tsc, s64 ns,
+				     bool user_initiated);
+
+static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz, bool init)
 {
+	bool freq_changed = (vcpu->arch.virtual_tsc_khz != user_tsc_khz);
 	u32 thresh_lo, thresh_hi;
+	u32 tsc_mult;
+	s8 tsc_shift;
 	int use_scaling = 0;
+	u64 tsc, host_tsc;
+	unsigned long flags;
+	s64 ns;
 
 	/* tsc_khz can be zero if TSC calibration fails */
 	if (user_tsc_khz == 0) {
@@ -1085,11 +1095,19 @@ static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz)
 		return -1;
 	}
 
+	/*
+	 * Setting the current frequency again is a no-op, as documented
+	 * — unconditionally, including a frequency the hardware could
+	 * not honour at vCPU creation (which commits it regardless):
+	 * re-running set_tsc_khz() would turn the documented no-op into
+	 * -EINVAL.
+	 */
+	if (!init && !freq_changed)
+		return 0;
+
 	/* Compute a scale to convert nanoseconds in TSC cycles */
 	kvm_get_time_scale(user_tsc_khz * 1000LL, NSEC_PER_SEC,
-			   &vcpu->arch.virtual_tsc_shift,
-			   &vcpu->arch.virtual_tsc_mult);
-	vcpu->arch.virtual_tsc_khz = user_tsc_khz;
+			   &tsc_shift, &tsc_mult);
 
 	/*
 	 * Compute the variation in TSC rate which is acceptable
@@ -1104,7 +1122,92 @@ static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz)
 			 user_tsc_khz, thresh_lo, thresh_hi);
 		use_scaling = 1;
 	}
-	return set_tsc_khz(vcpu, user_tsc_khz, use_scaling);
+
+	/*
+	 * Sample the guest TSC before changing the ratio, so continuity
+	 * can be preserved across the change — as a {guest TSC, host
+	 * clock} tuple, not a bare value: the synchronization below
+	 * computes the new offset against the tuple's own reference
+	 * point, so the guest TSC keeps counting through this function.
+	 * Preemption stays disabled from the sample until tsc_write_lock
+	 * is held — not for timing, but because this task's own preempt
+	 * notifier (kvm_sched_in -> kvm_arch_vcpu_load) is the one
+	 * writer which could apply a pending backwards-TSC compensation
+	 * to the offset the sample was read through; a compensation
+	 * still pending when the lock is taken is folded in there.
+	 */
+	if (!init) {
+		preempt_disable();
+#ifdef CONFIG_X86_64
+		if (!kvm_get_time_and_clockread(&ns, &host_tsc))
+#endif
+		{
+			unsigned long irqflags;
+
+			/*
+			 * The two reads must form one coherent reference
+			 * point.
+			 */
+			local_irq_save(irqflags);
+			host_tsc = rdtsc();
+			ns = get_kvmclock_base_ns();
+			local_irq_restore(irqflags);
+		}
+		tsc = kvm_read_l1_tsc(vcpu, host_tsc);
+	}
+
+	/*
+	 * Commit the new frequency only once set_tsc_khz() can no longer
+	 * fail: a failed ioctl must leave no trace, or a retry with the
+	 * still-active frequency would wrongly look like a change. The
+	 * exception is vCPU creation (@init), whose caller ignores the
+	 * return value: commit the requested frequency even when the
+	 * hardware cannot scale to it, as creation always has, rather
+	 * than leave the vCPU with the zeroed frequency from allocation
+	 * (KVM_GET_TSC_KHZ would report 0, and the catchup path would
+	 * compute with a zero mult).
+	 */
+	if (set_tsc_khz(vcpu, user_tsc_khz, use_scaling) && !init) {
+		preempt_enable();
+		return -1;
+	}
+
+	vcpu->arch.virtual_tsc_shift = tsc_shift;
+	vcpu->arch.virtual_tsc_mult = tsc_mult;
+	vcpu->arch.virtual_tsc_khz = user_tsc_khz;
+
+	/*
+	 * At vCPU creation (@init) the vCPU is not yet counted in
+	 * online_vcpus, so synchronizing here would corrupt the VM-wide
+	 * matching state (and leave it corrupted if creation subsequently
+	 * fails). The initial synchronization happens in
+	 * kvm_arch_vcpu_postcreate() instead.
+	 */
+	if (init)
+		return 0;
+
+	/*
+	 * Changing the frequency changes the vCPU's position on the
+	 * generation's cycle timeline, so re-synchronize. For a vCPU
+	 * which has run, preserve the guest TSC value read above across
+	 * the change; likewise if userspace has written a TSC on this
+	 * vCPU — a zero write would discard it. Otherwise the vCPU holds
+	 * only its creation-time synchronization, so synchronize as for
+	 * a zero write. When the generation is already at the new
+	 * frequency this places the vCPU on its timeline rather than
+	 * opening a gratuitous new generation and orphaning the matched
+	 * state of its siblings; the first vCPU to adopt a new frequency
+	 * necessarily opens a new generation, starting at zero — mixed
+	 * frequencies within a VM forfeit the master clock anyway.
+	 */
+	raw_spin_lock_irqsave(&vcpu->kvm->arch.tsc_write_lock, flags);
+	preempt_enable();
+	if (vcpu->arch.last_vmentry_cpu != -1 || vcpu->arch.user_set_tsc)
+		__kvm_synchronize_tsc_at(vcpu, &tsc, host_tsc, ns, false);
+	else
+		__kvm_synchronize_tsc_at(vcpu, NULL, host_tsc, ns, false);
+	raw_spin_unlock_irqrestore(&vcpu->kvm->arch.tsc_write_lock, flags);
+	return 0;
 }
 
 static u64 compute_guest_tsc(struct kvm_vcpu *vcpu, s64 kernel_ns)
@@ -1458,8 +1561,10 @@ static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc,
 		return;
 	}
 
-	if (user_set_tsc)
+	if (user_set_tsc) {
 		vcpu->kvm->arch.user_set_tsc = true;
+		vcpu->arch.user_set_tsc = true;
+	}
 
 	vcpu->arch.last_guest_tsc = tsc;
 
@@ -1511,40 +1616,84 @@ static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc,
 	kvm_track_tsc_matching(vcpu, !matched);
 }
 
-void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value)
+/*
+ * Synchronize the vCPU's TSC against an explicit {host_tsc, ns} reference
+ * point: @value (when non-NULL) is the guest TSC *at* @host_tsc, not at
+ * some unspecified "now". The offset is computed against that reference,
+ * so a sampled value stays valid however much time passes between the
+ * sampling and this call.
+ */
+static void __kvm_synchronize_tsc_at(struct kvm_vcpu *vcpu, u64 *value,
+				     u64 host_tsc, s64 ns,
+				     bool user_initiated)
 {
-	u64 data = user_value ? *user_value : 0;
+	u64 data = value ? *value : 0;
 	struct kvm *kvm = vcpu->kvm;
-	u64 offset, host_tsc, elapsed;
-	s64 ns;
-	unsigned long flags;
+	u64 elapsed_cycles = 0;
+	s64 elapsed;
+	u64 offset;
 	bool matched = false;
 	bool synchronizing = false;
 
-	raw_spin_lock_irqsave(&kvm->arch.tsc_write_lock, flags);
+	lockdep_assert_held(&kvm->arch.tsc_write_lock);
 
-#ifdef CONFIG_X86_64
-	if (!kvm_get_time_and_clockread(&ns, &host_tsc))
-#endif
-	{
-		host_tsc = rdtsc();
-		ns = get_kvmclock_base_ns();
-	}
+	/*
+	 * A backwards-TSC compensation pending now was not in a sampled
+	 * @value: it is applied to the vCPU's offset only at vcpu_load,
+	 * and the sample was read through the uncompensated offset. Fold
+	 * it in (scaled as adjust_tsc_offset_host() would), since
+	 * __kvm_synchronize_tsc() clears the pending adjustment as
+	 * superseded by the absolute offset installed here. A
+	 * user-initiated value is the desired guest TSC *now*, and needs
+	 * no correction.
+	 */
+	if (unlikely(vcpu->arch.tsc_offset_adjustment) &&
+	    value && !user_initiated)
+		data += kvm_scale_tsc(vcpu->arch.tsc_offset_adjustment,
+				      vcpu->arch.l1_tsc_scaling_ratio);
 
 	offset = kvm_compute_l1_tsc_offset(vcpu, host_tsc, data);
+
+	/*
+	 * The {host_tsc, ns} reference may have been sampled before a
+	 * concurrent synchronization (on another vCPU) opened a newer
+	 * generation, making the elapsed time negative. The anchored
+	 * offset above is still exact either way; extrapolate the
+	 * generation's line backwards for the matching comparison, so
+	 * that the error stays at |F1-F2| times the sampling skew.
+	 * Snapping to the generation's origin instead would inflate it
+	 * to the full guest frequency times the skew, spuriously
+	 * splitting the generation when two vCPUs race through
+	 * KVM_SET_TSC_KHZ.
+	 */
 	elapsed = ns - kvm->arch.cur_tsc_nsec;
 
 	if (vcpu->arch.virtual_tsc_khz) {
-		if (data == 0) {
+		/*
+		 * Keep the cycle count as a magnitude with the sign taken
+		 * from @elapsed itself: a genuinely huge positive elapsed
+		 * time (a u32-limit frequency and a weeks-old generation)
+		 * must extrapolate mod 2^64, as the guest's real TSC
+		 * would wrap, not read as negative.
+		 */
+		elapsed_cycles = nsec_to_cycles(vcpu, elapsed >= 0 ? elapsed
+								   : -elapsed);
+
+		if (!value || (data == 0 && user_initiated)) {
 			/*
-			 * Force synchronization when creating a vCPU, or when
-			 * userspace explicitly writes a zero value.
+			 * Force synchronization when creating a vCPU, or
+			 * when userspace explicitly writes a zero value —
+			 * but not for an internal preserved sample which
+			 * happens to be exactly zero: NULL, not zero, is
+			 * the no-value case.
 			 */
 			synchronizing = true;
-		} else if (kvm->arch.user_set_tsc) {
-			u64 tsc_exp = kvm->arch.cur_tsc_write +
-						nsec_to_cycles(vcpu, elapsed);
+		} else if (kvm->arch.user_set_tsc || !user_initiated) {
+			u64 tsc_exp = elapsed >= 0 ?
+				kvm->arch.cur_tsc_write + elapsed_cycles :
+				kvm->arch.cur_tsc_write - elapsed_cycles;
 			u64 tsc_hz = vcpu->arch.virtual_tsc_khz * 1000LL;
+			u64 window;
 			/*
 			 * Here lies UAPI baggage: when a user-initiated TSC write has
 			 * a small delta (1 second) of virtual cycle time against the
@@ -1559,9 +1708,35 @@ void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value)
 			 * to this 'correction' to make it sync up with values that only
 			 * come from the kernel's default vCPU creation. Make the 1-second
 			 * slop hack only trigger if the user_set_tsc flag is already set.
+			 *
+			 * A kernel-internal write (a frequency change preserving the
+			 * guest TSC) carries an exact value sampled with its own
+			 * reference point, not a racy userspace one; a vCPU on the
+			 * generation's line still matches it to within arithmetic
+			 * rounding, but one which genuinely diverged (e.g. by a guest
+			 * TSC_ADJUST write) must not be snapped back by up to a
+			 * second. Use a 1ms window for those, unconditionally —
+			 * the sample's exactness owes nothing to the legacy API's
+			 * flag. A frequency *change* still only matches siblings
+			 * converted within 1ms·F2/|F1-F2| of the first: values
+			 * preserved from the old frequency genuinely diverge from
+			 * the new generation's line at |F1-F2| per unit time.
 			 */
-			synchronizing = data < tsc_exp + tsc_hz &&
-					data + tsc_hz > tsc_exp;
+			/*
+			 * 1ms of cycles is virtual_tsc_khz by definition;
+			 * dividing tsc_hz would need libgcc's __udivdi3
+			 * on i386.
+			 */
+			window = user_initiated ? tsc_hz :
+						  vcpu->arch.virtual_tsc_khz;
+			/*
+			 * Wrap-safe |data - tsc_exp| < window: tsc_exp
+			 * may itself have wrapped when elapsed_cycles is
+			 * negative and exceeds the generation's origin
+			 * value; the difference is still correct mod 2^64.
+			 */
+			synchronizing = (u64)(data - tsc_exp + window) <
+					2 * window;
 		}
 	}
 
@@ -1579,10 +1754,19 @@ void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value)
 		 * "now": the matching window slides forward with wall time,
 		 * anchored on the generation's origin alone (a matched
 		 * write no longer updates any stored reference point).
+		 * A negative elapsed time (the tuple sampled just
+		 * before a sibling opened this generation) must not wrap
+		 * below zero: the wrapped value would land in
+		 * last_guest_tsc, and an unstable-TSC host would
+		 * re-anchor the guest TSC to it at the next vcpu_load.
 		 */
-		u64 delta = nsec_to_cycles(vcpu, elapsed);
-
-		data = kvm->arch.cur_tsc_write + delta;
+		if (elapsed < 0 &&
+		    elapsed_cycles > kvm->arch.cur_tsc_write)
+			data = 0;
+		else if (elapsed < 0)
+			data = kvm->arch.cur_tsc_write - elapsed_cycles;
+		else
+			data = kvm->arch.cur_tsc_write + elapsed_cycles;
 
 		if (!kvm_check_tsc_unstable())
 			offset = kvm->arch.cur_tsc_offset;
@@ -1591,8 +1775,38 @@ void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value)
 		matched = true;
 	}
 
-	__kvm_synchronize_tsc(vcpu, offset, data, ns, matched, !!user_value);
-	raw_spin_unlock_irqrestore(&kvm->arch.tsc_write_lock, flags);
+	__kvm_synchronize_tsc(vcpu, offset, data, ns, matched, user_initiated);
+}
+
+static void kvm_synchronize_tsc_at(struct kvm_vcpu *vcpu, u64 *value,
+				   u64 host_tsc, s64 ns, bool user_initiated)
+{
+	unsigned long flags;
+
+	raw_spin_lock_irqsave(&vcpu->kvm->arch.tsc_write_lock, flags);
+	__kvm_synchronize_tsc_at(vcpu, value, host_tsc, ns, user_initiated);
+	raw_spin_unlock_irqrestore(&vcpu->kvm->arch.tsc_write_lock, flags);
+}
+
+void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *value, bool user_initiated)
+{
+	u64 host_tsc;
+	s64 ns;
+
+#ifdef CONFIG_X86_64
+	if (!kvm_get_time_and_clockread(&ns, &host_tsc))
+#endif
+	{
+		unsigned long flags;
+
+		/* The two reads must form one coherent reference point. */
+		local_irq_save(flags);
+		host_tsc = rdtsc();
+		ns = get_kvmclock_base_ns();
+		local_irq_restore(flags);
+	}
+
+	kvm_synchronize_tsc_at(vcpu, value, host_tsc, ns, user_initiated);
 }
 
 #ifdef CONFIG_X86_64
@@ -4035,7 +4249,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 		if (user_tsc_khz == 0)
 			user_tsc_khz = tsc_khz;
 
-		if (!kvm_set_tsc_khz(vcpu, user_tsc_khz))
+		if (!kvm_set_tsc_khz(vcpu, user_tsc_khz, false))
 			r = 0;
 
 		goto out;
@@ -9814,7 +10028,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 	kvm_xen_init_vcpu(vcpu);
 	vcpu_load(vcpu);
 	kvm_vcpu_after_set_cpuid(vcpu);
-	kvm_set_tsc_khz(vcpu, vcpu->kvm->arch.default_tsc_khz);
+	kvm_set_tsc_khz(vcpu, vcpu->kvm->arch.default_tsc_khz, true);
 	kvm_vcpu_reset(vcpu, false);
 	kvm_init_mmu(vcpu);
 	vcpu_put(vcpu);
@@ -9873,7 +10087,7 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
 		return;
 	}
 	vcpu_load(vcpu);
-	kvm_synchronize_tsc(vcpu, NULL);
+	kvm_synchronize_tsc(vcpu, NULL, false);
 	if (kvm_check_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu))
 		kvm_update_masterclock(vcpu->kvm, NULL);
 	vcpu_put(vcpu);
diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
index 436e2fb1396c..de4c86ea5a7d 100644
--- a/arch/x86/kvm/x86.h
+++ b/arch/x86/kvm/x86.h
@@ -330,7 +330,7 @@ uint64_t kvm_get_wall_clock_epoch(struct kvm *kvm);
 bool kvm_get_monotonic_and_clockread(s64 *kernel_ns, u64 *tsc_timestamp);
 int kvm_guest_time_update(struct kvm_vcpu *v);
 
-void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value);
+void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *value, bool user_initiated);
 u64 kvm_scale_tsc(u64 tsc, u64 ratio);
 u64 kvm_read_l1_tsc(struct kvm_vcpu *vcpu, u64 host_tsc);
 u64 kvm_calc_nested_tsc_offset(u64 l1_offset, u64 l2_offset, u64 l2_multiplier);
-- 
2.43.0

[PATCH v13 11/26] KVM: x86: Cache host vclock_mode for masterclock eligibility checks

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The remaining users of pvclock_gtod_data only need to know whether
the host clocksource is TSC-based. Cache the vclock_mode value in a
simple kvm_host_vclock_mode variable, updated by the pvclock_gtod_notify
callback, and use gtod_is_based_on_tsc() for eligibility checks.

This is inherently racy (as it always was — kvm_track_tsc_matching
never held the gtod seqcount), relying on eventual consistency: the
notifier fires on every timekeeping update and will correct any
transient inconsistency within one tick.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/kvm/x86.c | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index cf35363366f1..f7dd4129dc51 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1224,6 +1224,8 @@ static u64 compute_guest_tsc(struct kvm_vcpu *vcpu, s64 kernel_ns)
 }
 
 #ifdef CONFIG_X86_64
+static int kvm_host_vclock_mode;
+
 static inline bool gtod_is_based_on_tsc(int mode)
 {
 	return mode == VDSO_CLOCKMODE_TSC || mode == VDSO_CLOCKMODE_HVCLOCK;
@@ -1253,7 +1255,6 @@ static bool kvm_use_master_clock(struct kvm *kvm)
 static void kvm_track_tsc_matching(struct kvm_vcpu *vcpu, bool update_mclock)
 {
 	struct kvm_arch *ka = &vcpu->kvm->arch;
-	struct pvclock_gtod_data *gtod = &pvclock_gtod_data;
 	bool prev_matched_tsc = ka->all_vcpus_matched_tsc;
 	bool prev_matched_freq = ka->all_vcpus_matched_freq;
 
@@ -1306,8 +1307,9 @@ static void kvm_track_tsc_matching(struct kvm_vcpu *vcpu, bool update_mclock)
 	 * are fine — each vCPU's pvclock has its own tsc_timestamp that
 	 * accounts for its offset.
 	 */
+	int host_vclock_mode = READ_ONCE(kvm_host_vclock_mode);
 	bool use_master_clock = kvm_use_master_clock(vcpu->kvm) &&
-				gtod_is_based_on_tsc(gtod->clock.vclock_mode);
+				gtod_is_based_on_tsc(host_vclock_mode);
 
 	/*
 	 * Request a masterclock update if needed: toggling master clock,
@@ -1320,7 +1322,7 @@ static void kvm_track_tsc_matching(struct kvm_vcpu *vcpu, bool update_mclock)
 
 	trace_kvm_track_tsc(vcpu->vcpu_id, ka->nr_vcpus_matched_tsc,
 			    atomic_read(&vcpu->kvm->online_vcpus),
-		            ka->use_master_clock, gtod->clock.vclock_mode);
+			    ka->use_master_clock, host_vclock_mode);
 }
 #else
 static inline void kvm_track_tsc_matching(struct kvm_vcpu *vcpu,
@@ -1491,7 +1493,7 @@ static inline bool kvm_check_tsc_unstable(void)
 	 * TSC is marked unstable when we're running on Hyper-V,
 	 * 'TSC page' clocksource is good.
 	 */
-	if (pvclock_gtod_data.clock.vclock_mode == VDSO_CLOCKMODE_HVCLOCK)
+	if (READ_ONCE(kvm_host_vclock_mode) == VDSO_CLOCKMODE_HVCLOCK)
 		return false;
 #endif
 	return check_tsc_unstable();
@@ -1935,7 +1937,6 @@ static void pvclock_update_vm_gtod_copy(struct kvm *kvm)
 {
 #ifdef CONFIG_X86_64
 	struct kvm_arch *ka = &kvm->arch;
-	int vclock_mode;
 	bool host_tsc_clocksource;
 
 	lockdep_assert_held(&kvm->arch.tsc_write_lock);
@@ -1975,9 +1976,9 @@ static void pvclock_update_vm_gtod_copy(struct kvm *kvm)
 			ka->use_master_clock = false;
 	}
 
-	vclock_mode = pvclock_gtod_data.clock.vclock_mode;
-	trace_kvm_update_master_clock(ka->use_master_clock, vclock_mode,
-					ka->all_vcpus_matched_freq);
+	trace_kvm_update_master_clock(ka->use_master_clock,
+				      READ_ONCE(kvm_host_vclock_mode),
+				      ka->all_vcpus_matched_freq);
 #endif
 }
 
@@ -7452,17 +7453,23 @@ static DEFINE_IRQ_WORK(pvclock_irq_work, pvclock_irq_work_fn);
 static int pvclock_gtod_notify(struct notifier_block *nb, unsigned long unused,
 			       void *priv)
 {
-	struct pvclock_gtod_data *gtod = &pvclock_gtod_data;
 	struct timekeeper *tk = priv;
 
 	update_pvclock_gtod(tk);
 
+	/*
+	 * Pairs with the READ_ONCE() in the lockless readers; the race is
+	 * benign by design, relying on eventual consistency (the notifier
+	 * fires on every timekeeping update).
+	 */
+	WRITE_ONCE(kvm_host_vclock_mode, tk->tkr_mono.clock->vdso_clock_mode);
+
 	/*
 	 * Disable master clock if host does not trust, or does not use,
 	 * TSC based clocksource. Delegate queue_work() to irq_work as
 	 * this is invoked with tk_core.seq write held.
 	 */
-	if (!gtod_is_based_on_tsc(gtod->clock.vclock_mode) &&
+	if (!gtod_is_based_on_tsc(kvm_host_vclock_mode) &&
 	    atomic_read(&kvm_guest_has_master_clock) != 0)
 		irq_work_queue(&pvclock_irq_work);
 	return 0;
-- 
2.43.0

[PATCH v13 12/26] KVM: x86: Remove pvclock_gtod_data and private timekeeping code

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Remove the now-unused KVM-private timekeeping infrastructure:

 - struct pvclock_clock and struct pvclock_gtod_data
 - update_pvclock_gtod() and its seqcount-protected state copy
   (the private readers were already removed when their last callers
   went away)

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/kvm/x86.c | 56 ----------------------------------------------
 1 file changed, 56 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index f7dd4129dc51..bd82c9b38233 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -889,60 +889,6 @@ static inline bool kvm_vcpu_exit_request(struct kvm_vcpu *vcpu)
 	       kvm_request_pending(vcpu) || xfer_to_guest_mode_work_pending();
 }
 
-#ifdef CONFIG_X86_64
-struct pvclock_clock {
-	int vclock_mode;
-	u64 cycle_last;
-	u64 mask;
-	u32 mult;
-	u32 shift;
-	u64 base_cycles;
-	u64 offset;
-};
-
-struct pvclock_gtod_data {
-	seqcount_t	seq;
-
-	struct pvclock_clock clock; /* extract of a clocksource struct */
-	struct pvclock_clock raw_clock; /* extract of a clocksource struct */
-
-	ktime_t		offs_boot;
-	u64		wall_time_sec;
-};
-
-static struct pvclock_gtod_data pvclock_gtod_data;
-
-static void update_pvclock_gtod(struct timekeeper *tk)
-{
-	struct pvclock_gtod_data *vdata = &pvclock_gtod_data;
-
-	write_seqcount_begin(&vdata->seq);
-
-	/* copy pvclock gtod data */
-	vdata->clock.vclock_mode	= tk->tkr_mono.clock->vdso_clock_mode;
-	vdata->clock.cycle_last		= tk->tkr_mono.cycle_last;
-	vdata->clock.mask		= tk->tkr_mono.mask;
-	vdata->clock.mult		= tk->tkr_mono.mult;
-	vdata->clock.shift		= tk->tkr_mono.shift;
-	vdata->clock.base_cycles	= tk->tkr_mono.xtime_nsec;
-	vdata->clock.offset		= tk->tkr_mono.base;
-
-	vdata->raw_clock.vclock_mode	= tk->tkr_raw.clock->vdso_clock_mode;
-	vdata->raw_clock.cycle_last	= tk->tkr_raw.cycle_last;
-	vdata->raw_clock.mask		= tk->tkr_raw.mask;
-	vdata->raw_clock.mult		= tk->tkr_raw.mult;
-	vdata->raw_clock.shift		= tk->tkr_raw.shift;
-	vdata->raw_clock.base_cycles	= tk->tkr_raw.xtime_nsec;
-	vdata->raw_clock.offset		= tk->tkr_raw.base;
-
-	vdata->wall_time_sec            = tk->xtime_sec;
-
-	vdata->offs_boot		= tk->offs_boot;
-
-	write_seqcount_end(&vdata->seq);
-}
-#endif
-
 static s64 get_kvmclock_base_ns(void)
 {
 	/* Count up from boot time, but with the frequency of the raw clock.  */
@@ -7455,8 +7401,6 @@ static int pvclock_gtod_notify(struct notifier_block *nb, unsigned long unused,
 {
 	struct timekeeper *tk = priv;
 
-	update_pvclock_gtod(tk);
-
 	/*
 	 * Pairs with the READ_ONCE() in the lockless readers; the race is
 	 * benign by design, relying on eventual consistency (the notifier
-- 
2.43.0

[PATCH v13 13/26] KVM: x86: Keep the master clock trustworthy while the VM has no vCPUs

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The master clock snapshot is taken at VM creation and, now that it no
longer requires matched vCPU TSCs, is active from that moment:
KVM_GET_CLOCK returns valid {host_tsc, realtime} fields before any
vCPU is created, as will KVM_[GS]ET_CLOCK_GUEST later in this series.
But nothing yet defends that snapshot through the window in which the
VM has no vCPUs.

Sample kvmclock_offset inside the tsc_write_lock section, immediately
adjacent to the snapshot it will forever be combined with, so the
kvmclock epoch is anchored to the same instant rather than to some
earlier point the task was preempted at.

No TSC generation is established at VM creation: cur_tsc_khz stays
zero, which no vCPU frequency ever equals, so the first vCPU opens its
own generation with its TSC starting at zero, as it always has. A
matchable VM-creation generation would be a silent guest-visible
change (a first-vCPU TSC equal to the host cycles elapsed since
KVM_CREATE_VM), and would hand the first vCPU a stale offset when a
host suspend resets the TSC before any vCPU exists — the backwards-TSC
compensation detects the reset only through existing vCPUs.

The same no-vCPU suspend window can leave the snapshot itself stale:
nothing refreshes it (pvclock_gtod_update_fn() also iterates vCPUs),
and the post-reset host TSC may read below master_cycle_now,
underflowing the delta, or above it, silently dropping the sleep
period. Make __get_kvmclock_master_clock() fall back to the raw clock
when the snapshot disagrees with it by more than a second — both sides
computed from one coherent {ns, TSC} sample, so a reader preempted
mid-ioctl cannot fake a discrepancy — but only while the VM still has
no vCPUs: the raw clock's and the masterclock's multipliers quantize
the frequency ~0.1ppm apart, which would trip the threshold after a
few months of an undisturbed snapshot and make the host-side clock
jump while the guest's pvclock sails on.

Finally, make the masterclock refresh in kvm_arch_vcpu_postcreate()
unconditional and synchronous for the first vCPU, so the snapshot is
valid even for userspace which queries the clock before ever running
a vCPU: a clocksource change notifies only VMs with vCPUs, and a
change back to TSC leaves nothing to disagree with the stale snapshot.
The same call services a KVM_REQ_MASTERCLOCK_UPDATE raised by any
later vCPU's postcreate synchronization, which would otherwise wait
until that vCPU first entered the guest.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/kvm/x86.c | 120 ++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 103 insertions(+), 17 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index bd82c9b38233..743da71cd84d 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1716,6 +1716,14 @@ static void __kvm_synchronize_tsc_at(struct kvm_vcpu *vcpu, u64 *value,
 		else
 			data = kvm->arch.cur_tsc_write + elapsed_cycles;
 
+		/*
+		 * The generation's offset can be reused directly: it is in
+		 * this vCPU's scaling domain, because every generation is
+		 * opened by a vCPU (the initial VM-creation generation is
+		 * not matchable) and set_tsc_khz() derives the scaling
+		 * ratio deterministically from the kHz value, so two vCPUs
+		 * at the same frequency carry the same ratio.
+		 */
 		if (!kvm_check_tsc_unstable())
 			offset = kvm->arch.cur_tsc_offset;
 		else
@@ -2107,6 +2115,7 @@ static bool __get_kvmclock_master_clock(struct kvm *kvm,
 	struct kvm_arch *ka = &kvm->arch;
 	struct pvclock_vcpu_time_info hv_clock;
 	struct timespec64 ts;
+	u64 clock;
 
 	if (!ka->use_master_clock)
 		return false;
@@ -2114,10 +2123,6 @@ static bool __get_kvmclock_master_clock(struct kvm *kvm,
 	if (!kvm_get_walltime_and_clockread(&ts, &data->host_tsc))
 		return false;
 
-	data->realtime = ts.tv_nsec + NSEC_PER_SEC * ts.tv_sec;
-	data->flags |= KVM_CLOCK_REALTIME | KVM_CLOCK_HOST_TSC |
-		       KVM_CLOCK_TSC_STABLE;
-
 	hv_clock.tsc_timestamp = ka->master_cycle_now;
 	hv_clock.system_time = ka->master_kernel_ns + ka->kvmclock_offset;
 
@@ -2132,13 +2137,63 @@ static bool __get_kvmclock_master_clock(struct kvm *kvm,
 		u64 tsc_delta = data->host_tsc - ka->master_cycle_now;
 
 		tsc_delta = kvm_scale_tsc(tsc_delta, ka->master_tsc_scaling_ratio);
-		data->clock = hv_clock.system_time +
-			      pvclock_scale_delta(tsc_delta,
-						  hv_clock.tsc_to_system_mul,
-						  hv_clock.tsc_shift);
+		clock = hv_clock.system_time +
+			pvclock_scale_delta(tsc_delta,
+					    hv_clock.tsc_to_system_mul,
+					    hv_clock.tsc_shift);
 	} else {
-		data->clock = __pvclock_read_cycles(&hv_clock, data->host_tsc);
+		clock = __pvclock_read_cycles(&hv_clock, data->host_tsc);
+	}
+
+	/*
+	 * While the VM has no vCPUs, nothing can refresh the snapshot
+	 * after a host suspend resets the TSC (both the resume
+	 * compensation and the clocksource notifier work through vCPUs),
+	 * and firmware may re-init the TSC to an arbitrary value —
+	 * including one below master_cycle_now, which would underflow the
+	 * delta. Sanity-check the snapshot against the raw clock in that
+	 * window, and use the fallback clock rather than return a
+	 * discontinuity flagged KVM_CLOCK_TSC_STABLE. Both sides of the
+	 * comparison derive from one coherent {ns, TSC} sample, so a
+	 * preempted reader cannot fake a discrepancy. (A reader racing
+	 * the first vCPU's creation can see online_vcpus already raised
+	 * before the postcreate refresh has run and skip the check
+	 * against a still-stale snapshot; that narrow window existed in
+	 * full before the check did.) Once vCPUs exist
+	 * the check must not run: the raw clock's and the masterclock's
+	 * multipliers quantize the frequency differently, drifting apart
+	 * by ~0.1ppm, which would trip a 1-second threshold after a few
+	 * months of an undisturbed snapshot and make the host-side clock
+	 * jump while the guest's pvclock sails on. A VM genuinely parked
+	 * with no vCPUs for those months serves the fallback clock from
+	 * then on — the conservative direction for a snapshot that old.
+	 */
+	if (unlikely(!atomic_read(&kvm->online_vcpus))) {
+		u64 chk_tsc, delta;
+		s64 chk_ns, diff;
+
+		if (!kvm_get_time_and_clockread(&chk_ns, &chk_tsc))
+			return false;
+
+		delta = chk_tsc - ka->master_cycle_now;
+		if (kvm_caps.has_tsc_control)
+			delta = kvm_scale_tsc(delta,
+					      ka->master_tsc_scaling_ratio);
+		/*
+		 * Two-sided compare rather than abs(): abs(S64_MIN) is
+		 * S64_MIN again, which would fail open on that one value.
+		 */
+		diff = (s64)pvclock_scale_delta(delta, ka->master_tsc_mul,
+						ka->master_tsc_shift) -
+		       (s64)(chk_ns - ka->master_kernel_ns);
+		if (diff > (s64)NSEC_PER_SEC || diff < -(s64)NSEC_PER_SEC)
+			return false;
 	}
+
+	data->clock = clock;
+	data->realtime = ts.tv_nsec + NSEC_PER_SEC * ts.tv_sec;
+	data->flags |= KVM_CLOCK_REALTIME | KVM_CLOCK_HOST_TSC |
+		       KVM_CLOCK_TSC_STABLE;
 	return true;
 #else
 	return false;
@@ -10039,8 +10094,19 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
 	}
 	vcpu_load(vcpu);
 	kvm_synchronize_tsc(vcpu, NULL, false);
-	if (kvm_check_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu))
+
+	/*
+	 * Refresh the VM-creation masterclock snapshot (see
+	 * kvm_arch_init_vm), synchronously so that it is valid even for
+	 * userspace which queries the clock before ever running a vCPU.
+	 * Likewise service any masterclock update the synchronization
+	 * above requested on this vCPU, which would otherwise languish
+	 * until it first enters the guest.
+	 */
+	if (kvm_check_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu) ||
+	    vcpu->vcpu_idx == 0)
 		kvm_update_masterclock(vcpu->kvm, NULL);
+
 	vcpu_put(vcpu);
 
 	/* poll control enabled by default */
@@ -10510,9 +10576,13 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	seqcount_raw_spinlock_init(&kvm->arch.pvclock_sc, &kvm->arch.tsc_write_lock);
 	ratelimit_state_init(&kvm->arch.kvmclock_update_rs, HZ, 10);
 	ratelimit_set_flags(&kvm->arch.kvmclock_update_rs, RATELIMIT_MSG_ON_RELEASE);
-	kvm->arch.kvmclock_offset = -get_kvmclock_base_ns();
-	kvm->arch.all_vcpus_matched_freq = true;
 
+	kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz;
+	kvm->arch.apic_bus_cycle_ns = APIC_BUS_CYCLE_NS_DEFAULT;
+	kvm->arch.guest_can_read_msr_platform_info = true;
+	kvm->arch.enable_pmu = enable_pmu && !kvm->arch.has_protected_pmu;
+
+	kvm->arch.all_vcpus_matched_freq = true;
 	/*
 	 * Start with all (zero) vCPUs considered matched, so that a VM whose
 	 * vCPUs never reach kvm_track_tsc_matching() — notably when the guest
@@ -10529,14 +10599,30 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	kvm->arch.cur_tsc_scaling_ratio = kvm_caps.default_tsc_scaling_ratio;
 
 	raw_spin_lock_irqsave(&kvm->arch.tsc_write_lock, flags);
+
+	/*
+	 * Take the master clock snapshot now, so that KVM_GET_CLOCK works
+	 * before any vCPU exists. It is refreshed again when the first
+	 * vCPU is created, in case the host clocksource changed (and
+	 * changed back) in between: pvclock_gtod_update_fn() does not
+	 * notify a VM with no vCPUs.
+	 *
+	 * No TSC generation is established here: cur_tsc_khz stays zero,
+	 * which no vCPU frequency ever equals, so the first vCPU opens
+	 * its own generation with its TSC starting at zero, exactly as
+	 * it always has. Letting it match a VM-creation generation would
+	 * make the first vCPU's TSC start at the cycles elapsed since
+	 * KVM_CREATE_VM — a silent guest-visible change over an unbounded
+	 * window — and would hand it a stale offset when a host suspend
+	 * reset the TSC in between, since the backwards-TSC compensation
+	 * in kvm_arch_enable_virtualization_cpu() detects the jump only
+	 * through existing vCPUs.
+	 */
+	kvm->arch.kvmclock_offset = -get_kvmclock_base_ns();
+
 	pvclock_update_vm_gtod_copy(kvm);
 	raw_spin_unlock_irqrestore(&kvm->arch.tsc_write_lock, flags);
 
-	kvm->arch.default_tsc_khz = max_tsc_khz ? : tsc_khz;
-	kvm->arch.apic_bus_cycle_ns = APIC_BUS_CYCLE_NS_DEFAULT;
-	kvm->arch.guest_can_read_msr_platform_info = true;
-	kvm->arch.enable_pmu = enable_pmu && !kvm->arch.has_protected_pmu;
-
 #if IS_ENABLED(CONFIG_HYPERV)
 	spin_lock_init(&kvm->arch.hv_root_tdp_lock);
 	kvm->arch.hv_root_tdp = INVALID_PAGE;
-- 
2.43.0

[PATCH v13 14/26] UAPI: x86: Move pvclock-abi to UAPI for x86 platforms

[David Woodhouse]

From: Jack Allister <jalliste@amazon.com>

A subsequent commit will provide a new KVM interface for performing a
fixup/correction of the KVM clock against the reference TSC. The
KVM_[GS]ET_CLOCK_GUEST API requires a pvclock_vcpu_time_info, as such
the caller must know about this definition.

Move the definition to the UAPI folder so that it is exported to
usermode, change the type definitions to use the standard for UAPI
exports, and add the Linux-syscall-note exception to the SPDX licence
identifier as is conventional for exported headers.

Define __KERNEL__ in scripts/xen-hypercalls.sh: the UAPI header now
pulls in linux/types.h, and the script's raw $CPP invocation with
$LINUXINCLUDE does not define it, which would trigger the "attempt to
use kernel headers from user space" warning from the UAPI type
headers.

Signed-off-by: Jack Allister <jalliste@amazon.com>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Reviewed-by: Paul Durrant <paul@xen.org>
Tested-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 MAINTAINERS                                   |  4 +--
 arch/x86/include/{ => uapi}/asm/pvclock-abi.h | 27 ++++++++++---------
 scripts/xen-hypercalls.sh                     |  2 +-
 3 files changed, 18 insertions(+), 15 deletions(-)
 rename arch/x86/include/{ => uapi}/asm/pvclock-abi.h (82%)

diff --git a/MAINTAINERS b/MAINTAINERS
index 9fadc98d3c49..773f7c800f6a 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -14663,8 +14663,8 @@ S:	Supported
 T:	git git://git.kernel.org/pub/scm/virt/kvm/kvm.git
 F:	arch/um/include/asm/kvm_para.h
 F:	arch/x86/include/asm/kvm_para.h
-F:	arch/x86/include/asm/pvclock-abi.h
 F:	arch/x86/include/uapi/asm/kvm_para.h
+F:	arch/x86/include/uapi/asm/pvclock-abi.h
 F:	arch/x86/kernel/kvm.c
 F:	arch/x86/kernel/kvmclock.c
 F:	include/asm-generic/kvm_para.h
@@ -29780,8 +29780,8 @@ R:	Boris Ostrovsky <boris.ostrovsky@oracle.com>
 L:	xen-devel@lists.xenproject.org (moderated for non-subscribers)
 S:	Supported
 F:	arch/x86/configs/xen.config
-F:	arch/x86/include/asm/pvclock-abi.h
 F:	arch/x86/include/asm/xen/
+F:	arch/x86/include/uapi/asm/pvclock-abi.h
 F:	arch/x86/platform/pvh/
 F:	arch/x86/xen/
 
diff --git a/arch/x86/include/asm/pvclock-abi.h b/arch/x86/include/uapi/asm/pvclock-abi.h
similarity index 82%
rename from arch/x86/include/asm/pvclock-abi.h
rename to arch/x86/include/uapi/asm/pvclock-abi.h
index b9fece5fc96d..6d70cf640362 100644
--- a/arch/x86/include/asm/pvclock-abi.h
+++ b/arch/x86/include/uapi/asm/pvclock-abi.h
@@ -1,6 +1,9 @@
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
 #ifndef _ASM_X86_PVCLOCK_ABI_H
 #define _ASM_X86_PVCLOCK_ABI_H
+
+#include <linux/types.h>
+
 #ifndef __ASSEMBLER__
 
 /*
@@ -24,20 +27,20 @@
  */
 
 struct pvclock_vcpu_time_info {
-	u32   version;
-	u32   pad0;
-	u64   tsc_timestamp;
-	u64   system_time;
-	u32   tsc_to_system_mul;
-	s8    tsc_shift;
-	u8    flags;
-	u8    pad[2];
+	__u32   version;
+	__u32   pad0;
+	__u64   tsc_timestamp;
+	__u64   system_time;
+	__u32   tsc_to_system_mul;
+	__s8    tsc_shift;
+	__u8    flags;
+	__u8    pad[2];
 } __attribute__((__packed__)); /* 32 bytes */
 
 struct pvclock_wall_clock {
-	u32   version;
-	u32   sec;
-	u32   nsec;
+	__u32   version;
+	__u32   sec;
+	__u32   nsec;
 } __attribute__((__packed__));
 
 #define PVCLOCK_TSC_STABLE_BIT	(1 << 0)
diff --git a/scripts/xen-hypercalls.sh b/scripts/xen-hypercalls.sh
index f18b00843df3..51a722198997 100755
--- a/scripts/xen-hypercalls.sh
+++ b/scripts/xen-hypercalls.sh
@@ -5,7 +5,7 @@ shift
 in="$@"
 
 for i in $in; do
-	eval $CPP $LINUXINCLUDE -dD -imacros "$i" -x c /dev/null
+	eval $CPP -D__KERNEL__ $LINUXINCLUDE -dD -imacros "$i" -x c /dev/null
 done | \
 awk '$1 == "#define" && $2 ~ /__HYPERVISOR_[a-z][a-z_0-9]*/ { v[$3] = $2 }
 	END {   print "/* auto-generated by scripts/xen-hypercall.sh */"
-- 
2.43.0

[PATCH v13 15/26] KVM: selftests: Use UAPI pvclock-abi.h in xen_shinfo_test

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Now that pvclock-abi.h is a UAPI header, use it in xen_shinfo_test.c
instead of locally duplicating the struct definitions.

Drop the tools/ mirror of the header: with the kernel copy now UAPI,
the selftests resolve <asm/pvclock-abi.h> from the installed kernel
headers, which they already depend on, and nothing else in tools/
builds against the mirror — keeping it would only be a file for
check-headers.sh to police with no consumer. The UAPI struct fields
are __u64 (unsigned long long), so switch the verbose printfs away
from PRIu64, which is "lu" on 64-bit.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 tools/arch/x86/include/asm/pvclock-abi.h      | 48 -------------------
 .../selftests/kvm/x86/xen_shinfo_test.c       | 33 ++++---------
 2 files changed, 10 insertions(+), 71 deletions(-)
 delete mode 100644 tools/arch/x86/include/asm/pvclock-abi.h

diff --git a/tools/arch/x86/include/asm/pvclock-abi.h b/tools/arch/x86/include/asm/pvclock-abi.h
deleted file mode 100644
index b9fece5fc96d..000000000000
--- a/tools/arch/x86/include/asm/pvclock-abi.h
+++ /dev/null
@@ -1,48 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _ASM_X86_PVCLOCK_ABI_H
-#define _ASM_X86_PVCLOCK_ABI_H
-#ifndef __ASSEMBLER__
-
-/*
- * These structs MUST NOT be changed.
- * They are the ABI between hypervisor and guest OS.
- * Both Xen and KVM are using this.
- *
- * pvclock_vcpu_time_info holds the system time and the tsc timestamp
- * of the last update. So the guest can use the tsc delta to get a
- * more precise system time.  There is one per virtual cpu.
- *
- * pvclock_wall_clock references the point in time when the system
- * time was zero (usually boot time), thus the guest calculates the
- * current wall clock by adding the system time.
- *
- * Protocol for the "version" fields is: hypervisor raises it (making
- * it uneven) before it starts updating the fields and raises it again
- * (making it even) when it is done.  Thus the guest can make sure the
- * time values it got are consistent by checking the version before
- * and after reading them.
- */
-
-struct pvclock_vcpu_time_info {
-	u32   version;
-	u32   pad0;
-	u64   tsc_timestamp;
-	u64   system_time;
-	u32   tsc_to_system_mul;
-	s8    tsc_shift;
-	u8    flags;
-	u8    pad[2];
-} __attribute__((__packed__)); /* 32 bytes */
-
-struct pvclock_wall_clock {
-	u32   version;
-	u32   sec;
-	u32   nsec;
-} __attribute__((__packed__));
-
-#define PVCLOCK_TSC_STABLE_BIT	(1 << 0)
-#define PVCLOCK_GUEST_STOPPED	(1 << 1)
-/* PVCLOCK_COUNTS_FROM_ZERO broke ABI and can't be used anymore. */
-#define PVCLOCK_COUNTS_FROM_ZERO (1 << 2)
-#endif /* __ASSEMBLER__ */
-#endif /* _ASM_X86_PVCLOCK_ABI_H */
diff --git a/tools/testing/selftests/kvm/x86/xen_shinfo_test.c b/tools/testing/selftests/kvm/x86/xen_shinfo_test.c
index aaa414c677ad..9045f03cf2d3 100644
--- a/tools/testing/selftests/kvm/x86/xen_shinfo_test.c
+++ b/tools/testing/selftests/kvm/x86/xen_shinfo_test.c
@@ -15,6 +15,8 @@
 
 #include <sys/eventfd.h>
 
+#include <asm/pvclock-abi.h>
+
 #define SHINFO_REGION_GVA	0xc0000000ULL
 #define SHINFO_REGION_GPA	0xc0000000ULL
 #define SHINFO_REGION_SLOT	10
@@ -98,23 +100,6 @@ struct sched_poll {
 	u64 timeout;
 };
 
-struct pvclock_vcpu_time_info {
-	u32   version;
-	u32   pad0;
-	u64   tsc_timestamp;
-	u64   system_time;
-	u32   tsc_to_system_mul;
-	s8    tsc_shift;
-	u8    flags;
-	u8    pad[2];
-} __attribute__((__packed__)); /* 32 bytes */
-
-struct pvclock_wall_clock {
-	u32   version;
-	u32   sec;
-	u32   nsec;
-} __attribute__((__packed__));
-
 struct vcpu_runstate_info {
 	u32 state;
 	u64 state_entry_time;
@@ -987,12 +972,14 @@ int main(int argc, char *argv[])
 
 	if (verbose) {
 		printf("Wall clock (v %d) %d.%09d\n", wc->version, wc->sec, wc->nsec);
-		printf("Time info 1: v %u tsc %" PRIu64 " time %" PRIu64 " mul %u shift %u flags %x\n",
-		       ti->version, ti->tsc_timestamp, ti->system_time, ti->tsc_to_system_mul,
-		       ti->tsc_shift, ti->flags);
-		printf("Time info 2: v %u tsc %" PRIu64 " time %" PRIu64 " mul %u shift %u flags %x\n",
-		       ti2->version, ti2->tsc_timestamp, ti2->system_time, ti2->tsc_to_system_mul,
-		       ti2->tsc_shift, ti2->flags);
+		printf("Time info 1: v %u tsc %llu time %llu mul %u shift %d flags %x\n",
+		       ti->version, (unsigned long long)ti->tsc_timestamp,
+		       (unsigned long long)ti->system_time,
+		       ti->tsc_to_system_mul, ti->tsc_shift, ti->flags);
+		printf("Time info 2: v %u tsc %llu time %llu mul %u shift %d flags %x\n",
+		       ti2->version, (unsigned long long)ti2->tsc_timestamp,
+		       (unsigned long long)ti2->system_time,
+		       ti2->tsc_to_system_mul, ti2->tsc_shift, ti2->flags);
 	}
 
 	TEST_ASSERT(wc->version && !(wc->version & 1),
-- 
2.43.0

[PATCH v13 16/26] KVM: x86: Add KVM_[GS]ET_CLOCK_GUEST for accurate KVM clock migration

[David Woodhouse]

From: Jack Allister <jalliste@amazon.com>

In the common case (where kvm->arch.use_master_clock is true), the KVM
clock is defined as a simple arithmetic function of the guest TSC, based
on a reference point stored in kvm->arch.master_kernel_ns and
kvm->arch.master_cycle_now.

The existing KVM_[GS]ET_CLOCK functionality does not allow for this
relationship to be precisely saved and restored by userspace. All it can
currently do is set the KVM clock at a given UTC reference time, which
is necessarily imprecise.

So on live update, the guest TSC can remain cycle accurate at precisely
the same offset from the host TSC, but there is no way for userspace to
restore the KVM clock accurately.

Even on live migration to a new host, where the accuracy of the guest
time-keeping is fundamentally limited by the accuracy of wallclock
synchronization between the source and destination hosts, the clock jump
experienced by the guest's TSC and its KVM clock should at least be
*consistent*. Even when the guest TSC suffers a discontinuity, its KVM
clock should still remain the *same* arithmetic function of the guest
TSC, and not suffer an *additional* discontinuity.

To allow for accurate migration of the KVM clock, add per-vCPU ioctls
which save and restore the actual PV clock info in
pvclock_vcpu_time_info.

The restoration in KVM_SET_CLOCK_GUEST works by creating a new reference
point in time just as kvm_update_masterclock() does, and calculating the
corresponding guest TSC value. This guest TSC value is then passed
through the user-provided pvclock structure to generate the *intended*
KVM clock value at that point in time, and through the *actual* KVM
clock calculation. Then kvm->arch.kvmclock_offset is adjusted to
eliminate the difference.

Both ioctls return -ENODATA when the KVM clock is not in master clock
mode (because the host TSC is unreliable, or the guest TSCs are
configured strangely); the KVM clock is then not a function of the
guest TSC at all, so userspace shall fall back to the legacy
KVM_[GS]ET_CLOCK, whose imprecision is acceptable in a mode which is
imprecise anyway. That means sending the results of *both* GET ioctls
with the migration data, unless the intent is to refuse to resume on a
host with a bad TSC. An unknown TSC frequency on the current CPU
(possible only without CONSTANT_TSC) also reads as -ENODATA. The
ioctls are advertised with KVM_CAP_CLOCK_GUEST.

KVM_GET_CLOCK_GUEST services a masterclock update pending on the vCPU
before reading: a backwards host TSC on resume parks the request and
leaves the snapshot stale, while vcpu_load() has already applied the
offset compensation — combining the two yields a value whose frequency
still matches, which the destination could not reject. Servicing the
update makes that case correctly return -ENODATA.

The set side validates everything it is given — padding, version (must
be zero), flags (nothing beyond PVCLOCK_TSC_STABLE_BIT) — so the fields
keep meaning for the future, and re-marks the Hyper-V TSC page once the
new clock is published, with a consumer queued behind the re-mark, as
kvm_update_masterclock() does.

Co-developed-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Jack Allister <jalliste@amazon.com>
Reviewed-by: Paul Durrant <paul@xen.org>
Cc: Dongli Zhang <dongli.zhang@oracle.com>
Tested-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 Documentation/virt/kvm/api.rst  |  56 ++++++++
 arch/x86/include/uapi/asm/kvm.h |   1 +
 arch/x86/kvm/x86.c              | 223 ++++++++++++++++++++++++++++++++
 include/uapi/linux/kvm.h        |   4 +
 4 files changed, 284 insertions(+)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 9487afc4431f..98f5da057b86 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -6767,6 +6767,62 @@ if the guest_memfd memory was pinned in IOMMU page tables.
 
 See also: :ref:`KVM_SET_MEMORY_ATTRIBUTES`.
 
+4.147 KVM_GET_CLOCK_GUEST
+----------------------------
+
+:Capability: KVM_CAP_CLOCK_GUEST
+:Architectures: x86_64
+:Type: vcpu ioctl
+:Parameters: struct pvclock_vcpu_time_info (out)
+:Returns: 0 on success, <0 on error
+
+Retrieves the current time information structure used for KVM/PV clocks,
+in precisely the form advertised to the guest vCPU, which gives parameters
+for a direct conversion from a guest TSC value to nanoseconds.
+
+The structure is in terms of the given vCPU's TSC. It must be passed back
+to KVM_SET_CLOCK_GUEST on the *same* vCPU (on the destination VM, for
+migration); vCPUs may legitimately have different TSC offsets, which would
+otherwise be silently folded into the restored KVM clock.
+
+When the KVM clock is not in "master clock" mode, for example because the
+host TSC is unreliable or the guest TSCs are oddly configured, the KVM clock
+is actually defined by the host CLOCK_MONOTONIC_RAW instead of the guest TSC.
+In this case, the KVM_GET_CLOCK_GUEST ioctl returns -ENODATA.
+
+4.148 KVM_SET_CLOCK_GUEST
+----------------------------
+
+:Capability: KVM_CAP_CLOCK_GUEST
+:Architectures: x86_64
+:Type: vcpu ioctl
+:Parameters: struct pvclock_vcpu_time_info (in)
+:Returns: 0 on success, <0 on error
+
+Sets the KVM clock (for the whole VM) in terms of the vCPU TSC, using the
+pvclock structure as returned by KVM_GET_CLOCK_GUEST. This allows the precise
+arithmetic relationship between guest TSC and KVM clock to be preserved by
+userspace across migration.
+
+When the KVM clock is not in "master clock" mode, and the KVM clock is actually
+defined by the host CLOCK_MONOTONIC_RAW, this ioctl returns -ENODATA. Userspace
+may choose to set the clock using the less precise KVM_SET_CLOCK ioctl, or may
+choose to fail, denying migration to a host whose TSC is misbehaving. A host
+whose CPU TSC frequency is unknown (only possible without
+X86_FEATURE_CONSTANT_TSC) is never in "master clock" mode, so it also returns
+-ENODATA.
+
+If the mul/shift fields of the provided pvclock structure encode a TSC
+frequency inconsistent with the vCPU's actual TSC frequency, this ioctl
+returns -ERANGE.
+
+If the vCPU's current TSC is behind the tsc_timestamp in the provided
+structure, this ioctl returns -EINVAL. Restore the guest TSC (which the
+documented migration procedure keeps monotonic) before invoking it.
+The padding fields must be zero, the version field must be zero, and
+no flags beyond PVCLOCK_TSC_STABLE_BIT are accepted; anything else
+returns -EINVAL.
+
 .. _kvm_run:
 
 5. The kvm_run structure
diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kvm.h
index 1585ec804066..9e3cdc3adb65 100644
--- a/arch/x86/include/uapi/asm/kvm.h
+++ b/arch/x86/include/uapi/asm/kvm.h
@@ -12,6 +12,7 @@
 #include <linux/types.h>
 #include <linux/ioctl.h>
 #include <linux/stddef.h>
+#include <asm/pvclock-abi.h>
 
 #define KVM_PIO_PAGE_OFFSET 1
 #define KVM_COALESCED_MMIO_PAGE_OFFSET 2
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 743da71cd84d..16383c71131d 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -2777,6 +2777,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_MEMORY_FAULT_INFO:
 	case KVM_CAP_X86_GUEST_MODE:
 	case KVM_CAP_ONE_REG:
+#ifdef CONFIG_X86_64
+	case KVM_CAP_CLOCK_GUEST:
+#endif
 		r = 1;
 		break;
 	case KVM_CAP_PRE_FAULT_MEMORY:
@@ -3941,6 +3944,218 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
 	}
 }
 
+#ifdef CONFIG_X86_64
+static int kvm_vcpu_ioctl_get_clock_guest(struct kvm_vcpu *v, void __user *argp)
+{
+	struct pvclock_vcpu_time_info hv_clock = {};
+	struct kvm_arch *ka = &v->kvm->arch;
+	bool use_master_clock;
+	unsigned int seq;
+
+	/*
+	 * Construct the pvclock purely from the master clock state. The
+	 * master mul/shift are computed for the guest TSC frequency, which
+	 * in master clock mode is the frequency of every vCPU; only the
+	 * tsc_timestamp is per-vCPU, translating the master snapshot into
+	 * this vCPU's TSC domain via its scaling ratio and offset.
+	 *
+	 * Note, this deliberately does NOT invoke kvm_guest_time_update(),
+	 * which would write the guest's pvclock pages: dirtying guest
+	 * memory after the final KVM_RUN would break post-copy migration
+	 * flows. The pvclock returned here describes the same clock as the
+	 * guest-visible copy (the same linear function of the guest TSC),
+	 * anchored at the current master clock snapshot.
+	 *
+	 * Translating the snapshot through the vCPU's *current* scaling
+	 * ratio and offset is correct even if they changed after the
+	 * snapshot was taken: the snapshot anchors a host TSC value, not
+	 * a guest one.
+	 */
+
+	/*
+	 * Service a pending masterclock update first. Events like a
+	 * backwards host TSC on resume, or a frequency change, park
+	 * KVM_REQ_MASTERCLOCK_UPDATE on the vCPUs and leave the snapshot
+	 * — and use_master_clock itself — stale until a guest entry.
+	 * vcpu_load() has already applied any pending backwards-TSC
+	 * offset adjustment to this vCPU, and combining that with the
+	 * pre-suspend snapshot would produce a value the destination
+	 * cannot reject: the frequency still matches.
+	 */
+	if (kvm_test_request(KVM_REQ_MASTERCLOCK_UPDATE, v))
+		kvm_update_masterclock(v->kvm, v);
+
+	do {
+		seq = read_seqcount_begin(&ka->pvclock_sc);
+
+		use_master_clock = ka->use_master_clock;
+		if (!use_master_clock)
+			continue;
+
+		hv_clock.tsc_timestamp = kvm_read_l1_tsc(v, ka->master_cycle_now);
+		hv_clock.system_time = ka->master_kernel_ns + ka->kvmclock_offset;
+		hv_clock.tsc_shift = ka->master_tsc_shift;
+		hv_clock.tsc_to_system_mul = ka->master_tsc_mul;
+		hv_clock.flags = READ_ONCE(ka->all_vcpus_matched_tsc) ?
+				 PVCLOCK_TSC_STABLE_BIT : 0;
+	} while (read_seqcount_retry(&ka->pvclock_sc, seq));
+
+	if (!use_master_clock)
+		return -ENODATA;
+
+	if (copy_to_user(argp, &hv_clock, sizeof(hv_clock)))
+		return -EFAULT;
+
+	return 0;
+}
+
+/*
+ * Reverse the calculation in the hv_clock definition.
+ *
+ * time_ns = ( (cycles << shift) * mul ) >> 32;
+ * (although shift can be negative, so that's bad C)
+ *
+ * So for a single second,
+ * NSEC_PER_SEC = ( ( FREQ_HZ << shift) * mul ) >> 32
+ * NSEC_PER_SEC << 32 = ( FREQ_HZ << shift ) * mul
+ * ( NSEC_PER_SEC << 32 ) / mul = FREQ_HZ << shift
+ * ( NSEC_PER_SEC << 32 ) / mul ) >> shift = FREQ_HZ
+ */
+static u64 hvclock_to_hz(u32 mul, s8 shift)
+{
+	u64 tm = NSEC_PER_SEC << 32;
+
+	/* Maximise precision. Shift left until the top bit is set */
+	tm <<= 2;
+	shift += 2;
+
+	/* While 'mul' is even, increase the shift *after* the division */
+	while (!(mul & 1)) {
+		shift++;
+		mul >>= 1;
+	}
+
+	tm /= mul;
+
+	if (shift >= 64)
+		return 0;
+	if (shift > 0)
+		return tm >> shift;
+
+	/*
+	 * A left shift can overflow for absurd user-supplied mul/shift
+	 * combinations; saturate (to a value which cannot itself overflow
+	 * the rounding in the caller's kHz conversion) so the frequency
+	 * check certainly rejects them.
+	 */
+	if (shift <= -64 || tm > (U64_MAX >> 1) >> -shift)
+		return U64_MAX >> 1;
+	return tm << -shift;
+}
+
+static int kvm_vcpu_ioctl_set_clock_guest(struct kvm_vcpu *v, void __user *argp)
+{
+	struct pvclock_vcpu_time_info user_hv_clock;
+	struct kvm *kvm = v->kvm;
+	struct kvm_arch *ka = &kvm->arch;
+	u64 curr_tsc_hz, user_tsc_hz;
+	u64 curr_tsc_khz, user_tsc_khz;
+	u64 user_clk_ns;
+	u64 guest_tsc;
+	int rc = 0;
+
+	if (copy_from_user(&user_hv_clock, argp, sizeof(user_hv_clock)))
+		return -EFAULT;
+
+	if (user_hv_clock.pad0 || user_hv_clock.pad[0] || user_hv_clock.pad[1])
+		return -EINVAL;
+
+	/*
+	 * KVM_GET_CLOCK_GUEST emits version 0 and at most
+	 * PVCLOCK_TSC_STABLE_BIT; reject anything else now, so the
+	 * fields keep meaning for the future rather than carrying
+	 * whatever garbage old userspace got away with.
+	 */
+	if (user_hv_clock.version ||
+	    (user_hv_clock.flags & ~PVCLOCK_TSC_STABLE_BIT))
+		return -EINVAL;
+
+	if (!user_hv_clock.tsc_to_system_mul)
+		return -EINVAL;
+
+	if (user_hv_clock.tsc_shift < -31 || user_hv_clock.tsc_shift > 31)
+		return -EINVAL;
+
+	user_tsc_hz = hvclock_to_hz(user_hv_clock.tsc_to_system_mul,
+				    user_hv_clock.tsc_shift);
+
+	kvm_hv_request_tsc_page_update(kvm);
+
+	/*
+	 * kvm_start_pvclock_update() takes tsc_write_lock and opens
+	 * the pvclock seqcount; kvm_end_pvclock_update() closes both.
+	 * All clock state modifications between them are atomic with
+	 * respect to readers in kvm_guest_time_update().
+	 */
+	kvm_start_pvclock_update(kvm);
+	pvclock_update_vm_gtod_copy(kvm);
+
+	if (!ka->use_master_clock) {
+		rc = -ENODATA;
+		goto out;
+	}
+
+	/*
+	 * Nonzero, or pvclock_update_vm_gtod_copy() — which read it on
+	 * this CPU, with interrupts still disabled — would have cleared
+	 * use_master_clock above.
+	 */
+	curr_tsc_hz = (u64)get_cpu_tsc_khz() * HZ_PER_KHZ;
+
+	if (kvm_caps.has_tsc_control)
+		curr_tsc_hz = kvm_scale_tsc(curr_tsc_hz,
+					    v->arch.l1_tsc_scaling_ratio);
+
+	/*
+	 * The mul/shift in the provided pvclock structure encode the guest
+	 * TSC frequency at which it was generated. Sanity-check that it is
+	 * consistent with this vCPU's effective TSC frequency, allowing a
+	 * discrepancy of 1 kHz either way since independently calibrated
+	 * hosts will not measure precisely the same value even for the
+	 * same nominal frequency. Compare in kHz: the reconstruction of
+	 * each side's frequency in Hz carries a few Hz of rounding, which
+	 * must not push two hosts calibrated exactly 1 kHz apart over the
+	 * line.
+	 */
+	user_tsc_khz = DIV_ROUND_CLOSEST_ULL(user_tsc_hz, HZ_PER_KHZ);
+	curr_tsc_khz = DIV_ROUND_CLOSEST_ULL(curr_tsc_hz, HZ_PER_KHZ);
+	if (user_tsc_khz + 1 < curr_tsc_khz ||
+	    user_tsc_khz > curr_tsc_khz + 1) {
+		rc = -ERANGE;
+		goto out;
+	}
+
+	/*
+	 * Calculate the guest TSC at the new reference point, and the
+	 * corresponding KVM clock value according to user_hv_clock.
+	 * Adjust kvmclock_offset so both definitions agree.
+	 */
+	guest_tsc = kvm_read_l1_tsc(v, ka->master_cycle_now);
+	if ((s64)(guest_tsc - user_hv_clock.tsc_timestamp) < 0) {
+		rc = -EINVAL;
+		goto out;
+	}
+	user_clk_ns = __pvclock_read_cycles(&user_hv_clock, guest_tsc);
+	ka->kvmclock_offset = user_clk_ns - ka->master_kernel_ns;
+
+out:
+	kvm_end_pvclock_update(kvm);
+
+	kvm_hv_remark_tsc_page(kvm);
+	return rc;
+}
+#endif
+
 long kvm_arch_vcpu_ioctl(struct file *filp,
 			 unsigned int ioctl, unsigned long arg)
 {
@@ -4334,6 +4549,14 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 		srcu_read_unlock(&vcpu->kvm->srcu, idx);
 		break;
 	}
+#ifdef CONFIG_X86_64
+	case KVM_SET_CLOCK_GUEST:
+		r = kvm_vcpu_ioctl_set_clock_guest(vcpu, argp);
+		break;
+	case KVM_GET_CLOCK_GUEST:
+		r = kvm_vcpu_ioctl_get_clock_guest(vcpu, argp);
+		break;
+#endif
 #ifdef CONFIG_KVM_HYPERV
 	case KVM_GET_SUPPORTED_HV_CPUID:
 		r = kvm_ioctl_get_supported_hv_cpuid(vcpu, argp);
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 8dff2fc1972e..1349387cceb0 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1000,6 +1000,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_PPC_COMPAT_CAPS 250
 #define KVM_CAP_ARM_PMU_V3_STRICT 251
 #define KVM_CAP_GUEST_MEMFD_MEMORY_ATTRIBUTES 252
+#define KVM_CAP_CLOCK_GUEST 253
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
@@ -1690,4 +1691,7 @@ struct kvm_pre_fault_memory {
 	__u64 padding[5];
 };
 
+#define KVM_SET_CLOCK_GUEST	_IOW(KVMIO, 0xd6, struct pvclock_vcpu_time_info)
+#define KVM_GET_CLOCK_GUEST	_IOR(KVMIO, 0xd7, struct pvclock_vcpu_time_info)
+
 #endif /* __LINUX_KVM_H */
-- 
2.43.0

[PATCH v13 17/26] KVM: selftests: Add library helpers for clock and TSC-offset ioctls

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Add wrappers for KVM_[GS]ET_CLOCK_GUEST, the KVM_VCPU_TSC_OFFSET device
attribute and KVM_[GS]ET_CLOCK to the x86 library header, so that the
KVM clock selftests don't each open-code the same ioctl invocations,
and convert the existing open-coded call sites in kvm_clock_test,
hyperv_clock, xen_shinfo_test and system_counter_offset_test.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 .../selftests/kvm/include/x86/processor.h     | 50 +++++++++++++++++++
 .../kvm/system_counter_offset_test.c          |  3 +-
 .../testing/selftests/kvm/x86/hyperv_clock.c  |  2 +-
 .../selftests/kvm/x86/kvm_clock_test.c        |  6 +--
 .../selftests/kvm/x86/xen_shinfo_test.c       |  2 +-
 5 files changed, 56 insertions(+), 7 deletions(-)

diff --git a/tools/testing/selftests/kvm/include/x86/processor.h b/tools/testing/selftests/kvm/include/x86/processor.h
index 5898a277a2e0..4062bdb32716 100644
--- a/tools/testing/selftests/kvm/include/x86/processor.h
+++ b/tools/testing/selftests/kvm/include/x86/processor.h
@@ -12,6 +12,7 @@
 
 #include <asm/msr-index.h>
 #include <asm/prctl.h>
+#include <asm/pvclock-abi.h>
 
 #include <linux/kvm_para.h>
 #include <linux/stringify.h>
@@ -1029,6 +1030,55 @@ static inline void vcpu_xcrs_set(struct kvm_vcpu *vcpu, struct kvm_xcrs *xcrs)
 	vcpu_ioctl(vcpu, KVM_SET_XCRS, xcrs);
 }
 
+static inline int __vcpu_get_clock_guest(struct kvm_vcpu *vcpu,
+					 struct pvclock_vcpu_time_info *pvti)
+{
+	return __vcpu_ioctl(vcpu, KVM_GET_CLOCK_GUEST, pvti);
+}
+
+static inline void vcpu_get_clock_guest(struct kvm_vcpu *vcpu,
+					struct pvclock_vcpu_time_info *pvti)
+{
+	vcpu_ioctl(vcpu, KVM_GET_CLOCK_GUEST, pvti);
+}
+
+static inline int __vcpu_set_clock_guest(struct kvm_vcpu *vcpu,
+					 struct pvclock_vcpu_time_info *pvti)
+{
+	return __vcpu_ioctl(vcpu, KVM_SET_CLOCK_GUEST, pvti);
+}
+
+static inline void vcpu_set_clock_guest(struct kvm_vcpu *vcpu,
+					struct pvclock_vcpu_time_info *pvti)
+{
+	vcpu_ioctl(vcpu, KVM_SET_CLOCK_GUEST, pvti);
+}
+
+static inline u64 vcpu_get_tsc_offset(struct kvm_vcpu *vcpu)
+{
+	u64 offset;
+
+	vcpu_device_attr_get(vcpu, KVM_VCPU_TSC_CTRL, KVM_VCPU_TSC_OFFSET,
+			     &offset);
+	return offset;
+}
+
+static inline void vcpu_set_tsc_offset(struct kvm_vcpu *vcpu, u64 offset)
+{
+	vcpu_device_attr_set(vcpu, KVM_VCPU_TSC_CTRL, KVM_VCPU_TSC_OFFSET,
+			     &offset);
+}
+
+static inline void vm_get_clock(struct kvm_vm *vm, struct kvm_clock_data *data)
+{
+	vm_ioctl(vm, KVM_GET_CLOCK, data);
+}
+
+static inline void vm_set_clock(struct kvm_vm *vm, struct kvm_clock_data *data)
+{
+	vm_ioctl(vm, KVM_SET_CLOCK, data);
+}
+
 const struct kvm_cpuid_entry2 *get_cpuid_entry(const struct kvm_cpuid2 *cpuid,
 					       u32 function, u32 index);
 const struct kvm_cpuid2 *kvm_get_supported_cpuid(void);
diff --git a/tools/testing/selftests/kvm/system_counter_offset_test.c b/tools/testing/selftests/kvm/system_counter_offset_test.c
index dc5e30b7b77f..f864e66dc789 100644
--- a/tools/testing/selftests/kvm/system_counter_offset_test.c
+++ b/tools/testing/selftests/kvm/system_counter_offset_test.c
@@ -35,8 +35,7 @@ static void check_preconditions(struct kvm_vcpu *vcpu)
 
 static void setup_system_counter(struct kvm_vcpu *vcpu, struct test_case *test)
 {
-	vcpu_device_attr_set(vcpu, KVM_VCPU_TSC_CTRL, KVM_VCPU_TSC_OFFSET,
-			     &test->tsc_offset);
+	vcpu_set_tsc_offset(vcpu, test->tsc_offset);
 }
 
 static u64 guest_read_system_counter(struct test_case *test)
diff --git a/tools/testing/selftests/kvm/x86/hyperv_clock.c b/tools/testing/selftests/kvm/x86/hyperv_clock.c
index 7a1b3b5d1e62..ced3d9873f9a 100644
--- a/tools/testing/selftests/kvm/x86/hyperv_clock.c
+++ b/tools/testing/selftests/kvm/x86/hyperv_clock.c
@@ -254,7 +254,7 @@ int main(void)
 		if (stage == 7 || stage == 8 || stage == 10) {
 			struct kvm_clock_data clock = {0};
 
-			vm_ioctl(vm, KVM_SET_CLOCK, &clock);
+			vm_set_clock(vm, &clock);
 		}
 	}
 
diff --git a/tools/testing/selftests/kvm/x86/kvm_clock_test.c b/tools/testing/selftests/kvm/x86/kvm_clock_test.c
index 5ad4aeb8e373..0028a40b8a3e 100644
--- a/tools/testing/selftests/kvm/x86/kvm_clock_test.c
+++ b/tools/testing/selftests/kvm/x86/kvm_clock_test.c
@@ -99,7 +99,7 @@ static void setup_clock(struct kvm_vm *vm, struct test_case *test_case)
 		data.realtime += test_case->realtime_offset;
 	}
 
-	vm_ioctl(vm, KVM_SET_CLOCK, &data);
+	vm_set_clock(vm, &data);
 }
 
 static void enter_guest(struct kvm_vcpu *vcpu)
@@ -112,10 +112,10 @@ static void enter_guest(struct kvm_vcpu *vcpu)
 	for (i = 0; i < ARRAY_SIZE(test_cases); i++) {
 		setup_clock(vm, &test_cases[i]);
 
-		vm_ioctl(vm, KVM_GET_CLOCK, &start);
+		vm_get_clock(vm, &start);
 
 		vcpu_run(vcpu);
-		vm_ioctl(vm, KVM_GET_CLOCK, &end);
+		vm_get_clock(vm, &end);
 
 		TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
 
diff --git a/tools/testing/selftests/kvm/x86/xen_shinfo_test.c b/tools/testing/selftests/kvm/x86/xen_shinfo_test.c
index 9045f03cf2d3..f251f8ea29ec 100644
--- a/tools/testing/selftests/kvm/x86/xen_shinfo_test.c
+++ b/tools/testing/selftests/kvm/x86/xen_shinfo_test.c
@@ -985,7 +985,7 @@ int main(int argc, char *argv[])
 	TEST_ASSERT(wc->version && !(wc->version & 1),
 		    "Bad wallclock version %x", wc->version);
 
-	vm_ioctl(vm, KVM_GET_CLOCK, &kcdata);
+	vm_get_clock(vm, &kcdata);
 
 	if (kcdata.flags & KVM_CLOCK_REALTIME) {
 		if (verbose) {
-- 
2.43.0

[PATCH v13 18/26] KVM: x86: Add KVM_VCPU_TSC_SCALE and fix the documentation on TSC migration

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The documentation on TSC migration using KVM_VCPU_TSC_OFFSET is woefully
inadequate. It ignores TSC scaling, and ignores the fact that the host
TSC may differ from one host to the next (and in fact because of the way
the kernel calibrates it, it generally differs from one boot to the next
even on the same hardware).

Add KVM_VCPU_TSC_SCALE to extract the actual scale ratio and frac_bits,
and attempt to document the process that userspace needs to follow to
preserve the TSC across migration. Add a self test to function as an
exemplar.

Only enumerate KVM_VCPU_TSC_SCALE when kvm_caps.has_tsc_control is true,
since the scaling ratio is only meaningful when hardware TSC scaling is
supported.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Reviewed-by: Paul Durrant <paul@xen.org>
---
 Documentation/virt/kvm/devices/vcpu.rst       | 144 +++--
 arch/x86/include/uapi/asm/kvm.h               |   6 +
 arch/x86/kvm/x86.c                            |  21 +
 tools/arch/x86/include/uapi/asm/kvm.h         |   6 +
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../kvm/x86/pvclock_migration_test.c          | 503 ++++++++++++++++++
 6 files changed, 651 insertions(+), 30 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/pvclock_migration_test.c

diff --git a/Documentation/virt/kvm/devices/vcpu.rst b/Documentation/virt/kvm/devices/vcpu.rst
index deb5c51bc00c..97174132285d 100644
--- a/Documentation/virt/kvm/devices/vcpu.rst
+++ b/Documentation/virt/kvm/devices/vcpu.rst
@@ -239,6 +239,7 @@ including the layout of the stolen time structure.
 :Architectures: x86
 
 4.1 ATTRIBUTE: KVM_VCPU_TSC_OFFSET
+----------------------------------
 
 :Parameters: 64-bit unsigned TSC offset
 
@@ -253,7 +254,10 @@ Returns:
 Specifies the guest's TSC offset relative to the host's TSC. The guest's
 TSC is then derived by the following equation:
 
-  guest_tsc = host_tsc + KVM_VCPU_TSC_OFFSET
+  guest_tsc = ((host_tsc * tsc_ratio) >> tsc_frac_bits) + KVM_VCPU_TSC_OFFSET
+
+The values of tsc_ratio and tsc_frac_bits can be obtained using
+the KVM_VCPU_TSC_SCALE attribute.
 
 This attribute is useful to adjust the guest's TSC on live migration,
 so that the TSC counts the time during which the VM was paused. The
@@ -261,44 +265,124 @@ following describes a possible algorithm to use for this purpose.
 
 From the source VMM process:
 
-1. Invoke the KVM_GET_CLOCK ioctl to record the host TSC (tsc_src),
+1. Invoke the KVM_GET_CLOCK ioctl to record the host TSC (host_tsc_src),
    kvmclock nanoseconds (guest_src), and host CLOCK_REALTIME nanoseconds
-   (host_src).
+   (time_src) at a given moment (Tsrc). The host_tsc field is only valid
+   when the returned flags include KVM_CLOCK_HOST_TSC.
+
+2. For each vCPU[i]:
+
+   a. Read the KVM_VCPU_TSC_OFFSET attribute to record the guest TSC offset
+      (ofs_src[i]).
+
+   b. Read the KVM_VCPU_TSC_SCALE attribute to record the guest TSC scaling
+      ratio (ratio_src[i], frac_bits_src[i]). If the attribute is not
+      supported (-ENXIO), there is no hardware TSC scaling and identity
+      scaling applies: use ratio 1, frac_bits 0.
+
+   c. Invoke the KVM_GET_TSC_KHZ ioctl to record the frequency of the
+      guest's TSC (freq[i]).
+
+   d. Use host_tsc_src and the scaling/offset factors to calculate this
+      vCPU's TSC at time Tsrc:
+
+      tsc_src[i] = ((host_tsc_src * ratio_src[i]) >> frac_bits_src[i]) + ofs_src[i]
 
-2. Read the KVM_VCPU_TSC_OFFSET attribute for every vCPU to record the
-   guest TSC offset (ofs_src[i]).
+      Note that frac_bits may be as high as 48, so this multiplication
+      needs a 128-bit intermediate to avoid overflow.
 
-3. Invoke the KVM_GET_TSC_KHZ ioctl to record the frequency of the
-   guest's TSC (freq).
+3. Invoke the KVM_GET_CLOCK_GUEST ioctl on the boot vCPU to return the KVM
+   clock as a function of the guest TSC (pvti_src). (This ioctl may not
+   succeed if the host and guest TSCs are not consistent and well-behaved.)
 
 From the destination VMM process:
 
-4. Invoke the KVM_SET_CLOCK ioctl, providing the source nanoseconds from
-   kvmclock (guest_src) and CLOCK_REALTIME (host_src) in their respective
-   fields.  Ensure that the KVM_CLOCK_REALTIME flag is set in the provided
-   structure.
+4. Invoke the KVM_SET_TSC_KHZ ioctl to set the scaled frequency of the
+   guest's TSC (freq[i]) — either on the VM before creating the vCPUs, or
+   on each vCPU after creating it (as the exemplar test does). When using
+   the per-vCPU form, create all the vCPUs before setting the frequency
+   on any of them (each vCPU's creation synchronizes it at the default
+   frequency, so interleaving creation with frequency changes fragments
+   the TSC matching), and set the frequency before performing step 7 on
+   that vCPU: KVM_VCPU_TSC_SCALE reports the ratio for the currently
+   configured frequency.
+
+5. Invoke the KVM_GET_CLOCK ioctl to record the host TSC (host_tsc_dst) and
+   host CLOCK_REALTIME nanoseconds (time_dst) at a given moment (Tdst),
+   again checking KVM_CLOCK_HOST_TSC in the returned flags.
+
+6. Calculate the number of nanoseconds elapsed between Tsrc and Tdst:
+
+   ΔT = time_dst - time_src
+
+7. As each vCPU[i] is created:
+
+   a. Read the KVM_VCPU_TSC_SCALE attribute to record the guest TSC scaling
+      ratio (ratio_dst[i], frac_bits_dst[i]). As in step 2b, -ENXIO means
+      identity scaling: use ratio 1, frac_bits 0.
+
+   b. Calculate the intended guest TSC value at time Tdst, taking care
+      with the units (ΔT is in nanoseconds; freq, as set by
+      KVM_SET_TSC_KHZ, is in kHz):
+
+      tsc_dst[i] = tsc_src[i] + (ΔT * freq[i] / 10^6)
 
-   KVM will advance the VM's kvmclock to account for elapsed time since
-   recording the clock values.  Note that this will cause problems in
-   the guest (e.g., timeouts) unless CLOCK_REALTIME is synchronized
-   between the source and destination, and a reasonably short time passes
-   between the source pausing the VMs and the destination executing
-   steps 4-7.
+      As in step 2d, this multiplication needs a 128-bit intermediate:
+      the product exceeds a signed 64-bit value once ΔT reaches about
+      51 minutes for a 3GHz guest.
+
+   c. Use host_tsc_dst and the scaling factors to calculate this vCPU's
+      raw scaled TSC at time Tdst without offsetting:
+
+      raw_dst[i] = ((host_tsc_dst * ratio_dst[i]) >> frac_bits_dst[i])
+
+   d. Calculate ofs_dst[i] = tsc_dst[i] - raw_dst[i] and set the resulting
+      offset using the KVM_VCPU_TSC_OFFSET attribute.
+
+8. If pvti_src was provided, invoke the KVM_SET_CLOCK_GUEST ioctl on the boot
+   vCPU to restore the KVM clock as a precise function of the guest TSC.
+
+9. If KVM_SET_CLOCK_GUEST was not available or failed (e.g. because the
+   master clock is not active), fall back to the KVM_SET_CLOCK ioctl,
+   providing the source nanoseconds from kvmclock (guest_src) and
+   CLOCK_REALTIME (time_src) in their respective fields. Ensure that the
+   KVM_CLOCK_REALTIME flag is set in the provided structure.
+
+   KVM will restore the VM's kvmclock, accounting for elapsed time since
+   the clock values were recorded. Note that this will cause problems in
+   the guest (e.g., timeouts) unless CLOCK_REALTIME is synchronized between
+   the source and destination, and a reasonably short time passes between
+   the source pausing the VMs and the destination resuming them.
+   Due to the KVM_[SG]ET_CLOCK API using CLOCK_REALTIME instead of
+   CLOCK_TAI, leap seconds during the migration may also introduce errors.
+
+4.2 ATTRIBUTE: KVM_VCPU_TSC_SCALE
+---------------------------------
+
+:Parameters: struct kvm_vcpu_tsc_scale
+
+Returns:
+
+	 ======= ======================================
+	 -EFAULT Error writing the result to the
+		 provided address.
+	 -ENXIO  Attribute not supported (no TSC scaling)
+	 -EINVAL Invalid request to write the attribute
+	 ======= ======================================
 
-5. Invoke the KVM_GET_CLOCK ioctl to record the host TSC (tsc_dest) and
-   kvmclock nanoseconds (guest_dest).
+This read-only attribute reports the guest's TSC scaling factor, in the form
+of a fixed-point number represented by the following structure::
 
-6. Adjust the guest TSC offsets for every vCPU to account for (1) time
-   elapsed since recording state and (2) difference in TSCs between the
-   source and destination machine:
+  struct kvm_vcpu_tsc_scale {
+	__u64 tsc_ratio;
+	__u64 tsc_frac_bits;
+  };
 
-   ofs_dst[i] = ofs_src[i] -
-     (guest_src - guest_dest) * freq +
-     (tsc_src - tsc_dest)
+The tsc_frac_bits field indicates the location of the fixed point, such that
+host TSC values are converted to guest TSC using the formula:
 
-   ("ofs[i] + tsc - guest * freq" is the guest TSC value corresponding to
-   a time of 0 in kvmclock.  The above formula ensures that it is the
-   same on the destination as it was on the source).
+  guest_tsc = ((host_tsc * tsc_ratio) >> tsc_frac_bits) + offset
 
-7. Write the KVM_VCPU_TSC_OFFSET attribute for every vCPU with the
-   respective value derived in the previous step.
+Userspace can use this to precisely calculate the guest TSC from the host
+TSC at any given moment. This is needed for accurate migration of guests,
+as described in the documentation for the KVM_VCPU_TSC_OFFSET attribute.
diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kvm.h
index 9e3cdc3adb65..f8274d8e471d 100644
--- a/arch/x86/include/uapi/asm/kvm.h
+++ b/arch/x86/include/uapi/asm/kvm.h
@@ -964,6 +964,12 @@ struct kvm_hyperv_eventfd {
 /* for KVM_{GET,SET,HAS}_DEVICE_ATTR */
 #define KVM_VCPU_TSC_CTRL 0 /* control group for the timestamp counter (TSC) */
 #define   KVM_VCPU_TSC_OFFSET 0 /* attribute for the TSC offset */
+#define   KVM_VCPU_TSC_SCALE  1 /* attribute for TSC scaling factor */
+
+struct kvm_vcpu_tsc_scale {
+	__u64 tsc_ratio;
+	__u64 tsc_frac_bits;
+};
 
 /* x86-specific KVM_EXIT_HYPERCALL flags. */
 #define KVM_EXIT_HYPERCALL_LONG_MODE	_BITULL(0)
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 16383c71131d..b9d3b2715403 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -3780,6 +3780,9 @@ static int kvm_arch_tsc_has_attr(struct kvm_vcpu *vcpu,
 	case KVM_VCPU_TSC_OFFSET:
 		r = 0;
 		break;
+	case KVM_VCPU_TSC_SCALE:
+		r = kvm_caps.has_tsc_control ? 0 : -ENXIO;
+		break;
 	default:
 		r = -ENXIO;
 	}
@@ -3800,6 +3803,21 @@ static int kvm_arch_tsc_get_attr(struct kvm_vcpu *vcpu,
 			break;
 		r = 0;
 		break;
+	case KVM_VCPU_TSC_SCALE: {
+		struct kvm_vcpu_tsc_scale scale;
+
+		r = -ENXIO;
+		if (!kvm_caps.has_tsc_control)
+			break;
+
+		scale.tsc_ratio = vcpu->arch.l1_tsc_scaling_ratio;
+		scale.tsc_frac_bits = kvm_caps.tsc_scaling_ratio_frac_bits;
+		r = -EFAULT;
+		if (copy_to_user(uaddr, &scale, sizeof(scale)))
+			break;
+		r = 0;
+		break;
+	}
 	default:
 		r = -ENXIO;
 	}
@@ -3855,6 +3873,9 @@ static int kvm_arch_tsc_set_attr(struct kvm_vcpu *vcpu,
 		r = 0;
 		break;
 	}
+	case KVM_VCPU_TSC_SCALE:
+		r = kvm_caps.has_tsc_control ? -EINVAL : -ENXIO;
+		break;
 	default:
 		r = -ENXIO;
 	}
diff --git a/tools/arch/x86/include/uapi/asm/kvm.h b/tools/arch/x86/include/uapi/asm/kvm.h
index 1585ec804066..7b3afccb6cb2 100644
--- a/tools/arch/x86/include/uapi/asm/kvm.h
+++ b/tools/arch/x86/include/uapi/asm/kvm.h
@@ -963,6 +963,12 @@ struct kvm_hyperv_eventfd {
 /* for KVM_{GET,SET,HAS}_DEVICE_ATTR */
 #define KVM_VCPU_TSC_CTRL 0 /* control group for the timestamp counter (TSC) */
 #define   KVM_VCPU_TSC_OFFSET 0 /* attribute for the TSC offset */
+#define   KVM_VCPU_TSC_SCALE  1 /* attribute for TSC scaling factor */
+
+struct kvm_vcpu_tsc_scale {
+	__u64 tsc_ratio;
+	__u64 tsc_frac_bits;
+};
 
 /* x86-specific KVM_EXIT_HYPERCALL flags. */
 #define KVM_EXIT_HYPERCALL_LONG_MODE	_BITULL(0)
diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 888a7530a280..18605318860e 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -113,6 +113,7 @@ TEST_GEN_PROGS_x86 += x86/pmu_counters_test
 TEST_GEN_PROGS_x86 += x86/pmu_event_filter_test
 TEST_GEN_PROGS_x86 += x86/private_mem_conversions_test
 TEST_GEN_PROGS_x86 += x86/private_mem_kvm_exits_test
+TEST_GEN_PROGS_x86 += x86/pvclock_migration_test
 TEST_GEN_PROGS_x86 += x86/save_restore_pf_stress_test
 TEST_GEN_PROGS_x86 += x86/set_boot_cpu_id
 TEST_GEN_PROGS_x86 += x86/set_sregs_test
diff --git a/tools/testing/selftests/kvm/x86/pvclock_migration_test.c b/tools/testing/selftests/kvm/x86/pvclock_migration_test.c
new file mode 100644
index 000000000000..a25522b5562f
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/pvclock_migration_test.c
@@ -0,0 +1,503 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Test KVM clock precision across simulated live migration.
+ *
+ * Verifies that the documented TSC migration procedure (using
+ * KVM_VCPU_TSC_OFFSET, KVM_VCPU_TSC_SCALE, KVM_GET_CLOCK, and
+ * KVM_SET_CLOCK_GUEST) preserves the kvmclock's relationship to
+ * CLOCK_MONOTONIC_RAW — including for vCPUs whose TSCs are offset
+ * from one another (same frequency), which no longer disables the
+ * master clock.
+ *
+ * The test:
+ * 1. Creates a VM with 3 vCPUs, runs each guest to enable kvmclock
+ * 2. Applies divergent TSC offsets to vCPUs 1 and 2
+ * 3. Verifies the master clock remains active (KVM_CLOCK_HOST_TSC),
+ *    that the guests see PVCLOCK_TSC_STABLE_BIT clear, and that all
+ *    vCPUs' pvclocks agree on "now" despite the divergent offsets
+ * 4. Does a PTP-like ABA measurement of kvmclock vs CLOCK_MONOTONIC_RAW
+ * 5. Follows the documented migration procedure (same host, 1s pause),
+ *    migrating each vCPU's TSC individually but setting the KVM clock
+ *    only once, via KVM_SET_CLOCK_GUEST on vCPU0
+ * 6. Does the same ABA measurement on the destination VM
+ * 7. Verifies the kvmclock-vs-monotonic delta is preserved, the
+ *    cross-vCPU TSC deltas are preserved, and all destination vCPUs'
+ *    pvclocks again agree on "now"
+ */
+#include <inttypes.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+
+#include <asm/pvclock.h>
+#include <asm/pvclock-abi.h>
+
+#define KVMCLOCK_GPA	0xc0000000ULL
+#define NR_VCPUS	3
+
+/*
+ * Divergent TSC offsets applied to vCPUs 1 and 2, in guest cycles.
+ * Roughly +1s and -0.5s at 1.5 GHz — large enough that any failure
+ * to account for them is orders of magnitude outside the ABA bounds.
+ */
+static const s64 tsc_deltas[NR_VCPUS] = { 0, 1500000000LL, -750000000LL };
+
+static void guest_code(int idx)
+{
+	wrmsr(MSR_KVM_SYSTEM_TIME_NEW, (KVMCLOCK_GPA + idx * PAGE_SIZE) | 1);
+
+	for (;;)
+		GUEST_SYNC(0);
+}
+
+static u64 read_kvmclock_ns(struct kvm_vm *vm)
+{
+	struct kvm_clock_data data = {};
+
+	vm_get_clock(vm, &data);
+	return data.clock;
+}
+
+/*
+ * ABA measurement: read CLOCK_MONOTONIC_RAW, kvmclock, CLOCK_MONOTONIC_RAW.
+ * Repeat 3 times, keep the reading with the smallest spread.
+ */
+static void aba_reading(struct kvm_vm *vm, u64 *lo, u64 *kvm_ns,
+			u64 *hi)
+{
+	u64 best_spread = UINT64_MAX;
+	int i;
+
+	for (i = 0; i < 3; i++) {
+		struct timespec ts1, ts2;
+		u64 m1, m2, clk;
+
+		clock_gettime(CLOCK_MONOTONIC_RAW, &ts1);
+		clk = read_kvmclock_ns(vm);
+		clock_gettime(CLOCK_MONOTONIC_RAW, &ts2);
+
+		m1 = ts1.tv_sec * 1000000000ULL + ts1.tv_nsec;
+		m2 = ts2.tv_sec * 1000000000ULL + ts2.tv_nsec;
+
+		if (m2 - m1 < best_spread) {
+			best_spread = m2 - m1;
+			*lo = m1;
+			*kvm_ns = clk;
+			*hi = m2;
+		}
+	}
+}
+
+static struct kvm_vm *create_vm(struct kvm_vcpu **vcpus, u32 freq_khz)
+{
+	struct kvm_vm *vm;
+	int i;
+
+	vm = vm_create_with_vcpus(NR_VCPUS, guest_code, vcpus);
+
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS,
+				    KVMCLOCK_GPA, 1, NR_VCPUS, 0);
+	virt_map(vm, KVMCLOCK_GPA, KVMCLOCK_GPA, NR_VCPUS);
+
+	for (i = 0; i < NR_VCPUS; i++) {
+		vcpu_args_set(vcpus[i], 1, i);
+		/* Set guest TSC frequency (may trigger scaling) */
+		vcpu_ioctl(vcpus[i], KVM_SET_TSC_KHZ,
+			   (void *)(unsigned long)freq_khz);
+	}
+	return vm;
+}
+
+static void run_to_sync(struct kvm_vcpu *vcpu)
+{
+	struct ucall uc;
+
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	TEST_ASSERT_EQ(get_ucall(vcpu, &uc), UCALL_SYNC);
+}
+
+/*
+ * Compute a vCPU's guest TSC for a given host TSC value, using the
+ * vCPU's scale and offset as saved from KVM_VCPU_TSC_SCALE/OFFSET.
+ */
+static u64 host_to_guest_tsc(struct kvm_vcpu_tsc_scale *scale, u64 ofs,
+			     u64 host_tsc)
+{
+	if (scale->tsc_frac_bits)
+		return ((unsigned __int128)host_tsc * scale->tsc_ratio
+			>> scale->tsc_frac_bits) + ofs;
+	return host_tsc + ofs;
+}
+
+/*
+ * Verify that all vCPUs agree on the KVM clock, despite having
+ * different TSC offsets. Take one atomic {host_tsc, kvmclock} pair
+ * for the whole VM, derive each vCPU's guest TSC at that instant,
+ * and evaluate each vCPU's pvti at its own guest TSC. All must give
+ * the same nanosecond value (same instant, same clock), and must
+ * match what KVM_GET_CLOCK itself said.
+ */
+static void check_cross_vcpu_coherence(struct kvm_vm *vm,
+				       struct kvm_vcpu **vcpus,
+				       struct pvclock_vcpu_time_info *pvtis,
+				       const char *what)
+{
+	struct kvm_clock_data now = {};
+	u64 clk[NR_VCPUS];
+	int i;
+
+	vm_get_clock(vm, &now);
+	TEST_ASSERT(now.flags & KVM_CLOCK_HOST_TSC,
+		    "%s: master clock inactive with divergent TSC offsets",
+		    what);
+
+	for (i = 0; i < NR_VCPUS; i++) {
+		struct kvm_vcpu_tsc_scale scale = {};
+		u64 ofs = vcpu_get_tsc_offset(vcpus[i]);
+		u64 guest_tsc;
+
+		vcpu_device_attr_get(vcpus[i], KVM_VCPU_TSC_CTRL,
+				     KVM_VCPU_TSC_SCALE, &scale);
+		guest_tsc = host_to_guest_tsc(&scale, ofs, now.host_tsc);
+		clk[i] = __pvclock_read_cycles(&pvtis[i], guest_tsc);
+
+		TEST_ASSERT(!(pvtis[i].flags & PVCLOCK_TSC_STABLE_BIT),
+			    "%s: vCPU%d still advertises TSC_STABLE with divergent offsets",
+			    what, i);
+	}
+
+	for (i = 1; i < NR_VCPUS; i++) {
+		s64 d = (s64)(clk[i] - clk[0]);
+
+		pr_info("  %s: vCPU%d vs vCPU0 kvmclock delta %" PRId64 " ns\n",
+			what, i, d);
+		TEST_ASSERT(d >= -2 && d <= 2,
+			    "%s: vCPU%d kvmclock disagrees with vCPU0 by %" PRId64 " ns",
+			    what, i, d);
+	}
+
+	/* And all of them must agree with KVM_GET_CLOCK itself */
+	s64 d0 = (s64)(clk[0] - now.clock);
+	pr_info("  %s: vCPU0 pvclock vs KVM_GET_CLOCK delta %" PRId64 " ns\n",
+		what, d0);
+	TEST_ASSERT(d0 >= -2 && d0 <= 2,
+		    "%s: vCPU0 pvclock disagrees with KVM_GET_CLOCK by %" PRId64 " ns",
+		    what, d0);
+}
+
+int main(void)
+{
+	struct pvclock_vcpu_time_info pvti_src[NR_VCPUS];
+	struct pvclock_vcpu_time_info pvti_dst_ioctl[NR_VCPUS];
+	struct kvm_clock_data clock_src, clock_dst;
+	struct kvm_vcpu_tsc_scale scale_src[NR_VCPUS], scale_dst;
+	struct kvm_vcpu *vcpus[NR_VCPUS];
+	struct kvm_vm *vm;
+	u64 mono_before, kvm_before, kvm_after;
+	s64 delta_before;
+	u64 ofs_src[NR_VCPUS], tsc_src[NR_VCPUS], tsc_dst, raw_dst;
+	u64 ofs_dst[NR_VCPUS];
+	u64 host_tsc_src, host_tsc_dst;
+	u64 time_src, time_dst;
+	s64 delta_t;
+	u32 freq_khz = 1500000; /* 1.5 GHz — forces TSC scaling */
+	int i, pass;
+
+	TEST_REQUIRE(sys_clocksource_is_based_on_tsc());
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_TSC_CONTROL));
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_CLOCK_GUEST));
+
+	/* === SOURCE SIDE === */
+	pr_info("=== Source VM ===\n");
+	vm = create_vm(vcpus, freq_khz);
+
+	/* Run each guest to enable its kvmclock */
+	for (i = 0; i < NR_VCPUS; i++)
+		run_to_sync(vcpus[i]);
+
+	/*
+	 * Apply divergent TSC offsets. Same frequency, different offsets:
+	 * this must no longer disable the master clock, but must clear
+	 * PVCLOCK_TSC_STABLE_BIT.
+	 */
+	for (i = 1; i < NR_VCPUS; i++)
+		vcpu_set_tsc_offset(vcpus[i],
+				    vcpu_get_tsc_offset(vcpus[i]) + tsc_deltas[i]);
+
+	/*
+	 * Re-run each vCPU twice. The offset writes leave a masterclock
+	 * update pending on vCPUs 1 and 2; the first pass processes it and
+	 * broadcasts KVM_REQ_CLOCK_UPDATE to every vCPU, and the second
+	 * pass has each vCPU rewrite its guest-visible pvclock page with
+	 * PVCLOCK_TSC_STABLE_BIT now clear. A single pass would leave
+	 * vCPU0's page stale: its rewrite happens before the broadcast.
+	 */
+	for (pass = 0; pass < 2; pass++)
+		for (i = 0; i < NR_VCPUS; i++)
+			run_to_sync(vcpus[i]);
+
+	/* Fetch each vCPU's pvti and verify cross-vCPU coherence */
+	for (i = 0; i < NR_VCPUS; i++)
+		vcpu_get_clock_guest(vcpus[i], &pvti_src[i]);
+	check_cross_vcpu_coherence(vm, vcpus, pvti_src, "source");
+
+	/* The guest-visible pvti pages must also show TSC_STABLE clear */
+	for (i = 0; i < NR_VCPUS; i++) {
+		struct pvclock_vcpu_time_info *pvti =
+			addr_gpa2hva(vm, KVMCLOCK_GPA + i * getpagesize());
+
+		TEST_ASSERT(!(pvti->flags & PVCLOCK_TSC_STABLE_BIT),
+			    "vCPU%d guest pvti advertises TSC_STABLE with divergent offsets", i);
+	}
+
+	/* ABA measurement: kvmclock vs CLOCK_MONOTONIC_RAW */
+	u64 src_mono_lo, src_mono_hi;
+	aba_reading(vm, &src_mono_lo, &kvm_before, &src_mono_hi);
+	mono_before = (src_mono_lo + src_mono_hi) / 2;
+	delta_before = (s64)(kvm_before - mono_before);
+	pr_info("  kvmclock - MONOTONIC_RAW = %" PRId64 " ns (±%" PRIu64 " ns)\n",
+		delta_before, (src_mono_hi - src_mono_lo) / 2);
+
+	/* Step 1: KVM_GET_CLOCK for atomic {host_tsc, realtime} */
+	memset(&clock_src, 0, sizeof(clock_src));
+	clock_src.flags = KVM_CLOCK_REALTIME;
+	vm_get_clock(vm, &clock_src);
+	TEST_ASSERT(clock_src.flags & KVM_CLOCK_HOST_TSC,
+		    "KVM_GET_CLOCK did not provide host_tsc (no master clock?)");
+	host_tsc_src = clock_src.host_tsc;
+	time_src = clock_src.realtime;
+
+	/* Step 2: Save each vCPU's TSC offset and scale */
+	for (i = 0; i < NR_VCPUS; i++) {
+		ofs_src[i] = vcpu_get_tsc_offset(vcpus[i]);
+		memset(&scale_src[i], 0, sizeof(scale_src[i]));
+		vcpu_device_attr_get(vcpus[i], KVM_VCPU_TSC_CTRL,
+				     KVM_VCPU_TSC_SCALE, &scale_src[i]);
+
+		/* Compute this vCPU's guest TSC at Tsrc */
+		tsc_src[i] = host_to_guest_tsc(&scale_src[i], ofs_src[i],
+					       host_tsc_src);
+		pr_info("  vCPU%d: TSC freq=%u kHz, offset=%" PRId64 "\n",
+			i, freq_khz, (s64)ofs_src[i]);
+	}
+
+	/* Step 3: KVM_GET_CLOCK_GUEST (already fetched above as pvti_src[]) */
+
+	kvm_vm_free(vm);
+
+	/* === PAUSE (simulate migration) === */
+	pr_info("=== Pausing 1 second ===\n");
+	sleep(1);
+
+	/* === DESTINATION SIDE === */
+	pr_info("=== Destination VM ===\n");
+	/* Step 4: create + KVM_SET_TSC_KHZ (in create_vm) */
+	vm = create_vm(vcpus, freq_khz);
+
+	/* Step 5: KVM_GET_CLOCK for atomic {host_tsc, realtime} pair.
+	 * Master clock is active from VM creation.
+	 */
+	memset(&clock_dst, 0, sizeof(clock_dst));
+	vm_get_clock(vm, &clock_dst);
+	TEST_ASSERT(clock_dst.flags & KVM_CLOCK_HOST_TSC,
+		    "KVM_GET_CLOCK did not provide host_tsc (no master clock?)");
+	host_tsc_dst = clock_dst.host_tsc;
+	time_dst = clock_dst.realtime;
+
+	/* Step 6: ΔT */
+	delta_t = (s64)(time_dst - time_src);
+
+	/* Step 7: Compute and set each vCPU's destination offset */
+	memset(&scale_dst, 0, sizeof(scale_dst));
+	vcpu_device_attr_get(vcpus[0], KVM_VCPU_TSC_CTRL, KVM_VCPU_TSC_SCALE,
+			     &scale_dst);
+
+	raw_dst = host_to_guest_tsc(&scale_dst, 0, host_tsc_dst);
+
+	for (i = 0; i < NR_VCPUS; i++) {
+		tsc_dst = tsc_src[i] + (u64)(delta_t * (__int128)freq_khz / 1000000);
+		ofs_dst[i] = tsc_dst - raw_dst;
+		vcpu_set_tsc_offset(vcpus[i], ofs_dst[i]);
+	}
+
+	/*
+	 * The cross-vCPU TSC relationships must be preserved exactly:
+	 * every vCPU's offset moved by the same amount, so the deltas
+	 * between them are untouched by the CLOCK_REALTIME imprecision.
+	 */
+	for (i = 1; i < NR_VCPUS; i++) {
+		s64 src_delta = (s64)(ofs_src[i] - ofs_src[0]);
+		s64 dst_delta = (s64)(ofs_dst[i] - ofs_dst[0]);
+
+		pr_info("  vCPU%d-vCPU0 TSC delta: src %" PRId64
+			", dst %" PRId64 " cycles\n", i, src_delta, dst_delta);
+		TEST_ASSERT(src_delta == dst_delta,
+			    "vCPU%d TSC delta not preserved: %" PRId64 " -> %" PRId64,
+			    i, src_delta, dst_delta);
+	}
+
+	/*
+	 * The TSC offset delta introduced by using CLOCK_REALTIME to
+	 * estimate elapsed time. On same host, the correct offset is
+	 * ofs_src; the difference is the CLOCK_REALTIME-vs-TSC error.
+	 */
+	s64 tsc_ofs_delta = (s64)(ofs_dst[0] - ofs_src[0]);
+	s64 tsc_ofs_delta_ns = tsc_ofs_delta * 1000000LL / freq_khz;
+	pr_info("  Destination TSC offset=%" PRId64
+		", imprecision from CLOCK_REALTIME: %" PRId64 " cycles = %"
+		PRId64 " ns\n", (s64)ofs_dst[0], tsc_ofs_delta, tsc_ofs_delta_ns);
+
+	/*
+	 * Step 8: KVM_SET_CLOCK_GUEST — once, via vCPU0. The other
+	 * vCPUs' clocks must come out right too: the KVM clock is
+	 * VM-wide, and each vCPU's pvti relates it to that vCPU's
+	 * own TSC.
+	 */
+	vcpu_set_clock_guest(vcpus[0], &pvti_src[0]);
+
+	/* Run each guest to write its pvclock page on the destination */
+	for (i = 0; i < NR_VCPUS; i++)
+		run_to_sync(vcpus[i]);
+
+	/* Verify cross-vCPU coherence on the destination */
+	for (i = 0; i < NR_VCPUS; i++)
+		vcpu_get_clock_guest(vcpus[i], &pvti_dst_ioctl[i]);
+	check_cross_vcpu_coherence(vm, vcpus, pvti_dst_ioctl, "dest");
+
+	/* ABA measurement on destination */
+	u64 mono_lo, mono_hi;
+	aba_reading(vm, &mono_lo, &kvm_after, &mono_hi);
+
+	/*
+	 * The kvmclock is tied to the guest TSC via SET_CLOCK_GUEST.
+	 * The guest TSC is offset from the correct value by tsc_ofs_delta_ns
+	 * (due to CLOCK_REALTIME imprecision). So the kvmclock should be
+	 * offset from CLOCK_MONOTONIC_RAW by exactly:
+	 *   (original delta) + tsc_ofs_delta_ns
+	 *
+	 * The "original delta" has uncertainty from the source ABA spread,
+	 * and the measurement has uncertainty from the destination ABA spread.
+	 * Verify the expected value falls within the combined bounds.
+	 */
+	s64 delta_before_lo = (s64)(kvm_before - src_mono_hi);
+	s64 delta_before_hi = (s64)(kvm_before - src_mono_lo);
+	s64 expected_lo = delta_before_lo + tsc_ofs_delta_ns;
+	s64 expected_hi = delta_before_hi + tsc_ofs_delta_ns;
+	s64 actual_lo = (s64)(kvm_after - mono_hi);
+	s64 actual_hi = (s64)(kvm_after - mono_lo);
+
+	/* Show the shift relative to the source measurement */
+	s64 expected_mid = tsc_ofs_delta_ns;
+	s64 expected_err = (s64)(src_mono_hi - src_mono_lo) / 2;
+	s64 actual_mid = ((actual_lo + actual_hi) / 2) - delta_before;
+	s64 actual_err = (s64)(mono_hi - mono_lo) / 2;
+	pr_info("  kvmclock-mono shift: expected %" PRId64 " ns (±%" PRId64
+		"), measured %" PRId64 " ns (±%" PRId64 ")\n",
+		expected_mid, expected_err, actual_mid, actual_err);
+
+	/* The ranges must overlap */
+	TEST_ASSERT(expected_hi >= actual_lo && expected_lo <= actual_hi,
+		    "Ranges don't overlap: expected [%" PRId64 ", %" PRId64
+		    "] measured [%" PRId64 ", %" PRId64 "]",
+		    expected_lo, expected_hi, actual_lo, actual_hi);
+
+	/*
+	 * Direct pvclock verification: read each destination pvclock page
+	 * (as written for the guest) and verify that computing kvmclock
+	 * from the source pvti and the destination pvti at the same
+	 * instant gives the same result, for every vCPU.
+	 *
+	 * Get one atomic {host_tsc, kvmclock} pair, derive each vCPU's
+	 * guest TSC at that instant, then compute kvmclock from both
+	 * pvclock structs. The source pvti evaluates at the *migrated*
+	 * guest TSC, i.e. shifted by the CLOCK_REALTIME imprecision that
+	 * was baked into the offsets — which is exactly what "the clock
+	 * follows the TSC" means.
+	 */
+	struct kvm_clock_data clock_now = {};
+	vm_get_clock(vm, &clock_now);
+	pr_info("  KVM_GET_CLOCK:  %" PRIu64 " ns\n", (u64)clock_now.clock);
+
+	u64 host_tsc_now = clock_now.host_tsc;
+
+	for (i = 0; i < NR_VCPUS; i++) {
+		struct pvclock_vcpu_time_info *pvti_page =
+			addr_gpa2hva(vm, KVMCLOCK_GPA + i * getpagesize());
+		u64 guest_tsc_now = host_to_guest_tsc(&scale_dst, ofs_dst[i],
+						      host_tsc_now);
+		u64 clk_from_src = __pvclock_read_cycles(&pvti_src[i],
+						       guest_tsc_now);
+		u64 clk_from_dst = __pvclock_read_cycles(pvti_page,
+						       guest_tsc_now);
+		s64 pvclock_delta = (s64)(clk_from_src - clk_from_dst);
+
+		pr_info("  vCPU%d pvclock direct: src=%" PRIu64 " dst=%" PRIu64
+			" delta=%" PRId64 " ns\n", i, clk_from_src,
+			clk_from_dst, pvclock_delta);
+
+		TEST_ASSERT(pvclock_delta >= -2 && pvclock_delta <= 2,
+			    "vCPU%d pvclock src vs dst disagree by %" PRId64 " ns",
+			    i, pvclock_delta);
+	}
+
+	/*
+	 * Tight ABA: compare pvclock_read() directly (no ioctl) against
+	 * CLOCK_MONOTONIC_RAW. The spread should be much smaller since
+	 * there's no syscall between the two clock_gettime calls — just
+	 * rdtsc + userspace mul/shift.
+	 */
+	struct pvclock_vcpu_time_info *pvti0 = addr_gpa2hva(vm, KVMCLOCK_GPA);
+	u64 tight_mono_lo = 0, tight_mono_hi = 0, tight_kvm = 0;
+	u64 tight_best_spread = UINT64_MAX;
+	for (i = 0; i < 3; i++) {
+		struct timespec ts1, ts2;
+		u64 m1, m2, tsc, clk;
+
+		clock_gettime(CLOCK_MONOTONIC_RAW, &ts1);
+		tsc = rdtsc();
+		clock_gettime(CLOCK_MONOTONIC_RAW, &ts2);
+
+		m1 = ts1.tv_sec * 1000000000ULL + ts1.tv_nsec;
+		m2 = ts2.tv_sec * 1000000000ULL + ts2.tv_nsec;
+
+		/* Scale host TSC to vCPU0's guest TSC */
+		tsc = host_to_guest_tsc(&scale_dst, ofs_dst[0], tsc);
+
+		clk = __pvclock_read_cycles(pvti0, tsc);
+
+		if (m2 - m1 < tight_best_spread) {
+			tight_best_spread = m2 - m1;
+			tight_mono_lo = m1;
+			tight_mono_hi = m2;
+			tight_kvm = clk;
+		}
+	}
+	pr_info("  Tight ABA spread: %" PRIu64 " ns (best of 3)\n", tight_best_spread);
+
+	s64 tight_expected_lo = delta_before_lo + tsc_ofs_delta_ns;
+	s64 tight_expected_hi = delta_before_hi + tsc_ofs_delta_ns;
+	s64 tight_actual_lo = (s64)(tight_kvm - tight_mono_hi);
+	s64 tight_actual_hi = (s64)(tight_kvm - tight_mono_lo);
+	s64 tight_actual_mid = ((tight_actual_lo + tight_actual_hi) / 2) - delta_before;
+	s64 tight_actual_err = (s64)(tight_mono_hi - tight_mono_lo) / 2;
+
+	pr_info("  Tight kvmclock-mono shift: expected %" PRId64
+		" ns (±%" PRId64 "), measured %" PRId64 " ns (±%" PRId64 ")\n",
+		expected_mid, expected_err, tight_actual_mid, tight_actual_err);
+
+	TEST_ASSERT(tight_expected_hi >= tight_actual_lo &&
+		    tight_expected_lo <= tight_actual_hi,
+		    "Tight ABA ranges don't overlap");
+
+	kvm_vm_free(vm);
+	pr_info("PASS: kvmclock offset matches TSC delta from CLOCK_REALTIME"
+		" (%" PRId64 " ns) within ABA bounds\n", tsc_ofs_delta_ns);
+	return 0;
+}
-- 
2.43.0

[PATCH v13 19/26] KVM: selftests: Add KVM/PV clock selftest to prove timer correction

[David Woodhouse]

From: Jack Allister <jalliste@amazon.com>

A VM's KVM/PV clock has an inherent relationship to its TSC. When either
the host system live-updates or the VM is live-migrated this pairing of
the two clock sources should stay the same. In reality this is not the
case without some correction taking place.

The KVM_GET_CLOCK_GUEST/KVM_SET_CLOCK_GUEST ioctls can be used to
perform a correction on the PVTI (PV time information) structure held by
KVM to effectively fix up the kvmclock_offset prior to the guest VM
resuming in either a live-update/migration scenario.

This test proves that without the necessary fixup there is a perceived
change in the guest TSC and KVM/PV clock relationship before and after a
simulated LU/LM takes place, and that the correction eliminates it.

The test:
  1. Snapshots the PVTI at boot (PVTI0).
  2. Steps the KVM clock forward 10 seconds with KVM_SET_CLOCK and
     induces a change in PVTI data, so the divergence the correction
     must eliminate is deterministic.
  3. Snapshots the PVTI after the change (PVTI1).
  4. Requests correction via KVM_SET_CLOCK_GUEST using PVTI0.
  5. Snapshots the PVTI after correction (PVTI2).

Then samples the TSC at a single point in time and calculates the KVM
clock using each PVTI snapshot, verifying that the uncorrected clock
diverged and that the corrected clock matches the boot clock to
within ±2ns.

Also test PVCLOCK_TSC_STABLE_BIT across matched, offset-mismatched
and frequency-mismatched vCPUs, and KVM_[GS]ET_CLOCK_GUEST on vCPUs
with divergent TSC offsets.

The test enumerates multiple TSC frequencies from 1GHz to 5GHz at 500MHz
steps, crossing the 32-bit boundary, to exercise the scaling path at
various ratios. The sleep duration between snapshots is configurable via
the -s/--sleep command line option.

The divergent-offset test steps the VM clock forward between the GET
and the SET and asserts the divergence, so a silently ignored SET
cannot pass — the same discrimination the single-vCPU path already
has. The pvclock arithmetic and guest-side snapshot protocol come from
the shared tools/arch/x86/include/asm/pvclock.h helpers, and only the
scaled-frequency sweep and frequency-mismatch case require
KVM_CAP_TSC_CONTROL; the correction and offset tests still run without
hardware TSC scaling.

Co-developed-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Signed-off-by: Jack Allister <jalliste@amazon.com>
Reviewed-by: Paul Durrant <paul@xen.org>
Cc: Dongli Zhang <dongli.zhang@oracle.com>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../testing/selftests/kvm/x86/pvclock_test.c  | 462 ++++++++++++++++++
 2 files changed, 463 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/pvclock_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 18605318860e..15a1a8f64723 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -114,6 +114,7 @@ TEST_GEN_PROGS_x86 += x86/pmu_event_filter_test
 TEST_GEN_PROGS_x86 += x86/private_mem_conversions_test
 TEST_GEN_PROGS_x86 += x86/private_mem_kvm_exits_test
 TEST_GEN_PROGS_x86 += x86/pvclock_migration_test
+TEST_GEN_PROGS_x86 += x86/pvclock_test
 TEST_GEN_PROGS_x86 += x86/save_restore_pf_stress_test
 TEST_GEN_PROGS_x86 += x86/set_boot_cpu_id
 TEST_GEN_PROGS_x86 += x86/set_sregs_test
diff --git a/tools/testing/selftests/kvm/x86/pvclock_test.c b/tools/testing/selftests/kvm/x86/pvclock_test.c
new file mode 100644
index 000000000000..cb57f5e2bf1e
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/pvclock_test.c
@@ -0,0 +1,462 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright © Amazon.com, Inc. or its affiliates.
+ *
+ * Tests for pvclock API
+ * KVM_SET_CLOCK_GUEST/KVM_GET_CLOCK_GUEST
+ */
+#include <getopt.h>
+#include <stdint.h>
+#include <string.h>
+#include <time.h>
+#include <unistd.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+
+#include <asm/pvclock.h>
+#include <asm/pvclock-abi.h>
+
+static inline void pvti_snapshot(struct pvclock_vcpu_time_info *dst,
+				 volatile struct pvclock_vcpu_time_info *src)
+{
+	const struct pvclock_vcpu_time_info *p = (void *)src;
+	u32 version;
+
+	do {
+		version = pvclock_read_begin(p);
+		*dst = *p;
+	} while (pvclock_read_retry(p, version));
+}
+
+enum {
+	STAGE_FIRST_BOOT,
+	STAGE_UNCORRECTED,
+	STAGE_CORRECTED
+};
+
+#define KVMCLOCK_GPA	0xc0000000ull
+#define KVMCLOCK_SIZE	sizeof(struct pvclock_vcpu_time_info)
+
+static void trigger_pvti_update(void)
+{
+	/*
+	 * Toggle between KVM's old and new system time methods to coerce KVM
+	 * into updating the fields in the PV time info struct.
+	 */
+	wrmsr(MSR_KVM_SYSTEM_TIME, KVMCLOCK_GPA | KVM_MSR_ENABLED);
+	wrmsr(MSR_KVM_SYSTEM_TIME_NEW, KVMCLOCK_GPA | KVM_MSR_ENABLED);
+}
+
+static void guest_code(void)
+{
+	struct pvclock_vcpu_time_info *pvti =
+		(void *)(unsigned long)KVMCLOCK_GPA;
+	struct pvclock_vcpu_time_info pvti_boot;
+	struct pvclock_vcpu_time_info pvti_uncorrected;
+	struct pvclock_vcpu_time_info pvti_corrected;
+	u64 tsc_guest;
+	u64 clk_boot, clk_uncorrected, clk_corrected;
+	s64 delta_corrected;
+
+	/* Set up kvmclock and snapshot the initial pvclock parameters. */
+	wrmsr(MSR_KVM_SYSTEM_TIME_NEW, KVMCLOCK_GPA | KVM_MSR_ENABLED);
+	pvti_snapshot(&pvti_boot, pvti);
+	GUEST_SYNC(STAGE_FIRST_BOOT);
+
+	/*
+	 * Trigger an update of the PVTI. Calculating the KVM clock using this
+	 * updated structure will show a delta from the original.
+	 */
+	trigger_pvti_update();
+	pvti_snapshot(&pvti_uncorrected, pvti);
+	GUEST_SYNC(STAGE_UNCORRECTED);
+
+	/*
+	 * Snapshot the corrected time (the host does KVM_SET_CLOCK_GUEST when
+	 * handling STAGE_UNCORRECTED).
+	 */
+	pvti_snapshot(&pvti_corrected, pvti);
+
+	/*
+	 * Sample the TSC at a single point in time, then calculate the
+	 * effective KVM clock using the PVTI from each stage. Verify that the
+	 * corrected clock matches the boot clock to within ±2ns.
+	 */
+	tsc_guest = rdtsc();
+
+	clk_boot = __pvclock_read_cycles(&pvti_boot, tsc_guest);
+	clk_uncorrected = __pvclock_read_cycles(&pvti_uncorrected, tsc_guest);
+	clk_corrected = __pvclock_read_cycles(&pvti_corrected, tsc_guest);
+
+	/*
+	 * The host stepped the KVM clock forward by 10 seconds at
+	 * STAGE_FIRST_BOOT, so the uncorrected clock must diverge; without
+	 * this, a no-op KVM_SET_CLOCK_GUEST would pass the test below.
+	 */
+	__GUEST_ASSERT((s64)(clk_uncorrected - clk_boot) > NSEC_PER_SEC,
+		       "uncorrected clock did not diverge (boot=%lu uncorrected=%lu)",
+		       clk_boot, clk_uncorrected);
+
+	delta_corrected = clk_boot - clk_corrected;
+
+	__GUEST_ASSERT(delta_corrected >= -2 && delta_corrected <= 2,
+		       "corrected delta %ld out of range (boot=%lu uncorrected=%lu corrected=%lu)",
+		       delta_corrected, clk_boot, clk_uncorrected, clk_corrected);
+
+	GUEST_SYNC(STAGE_CORRECTED);
+}
+
+static void run_test(struct kvm_vm *vm, struct kvm_vcpu *vcpu,
+		     unsigned int sleep_sec)
+{
+	struct pvclock_vcpu_time_info pvti_before;
+	struct kvm_clock_data clock_step;
+	struct ucall uc;
+
+	for (;;) {
+		vcpu_run(vcpu);
+		TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+
+		switch (get_ucall(vcpu, &uc)) {
+		case UCALL_ABORT:
+			REPORT_GUEST_ASSERT(uc);
+			break;
+		case UCALL_SYNC:
+			break;
+		default:
+			TEST_FAIL("Unexpected ucall");
+		}
+
+		switch (uc.args[1]) {
+		case STAGE_FIRST_BOOT:
+			/* Save the pvclock parameters before the update. */
+			vcpu_get_clock_guest(vcpu, &pvti_before);
+
+			/* Sleep to let the clocks diverge. */
+			sleep(sleep_sec);
+
+			/*
+			 * Step the KVM clock forward 10 seconds so that the
+			 * divergence the correction must eliminate is
+			 * deterministic, not at the mercy of accumulated
+			 * drift during the sleep.
+			 */
+			vm_get_clock(vm, &clock_step);
+			clock_step.clock += 10ull * NSEC_PER_SEC;
+			clock_step.flags = 0;
+			vm_set_clock(vm, &clock_step);
+			break;
+
+		case STAGE_UNCORRECTED:
+			/* Restore the original pvclock parameters. */
+			vcpu_set_clock_guest(vcpu, &pvti_before);
+			break;
+
+		case STAGE_CORRECTED:
+			/* Guest verified the delta in-guest. */
+			return;
+
+		default:
+			TEST_FAIL("Unknown stage %lu", uc.args[1]);
+		}
+	}
+}
+
+static void configure_pvclock(struct kvm_vm *vm)
+{
+	unsigned int nr_pages;
+
+	nr_pages = vm_calc_num_guest_pages(VM_MODE_DEFAULT, getpagesize());
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS,
+				    KVMCLOCK_GPA, 1, nr_pages, 0);
+	virt_map(vm, KVMCLOCK_GPA, KVMCLOCK_GPA, nr_pages);
+}
+
+static void run_at_frequency(u64 tsc_khz, unsigned int sleep_sec)
+{
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+
+	pr_info("Testing at TSC frequency %lu kHz\n", tsc_khz);
+	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	configure_pvclock(vm);
+	vcpu_ioctl(vcpu, KVM_SET_TSC_KHZ, (void *)tsc_khz);
+	run_test(vm, vcpu, sleep_sec);
+	kvm_vm_free(vm);
+}
+
+static void test_tsc_stable_bit(void);
+static void test_clock_guest_with_offsets(void);
+
+static void usage(const char *name)
+{
+	printf("Usage: %s [options]\n"
+	       "  -s, --sleep SEC     sleep duration between snapshots (default: 2)\n"
+	       "  -h, --help          show this help\n", name);
+}
+
+int main(int argc, char *argv[])
+{
+	static const struct option long_opts[] = {
+		{ "sleep", required_argument, NULL, 's' },
+		{ "help",  no_argument,       NULL, 'h' },
+		{ NULL,    0,                  NULL,  0  },
+	};
+	unsigned int sleep_sec = 2;
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	u64 host_khz;
+	u64 freq;
+	int opt;
+
+	while ((opt = getopt_long(argc, argv, "s:h", long_opts, NULL)) != -1) {
+		switch (opt) {
+		case 's':
+			sleep_sec = atoi_non_negative("sleep", optarg);
+			break;
+		case 'h':
+		default:
+			usage(argv[0]);
+			return opt == 'h' ? 0 : 1;
+		}
+	}
+
+	TEST_REQUIRE(sys_clocksource_is_based_on_tsc());
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_CLOCK_GUEST));
+
+	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	configure_pvclock(vm);
+
+	/* First run at native frequency (no scaling). */
+	run_test(vm, vcpu, sleep_sec);
+
+	/*
+	 * Then enumerate a range of TSC frequencies crossing the 32-bit
+	 * boundary, to exercise the scaling path at various ratios.
+	 */
+	host_khz = __vcpu_ioctl(vcpu, KVM_GET_TSC_KHZ, NULL);
+	kvm_vm_free(vm);
+
+	if (kvm_has_cap(KVM_CAP_TSC_CONTROL)) {
+		for (freq = 1000000; freq <= 5000000; freq += 500000) {
+			if (freq == host_khz)
+				continue;
+			run_at_frequency(freq, sleep_sec);
+		}
+	} else {
+		pr_info("Skipping scaled frequencies (no KVM_CAP_TSC_CONTROL)\n");
+	}
+
+	test_tsc_stable_bit();
+	test_clock_guest_with_offsets();
+
+	return 0;
+}
+
+static volatile u32 vcpu_counter;
+static void guest_code_stable_bit(void)
+{
+	u32 idx = __atomic_fetch_add(&vcpu_counter, 1, __ATOMIC_SEQ_CST);
+	u64 gpa = KVMCLOCK_GPA + idx * KVMCLOCK_SIZE;
+
+	wrmsr(MSR_KVM_SYSTEM_TIME_NEW, gpa | KVM_MSR_ENABLED);
+	GUEST_SYNC(0);
+	GUEST_SYNC(0);
+	GUEST_SYNC(0);
+}
+
+static void run_vcpu_once(struct kvm_vcpu *vcpu)
+{
+	struct ucall uc;
+
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	switch (get_ucall(vcpu, &uc)) {
+	case UCALL_ABORT:
+		REPORT_GUEST_ASSERT(uc);
+		break;
+	case UCALL_SYNC:
+		break;
+	default:
+		TEST_FAIL("Unexpected ucall");
+	}
+}
+
+static void test_tsc_stable_bit(void)
+{
+	struct pvclock_vcpu_time_info pvti;
+	struct kvm_vcpu *vcpus[2];
+	struct kvm_vm *vm;
+	int ret;
+
+	pr_info("Testing PVCLOCK_TSC_STABLE_BIT with matched/unmatched TSCs\n");
+
+	vm = vm_create_with_vcpus(2, guest_code_stable_bit, vcpus);
+	configure_pvclock(vm);
+
+	/*
+	 * Case 1: All TSCs matched (same frequency and offset).
+	 * Master clock should be active, PVCLOCK_TSC_STABLE_BIT set.
+	 */
+	run_vcpu_once(vcpus[0]);
+
+	vcpu_get_clock_guest(vcpus[0], &pvti);
+	TEST_ASSERT(pvti.flags & PVCLOCK_TSC_STABLE_BIT,
+		    "PVCLOCK_TSC_STABLE_BIT should be set with matched TSCs");
+
+	/*
+	 * Case 2: Different TSC offset, same frequency.
+	 * Master clock should still be active (frequency matches), but
+	 * PVCLOCK_TSC_STABLE_BIT should be cleared (offsets differ).
+	 */
+	vcpu_set_tsc_offset(vcpus[1], 12345678);
+	run_vcpu_once(vcpus[1]);
+	run_vcpu_once(vcpus[0]);
+
+	ret = __vcpu_get_clock_guest(vcpus[0], &pvti);
+	if (ret) {
+		/*
+		 * Master clock disabled by offset mismatch — old kernel.
+		 * -ENODATA is the only legitimate reason to skip; anything
+		 * else is a regression, not an old kernel.
+		 */
+		TEST_ASSERT(errno == ENODATA,
+			    "GET_CLOCK_GUEST failed with unexpected errno %d", errno);
+		pr_info("  Skipping offset test (master clock requires matched offsets)\n");
+		goto case_freq;
+	}
+	TEST_ASSERT(!(pvti.flags & PVCLOCK_TSC_STABLE_BIT),
+		    "PVCLOCK_TSC_STABLE_BIT should be clear with offset-mismatched TSCs");
+
+case_freq:
+	/*
+	 * Case 3: Different TSC frequency.
+	 * Master clock should be disabled entirely.
+	 */
+	if (!kvm_has_cap(KVM_CAP_TSC_CONTROL)) {
+		pr_info("  Skipping frequency-mismatch case (no KVM_CAP_TSC_CONTROL)\n");
+		kvm_vm_free(vm);
+		return;
+	}
+	vcpu_ioctl(vcpus[1], KVM_SET_TSC_KHZ,
+		   (void *)(unsigned long)(__vcpu_ioctl(vcpus[1], KVM_GET_TSC_KHZ, NULL) / 2));
+	/* Write TSC to trigger kvm_synchronize_tsc / kvm_track_tsc_matching */
+	vcpu_set_msr(vcpus[1], MSR_IA32_TSC, 0);
+	run_vcpu_once(vcpus[1]);
+
+	ret = __vcpu_get_clock_guest(vcpus[0], &pvti);
+	TEST_ASSERT(ret && errno == ENODATA,
+		    "GET_CLOCK_GUEST should fail with frequency-mismatched TSCs, got %d (errno %d)",
+		    ret, errno);
+
+	kvm_vm_free(vm);
+}
+
+static void test_clock_guest_with_offsets(void)
+{
+	struct pvclock_vcpu_time_info pvti0, pvti1, pvti1_after;
+	struct kvm_vcpu *vcpus[2];
+	struct kvm_vm *vm;
+	s64 delta;
+	int ret;
+
+	pr_info("Testing KVM_[GS]ET_CLOCK_GUEST with different TSC offsets\n");
+
+	vm = vm_create_with_vcpus(2, guest_code_stable_bit, vcpus);
+	configure_pvclock(vm);
+
+	/* Set different TSC offsets on the two vCPUs */
+	vcpu_set_tsc_offset(vcpus[0], 0);
+	vcpu_set_tsc_offset(vcpus[1], 1000000000ull);
+
+	/* Run both to establish kvmclock */
+	run_vcpu_once(vcpus[0]);
+	run_vcpu_once(vcpus[1]);
+
+	/* GET_CLOCK_GUEST on both — should succeed (master clock active) */
+	ret = __vcpu_get_clock_guest(vcpus[0], &pvti0);
+	if (ret) {
+		TEST_ASSERT(errno == ENODATA,
+			    "GET_CLOCK_GUEST failed with unexpected errno %d", errno);
+		pr_info("  Skipping (master clock requires matched offsets on this kernel)\n");
+		kvm_vm_free(vm);
+		return;
+	}
+	vcpu_get_clock_guest(vcpus[1], &pvti1);
+
+	/* The tsc_timestamps should differ (different offsets) */
+	TEST_ASSERT(pvti0.tsc_timestamp != pvti1.tsc_timestamp,
+		    "tsc_timestamps should differ with different offsets");
+
+	/*
+	 * Step the VM clock forward so that a silently ignored
+	 * KVM_SET_CLOCK_GUEST cannot pass the delta check below: after
+	 * the step, the uncorrected clock computed from pvti0 must
+	 * diverge from the stepped clock, and only an applied SET
+	 * restores the original timeline.
+	 */
+	{
+		struct kvm_clock_data clock_step;
+
+		vm_get_clock(vm, &clock_step);
+		clock_step.clock += 10ull * NSEC_PER_SEC;
+		clock_step.flags = 0;
+		vm_set_clock(vm, &clock_step);
+	}
+
+	{
+		struct pvclock_vcpu_time_info pvti_stepped;
+		s64 diverged;
+
+		vcpu_get_clock_guest(vcpus[0], &pvti_stepped);
+		diverged = (s64)(__pvclock_read_cycles(&pvti_stepped,
+						     pvti_stepped.tsc_timestamp) -
+				 __pvclock_read_cycles(&pvti0,
+						     pvti_stepped.tsc_timestamp));
+		TEST_ASSERT(diverged > (s64)NSEC_PER_SEC,
+			    "clock did not diverge after step, got %ld ns",
+			    diverged);
+	}
+
+	/* Restore vcpu0's original clock */
+	vcpu_set_clock_guest(vcpus[0], &pvti0);
+
+	/* Run vcpu0 to process the clock update */
+	run_vcpu_once(vcpus[0]);
+
+	/*
+	 * After SET on vcpu0, verify the correction worked by getting
+	 * the clock on vcpu0 again. The mul/shift should be the same,
+	 * and computing kvmclock at the same TSC should give the same
+	 * result as the original (within ±2ns).
+	 */
+	{
+		struct pvclock_vcpu_time_info pvti0_after;
+		u64 tsc_now, clk_from_old, clk_from_new;
+
+		vcpu_get_clock_guest(vcpus[0], &pvti0_after);
+
+		TEST_ASSERT(pvti0_after.tsc_to_system_mul == pvti0.tsc_to_system_mul &&
+			    pvti0_after.tsc_shift == pvti0.tsc_shift,
+			    "mul/shift changed across KVM_SET_CLOCK_GUEST");
+
+		tsc_now = pvti0_after.tsc_timestamp;
+		clk_from_old = __pvclock_read_cycles(&pvti0, tsc_now);
+		clk_from_new = __pvclock_read_cycles(&pvti0_after, tsc_now);
+
+		delta = (s64)clk_from_new - (s64)clk_from_old;
+		TEST_ASSERT(delta >= -2 && delta <= 2,
+			    "clock correction delta should be <=2ns, got %ld ns",
+			    delta);
+	}
+
+	/*
+	 * Also verify that vcpu1's clock is still accessible (master
+	 * clock still active with different offsets).
+	 */
+	vcpu_get_clock_guest(vcpus[1], &pvti1_after);
+
+	kvm_vm_free(vm);
+}
-- 
2.43.0

[PATCH v13 20/26] KVM: selftests: Add master clock offset test

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Verify that KVM master clock mode remains active when vCPUs have
different TSC offsets but the same frequency. Creates three vCPUs,
sets one to a different TSC value, and confirms:

 - KVM_CLOCK_HOST_TSC is set (master clock active)
 - KVM_CLOCK_TSC_STABLE IS set (master clock ⇒ stable flag)
 - PVCLOCK_TSC_STABLE_BIT is NOT set (offsets differ)

Then set the KVM clock forward one hour with KVM_SET_CLOCK and verify
that all vCPUs still agree on the KVM clock computed from their own
pvclock parameters.

This remains a separate binary from pvclock_test deliberately: the
rdtsc()-plus-offset arithmetic here is exact only at the default host
frequency, while pvclock_test sweeps scaled frequencies, and the two
must not inherit each other's constraints.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../kvm/x86/masterclock_offset_test.c         | 249 ++++++++++++++++++
 2 files changed, 250 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/masterclock_offset_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 15a1a8f64723..4d4eff41c525 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -95,6 +95,7 @@ TEST_GEN_PROGS_x86 += x86/hyperv_tlb_flush
 TEST_GEN_PROGS_x86 += x86/kvm_clock_test
 TEST_GEN_PROGS_x86 += x86/kvm_pv_test
 TEST_GEN_PROGS_x86 += x86/kvm_buslock_test
+TEST_GEN_PROGS_x86 += x86/masterclock_offset_test
 TEST_GEN_PROGS_x86 += x86/monitor_mwait_test
 TEST_GEN_PROGS_x86 += x86/msrs_test
 TEST_GEN_PROGS_x86 += x86/nested_close_kvm_test
diff --git a/tools/testing/selftests/kvm/x86/masterclock_offset_test.c b/tools/testing/selftests/kvm/x86/masterclock_offset_test.c
new file mode 100644
index 000000000000..7d0fe97cedc4
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/masterclock_offset_test.c
@@ -0,0 +1,249 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Test that KVM master clock mode works with different TSC offsets
+ * as long as all vCPUs have the same TSC frequency.
+ */
+#include <string.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+
+#include <asm/pvclock.h>
+#include <asm/pvclock-abi.h>
+
+#define KVMCLOCK_GPA	0xc0000000ull
+#define TSC_OFFSET	(1000000000ULL)
+
+#define NR_VCPUS	3
+
+/* Read the per-vCPU pvclocks in an order other than creation order. */
+static const int vcpu_order[NR_VCPUS] = {0, 2, 1};
+
+static void guest_code(u64 pvti_gpa)
+{
+	wrmsr(MSR_KVM_SYSTEM_TIME_NEW, pvti_gpa | KVM_MSR_ENABLED);
+	for (;;)
+		GUEST_SYNC(0);
+}
+
+static void get_pvtis(struct kvm_vcpu **vcpus,
+		      struct pvclock_vcpu_time_info *pvti)
+{
+	int i;
+
+	for (i = 0; i < NR_VCPUS; i++) {
+		int idx = vcpu_order[i];
+
+		vcpu_get_clock_guest(vcpus[idx], &pvti[idx]);
+		pr_info("vCPU %d: tsc_timestamp=%lu system_time=%lu "
+			"mul=%u shift=%d flags=0x%x\n",
+			idx, (unsigned long)pvti[idx].tsc_timestamp,
+			(unsigned long)pvti[idx].system_time,
+			pvti[idx].tsc_to_system_mul, pvti[idx].tsc_shift,
+			pvti[idx].flags);
+	}
+}
+
+static void run_all_vcpus(struct kvm_vcpu **vcpus)
+{
+	int i;
+
+	for (i = 0; i < NR_VCPUS; i++) {
+		vcpu_run(vcpus[vcpu_order[i]]);
+		TEST_ASSERT_KVM_EXIT_REASON(vcpus[vcpu_order[i]], KVM_EXIT_IO);
+	}
+}
+
+/*
+ * The guest-visible pvclock pages must carry exactly what the ioctl
+ * reports: in master clock mode both are derived from the same master
+ * snapshot. Requires every vCPU to have run since the last masterclock
+ * update, so the pages are written from the settled snapshot.
+ */
+static void assert_guest_pages_match(struct kvm_vm *vm,
+				     struct pvclock_vcpu_time_info *pvti,
+				     const char *what)
+{
+	int i;
+
+	for (i = 0; i < NR_VCPUS; i++) {
+		struct pvclock_vcpu_time_info *guest_pvti =
+			addr_gpa2hva(vm, KVMCLOCK_GPA + i *
+				     sizeof(struct pvclock_vcpu_time_info) * 2);
+
+		TEST_ASSERT(guest_pvti->tsc_timestamp == pvti[i].tsc_timestamp &&
+			    guest_pvti->system_time == pvti[i].system_time &&
+			    guest_pvti->tsc_to_system_mul == pvti[i].tsc_to_system_mul &&
+			    guest_pvti->tsc_shift == pvti[i].tsc_shift,
+			    "%s: vCPU%d guest page pvclock should match KVM_GET_CLOCK_GUEST",
+			    what, i);
+	}
+}
+
+/*
+ * Read the host TSC once and feed it through each vCPU's pvclock
+ * (adding each vCPU's TSC offset). All must compute exactly the same
+ * kvmclock value.
+ */
+static void assert_pvclocks_match(struct pvclock_vcpu_time_info *pvti,
+				  u64 offset0, const char *what)
+{
+	u64 host_tsc, clk0, clk1, clk2;
+
+	host_tsc = rdtsc();
+	clk0 = __pvclock_read_cycles(&pvti[0], host_tsc + offset0);
+	clk1 = __pvclock_read_cycles(&pvti[1], host_tsc + offset0);
+	clk2 = __pvclock_read_cycles(&pvti[2], host_tsc + offset0 + TSC_OFFSET);
+
+	pr_info("%s: kvmclock via vCPU 0: %lu ns, vCPU 1: %lu ns, vCPU 2: %lu ns\n",
+		what, (unsigned long)clk0, (unsigned long)clk1,
+		(unsigned long)clk2);
+	TEST_ASSERT(clk0 == clk1 && clk0 == clk2,
+		    "%s: kvmclock from all vCPUs should match exactly "
+		    "(0: %lu, 1: %lu, 2: %lu)", what, (unsigned long)clk0,
+		    (unsigned long)clk1, (unsigned long)clk2);
+}
+
+static void test_masterclock_active(struct kvm_vm *vm,
+				    struct kvm_clock_data *clock)
+{
+	memset(clock, 0, sizeof(*clock));
+	vm_get_clock(vm, clock);
+	pr_info("KVM_GET_CLOCK flags: 0x%x\n", clock->flags);
+	TEST_ASSERT(clock->flags & KVM_CLOCK_HOST_TSC,
+		    "Master clock should be active, flags=0x%x", clock->flags);
+	TEST_ASSERT(clock->flags & KVM_CLOCK_TSC_STABLE,
+		    "KVM_CLOCK_TSC_STABLE should be set, flags=0x%x",
+		    clock->flags);
+}
+
+static void test_offset_consistency(struct kvm_vcpu **vcpus,
+				    struct pvclock_vcpu_time_info *pvti,
+				    u64 offset0)
+{
+	u64 gtsc0, gtsc1, gtsc2;
+
+	get_pvtis(vcpus, pvti);
+
+	/* Read guest TSCs: should see (0+OFF) < 2 < (1+OFF) */
+	gtsc0 = vcpu_get_msr(vcpus[0], MSR_IA32_TSC);
+	gtsc2 = vcpu_get_msr(vcpus[2], MSR_IA32_TSC);
+	gtsc1 = vcpu_get_msr(vcpus[1], MSR_IA32_TSC);
+	pr_info("Guest TSCs: vcpu0=%lu vcpu2=%lu vcpu1=%lu\n",
+		(unsigned long)gtsc0, (unsigned long)gtsc2,
+		(unsigned long)gtsc1);
+	TEST_ASSERT(gtsc0 + TSC_OFFSET < gtsc2 && gtsc2 < gtsc1 + TSC_OFFSET,
+		    "Expected (vcpu0+OFF) < vcpu2 < (vcpu1+OFF)");
+
+	/* PVCLOCK_TSC_STABLE_BIT should NOT be set (offsets differ) */
+	TEST_ASSERT(!(pvti[2].flags & PVCLOCK_TSC_STABLE_BIT),
+		    "PVCLOCK_TSC_STABLE_BIT should NOT be set, flags=0x%x",
+		    pvti[2].flags);
+
+	/* Same mul/shift */
+	TEST_ASSERT(pvti[0].tsc_to_system_mul == pvti[1].tsc_to_system_mul &&
+		    pvti[0].tsc_to_system_mul == pvti[2].tsc_to_system_mul &&
+		    pvti[0].tsc_shift == pvti[1].tsc_shift &&
+		    pvti[0].tsc_shift == pvti[2].tsc_shift,
+		    "All vCPUs should have same mul/shift");
+
+	assert_pvclocks_match(pvti, offset0, "Initial");
+}
+
+#define ONE_HOUR_NS (3600ULL * NSEC_PER_SEC)
+
+static void test_set_clock_consistency(struct kvm_vm *vm,
+				       struct kvm_vcpu **vcpus,
+				       struct pvclock_vcpu_time_info *pvti,
+				       struct kvm_clock_data *clock,
+				       u64 offset0)
+{
+	struct kvm_clock_data setclk = { .clock = clock->clock + ONE_HOUR_NS };
+	u64 clk0;
+
+	/*
+	 * Add an hour to the VM kvmclock via KVM_SET_CLOCK, run each
+	 * vCPU to pick up the update, and check they're still in sync.
+	 */
+	vm_set_clock(vm, &setclk);
+
+	/*
+	 * KVM_SET_CLOCK refreshes the master snapshot synchronously and
+	 * broadcasts a clock update, so a single pass rewrites every
+	 * guest pvclock page from the settled snapshot. Run two passes
+	 * anyway, so this flow doesn't silently depend on that: after a
+	 * per-vCPU write which parks a masterclock update on one vCPU
+	 * (as the TSC-offset writes in main() do), the first pass
+	 * services it and only the second rewrites every page.
+	 */
+	run_all_vcpus(vcpus);
+	run_all_vcpus(vcpus);
+
+	get_pvtis(vcpus, pvti);
+	assert_guest_pages_match(vm, pvti, "After +1h");
+
+	TEST_ASSERT(pvti[0].system_time == pvti[1].system_time &&
+		    pvti[0].system_time == pvti[2].system_time,
+		    "system_time should still match after KVM_SET_CLOCK");
+
+	assert_pvclocks_match(pvti, offset0, "After +1h");
+
+	/* Verify the clock actually moved by ~1 hour */
+	clk0 = __pvclock_read_cycles(&pvti[0], rdtsc() + offset0);
+	TEST_ASSERT(clk0 > ONE_HOUR_NS,
+		    "Clock should be > 1 hour after set, got %lu ns",
+		    (unsigned long)clk0);
+}
+
+int main(void)
+{
+	struct pvclock_vcpu_time_info pvti[NR_VCPUS];
+	struct kvm_vcpu *vcpus[NR_VCPUS];
+	struct kvm_clock_data clock;
+	struct kvm_vm *vm;
+	u64 offset0;
+	int nr_pages;
+	int i;
+
+	TEST_REQUIRE(sys_clocksource_is_based_on_tsc());
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_CLOCK_GUEST));
+
+	vm = vm_create_with_vcpus(NR_VCPUS, guest_code, vcpus);
+
+	TEST_REQUIRE(!__vcpu_has_device_attr(vcpus[0], KVM_VCPU_TSC_CTRL,
+					     KVM_VCPU_TSC_OFFSET));
+
+	nr_pages = vm_calc_num_guest_pages(VM_MODE_DEFAULT, getpagesize());
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS,
+				    KVMCLOCK_GPA, 1, nr_pages, 0);
+
+	/* Give each vCPU its own pvclock structure within the page */
+	for (i = 0; i < NR_VCPUS; i++)
+		vcpu_args_set(vcpus[i], 1, KVMCLOCK_GPA +
+			      i * sizeof(struct pvclock_vcpu_time_info) * 2);
+
+	/* Get vCPU 0's default offset and set vCPU 2's offset higher */
+	offset0 = vcpu_get_tsc_offset(vcpus[0]);
+	vcpu_set_tsc_offset(vcpus[2], offset0 + TSC_OFFSET);
+
+	/*
+	 * Run each vCPU twice to enable kvmclock (with the offset already
+	 * set): the first pass services the masterclock update requested
+	 * by the offset write, the second rewrites every guest pvclock
+	 * page from the settled snapshot.
+	 */
+	run_all_vcpus(vcpus);
+	run_all_vcpus(vcpus);
+
+	test_masterclock_active(vm, &clock);
+	test_offset_consistency(vcpus, pvti, offset0);
+	assert_guest_pages_match(vm, pvti, "Initial");
+	pr_info("PASSED: pvclock consistent across offset vCPUs\n");
+
+	test_set_clock_consistency(vm, vcpus, pvti, &clock, offset0);
+	pr_info("PASSED: pvclock still consistent after KVM_SET_CLOCK +1h\n");
+
+	kvm_vm_free(vm);
+	return 0;
+}
-- 
2.43.0

[PATCH v13 21/26] KVM: selftests: Add Xen/generic CPUID timing leaf test

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Verify that userspace can correctly populate Xen and generic CPUID
timing leaves using the KVM_GET_TSC_KHZ ioctl and
KVM_VCPU_TSC_SCALE attribute.

This validates that commit 7d3bd21e457b ("KVM: x86: Remove runtime Xen
TSC frequency CPUID update") doesn't break guests: userspace queries
the effective TSC and bus frequencies, computes the pvclock mul/shift,
populates the CPUID leaves, and the guest verifies the values match.

The test exercises:
 - KVM_GET_TSC_KHZ at native and scaled frequencies
 - KVM_VCPU_TSC_SCALE ratio verification against effective frequency
 - Generic timing leaf 0x40000010 (EAX=tsc_khz, EBX=bus_khz)
 - Xen leaf 3 sub-leaf 0 (ECX=guest TSC kHz)
 - Xen leaf 3 sub-leaf 1 (ECX=mul, EDX=shift)

Where KVM_CAP_XEN_HVM is available, the guest is given the Xen
signature leaf and the Xen hypercall MSR is enabled — the exact
configuration in which the removed runtime update used to fire — and
a sentinel is planted in leaf 3 sub-leaf 2 EAX, where it used to
write, which the guest verifies survives untouched.

The scaled runs are skipped on hardware without TSC scaling. For a
requested frequency within tsc_tolerance_ppm of the host's (where no
scaling engages and the ratio stays identity), only the ratio
cross-check is skipped, and the skip is gated on the frequencies
themselves — read from the kernel's actual tsc_tolerance_ppm — not on
the reported ratio, which is exactly what a broken KVM would get
wrong. Results go through the kselftest TAP machinery, a fixed plan
of five.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/x86/xen_cpuid_timing_test.c | 293 ++++++++++++++++++
 2 files changed, 294 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/xen_cpuid_timing_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 4d4eff41c525..79d26de18bf3 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -152,6 +152,7 @@ TEST_GEN_PROGS_x86 += x86/xss_msr_test
 TEST_GEN_PROGS_x86 += x86/debug_regs
 TEST_GEN_PROGS_x86 += x86/tsc_msrs_test
 TEST_GEN_PROGS_x86 += x86/vmx_pmu_caps_test
+TEST_GEN_PROGS_x86 += x86/xen_cpuid_timing_test
 TEST_GEN_PROGS_x86 += x86/xen_shinfo_test
 TEST_GEN_PROGS_x86 += x86/xen_vmcall_test
 TEST_GEN_PROGS_x86 += x86/sev_dbg_test
diff --git a/tools/testing/selftests/kvm/x86/xen_cpuid_timing_test.c b/tools/testing/selftests/kvm/x86/xen_cpuid_timing_test.c
new file mode 100644
index 000000000000..33b5300efa03
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/xen_cpuid_timing_test.c
@@ -0,0 +1,293 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Test that userspace can correctly populate Xen and generic CPUID
+ * timing leaves using KVM_GET_TSC_KHZ and KVM_VCPU_TSC_SCALE.
+ *
+ * This validates that the removal of KVM's runtime Xen CPUID modification
+ * doesn't break guests, because userspace has all the information needed.
+ */
+#include <stdint.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+
+
+#define XEN_CPUID_BASE		0x40000100
+#define XEN_CPUID_LEAF(n)	(XEN_CPUID_BASE + (n))
+#define GENERIC_TIMING_LEAF	0x40000010
+
+/*
+ * Sentinel in Xen leaf 3 sub-leaf 2 EAX (where the removed runtime
+ * update used to write the vCPU's tsc_khz): it must survive untouched.
+ */
+#define XEN_SENTINEL		0xdeadbeef
+
+/* Values set by host, verified by guest */
+static u32 expected_tsc_khz;
+static u32 expected_bus_khz;
+static u32 expected_tsc_mul;
+static s8   expected_tsc_shift;
+static u64 host_khz;
+static bool xen_enabled;
+
+static void guest_code(void)
+{
+	u32 eax, ebx, ecx, edx;
+
+	/* Check generic timing leaf 0x40000010 */
+	__cpuid(GENERIC_TIMING_LEAF, 0, &eax, &ebx, &ecx, &edx);
+	GUEST_ASSERT_EQ(eax, expected_tsc_khz);
+	GUEST_ASSERT_EQ(ebx, expected_bus_khz);
+
+	/* Check Xen leaf 3, sub-leaf 0: ECX = guest TSC frequency */
+	__cpuid(XEN_CPUID_LEAF(3), 0, &eax, &ebx, &ecx, &edx);
+	GUEST_ASSERT_EQ(ecx, expected_tsc_khz);
+
+	/* Check Xen leaf 3, sub-leaf 1: ECX = mul, EDX = shift */
+	__cpuid(XEN_CPUID_LEAF(3), 1, &eax, &ebx, &ecx, &edx);
+	GUEST_ASSERT_EQ(ecx, expected_tsc_mul);
+	GUEST_ASSERT_EQ((s8)edx, expected_tsc_shift);
+
+	if (xen_enabled) {
+		/*
+		 * With the Xen signature present and Xen enabled, verify the
+		 * sentinel survives: KVM's removed runtime CPUID update used
+		 * to clobber this sub-leaf for exactly this configuration.
+		 */
+		__cpuid(XEN_CPUID_LEAF(3), 2, &eax, &ebx, &ecx, &edx);
+		GUEST_ASSERT_EQ(eax, XEN_SENTINEL);
+	}
+
+	GUEST_SYNC(0);
+}
+
+static void add_cpuid_entry(struct kvm_vcpu *vcpu, u32 function,
+			    u32 index, u32 eax, u32 ebx,
+			    u32 ecx, u32 edx)
+{
+	struct kvm_cpuid2 *cpuid = vcpu->cpuid;
+	struct kvm_cpuid_entry2 *entry;
+	int n = cpuid->nent;
+
+	cpuid = realloc(vcpu->cpuid,
+			sizeof(*cpuid) + (n + 1) * sizeof(*entry));
+	TEST_ASSERT(cpuid, "Failed to grow vCPU CPUID array");
+	vcpu->cpuid = cpuid;
+	cpuid->nent = n + 1;
+
+	entry = &cpuid->entries[n];
+	memset(entry, 0, sizeof(*entry));
+	entry->function = function;
+	entry->index = index;
+	entry->flags = KVM_CPUID_FLAG_SIGNIFCANT_INDEX;
+	entry->eax = eax;
+	entry->ebx = ebx;
+	entry->ecx = ecx;
+	entry->edx = edx;
+}
+
+/*
+ * Compute pvclock mul/shift from frequency, matching kvm_get_time_scale().
+ */
+static void compute_tsc_mul_shift(u64 tsc_hz, u32 *mul, s8 *shift)
+{
+	u64 scaled = 1000000000ULL;
+	u64 base = tsc_hz;
+	s32 s = 0;
+	u32 base32;
+
+	while (base > scaled * 2 || base >> 32) {
+		base >>= 1;
+		s--;
+	}
+	base32 = (u32)base;
+	while (base32 <= scaled || scaled >> 32) {
+		if (scaled >> 32 || base32 & (1U << 31))
+			scaled >>= 1;
+		else
+			base32 <<= 1;
+		s++;
+	}
+	*mul = (u32)((scaled << 32) / base32);
+	*shift = (s8)s;
+}
+
+static void run_test(u64 tsc_khz)
+{
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	struct ucall uc;
+	u32 effective_tsc_khz, effective_bus_khz;
+	int bus_cycle_ns, ret;
+
+	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+
+	if (tsc_khz) {
+		pr_info("Testing at TSC frequency %lu kHz\n", tsc_khz);
+		vcpu_ioctl(vcpu, KVM_SET_TSC_KHZ, (void *)(unsigned long)tsc_khz);
+	} else {
+		pr_info("Testing at native TSC frequency\n");
+	}
+
+	ret = __vcpu_ioctl(vcpu, KVM_GET_TSC_KHZ, NULL);
+	TEST_ASSERT(ret > 0, "KVM_GET_TSC_KHZ failed: %d", ret);
+	effective_tsc_khz = ret;
+	bus_cycle_ns = vm_check_cap(vm, KVM_CAP_X86_APIC_BUS_CYCLES_NS);
+	effective_bus_khz = bus_cycle_ns > 0 ? 1000000 / bus_cycle_ns : 1000000;
+
+	pr_info("  Effective TSC: %u kHz, Bus: %u kHz\n", effective_tsc_khz, effective_bus_khz);
+
+	/* Also exercise KVM_VCPU_TSC_SCALE if available */
+	if (!__vcpu_has_device_attr(vcpu, KVM_VCPU_TSC_CTRL,
+				    KVM_VCPU_TSC_SCALE)) {
+		struct kvm_vcpu_tsc_scale scale;
+
+		vcpu_device_attr_get(vcpu, KVM_VCPU_TSC_CTRL,
+				     KVM_VCPU_TSC_SCALE, &scale);
+		pr_info("  TSC scale: ratio=%llu frac_bits=%llu\n",
+			scale.tsc_ratio, scale.tsc_frac_bits);
+
+		/*
+		 * A requested frequency within tsc_tolerance_ppm of the
+		 * host's does not engage hardware scaling: the effective
+		 * frequency is the requested one but the ratio stays 1.0,
+		 * so the ratio cross-check cannot hold. Skip only that
+		 * check — the CPUID leaf population and guest verification
+		 * below are about the effective frequency and remain valid.
+		 *
+		 * Judge "within tolerance" from the frequencies, not from
+		 * the reported ratio: an identity ratio for a frequency
+		 * far outside tolerance is exactly the regression the
+		 * cross-check exists to catch, and must not take the skip
+		 * path. Read the kernel's actual tsc_tolerance_ppm, and
+		 * compute the bounds exactly as kvm_set_tsc_khz() does
+		 * (multiply then truncate), so the window matches even at
+		 * its edges and when the parameter has been tuned.
+		 */
+		u64 tol_ppm = get_kvm_param_integer("tsc_tolerance_ppm");
+		u64 thresh_lo = host_khz * (1000000 - tol_ppm) / 1000000;
+		u64 thresh_hi = host_khz * (1000000 + tol_ppm) / 1000000;
+		bool within_tol = tsc_khz >= thresh_lo &&
+				  tsc_khz <= thresh_hi;
+
+		if (tsc_khz && within_tol &&
+		    scale.tsc_ratio == (1ULL << scale.tsc_frac_bits)) {
+			pr_info("  Requested frequency within host tolerance; scaling not engaged, skipping ratio cross-check\n");
+		} else if (tsc_khz) {
+			/*
+			 * Verify: applying the ratio to the host TSC frequency
+			 * should give approximately the effective frequency.
+			 */
+			u64 computed = ((__uint128_t)host_khz * scale.tsc_ratio) >> scale.tsc_frac_bits;
+			s64 diff = (s64)computed - (s64)effective_tsc_khz;
+
+			TEST_ASSERT(diff >= -1 && diff <= 1,
+				    "TSC_SCALE ratio mismatch: computed %lu vs effective %u (diff %ld)",
+				    computed, effective_tsc_khz, diff);
+		}
+	}
+
+	compute_tsc_mul_shift((u64)effective_tsc_khz * 1000,
+			      &expected_tsc_mul, &expected_tsc_shift);
+
+	expected_tsc_khz = effective_tsc_khz;
+	expected_bus_khz = effective_bus_khz;
+
+	sync_global_to_guest(vm, expected_tsc_khz);
+	sync_global_to_guest(vm, expected_bus_khz);
+	sync_global_to_guest(vm, expected_tsc_mul);
+	sync_global_to_guest(vm, expected_tsc_shift);
+
+	/* Populate CPUID leaves as a VMM would */
+	add_cpuid_entry(vcpu, GENERIC_TIMING_LEAF, 0,
+			effective_tsc_khz, effective_bus_khz, 0, 0);
+	add_cpuid_entry(vcpu, XEN_CPUID_LEAF(3), 0,
+			0, 0, effective_tsc_khz, 0);
+	add_cpuid_entry(vcpu, XEN_CPUID_LEAF(3), 1,
+			0, 0, expected_tsc_mul,
+			(u32)(s32)expected_tsc_shift);
+
+	/*
+	 * Where possible, make this a real Xen guest: install the Xen
+	 * signature leaf and enable the Xen hypercall MSR, which is the
+	 * configuration in which KVM's removed runtime CPUID update used
+	 * to fire, and plant a sentinel where it used to write.
+	 */
+	xen_enabled = kvm_has_cap(KVM_CAP_XEN_HVM);
+	if (xen_enabled) {
+		struct kvm_xen_hvm_config cfg = {
+			.msr = 0x40000200,
+		};
+
+		vm_ioctl(vm, KVM_XEN_HVM_CONFIG, &cfg);
+		add_cpuid_entry(vcpu, XEN_CPUID_BASE, 0, XEN_CPUID_LEAF(4),
+				0x566e6558 /* "XenV" */,
+				0x65584d4d /* "MMXe" */,
+				0x4d4d566e /* "nVMM" */);
+		add_cpuid_entry(vcpu, XEN_CPUID_LEAF(3), 2,
+				XEN_SENTINEL, 0, 0, 0);
+	}
+	sync_global_to_guest(vm, xen_enabled);
+
+	vcpu_set_cpuid(vcpu);
+
+	pr_info("  pvclock mul=%u shift=%d\n", expected_tsc_mul, expected_tsc_shift);
+
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+
+	switch (get_ucall(vcpu, &uc)) {
+	case UCALL_ABORT:
+		REPORT_GUEST_ASSERT(uc);
+		break;
+	case UCALL_SYNC:
+		break;
+	default:
+		TEST_FAIL("Unexpected ucall");
+	}
+
+	kvm_vm_free(vm);
+}
+
+int main(void)
+{
+	u64 freq;
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	bool have_tsc_scale;
+
+	TEST_REQUIRE(sys_clocksource_is_based_on_tsc());
+
+	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	have_tsc_scale = !__vcpu_has_device_attr(vcpu, KVM_VCPU_TSC_CTRL,
+						 KVM_VCPU_TSC_SCALE);
+	host_khz = __vcpu_ioctl(vcpu, KVM_GET_TSC_KHZ, NULL);
+	kvm_vm_free(vm);
+
+	ksft_print_header();
+	ksft_set_plan(5);
+
+	/* Native frequency */
+	run_test(0);
+	ksft_test_result_pass("native frequency (%lu kHz)\n",
+			      (unsigned long)host_khz);
+
+	/* Scaled frequencies, when hardware TSC scaling is available */
+	for (freq = 1000000; freq <= 4000000; freq += 1000000) {
+		if (!have_tsc_scale) {
+			ksft_test_result_skip("%lu kHz (no TSC scaling)\n",
+					      (unsigned long)freq);
+		} else if (freq == host_khz) {
+			ksft_test_result_skip("%lu kHz is the host frequency\n",
+					      (unsigned long)freq);
+		} else {
+			run_test(freq);
+			ksft_test_result_pass("%lu kHz\n", (unsigned long)freq);
+		}
+	}
+
+	ksft_finished();
+}
-- 
2.43.0

[PATCH v13 22/26] KVM: selftests: Add Xen runstate migration test

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Test that Xen runstate (steal time) is correctly accounted across a
simulated live migration using KVM_XEN_VCPU_ATTR and KVM_[GS]ET_CLOCK_GUEST.

The test simulates what a real VMM does during migration:
1. Creates a VM with Xen HVM config and runstate tracking
2. Runs the guest to accumulate some kvmclock time
3. Saves clock (KVM_GET_CLOCK_GUEST), TSC offset, and runstate
4. Marks the saved state as RUNSTATE_runnable (vCPU not running)
5. Destroys the source VM
6. Sleeps 10ms (simulating migration network transfer time)
7. Creates a new VM and restores all state precisely as saved
8. Runs the guest and verifies the migration gap appears as steal

The VM is placed in long mode explicitly (KVM_XEN_ATTR_TYPE_LONG_MODE)
so the kernel writes the 64-bit runstate layout; nothing in this test
writes the hypercall-page MSR, which is what would otherwise latch the
mode from the guest's EFER.

The guest TSC is transferred via the KVM_VCPU_TSC_OFFSET attribute. As
the "migration" is to a new VM on the same host, restoring the same
offset makes the guest TSC continue uninterrupted across the gap, so the
restored pvclock yields a kvmclock which has kept running too. That is
what makes the migration gap visible: the destination kvmclock is ahead
of the saved state_entry_time by (at least) the gap duration, rather
than restarting from the destination VM's creation time.

The kernel accounts the gap because: on vcpu_load, it transitions from
RUNSTATE_runnable to RUNSTATE_running, computing delta = kvmclock_now -
state_entry_time. Since kvmclock has advanced past the saved entry time
(real time elapsed during migration), the delta is added to time_runnable.

The result is reported through the kselftest TAP machinery.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/x86/xen_migration_test.c    | 227 ++++++++++++++++++
 2 files changed, 228 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/xen_migration_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 79d26de18bf3..1edc78845fd3 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -153,6 +153,7 @@ TEST_GEN_PROGS_x86 += x86/debug_regs
 TEST_GEN_PROGS_x86 += x86/tsc_msrs_test
 TEST_GEN_PROGS_x86 += x86/vmx_pmu_caps_test
 TEST_GEN_PROGS_x86 += x86/xen_cpuid_timing_test
+TEST_GEN_PROGS_x86 += x86/xen_migration_test
 TEST_GEN_PROGS_x86 += x86/xen_shinfo_test
 TEST_GEN_PROGS_x86 += x86/xen_vmcall_test
 TEST_GEN_PROGS_x86 += x86/sev_dbg_test
diff --git a/tools/testing/selftests/kvm/x86/xen_migration_test.c b/tools/testing/selftests/kvm/x86/xen_migration_test.c
new file mode 100644
index 000000000000..e316e6cd82de
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/xen_migration_test.c
@@ -0,0 +1,227 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Test Xen runstate (steal time) preservation across simulated migration.
+ *
+ * Verifies that the kernel correctly accounts the migration gap as
+ * steal time (runnable) when runstate data is saved and restored
+ * precisely, but real time elapses during the migration.
+ *
+ * The key insight: userspace saves the runstate with state=RUNSTATE_runnable
+ * (the vCPU is not running during migration). On restore, the kernel sees
+ * that kvmclock has advanced past state_entry_time, and accounts the
+ * difference as time spent in the runnable state.
+ */
+#include <inttypes.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+
+#include <asm/pvclock-abi.h>
+
+#define SHINFO_GPA	0xc0000000ULL
+#define RUNSTATE_GPA	(SHINFO_GPA + 0x1000)
+
+#define RUNSTATE_running  0
+#define RUNSTATE_runnable 1
+#define RUNSTATE_blocked  2
+#define RUNSTATE_offline  3
+
+/*
+ * The 64-bit (long mode) layout: 'state' is followed by 4 bytes of
+ * padding so state_entry_time is naturally aligned. The VM is placed
+ * in long mode explicitly below; without that, KVM would latch 32-bit
+ * mode (no hypercall-page write ever happens here to latch it from
+ * EFER) and write the packed compat layout instead.
+ */
+struct vcpu_runstate_info {
+	u32 state;
+	u64 state_entry_time;
+	u64 time[4];
+};
+
+static void guest_code(void)
+{
+	volatile struct vcpu_runstate_info *rs =
+		(void *)(unsigned long)RUNSTATE_GPA;
+
+	/*
+	 * Report runstate times — no need to enable the kvmclock MSR,
+	 * the kernel writes runstate using its internal kvmclock.
+	 */
+	GUEST_SYNC_ARGS(0, rs->time[RUNSTATE_runnable],
+			rs->time[RUNSTATE_running], 0, 0);
+}
+
+static struct kvm_vm *create_xen_vm(struct kvm_vcpu **vcpu)
+{
+	struct kvm_vm *vm;
+	int xen_caps;
+
+	vm = vm_create_with_one_vcpu(vcpu, guest_code);
+
+	xen_caps = kvm_check_cap(KVM_CAP_XEN_HVM);
+	TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_SHARED_INFO);
+	TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_RUNSTATE);
+
+	/* Map pages */
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS,
+				    SHINFO_GPA, 1, 2, 0);
+	virt_map(vm, SHINFO_GPA, SHINFO_GPA, 2);
+
+	/* Enable Xen HVM with MSR interception (enables runstate tracking) */
+	struct kvm_xen_hvm_config cfg = {
+		.flags = KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL,
+		.msr = 0x40000000,
+	};
+	vm_ioctl(vm, KVM_XEN_HVM_CONFIG, &cfg);
+
+	/* The guest runs in long mode; make the runstate layout match. */
+	struct kvm_xen_hvm_attr lm = {
+		.type = KVM_XEN_ATTR_TYPE_LONG_MODE,
+		.u.long_mode = 1,
+	};
+	vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &lm);
+
+	/* Set shared_info */
+	struct kvm_xen_hvm_attr ha = {
+		.type = KVM_XEN_ATTR_TYPE_SHARED_INFO,
+		.u.shared_info.gfn = SHINFO_GPA >> 12,
+	};
+	vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &ha);
+
+	/* Set runstate address */
+	struct kvm_xen_vcpu_attr rs_addr = {
+		.type = KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_ADDR,
+		.u.gpa = RUNSTATE_GPA,
+	};
+	vcpu_ioctl(*vcpu, KVM_XEN_VCPU_SET_ATTR, &rs_addr);
+
+	return vm;
+}
+
+int main(void)
+{
+	struct pvclock_vcpu_time_info pvti;
+	struct kvm_xen_vcpu_attr runstate_save;
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	struct ucall uc;
+	u64 tsc_offset;
+	int ret;
+
+	TEST_REQUIRE(sys_clocksource_is_based_on_tsc());
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_CLOCK_GUEST));
+
+	ksft_print_header();
+	ksft_set_plan(1);
+
+	/* === SOURCE SIDE === */
+	pr_info("=== Source: create VM and run guest ===\n");
+	vm = create_xen_vm(&vcpu);
+
+	/* Run guest once to accumulate some runstate time */
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	TEST_ASSERT_EQ(get_ucall(vcpu, &uc), UCALL_SYNC);
+
+	pr_info("  Guest sees: runnable=%" PRIu64 " running=%" PRIu64 "\n",
+		uc.args[2], uc.args[3]);
+
+	/* Save clock state */
+	vcpu_get_clock_guest(vcpu, &pvti);
+
+	/*
+	 * Save the guest TSC offset. This test migrates to a new VM on
+	 * the same host, so restoring the same offset makes the guest
+	 * TSC continue uninterrupted across the migration gap, and the
+	 * restored pvclock (KVM_SET_CLOCK_GUEST) then yields a kvmclock
+	 * which has also kept running across the gap. (A real migration
+	 * to a different host would compute the destination offset from
+	 * KVM_GET_CLOCK's {host_tsc, realtime} pair as described in
+	 * Documentation/virt/kvm/devices/vcpu.rst.)
+	 */
+	tsc_offset = vcpu_get_tsc_offset(vcpu);
+
+	/* Save runstate — the vCPU is now "runnable" (not running) */
+	runstate_save.type = KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_DATA;
+	vcpu_ioctl(vcpu, KVM_XEN_VCPU_GET_ATTR, &runstate_save);
+
+	/*
+	 * Transition to runnable state before saving — the vCPU is
+	 * not running during migration.
+	 */
+	runstate_save.u.runstate.state = RUNSTATE_runnable;
+
+	pr_info("  Saved runstate: running=%" PRIu64 " runnable=%" PRIu64
+		" entry=%" PRIu64 "\n",
+		(u64)runstate_save.u.runstate.time_running,
+		(u64)runstate_save.u.runstate.time_runnable,
+		(u64)runstate_save.u.runstate.state_entry_time);
+
+	u64 saved_runnable = runstate_save.u.runstate.time_runnable;
+
+	kvm_vm_free(vm);
+
+	/* === MIGRATION GAP === */
+	pr_info("=== Simulating migration (sleeping 10ms) ===\n");
+	usleep(10000);
+
+	/* === DESTINATION SIDE === */
+	pr_info("=== Destination: create new VM and restore ===\n");
+	vm = create_xen_vm(&vcpu);
+
+	/* Restore the guest TSC offset (same host: TSC continues) */
+	vcpu_set_tsc_offset(vcpu, tsc_offset);
+
+	/* Restore clock — kvmclock will now be ~10ms ahead of the snapshot */
+	vcpu_set_clock_guest(vcpu, &pvti);
+
+	/* Restore runstate exactly as saved (state=runnable) */
+	runstate_save.type = KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_DATA;
+	ret = __vcpu_ioctl(vcpu, KVM_XEN_VCPU_SET_ATTR, &runstate_save);
+	TEST_ASSERT(!ret, "Restore runstate failed: errno %d", errno);
+
+	/*
+	 * Run the guest. When the vCPU enters vcpu_run, the kernel
+	 * transitions from RUNSTATE_runnable to RUNSTATE_running.
+	 * It computes: delta = kvmclock_now - state_entry_time
+	 * This delta (which includes the migration gap) is added to
+	 * time_runnable (steal time).
+	 */
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	TEST_ASSERT_EQ(get_ucall(vcpu, &uc), UCALL_SYNC);
+
+	u64 guest_runnable = uc.args[2];
+	u64 guest_running = uc.args[3];
+
+	pr_info("  Guest sees: runnable=%" PRIu64 " running=%" PRIu64 "\n",
+		guest_runnable, guest_running);
+
+	u64 steal_increase = guest_runnable - saved_runnable;
+	pr_info("  Steal time increase: %" PRIu64 " ns (migration gap)\n",
+		steal_increase);
+
+	/*
+	 * The steal time increase should be at least 10ms (the sleep).
+	 * The actual gap runs from the source's state_entry_time to the
+	 * destination's kvmclock "now" at vcpu_load time, which includes
+	 * everything from the source guest's last runstate transition
+	 * through VM teardown and re-creation — so the upper bound exists
+	 * only to catch wild miscalculation (e.g. an unscaled TSC delta),
+	 * and is deliberately generous for slow or loaded hosts.
+	 */
+	TEST_ASSERT(steal_increase >= 10000000ULL &&
+		    steal_increase < 60000000000ULL,
+		    "Steal time increase %" PRIu64 " ns not in expected range "
+		    "[10ms, 60s]", steal_increase);
+
+	kvm_vm_free(vm);
+	ksft_test_result_pass("migration gap accounted as steal time\n");
+	ksft_finished();
+}
-- 
2.43.0

[PATCH v13 23/26] KVM: selftests: Test that the first user TSC write is honoured exactly

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The 1-second "slop" heuristic in kvm_synchronize_tsc() is only
supposed to apply between successive userspace TSC writes, i.e. once
kvm->arch.user_set_tsc has been set by a previous write. The kernel's
own internal TSC synchronization on vCPU creation (and frequency
changes) must not arm it, otherwise the first genuine userspace write
falling within 1 second of virtual cycle time of the kernel's default
generation is "corrected" back to that generation instead of being
honoured — breaking the restore of a guest with less than one second
of runtime.

Add a test which creates a vCPU and immediately performs a first
host-initiated write of a TSC value corresponding to ~500ms of guest
runtime, then verifies the value reads back exactly (modulo elapsed
time) rather than being snapped back to the kernel's default
generation near zero.

In the frequency-change case, a buggy kernel carries the creation-rate
TSC across the change, where (expressed at the halved rate) it reads as
the creation-to-change plus creation-to-read time; a further timing
gate keeps that sum comfortably below the written 500ms so a
snapped-back value can never sit above the written one.

Both cases report through the kselftest TAP machinery; an
inconclusive case (host timing repeatedly too slow, a property of
host load rather than of the kernel under test) is recorded as a skip
without discarding the other case's result. A conclusively buggy
kernel still fails via TEST_ASSERT before any skip is reachable.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/x86/tsc_user_set_test.c     | 193 ++++++++++++++++++
 2 files changed, 194 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/tsc_user_set_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 1edc78845fd3..fd063806a7fb 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -151,6 +151,7 @@ TEST_GEN_PROGS_x86 += x86/xcr0_cpuid_test
 TEST_GEN_PROGS_x86 += x86/xss_msr_test
 TEST_GEN_PROGS_x86 += x86/debug_regs
 TEST_GEN_PROGS_x86 += x86/tsc_msrs_test
+TEST_GEN_PROGS_x86 += x86/tsc_user_set_test
 TEST_GEN_PROGS_x86 += x86/vmx_pmu_caps_test
 TEST_GEN_PROGS_x86 += x86/xen_cpuid_timing_test
 TEST_GEN_PROGS_x86 += x86/xen_migration_test
diff --git a/tools/testing/selftests/kvm/x86/tsc_user_set_test.c b/tools/testing/selftests/kvm/x86/tsc_user_set_test.c
new file mode 100644
index 000000000000..5e6eb0a26c8e
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/tsc_user_set_test.c
@@ -0,0 +1,193 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Test that the first userspace write to MSR_IA32_TSC takes effect
+ * precisely, and is not subjected to the 1-second "slop" heuristic
+ * which is only supposed to apply between successive *userspace*
+ * writes (kvm->arch.user_set_tsc).
+ *
+ * The kernel synchronizes the TSC internally when a vCPU is created
+ * (and when its frequency is set). If any of those kernel-internal
+ * paths incorrectly sets user_set_tsc, then the first genuine write
+ * from userspace which happens to fall within 1 second of virtual
+ * cycle time of the kernel's default generation is "corrected" to
+ * match that generation instead of being honoured exactly. That
+ * breaks the restore of a guest which has genuinely been running
+ * for less than one second.
+ *
+ * Simulate exactly that: create a vCPU (kernel establishes a default
+ * TSC generation starting near zero), then perform the first
+ * host-initiated write with a value corresponding to ~500ms of guest
+ * runtime. If the slop heuristic fires, the written value is snapped
+ * back to the kernel's generation (near zero) and the TSC reads back
+ * hundreds of milliseconds low. Repeat with an explicit
+ * KVM_SET_TSC_KHZ on the never-run vCPU before the write: that is
+ * another kernel-internal synchronization which must not arm the
+ * heuristic (nor open a generation that swallows the first write).
+ */
+#include <time.h>
+
+#include "kvm_util.h"
+#include "processor.h"
+
+/*
+ * Returns false when the attempt was inconclusive: a snapped-back value
+ * tracks time since vCPU creation, so if creation-to-write time gets
+ * anywhere near the written 500ms, a buggy kernel's value could land
+ * inside the assertions' tolerance and pass. The caller retries.
+ */
+static bool test_first_write(bool set_freq)
+{
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	u64 freq_khz, t_write, t_read, slack_cycles, mono_ns, create_ns, read_ns;
+	u64 freq_ns;
+	struct timespec ts;
+	int r;
+
+	clock_gettime(CLOCK_MONOTONIC_RAW, &ts);
+	create_ns = ts.tv_sec * 1000000000ULL + ts.tv_nsec;
+	freq_ns = create_ns;
+
+	vm = vm_create_with_one_vcpu(&vcpu, NULL);
+
+	if (set_freq) {
+		/*
+		 * Halve the frequency so this is a real change, exercising
+		 * the frequency resynchronization path rather than the
+		 * unchanged-frequency no-op.
+		 */
+		r = __vcpu_ioctl(vcpu, KVM_GET_TSC_KHZ, NULL);
+		TEST_ASSERT(r > 0, "KVM_GET_TSC_KHZ failed");
+		vcpu_ioctl(vcpu, KVM_SET_TSC_KHZ,
+			   (void *)(unsigned long)(r / 2));
+		clock_gettime(CLOCK_MONOTONIC_RAW, &ts);
+		freq_ns = ts.tv_sec * 1000000000ULL + ts.tv_nsec;
+	}
+
+	r = __vcpu_ioctl(vcpu, KVM_GET_TSC_KHZ, NULL);
+	TEST_ASSERT(r > 0, "KVM_GET_TSC_KHZ failed");
+	freq_khz = r;
+
+	/*
+	 * The first user write: a guest TSC corresponding to 500ms of
+	 * runtime. This is within 1s (of virtual cycle time) of the
+	 * kernel's default generation, so it is exactly the case that
+	 * the slop heuristic would destroy if it were (incorrectly)
+	 * armed by the kernel's own internal TSC synchronization.
+	 */
+	t_write = freq_khz * 500ULL;		/* khz * ms = cycles */
+
+	/*
+	 * Measure only the write..read window (a window measured from
+	 * before creation would also bound a snapped-back value), and
+	 * insist the write lands well before 500ms of creation time so
+	 * a snapped-back value cannot sit inside the tolerance.
+	 */
+	clock_gettime(CLOCK_MONOTONIC_RAW, &ts);
+	mono_ns = ts.tv_sec * 1000000000ULL + ts.tv_nsec;
+
+	if (mono_ns - create_ns > 250000000ULL) {
+		pr_info("  creation took %lu ms, retrying for a conclusive run\n",
+			(mono_ns - create_ns) / 1000000);
+		kvm_vm_free(vm);
+		return false;
+	}
+
+	vcpu_set_msr(vcpu, MSR_IA32_TSC, t_write);
+
+	t_read = vcpu_get_msr(vcpu, MSR_IA32_TSC);
+
+	clock_gettime(CLOCK_MONOTONIC_RAW, &ts);
+	read_ns = ts.tv_sec * 1000000000ULL + ts.tv_nsec;
+
+	/*
+	 * Bound the whole creation..read interval too: a snapped-back
+	 * value tracks it, so letting it approach the written 500ms
+	 * (via a stall between the two MSR ioctls, which the slack
+	 * below would otherwise absorb) would let a buggy kernel pass.
+	 */
+	if (read_ns - create_ns > 400000000ULL) {
+		pr_info("  creation..read took %lu ms, retrying for a conclusive run\n",
+			(read_ns - create_ns) / 1000000);
+		kvm_vm_free(vm);
+		return false;
+	}
+
+	/*
+	 * In the set_freq case a buggy kernel carries the creation-rate
+	 * TSC across the frequency change, where it reads (expressed at
+	 * the halved rate) as the creation..freq-change plus the
+	 * creation..read time in milliseconds. Keep that sum comfortably
+	 * below the written 500ms, or a snapped-back value could sit
+	 * above t_write and pass the assertions.
+	 */
+	if ((freq_ns - create_ns) + (read_ns - create_ns) > 430000000ULL) {
+		pr_info("  freq-change at %lu ms + read at %lu ms, retrying for a conclusive run\n",
+			(freq_ns - create_ns) / 1000000,
+			(read_ns - create_ns) / 1000000);
+		kvm_vm_free(vm);
+		return false;
+	}
+
+	mono_ns = read_ns - mono_ns;
+
+	/*
+	 * The TSC advances between the write and the read; bound the
+	 * difference by the measured elapsed time around the two ioctls
+	 * plus 10ms of slop, so a descheduled test cannot flake. If the
+	 * write was snapped to the kernel's default generation, the
+	 * value read back is ~500ms low (i.e. below t_write), which is
+	 * unambiguous.
+	 */
+	slack_cycles = freq_khz * (mono_ns / 1000000ULL + 10);
+
+	TEST_ASSERT(t_read >= t_write,
+		    "TSC went backwards from explicit write: wrote %lu, read %lu"
+		    " (first userspace write was subjected to 1-second slop?)",
+		    t_write, t_read);
+	TEST_ASSERT(t_read - t_write < slack_cycles,
+		    "TSC read back too far from written value: wrote %lu, read %lu",
+		    t_write, t_read);
+
+	kvm_vm_free(vm);
+	return true;
+}
+
+static void run_first_write(bool set_freq)
+{
+	const char *name = set_freq ? "first write after KVM_SET_TSC_KHZ"
+				    : "first write";
+	int i;
+
+	for (i = 0; i < 5; i++) {
+		if (test_first_write(set_freq)) {
+			ksft_test_result_pass("%s\n", name);
+			return;
+		}
+	}
+
+	/*
+	 * Inconclusive runs — either gate: slow vCPU creation, or a
+	 * stall between the MSR write and read — reflect host
+	 * scheduling load, not the behaviour under test; record a skip
+	 * rather than report a kernel regression, without discarding a
+	 * conclusive result the other case already established.
+	 */
+	ksft_test_result_skip("%s: host timing repeatedly too slow for a conclusive run\n",
+			      name);
+}
+
+int main(void)
+{
+	ksft_print_header();
+	ksft_set_plan(2);
+
+	run_first_write(false);
+
+	if (kvm_has_cap(KVM_CAP_TSC_CONTROL))
+		run_first_write(true);
+	else
+		ksft_test_result_skip("first write after KVM_SET_TSC_KHZ: no KVM_CAP_TSC_CONTROL\n");
+
+	ksft_finished();
+}
-- 
2.43.0

[PATCH v13 24/26] KVM: x86/xen: Drop open-coded fallback clock read in kvm_xen_start_timer()

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

When no coherent (CLOCK_MONOTONIC, TSC) pair was available,
kvm_xen_start_timer() computed one from separate ktime_get() and
rdtsc() calls rather than using get_kvmclock_ns(), with a comment
explaining that the latter had a systemic error from scaling host TSC
directly to nanoseconds instead of scaling to the guest TSC first.

Commit abdef42276d8 ("KVM: x86: Fix KVM clock precision in
get_kvmclock() with TSC scaling") removed that systemic error: in
master clock mode, get_kvmclock_ns() now scales the host TSC delta to
the guest TSC and converts it with the guest's own mul/shift,
computing precisely what the guest would.

The fallback is also nearly unreachable. It sits behind the
use_master_clock check, so on 32-bit (where the master clock is never
active) it is dead code, and on 64-bit kvm_get_monotonic_and_clockread()
can only fail there if the host clocksource changed after the last
master clock update, in the window before the resulting masterclock
update is processed. In exactly that window get_kvmclock_ns() takes
its raw-clock fallback path rather than the guest TSC computation,
which is continuous with the last valid snapshot — though up to
~0.1ppm of the snapshot's age (multiplier quantization) away from the
guest's own pvclock arithmetic — and tolerable for the timers that
can be started before the pending masterclock update lands (the
clocksource change is noticed via a two-stage deferral, so it is not
strictly one).

Drop it and fall back to the existing get_kvmclock_ns() path.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/kvm/xen.c | 50 +++++++++++++++++++++++-----------------------
 1 file changed, 25 insertions(+), 25 deletions(-)

diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c
index 6aa00f746cba..2a157cba1c7c 100644
--- a/arch/x86/kvm/xen.c
+++ b/arch/x86/kvm/xen.c
@@ -188,7 +188,7 @@ static void kvm_xen_start_timer(struct kvm_vcpu *vcpu, u64 guest_abs,
 	struct kvm_vcpu_xen *xen = &vcpu->arch.xen;
 	int64_t kernel_now, delta;
 	uint64_t guest_now;
-	int r = -EOPNOTSUPP;
+	bool fallback = true;
 
 	/*
 	 * The guest provides the requested timeout in absolute nanoseconds
@@ -210,6 +210,7 @@ static void kvm_xen_start_timer(struct kvm_vcpu *vcpu, u64 guest_abs,
 	do {
 		struct pvclock_vcpu_time_info hv_clock;
 		uint64_t host_tsc, guest_tsc;
+		int r = -EOPNOTSUPP;
 
 		if (!cpu_feature_enabled(X86_FEATURE_CONSTANT_TSC) ||
 		    !vcpu->kvm->arch.use_master_clock)
@@ -230,43 +231,42 @@ static void kvm_xen_start_timer(struct kvm_vcpu *vcpu, u64 guest_abs,
 		if (r)
 			break;
 
+		/*
+		 * With use_master_clock set, the clocksource was TSC-based
+		 * when the master clock was last updated. The clockread can
+		 * only fail here if the clocksource has changed since, in
+		 * which case a masterclock update is already pending and
+		 * get_kvmclock_ns() returns the raw-clock fallback value —
+		 * continuous with the last valid snapshot, though it may
+		 * sit up to ~0.1ppm of the snapshot's age away from the
+		 * guest's own pvclock arithmetic. Only timers armed
+		 * before the pending update is consumed see it.
+		 */
 		if (!IS_ENABLED(CONFIG_64BIT) ||
-		    !kvm_get_monotonic_and_clockread(&kernel_now, &host_tsc)) {
-			/*
-			 * Don't fall back to get_kvmclock_ns() because it's
-			 * broken; it has a systemic error in its results
-			 * because it scales directly from host TSC to
-			 * nanoseconds, and doesn't scale first to guest TSC
-			 * and *then* to nanoseconds as the guest does.
-			 *
-			 * There is a small error introduced here because time
-			 * continues to elapse between the ktime_get() and the
-			 * subsequent rdtsc(). But not the systemic drift due
-			 * to get_kvmclock_ns().
-			 */
-			kernel_now = ktime_get(); /* This is CLOCK_MONOTONIC */
-			host_tsc = rdtsc();
-		}
+		    !kvm_get_monotonic_and_clockread(&kernel_now, &host_tsc))
+			break;
 
 		/* Calculate the guest kvmclock as the guest would do it. */
 		guest_tsc = kvm_read_l1_tsc(vcpu, host_tsc);
 		guest_now = __pvclock_read_cycles(&hv_clock, guest_tsc);
+		fallback = false;
 	} while (0);
 
-	if (r) {
+	if (fallback) {
 		/*
-		 * Without CONSTANT_TSC, get_kvmclock_ns() is the only option.
+		 * Without CONSTANT_TSC and the master clock,
+		 * get_kvmclock_ns() is the only option.
 		 *
 		 * Also if the guest PV clock hasn't been set up yet, as is
 		 * likely to be the case during migration when the vCPU has
 		 * not been run yet. It would be possible to calculate the
 		 * scaling factors properly in that case but there's not much
-		 * point in doing so. The get_kvmclock_ns() drift accumulates
-		 * over time, so it's OK to use it at startup. Besides, on
-		 * migration there's going to be a little bit of skew in the
-		 * precise moment at which timers fire anyway. Often they'll
-		 * be in the "past" by the time the VM is running again after
-		 * migration.
+		 * point in doing so. Without the master clock, the
+		 * get_kvmclock_ns() drift accumulates over time, so it's OK
+		 * to use it at startup. Besides, on migration there's going
+		 * to be a little bit of skew in the precise moment at which
+		 * timers fire anyway. Often they'll be in the "past" by the
+		 * time the VM is running again after migration.
 		 */
 		guest_now = get_kvmclock_ns(vcpu->kvm);
 		kernel_now = ktime_get();
-- 
2.43.0

[PATCH v13 25/26] KVM: x86: Use a single time snapshot in kvm_vm_ioctl_set_clock()

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

The KVM_CLOCK_REALTIME handling read CLOCK_REALTIME with
ktime_get_real_ns() and then anchored the result at a different
instant: the master clock reference point captured a few lines
earlier, or a separate get_kvmclock_base_ns() read. The time elapsed
between the two reads is added to the restored clock as error --
bounded by the interrupts-off critical section, but gratuitous now
that the timekeeping core can snapshot CLOCK_REALTIME and
CLOCK_MONOTONIC_RAW at the same instant.

Take one ktime_get_snapshot_id(CLOCK_REALTIME) snapshot and derive
both the realtime correction and the raw reference point from it. In
master clock mode, keep anchoring at master_kernel_ns -- so that
'system_time' cannot go negative for a very small data.clock, as the
existing comment explains -- by projecting the snapshot's realtime
value back to that reference point; CLOCK_REALTIME and
CLOCK_MONOTONIC_RAW tick at the same rate across the few ns in
between, give or take NTP slew.

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/kvm/x86.c | 40 +++++++++++++++++++++++++++++++++++-----
 1 file changed, 35 insertions(+), 5 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index b9d3b2715403..921d834b9682 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -5143,19 +5143,49 @@ static int kvm_vm_ioctl_set_clock(struct kvm *kvm, void __user *argp)
 	 * 'system_time' when 'data.clock' is very small.
 	 */
 	if (data.flags & KVM_CLOCK_REALTIME) {
-		u64 now_real_ns = ktime_get_real_ns();
+		struct system_time_snapshot snap = {};
+		u64 now_real_ns;
+
+		/*
+		 * Snapshot CLOCK_REALTIME and CLOCK_MONOTONIC_RAW at the
+		 * same instant, so that the elapsed time applied to
+		 * data.clock and the reference point the result is anchored
+		 * to are coherent.
+		 */
+		ktime_get_snapshot_id(CLOCK_REALTIME, &snap);
+		now_real_ns = ktime_to_ns(snap.systime);
+		now_raw_ns = ktime_to_ns(ktime_mono_to_any(snap.monoraw,
+							   TK_OFFS_BOOT));
+
+		/*
+		 * When the master clock is in use, anchor at its reference
+		 * point instead, projecting the realtime value back to the
+		 * same instant. The two clocks tick at the same rate across
+		 * the few ns in between, give or take NTP slew. In the
+		 * absurd case that the projection would take a
+		 * CLOCK_REALTIME set to within moments of the 1970 epoch
+		 * (legal with CAP_SYS_TIME) below zero, clamp it to zero;
+		 * the anchor must still move to master_kernel_ns, from
+		 * which the master clock consumers compute system_time,
+		 * lest a near-zero data.clock wrap it below zero.
+		 */
+		if (ka->use_master_clock) {
+			u64 delta = now_raw_ns - ka->master_kernel_ns;
+
+			now_real_ns = now_real_ns > delta ? now_real_ns - delta : 0;
+			now_raw_ns = ka->master_kernel_ns;
+		}
 
 		/*
 		 * Avoid stepping the kvmclock backwards.
 		 */
 		if (now_real_ns > data.realtime)
 			data.clock += now_real_ns - data.realtime;
-	}
-
-	if (ka->use_master_clock)
+	} else if (ka->use_master_clock) {
 		now_raw_ns = ka->master_kernel_ns;
-	else
+	} else {
 		now_raw_ns = get_kvmclock_base_ns();
+	}
 	ka->kvmclock_offset = data.clock - now_raw_ns;
 	kvm_end_pvclock_update(kvm);
 
-- 
2.43.0

[PATCH v13 26/26] KVM: x86: Keep master clock for guests using legacy MSR_KVM_SYSTEM_TIME

[David Woodhouse]

From: David Woodhouse <dwmw@amazon.co.uk>

Commit 54750f2cf042 ("KVM: x86: workaround SuSE's 2.6.16 pvclock vs
masterclock issue") disables the master clock entirely for any guest
whose boot vCPU uses the legacy MSR. SUSE's 2.6.16 kernel treats a
pvclock tsc_timestamp more than 1/5 jiffie behind the TSC as suspect
and re-requests the time by rewriting the MSR; against the master
clock's long-lived reference point the delta never shrinks, so it
re-requests in a loop and fails to boot.

Satisfy the guest's freshness heuristic instead of capitulating to it:
a guest write to the legacy MSR now requests a masterclock update,
sliding the reference point forward before the pvclock is written (the
masterclock request is serviced before KVM_REQ_CLOCK_UPDATE on the
same guest entry). The re-requested pvclock then carries a recent
tsc_timestamp and the heuristic is content. Each re-anchor steps the
pvclock function by the sub-nanosecond (possibly backwards)
mult-rounding divergence accumulated since the previous anchor — the
same step every pre-existing masterclock update already takes — and
legacy-MSR guests now see PVCLOCK_TSC_STABLE_BIT like anyone else.

A guest hammering the legacy MSR forces repeated VM-wide masterclock
updates, but only its own VM's: a guest has always been able to burn
its own cycles. Re-anchoring just the writing vCPU's pvclock along the
masterclock line would avoid even that, but isn't worth the extra
plumbing for a 2006 guest kernel.

This removes the last way a guest could force the master clock off;
the remaining disable, backwards_tsc_observed, is the host's own
doing. The pvclock_test selftest grows a legacy-MSR case proving the
master clock survives and the rewritten pvclock is fresh.

Not tested against a real SLES10 image; the behaviour it demands is
reconstructed from the guest comment quoted in 54750f2cf042. The
freshness that guest observes can be no worse than before: with the
master clock disabled it saw a tsc_timestamp one exit-to-read latency
old, exactly what the rewrite now produces, and the update's own cost
does not age the timestamp, which is written only after the update
completes. A host loaded enough to defeat that would have livelocked
the old scheme equally. Should a real SLES10 guest prove otherwise,
the revert path is to restore the boot_vcpu_runs_old_kvmclock gate in
kvm_use_master_clock().

Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
---
 arch/x86/include/asm/kvm_host.h               |  1 -
 arch/x86/kvm/msrs.c                           | 26 ++++----
 arch/x86/kvm/x86.c                            | 15 +----
 .../testing/selftests/kvm/x86/pvclock_test.c  | 63 +++++++++++++++++++
 4 files changed, 77 insertions(+), 28 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 773df58898ea..0db8169612a4 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1275,7 +1275,6 @@ struct kvm_arch {
 #endif
 
 	bool backwards_tsc_observed;
-	bool boot_vcpu_runs_old_kvmclock;
 	u32 bsp_vcpu_id;
 
 	u64 disabled_quirks;
diff --git a/arch/x86/kvm/msrs.c b/arch/x86/kvm/msrs.c
index d07831b5dfb8..94cd4d73689a 100644
--- a/arch/x86/kvm/msrs.c
+++ b/arch/x86/kvm/msrs.c
@@ -1275,19 +1275,19 @@ static void kvm_write_wall_clock(struct kvm *kvm, gpa_t wall_clock, int sec_hi_o
 static void kvm_write_system_time(struct kvm_vcpu *vcpu, gpa_t system_time,
 				  bool old_msr, bool host_initiated)
 {
-	struct kvm_arch *ka = &vcpu->kvm->arch;
-
-	if (vcpu->vcpu_id == 0 && !host_initiated) {
-		/*
-		 * Store the flag before requesting the masterclock update:
-		 * the update may now be performed by another vCPU (which
-		 * consumes this request), and it must observe the new value.
-		 */
-		if (ka->boot_vcpu_runs_old_kvmclock != old_msr) {
-			ka->boot_vcpu_runs_old_kvmclock = old_msr;
-			kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu);
-		}
-	}
+	/*
+	 * Slide the master clock reference point forward, so that the
+	 * pvclock written below carries a recent tsc_timestamp. SUSE's
+	 * 2.6.16 kernel (the only known user of the legacy MSR with a
+	 * reason to care) re-requests the time by rewriting this MSR
+	 * whenever tsc_timestamp falls more than 1/5 jiffie behind the
+	 * TSC, and would livelock against a fixed reference point; each
+	 * rewrite now genuinely freshens it. A write with the enable
+	 * bit clear deactivates the pvclock; there is nothing to
+	 * freshen.
+	 */
+	if (old_msr && !host_initiated && (system_time & 1))
+		kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu);
 
 	vcpu->arch.time = system_time;
 
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 921d834b9682..191f811a0cdd 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1181,21 +1181,8 @@ static bool kvm_use_master_clock(struct kvm *kvm)
 {
 	struct kvm_arch *ka = &kvm->arch;
 
-	/*
-	 * The 'old kvmclock' check is a workaround (from 2015) for a
-	 * SUSE 2.6.16 kernel that didn't boot if the system_time in
-	 * its kvmclock was too far behind the current time. So the
-	 * mode of just setting the reference point and allowing time
-	 * to proceed linearly from there makes it fail to boot.
-	 * Despite that being kind of the *point* of the way the clock
-	 * is exposed to the guest. By coincidence, the offending
-	 * kernels used the old MSR_KVM_SYSTEM_TIME, which was moved
-	 * only because it resided in the wrong number range. So the
-	 * workaround is activated for *all* guests using the old MSR.
-	 */
 	return ka->all_vcpus_matched_freq &&
-		!ka->backwards_tsc_observed &&
-		!ka->boot_vcpu_runs_old_kvmclock;
+		!ka->backwards_tsc_observed;
 }
 
 static void kvm_track_tsc_matching(struct kvm_vcpu *vcpu, bool update_mclock)
diff --git a/tools/testing/selftests/kvm/x86/pvclock_test.c b/tools/testing/selftests/kvm/x86/pvclock_test.c
index cb57f5e2bf1e..f3c8d9d8303a 100644
--- a/tools/testing/selftests/kvm/x86/pvclock_test.c
+++ b/tools/testing/selftests/kvm/x86/pvclock_test.c
@@ -189,6 +189,8 @@ static void run_at_frequency(u64 tsc_khz, unsigned int sleep_sec)
 
 static void test_tsc_stable_bit(void);
 static void test_clock_guest_with_offsets(void);
+static void test_legacy_msr(void);
+static void run_vcpu_once(struct kvm_vcpu *vcpu);
 
 static void usage(const char *name)
 {
@@ -251,10 +253,71 @@ int main(int argc, char *argv[])
 
 	test_tsc_stable_bit();
 	test_clock_guest_with_offsets();
+	test_legacy_msr();
 
 	return 0;
 }
 
+static void guest_code_legacy_msr(void)
+{
+	wrmsr(MSR_KVM_SYSTEM_TIME, KVMCLOCK_GPA | KVM_MSR_ENABLED);
+	GUEST_SYNC(0);
+	/* Re-request the time, as SUSE's 2.6.16 kernel would. */
+	wrmsr(MSR_KVM_SYSTEM_TIME, KVMCLOCK_GPA | KVM_MSR_ENABLED);
+	GUEST_SYNC(1);
+}
+
+/*
+ * A guest using the legacy MSR_KVM_SYSTEM_TIME must no longer lose the
+ * master clock, and rewriting the MSR must slide the reference point
+ * forward so the pvclock's tsc_timestamp is recent — that freshness is
+ * what SUSE's 2.6.16 kernel demands instead of tolerating a fixed
+ * reference point.
+ */
+static void test_legacy_msr(void)
+{
+	struct pvclock_vcpu_time_info *pvti;
+	struct kvm_clock_data clock;
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	u64 tsc_now, age_limit;
+
+	pr_info("Testing the legacy MSR_KVM_SYSTEM_TIME\n");
+
+	vm = vm_create_with_one_vcpu(&vcpu, guest_code_legacy_msr);
+	configure_pvclock(vm);
+
+	run_vcpu_once(vcpu);
+	vm_ioctl(vm, KVM_GET_CLOCK, &clock);
+	TEST_ASSERT(clock.flags & KVM_CLOCK_TSC_STABLE,
+		    "Legacy MSR_KVM_SYSTEM_TIME should not disable the master clock");
+
+	/* Age the master clock reference point, then re-request. */
+	usleep(300 * 1000);
+	run_vcpu_once(vcpu);
+
+	pvti = addr_gpa2hva(vm, KVMCLOCK_GPA);
+	tsc_now = vcpu_get_msr(vcpu, MSR_IA32_TSC);
+
+	/*
+	 * The rewrite slid the reference point, so the pvclock must be
+	 * anchored after the 300ms sleep: allow 250ms of slack for the
+	 * ioctls around it. SUSE's real threshold was 1/5 jiffie, but
+	 * asserting that tightly would flake on a loaded host.
+	 */
+	age_limit = (u64)__vcpu_ioctl(vcpu, KVM_GET_TSC_KHZ, NULL) * 250;
+	TEST_ASSERT(tsc_now - pvti->tsc_timestamp < age_limit,
+		    "pvclock tsc_timestamp should be fresh after legacy MSR rewrite (age %llu cycles, limit %llu)",
+		    (unsigned long long)(tsc_now - pvti->tsc_timestamp),
+		    (unsigned long long)age_limit);
+
+	vm_ioctl(vm, KVM_GET_CLOCK, &clock);
+	TEST_ASSERT(clock.flags & KVM_CLOCK_TSC_STABLE,
+		    "Master clock should survive a legacy MSR rewrite");
+
+	kvm_vm_free(vm);
+}
+
 static volatile u32 vcpu_counter;
 static void guest_code_stable_bit(void)
 {
-- 
2.43.0

Re: [PATCH v13 03/26] KVM: x86: Replace nr_vcpus_matched_tsc count with all_vcpus_matched_tsc bool

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] KVM: x86: race clearing matched TSC state in
  kvm_arch_vcpu_postcreate()
--

commit 0772c4b5400b34e15b8efeba1e351501ea33af4a
Author: David Woodhouse <dwmw@amazon.co.uk>

KVM: x86: Replace nr_vcpus_matched_tsc count with all_vcpus_matched_tsc bool

This commit replaces read-time comparisons of nr_vcpus_matched_tsc against
online_vcpus with a boolean all_vcpus_matched_tsc maintained under
tsc_write_lock, initializing it to true at VM creation and clearing it in
kvm_arch_vcpu_postcreate() before synchronizing the new vCPU.

> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 12f465040efc2..1bb0136042a65 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
[ ... ]
> @@ -9584,8 +9589,38 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
>  
>  void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
>  {
> -	if (mutex_lock_killable(&vcpu->mutex))
> +	struct kvm_arch *ka = &vcpu->kvm->arch;
> +	unsigned long flags;
> +
> +	/*
> +	 * The new vCPU is already counted in online_vcpus but has not
> +	 * been synchronized. Clear the cached matched state before
> +	 * trying the mutex: the old read-time comparison failed
> +	 * conservatively false throughout this window, and userspace
> +	 * can hold it open across a whole first KVM_RUN by taking
> +	 * vcpu->mutex before we do. The synchronization below (or any
> +	 * racing host-initiated TSC write on the already-live fd)
> +	 * re-latches the flags. A protected guest TSC never
> +	 * participates in offset matching, so for it there is nothing
> +	 * to clear — and no later sync to re-latch.
> +	 */
> +	if (!vcpu->arch.guest_tsc_protected) {
> +		raw_spin_lock_irqsave(&ka->tsc_write_lock, flags);
> +		WRITE_ONCE(ka->all_vcpus_matched_tsc, false);
> +		raw_spin_unlock_irqrestore(&ka->tsc_write_lock, flags);
> +	}
> +
> +	if (mutex_lock_killable(&vcpu->mutex)) {
> +		/*
> +		 * Aborted by a fatal signal: the synchronization never
> +		 * ran. Propagate whatever the matched state now is (a
> +		 * racing TSC write may have re-latched it), so the
> +		 * master clock does not stay in its previous mode with
> +		 * running siblings keeping PVCLOCK_TSC_STABLE_BIT.
> +		 */
> +		kvm_update_masterclock(vcpu->kvm);
> +		return;
> +	}

[Severity: Medium]
Can clearing ka->all_vcpus_matched_tsc here race with vCPU ioctls on the
newly created fd, or leave ka->use_master_clock stale while waiting for
vcpu->mutex?

In kvm_vm_ioctl_create_vcpu(), create_vcpu_fd(), atomic_inc(&kvm->online_vcpus),
and mutex_unlock(&vcpu->mutex) all run before kvm_arch_vcpu_postcreate():

CPU0                                    CPU1
kvm_vm_ioctl_create_vcpu()
  create_vcpu_fd(vcpu)
  atomic_inc(&kvm->online_vcpus)
  mutex_unlock(&vcpu->mutex)
                                        kvm_vcpu_ioctl(KVM_SET_MSRS)
                                          mutex_lock(&vcpu->mutex)
                                          ...
                                          __kvm_synchronize_tsc()
                                            ka->nr_vcpus_matched_tsc++
                                            vcpu->arch.this_tsc_generation =
                                              ka->cur_tsc_generation
                                            ka->all_vcpus_matched_tsc = true
  kvm_arch_vcpu_postcreate()
    raw_spin_lock_irqsave(&ka->tsc_write_lock)
    WRITE_ONCE(ka->all_vcpus_matched_tsc, false) <-- clobbers matched state
    raw_spin_unlock_irqrestore(&ka->tsc_write_lock)
    mutex_lock_killable(&vcpu->mutex)   <-- aborted by fatal signal
    kvm_update_masterclock()            <-- disables masterclock permanently

Because vcpu->arch.this_tsc_generation already matches
ka->cur_tsc_generation after CPU1's write, even if mutex_lock_killable()
succeeds, a subsequent matching sync in postcreate will not increment
nr_vcpus_matched_tsc again (and in the later commit "KVM: x86: Allow KVM
master clock mode when TSCs are offset from each other", a non-matching
offset write on CPU1 followed by the clear on CPU0 leaves
ka->all_vcpus_matched_freq false).

Additionally, when another thread takes vcpu->mutex for KVM_RUN before
postcreate does, clearing ka->all_vcpus_matched_tsc without calling
kvm_update_masterclock() leaves ka->use_master_clock true across that
entire first KVM_RUN (and even if CPU1 runs kvm_guest_time_update() before
CPU0 acquires tsc_write_lock, both ka->all_vcpus_matched_tsc and
ka->use_master_clock are still true).

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261111180922.4177927-1-dwmw2@infradead.org?part=3

Re: [PATCH v13 10/26] KVM: x86: Re-synchronize TSC after KVM_SET_TSC_KHZ

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [Medium] KVM: x86: wrong scaling of tsc_offset_adjustment in
  __kvm_synchronize_tsc_at()
- [Medium] KVM: x86: uninitialized l1_tsc_scaling_ratio in
  kvm_set_tsc_khz()
- [Medium] KVM: x86: kvm_set_tsc_khz() sync breaks per-vCPU init and
  restore
--

commit 28669172288cdaaabe3c3f647bfb5b2ebfc1d816
Author: David Woodhouse <dwmw@amazon.co.uk>

KVM: x86: Re-synchronize TSC after KVM_SET_TSC_KHZ

This commit re-synchronizes a vCPU's TSC when its frequency is changed via
KVM_SET_TSC_KHZ, sampling the current guest TSC and host clock before
updating the scaling ratio and computing the new offset against that
reference point.

> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 1d4b66257b35f..cf35363366f14 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -1073,10 +1073,20 @@ static int set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz, bool scale)
>  	return 0;
>  }
>  
> -static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz)
> +static void __kvm_synchronize_tsc_at(struct kvm_vcpu *vcpu, u64 *value,
> +				     u64 host_tsc, s64 ns,
> +				     bool user_initiated);
> +
> +static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz, bool init)
>  {
> +	bool freq_changed = (vcpu->arch.virtual_tsc_khz != user_tsc_khz);
>  	u32 thresh_lo, thresh_hi;
> +	u32 tsc_mult;
> +	s8 tsc_shift;
>  	int use_scaling = 0;
> +	u64 tsc, host_tsc;
> +	unsigned long flags;
> +	s64 ns;
>  
>  	/* tsc_khz can be zero if TSC calibration fails */
>  	if (user_tsc_khz == 0) {
> @@ -1085,11 +1095,19 @@ static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz)
>  		return -1;
>  	}
>  
> +	/*
> +	 * Setting the current frequency again is a no-op, as documented
> +	 * — unconditionally, including a frequency the hardware could
> +	 * not honour at vCPU creation (which commits it regardless):
> +	 * re-running set_tsc_khz() would turn the documented no-op into
> +	 * -EINVAL.
> +	 */
> +	if (!init && !freq_changed)
> +		return 0;
> +
>  	/* Compute a scale to convert nanoseconds in TSC cycles */
>  	kvm_get_time_scale(user_tsc_khz * 1000LL, NSEC_PER_SEC,
> -			   &vcpu->arch.virtual_tsc_shift,
> -			   &vcpu->arch.virtual_tsc_mult);
> -	vcpu->arch.virtual_tsc_khz = user_tsc_khz;
> +			   &tsc_shift, &tsc_mult);
>  
>  	/*
>  	 * Compute the variation in TSC rate which is acceptable
> @@ -1104,7 +1122,92 @@ static int kvm_set_tsc_khz(struct kvm_vcpu *vcpu, u32 user_tsc_khz)
>  			 user_tsc_khz, thresh_lo, thresh_hi);
>  		use_scaling = 1;
>  	}
> -	return set_tsc_khz(vcpu, user_tsc_khz, use_scaling);
> +
> +	/*
> +	 * Sample the guest TSC before changing the ratio, so continuity
> +	 * can be preserved across the change — as a {guest TSC, host
> +	 * clock} tuple, not a bare value: the synchronization below
> +	 * computes the new offset against the tuple's own reference
> +	 * point, so the guest TSC keeps counting through this function.
> +	 * Preemption stays disabled from the sample until tsc_write_lock
> +	 * is held — not for timing, but because this task's own preempt
> +	 * notifier (kvm_sched_in -> kvm_arch_vcpu_load) is the one
> +	 * writer which could apply a pending backwards-TSC compensation
> +	 * to the offset the sample was read through; a compensation
> +	 * still pending when the lock is taken is folded in there.
> +	 */
> +	if (!init) {
> +		preempt_disable();
> +#ifdef CONFIG_X86_64
> +		if (!kvm_get_time_and_clockread(&ns, &host_tsc))
> +#endif
> +		{
> +			unsigned long irqflags;
> +
> +			/*
> +			 * The two reads must form one coherent reference
> +			 * point.
> +			 */
> +			local_irq_save(irqflags);
> +			host_tsc = rdtsc();
> +			ns = get_kvmclock_base_ns();
> +			local_irq_restore(irqflags);
> +		}
> +		tsc = kvm_read_l1_tsc(vcpu, host_tsc);
> +	}
> +
> +	/*
> +	 * Commit the new frequency only once set_tsc_khz() can no longer
> +	 * fail: a failed ioctl must leave no trace, or a retry with the
> +	 * still-active frequency would wrongly look like a change. The
> +	 * exception is vCPU creation (@init), whose caller ignores the
> +	 * return value: commit the requested frequency even when the
> +	 * hardware cannot scale to it, as creation always has, rather
> +	 * than leave the vCPU with the zeroed frequency from allocation
> +	 * (KVM_GET_TSC_KHZ would report 0, and the catchup path would
> +	 * compute with a zero mult).
> +	 */
> +	if (set_tsc_khz(vcpu, user_tsc_khz, use_scaling) && !init) {
> +		preempt_enable();
> +		return -1;
> +	}
> +
> +	vcpu->arch.virtual_tsc_shift = tsc_shift;
> +	vcpu->arch.virtual_tsc_mult = tsc_mult;
> +	vcpu->arch.virtual_tsc_khz = user_tsc_khz;

[Severity: Medium]
When init is true and set_tsc_khz() fails or enables catchup mode, does
this leave vcpu->arch.l1_tsc_scaling_ratio and vcpu->arch.tsc_scaling_ratio
at 0 instead of kvm_caps.default_tsc_scaling_ratio?

arch/x86/kvm/x86.c:set_tsc_khz() {
	...
	/* TSC scaling supported? */
	if (!kvm_caps.has_tsc_control) {
		if (user_tsc_khz > tsc_khz) {
			vcpu->arch.tsc_catchup = 1;
			vcpu->arch.tsc_always_catchup = 1;
			return 0;
		} else {
			pr_warn_ratelimited("user requested TSC rate below hardware speed\n");
			return -1;
		}
	}

	/* TSC scaling required  - calculate ratio */
	ratio = mul_u64_u32_div(1ULL << kvm_caps.tsc_scaling_ratio_frac_bits,
				user_tsc_khz, tsc_khz);

	if (ratio == 0 || ratio >= kvm_caps.max_tsc_scaling_ratio) {
		pr_warn_ratelimited("Invalid TSC scaling ratio - virtual-tsc-khz=%u\n",
			            user_tsc_khz);
		return -1;
	}
}

Because a newly allocated vCPU starts with a zeroed arch struct, neither
the !kvm_caps.has_tsc_control path nor the invalid ratio path calls
kvm_vcpu_write_tsc_multiplier().  A zero ratio causes kvm_scale_tsc() to
scale host_tsc by 0, sets kvm->arch.cur_tsc_scaling_ratio to 0 in
kvm_arch_vcpu_postcreate() (disabling the masterclock), and leaves VMX's
TSC_MULTIPLIER VMCS field at 0 (causing KVM_EXIT_FAIL_ENTRY on VM-Entry).
In addition, because virtual_tsc_khz is committed on creation even when
set_tsc_khz() failed, the new "!init && !freq_changed" early return above
causes a later vCPU KVM_SET_TSC_KHZ ioctl with that same unsupported
default_tsc_khz to return 0 instead of -EINVAL.

> +
> +	/*
> +	 * At vCPU creation (@init) the vCPU is not yet counted in
> +	 * online_vcpus, so synchronizing here would corrupt the VM-wide
> +	 * matching state (and leave it corrupted if creation subsequently
> +	 * fails). The initial synchronization happens in
> +	 * kvm_arch_vcpu_postcreate() instead.
> +	 */
> +	if (init)
> +		return 0;
> +
> +	/*
> +	 * Changing the frequency changes the vCPU's position on the
> +	 * generation's cycle timeline, so re-synchronize. For a vCPU
> +	 * which has run, preserve the guest TSC value read above across
> +	 * the change; likewise if userspace has written a TSC on this
> +	 * vCPU — a zero write would discard it. Otherwise the vCPU holds
> +	 * only its creation-time synchronization, so synchronize as for
> +	 * a zero write. When the generation is already at the new
> +	 * frequency this places the vCPU on its timeline rather than
> +	 * opening a gratuitous new generation and orphaning the matched
> +	 * state of its siblings; the first vCPU to adopt a new frequency
> +	 * necessarily opens a new generation, starting at zero — mixed
> +	 * frequencies within a VM forfeit the master clock anyway.
> +	 */
> +	raw_spin_lock_irqsave(&vcpu->kvm->arch.tsc_write_lock, flags);
> +	preempt_enable();
> +	if (vcpu->arch.last_vmentry_cpu != -1 || vcpu->arch.user_set_tsc)
> +		__kvm_synchronize_tsc_at(vcpu, &tsc, host_tsc, ns, false);
> +	else
> +		__kvm_synchronize_tsc_at(vcpu, NULL, host_tsc, ns, false);
> +	raw_spin_unlock_irqrestore(&vcpu->kvm->arch.tsc_write_lock, flags);
> +	return 0;
>  }

[Severity: Medium]
Does synchronizing in kvm_set_tsc_khz() break TSC and frequency matching
when userspace initializes or restores vCPUs one by one?

If a VMM creates and configures each vCPU in a loop (KVM_CREATE_VCPU
followed by KVM_SET_TSC_KHZ(F_new) per vCPU), vCPU 0's KVM_SET_TSC_KHZ sets
kvm->arch.cur_tsc_khz to F_new in generation 2.  Creating vCPU 1 then runs
kvm_arch_vcpu_postcreate() -> kvm_synchronize_tsc() at default_tsc_khz,
which sees virtual_tsc_khz != cur_tsc_khz, bumps cur_tsc_freq_generation
and cur_tsc_generation to 3, and resets cur_tsc_khz back to
default_tsc_khz.  When KVM_SET_TSC_KHZ(F_new) then runs on vCPU 1,
virtual_tsc_khz == cur_tsc_khz is false in __kvm_synchronize_tsc_at(), so
vCPU 1 opens generation 4 at data = 0 with a new offset and leaves vCPU 0
permanently orphaned in generation 2 with all_vcpus_matched_freq and
all_vcpus_matched_tsc both false (disabling the masterclock and
PVCLOCK_TSC_STABLE_BIT).  The same generation ping-pong happens when all
vCPUs are created first at default_tsc_khz and userspace restores each vCPU
in turn by writing MSR_IA32_TSC followed by KVM_SET_TSC_KHZ(F_new),
Despite commit "KVM: x86: Add KVM_VCPU_TSC_SCALE and fix the documentation
on TSC migration" later in the series documenting that userspace should
create all vCPUs before setting their frequency, existing VMMs already
perform per-vCPU setup and restore in a loop.

[ ... ]

> @@ -1511,40 +1616,84 @@ static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc,
>  	kvm_track_tsc_matching(vcpu, !matched);
>  }
>  
> -void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value)
> +/*
> + * Synchronize the vCPU's TSC against an explicit {host_tsc, ns} reference
> + * point: @value (when non-NULL) is the guest TSC *at* @host_tsc, not at
> + * some unspecified "now". The offset is computed against that reference,
> + * so a sampled value stays valid however much time passes between the
> + * sampling and this call.
> + */
> +static void __kvm_synchronize_tsc_at(struct kvm_vcpu *vcpu, u64 *value,
> +				     u64 host_tsc, s64 ns,
> +				     bool user_initiated)
>  {
> -	u64 data = user_value ? *user_value : 0;
> +	u64 data = value ? *value : 0;
>  	struct kvm *kvm = vcpu->kvm;
> -	u64 offset, host_tsc, elapsed;
> -	s64 ns;
> -	unsigned long flags;
> +	u64 elapsed_cycles = 0;
> +	s64 elapsed;
> +	u64 offset;
>  	bool matched = false;
>  	bool synchronizing = false;
>  
> -	raw_spin_lock_irqsave(&kvm->arch.tsc_write_lock, flags);
> +	lockdep_assert_held(&kvm->arch.tsc_write_lock);
>  
> -#ifdef CONFIG_X86_64
> -	if (!kvm_get_time_and_clockread(&ns, &host_tsc))
> -#endif
> -	{
> -		host_tsc = rdtsc();
> -		ns = get_kvmclock_base_ns();
> -	}
> +	/*
> +	 * A backwards-TSC compensation pending now was not in a sampled
> +	 * @value: it is applied to the vCPU's offset only at vcpu_load,
> +	 * and the sample was read through the uncompensated offset. Fold
> +	 * it in (scaled as adjust_tsc_offset_host() would), since
> +	 * __kvm_synchronize_tsc() clears the pending adjustment as
> +	 * superseded by the absolute offset installed here. A
> +	 * user-initiated value is the desired guest TSC *now*, and needs
> +	 * no correction.
> +	 */
> +	if (unlikely(vcpu->arch.tsc_offset_adjustment) &&
> +	    value && !user_initiated)
> +		data += kvm_scale_tsc(vcpu->arch.tsc_offset_adjustment,
> +				      vcpu->arch.l1_tsc_scaling_ratio);

[Severity: Medium]
Doesn't this scale vcpu->arch.tsc_offset_adjustment with the new scaling
ratio instead of the old one?

In kvm_set_tsc_khz(), tsc is sampled via kvm_read_l1_tsc() before
set_tsc_khz() runs, so the sample reflects the uncompensated offset under
the old l1_tsc_scaling_ratio.  set_tsc_khz() then overwrites
vcpu->arch.l1_tsc_scaling_ratio with the new ratio before calling
__kvm_synchronize_tsc_at().  If kvm_arch_enable_virtualization_cpu() queued
a backwards-TSC adjustment before tsc_write_lock was taken, scaling
tsc_offset_adjustment by the new l1_tsc_scaling_ratio here introduces a
step jump of kvm_scale_tsc(adj, R_new) - kvm_scale_tsc(adj, R_old) into the
preserved guest TSC and desynchronizes the vCPU from its siblings.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261111180922.4177927-1-dwmw2@infradead.org?part=10

Re: [PATCH v13 14/26] UAPI: x86: Move pvclock-abi to UAPI for x86 platforms

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] x86/xen: missing asm/types.h in xen-hypercalls.sh truncates
  xen-hypercalls.h
--

commit 02c9335f9071a6232c46010946be3c3ff37343b0
Author: Jack Allister <jalliste@amazon.com>

UAPI: x86: Move pvclock-abi to UAPI for x86 platforms

This commit moves arch/x86/include/asm/pvclock-abi.h to UAPI so userspace
can access struct pvclock_vcpu_time_info, switches its field types to UAPI
types via linux/types.h, and passes -D__KERNEL__ in
scripts/xen-hypercalls.sh.

> diff --git a/arch/x86/include/asm/pvclock-abi.h b/arch/x86/include/uapi/asm/pvclock-abi.h
> similarity index 82%
> rename from arch/x86/include/asm/pvclock-abi.h
> rename to arch/x86/include/uapi/asm/pvclock-abi.h
> index b9fece5fc96d6..6d70cf6403629 100644
> --- a/arch/x86/include/asm/pvclock-abi.h
> +++ b/arch/x86/include/uapi/asm/pvclock-abi.h
> @@ -1,6 +1,9 @@
> -/* SPDX-License-Identifier: GPL-2.0 */
> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
>  #ifndef _ASM_X86_PVCLOCK_ABI_H
>  #define _ASM_X86_PVCLOCK_ABI_H
> +
> +#include <linux/types.h>
> +
>  #ifndef __ASSEMBLER__

[ ... ]

> diff --git a/scripts/xen-hypercalls.sh b/scripts/xen-hypercalls.sh
> index f18b00843df3f..51a722198997d 100755
> --- a/scripts/xen-hypercalls.sh
> +++ b/scripts/xen-hypercalls.sh
> @@ -5,7 +5,7 @@ shift
>  in="$@"
>  
>  for i in $in; do
> -	eval $CPP $LINUXINCLUDE -dD -imacros "$i" -x c /dev/null
> +	eval $CPP -D__KERNEL__ $LINUXINCLUDE -dD -imacros "$i" -x c /dev/null
>  done | \
>  awk '$1 == "#define" && $2 ~ /__HYPERVISOR_[a-z][a-z_0-9]*/ { v[$3] = $2 }
>  	END {   print "/* auto-generated by scripts/xen-hypercall.sh */"

[Severity: Medium]
Does scripts/xen-hypercalls.sh fail to find asm/types.h on a clean x86
build with CONFIG_XEN=y, producing a truncated
arch/x86/include/generated/asm/xen-hypercalls.h?

When scripts/xen-hypercalls.sh preprocesses include/xen/interface/xen.h, it
pulls in asm/xen/interface.h -> asm/pvclock-abi.h -> linux/types.h ->
uapi/linux/types.h -> asm/types.h before the __HYPERVISOR_* definitions in
include/xen/interface/xen.h.  On x86, asm/types.h is generated in
arch/x86/include/generated/uapi/asm/types.h by the uapi-asm-generic target,
which is a prerequisite of asm-generic.  However, scripts/xen-hypercalls.sh
runs during archheaders, which archprepare orders before asm-generic
without a dependency on uapi-asm-generic:

Makefile:
archprepare: outputmakefile archheaders archscripts scripts include/config/kernel.release \
	asm-generic $(version_h) include/generated/utsrelease.h \

Because the pipeline in scripts/xen-hypercalls.sh ignores $CPP failures, the
build continues after $CPP aborts on missing asm/types.h and writes a
xen-hypercalls.h containing only HYPERCALL(mca) from xen-mca.h, which leaves
xen_hypercall_name() in arch/x86/xen/trace.c unable to resolve standard Xen
hypercall names for tracepoints.

Having archheaders depend on uapi-asm-generic (or avoiding the linux/types.h
dependency when preprocessing the hypercall headers) would ensure
asm/types.h is present before scripts/xen-hypercalls.sh runs.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261111180922.4177927-1-dwmw2@infradead.org?part=14

Re: [PATCH v13 16/26] KVM: x86: Add KVM_[GS]ET_CLOCK_GUEST for accurate KVM clock migration

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] KVM: x86: stale masterclock state in
  kvm_vcpu_ioctl_get_clock_guest()
--

commit 06987440f414497c2402ac529d8fa56884dbca59
Author: Jack Allister <jalliste@amazon.com>

KVM: x86: Add KVM_[GS]ET_CLOCK_GUEST for accurate KVM clock migration

This commit adds the KVM_GET_CLOCK_GUEST and KVM_SET_CLOCK_GUEST per-vCPU
ioctls to save and restore the PV clock parameters in terms of the guest
TSC across migration.

> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 743da71cd84d7..16383c71131d0 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
[ ... ]
> @@ -3941,6 +3944,218 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
>  	}
>  }
>  
> +#ifdef CONFIG_X86_64
> +static int kvm_vcpu_ioctl_get_clock_guest(struct kvm_vcpu *v, void __user *argp)
> +{
> +	struct pvclock_vcpu_time_info hv_clock = {};
> +	struct kvm_arch *ka = &v->kvm->arch;
> +	bool use_master_clock;
> +	unsigned int seq;
[ ... ]
> +	if (kvm_test_request(KVM_REQ_MASTERCLOCK_UPDATE, v))
> +		kvm_update_masterclock(v->kvm, v);

[Severity: Medium]
Can kvm_vcpu_ioctl_get_clock_guest() read stale VM-wide masterclock state
when KVM_REQ_MASTERCLOCK_UPDATE is queued on a different vCPU?

While host resume and gtod updates broadcast KVM_REQ_MASTERCLOCK_UPDATE
to all vCPUs, per-vCPU TSC and frequency synchronization paths queue it
only on the vCPU being modified:

arch/x86/kvm/x86.c:kvm_track_tsc_matching() {
    ...
	if ((ka->use_master_clock && update_mclock) ||
	    (ka->use_master_clock != use_master_clock))
		kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu);
}

If userspace changes vCPU 1's TSC frequency via KVM_SET_TSC_KHZ so that
ka->all_vcpus_matched_freq becomes false, KVM_REQ_MASTERCLOCK_UPDATE is
set only on vCPU 1 while ka->use_master_clock remains true.  Calling
KVM_GET_CLOCK_GUEST on vCPU 0 before vCPU 1 enters the guest sees no
pending request on vCPU 0, skips kvm_update_masterclock(), and returns
the stale masterclock parameters instead of -ENODATA (or conversely
returns -ENODATA after vCPU 1 restores frequency matching).

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261111180922.4177927-1-dwmw2@infradead.org?part=16