Re: [RFC PATCH 1/2] Hyper-H reference counter

Peter Lieven
2013-05-20T10:06:46Z

Hi all,

sorry that I am a bit unresponsive about this series. I have a few days off and can't spend much time in this.
If I read that the REFERENCE TSC breaks migration I don't think its a good option to include it at all.
I have this hyperv_refcnt MSR in an internal patch I sent over about 1.5 years ago and its working flawlessly
with Win2k8R2, Win7, Win8 + Win2012. I set the reference TSC to 0x00 and this seems to work with all
the above Windows versions. Some of the early Alphas of Windows 8 didn't work with this patch, but the
final is running smoothly also with migration etc.

I crafted this patch to avoid the heavy calls to PM Timer during high I/O which slowed down Windows
approx. by 30% compared to Hyper-V.

I reinclude this patch for reference. Its unchanged since mid 2012 and it might not apply.

Cheers,
Peter

diff -Npur kvm-kmod-3.3/include/asm-x86/hyperv.h kvm-kmod-3.3-hyperv-refcnt/include/asm-x86/hyperv.h
--- kvm-kmod-3.3/include/asm-x86/hyperv.h 2012-03-19 23:00:49.000000000 +0100
+++ kvm-kmod-3.3-hyperv-refcnt/include/asm-x86/hyperv.h 2012-03-28 12:23:02.000000000 +0200
@@ -169,7 +169,8 @@

/* MSR used to read the per-partition time reference counter */
#define HV_X64_MSR_TIME_REF_COUNT 0x40000020
-
+#define HV_X64_MSR_REFERENCE_TSC 0x40000021
+
/* Define the virtual APIC registers */
#define HV_X64_MSR_EOI 0x40000070
#define HV_X64_MSR_ICR 0x40000071
diff -Npur kvm-kmod-3.3/include/asm-x86/kvm_host.h kvm-kmod-3.3-hyperv-refcnt/include/asm-x86/kvm_host.h
--- kvm-kmod-3.3/include/asm-x86/kvm_host.h 2012-03-19 23:00:49.000000000 +0100
+++ kvm-kmod-3.3-hyperv-refcnt/include/asm-x86/kvm_host.h 2012-03-28 15:08:24.000000000 +0200
@@ -553,6 +553,8 @@ struct kvm_arch {
/* fields used by HYPER-V emulation */
u64 hv_guest_os_id;
u64 hv_hypercall;
+ u64 hv_ref_count;
+ u64 hv_reference_tsc;

atomic_t reader_counter;

diff -Npur kvm-kmod-3.3/x86/x86.c kvm-kmod-3.3-hyperv-refcnt/x86/x86.c
--- kvm-kmod-3.3/x86/x86.c 2012-03-19 23:00:56.000000000 +0100
+++ kvm-kmod-3.3-hyperv-refcnt/x86/x86.c 2012-03-28 16:27:46.000000000 +0200
@@ -826,7 +826,7 @@ EXPORT_SYMBOL_GPL(kvm_rdpmc);
static u32 msrs_to_save[] = {
MSR_KVM_SYSTEM_TIME, MSR_KVM_WALL_CLOCK,
MSR_KVM_SYSTEM_TIME_NEW, MSR_KVM_WALL_CLOCK_NEW,
- HV_X64_MSR_GUEST_OS_ID, HV_X64_MSR_HYPERCALL,
+ HV_X64_MSR_GUEST_OS_ID, HV_X64_MSR_HYPERCALL, HV_X64_MSR_TIME_REF_COUNT,
HV_X64_MSR_APIC_ASSIST_PAGE, MSR_KVM_ASYNC_PF_EN, MSR_KVM_STEAL_TIME,
MSR_IA32_SYSENTER_CS, MSR_IA32_SYSENTER_ESP, MSR_IA32_SYSENTER_EIP,
MSR_STAR,
@@ -1387,6 +1387,8 @@ static bool kvm_hv_msr_partition_wide(u3
switch (msr) {
case HV_X64_MSR_GUEST_OS_ID:
case HV_X64_MSR_HYPERCALL:
+ case HV_X64_MSR_REFERENCE_TSC:
+ case HV_X64_MSR_TIME_REF_COUNT:
r = true;
break;
}
@@ -1426,6 +1428,21 @@ static int set_msr_hyperv_pw(struct kvm_
if (__copy_to_user((void *)addr, instructions, 4))
return 1;
kvm->arch.hv_hypercall = data;
+ kvm->arch.hv_ref_count = get_kernel_ns();
+ break;
+ }
+ case HV_X64_MSR_REFERENCE_TSC: {
+ u64 gfn;
+ unsigned long addr;
+ u32 hv_tsc_sequence;
+ gfn = data >> HV_X64_MSR_HYPERCALL_PAGE_ADDRESS_SHIFT;
+ addr = gfn_to_hva(kvm, gfn);
+ if (kvm_is_error_hva(addr))
+ return 1;
+ hv_tsc_sequence = 0x0; //invalid
+ if (__copy_to_user((void *)addr, (void __user *) &hv_tsc_sequence, sizeof(hv_tsc_sequence)))
+ return 1;
+ kvm->arch.hv_reference_tsc = data;
break;
}
default:
@@ -1826,6 +1843,17 @@ static int get_msr_hyperv_pw(struct kvm_
case HV_X64_MSR_HYPERCALL:
data = kvm->arch.hv_hypercall;
break;
+ case HV_X64_MSR_TIME_REF_COUNT: {
+ u64 now_ns;
+ local_irq_disable();
+ now_ns = get_kernel_ns();
+ data = div_u64(now_ns + kvm->arch.kvmclock_offset - kvm->arch.hv_ref_count,100);
+ local_irq_enable();
+ break;
+ }
+ case HV_X64_MSR_REFERENCE_TSC:
+ data = kvm->arch.hv_reference_tsc;
+ break;
default:
pr_unimpl(vcpu, "Hyper-V unhandled rdmsr: 0x%x\n", msr);
return 1;

Am 20.05.2013 um 11:41 schrieb Gleb Natapov <gleb@redhat.com>:

> On Mon, May 20, 2013 at 11:32:27AM +0200, Paolo Bonzini wrote:
>> Il 20/05/2013 11:25, Gleb Natapov ha scritto:
>>> So in Hyper-V spec they
>>> say:
>>>
>>> Special value of 0xFFFFFFFF is used to indicate that this facility is no
>>> longer a reliable source of reference time and the virtual machine must
>>> fall back to a different source (for example, the virtual PM timer).
>>>
>>> May be they really mean "virtual PM timer" here and reference counter is
>>> not considered as a fall back source, but this is not what we want.
>>>
>>> On the other hand in API specification [1] they have:
>>>
>>> #define HV_REFERENCE_TSC_SEQUENCE_INVALID (0x00000000)
>>>
>>> which is not even documented in hyper-v spec. Actually 0 is specified as
>>> valid value there. Go figure.
>>>
>>> [1] http://msdn.microsoft.com/en-us/library/windows/hardware/ff540244%28v=vs.85%29.aspx
>>
>> Ok, if the API document is right then we should use
>> HV_REFERENCE_TSC_SEQUENCE_INVALID instead of 0, with a comment
>> explaining why we use 0 and not 0xFFFFFFFF.
>>
> Using define is always a good idea no matter if API doc is right or
> hyper-v spec is, it's just the "decent documentation" part that got me :)
> Definitely better than nothing and thanks them for that.
>
> --
> Gleb.