Re: [RFC PATCH v3 2/2] add support for Hyper-V partition reference time enlightenment

[Marcelo Tosatti <mtosatti@redhat.com>]

On Mon, Jan 13, 2014 at 11:11:40PM -0500, Vadim Rozenfeld wrote:
> 
> 
> ----- Original Message -----
> From: "Marcelo Tosatti" <mtosatti@redhat.com>
> To: "Vadim Rozenfeld" <vrozenfe@redhat.com>
> Cc: kvm@vger.kernel.org, pl@dlhnet.de, pbonzini@redhat.com
> Sent: Thursday, December 12, 2013 6:27:00 AM
> Subject: Re: [RFC PATCH v3 2/2] add support for Hyper-V partition reference time enlightenment
> 
> On Sun, Dec 08, 2013 at 10:33:39PM +1100, Vadim Rozenfeld wrote:
> > The following patch allows to activate a partition reference
> > time enlightenment that is based on the host platform's support
> > for an Invariant Time Stamp Counter (iTSC).
> > 
> > v2 -> v3
> > Handle TSC sequence, scale, and offest changing during migration.
> > 
> > ---
> >  arch/x86/include/asm/kvm_host.h |  1 +
> >  arch/x86/kvm/x86.c              | 29 +++++++++++++++++++++++++++--
> >  2 files changed, 28 insertions(+), 2 deletions(-)
> > 
> > diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> > index 2fd0753..81fdff0 100644
> > --- a/arch/x86/include/asm/kvm_host.h
> > +++ b/arch/x86/include/asm/kvm_host.h
> > @@ -607,6 +607,7 @@ struct kvm_arch {
> >  	u64 hv_hypercall;
> >  	u64 hv_ref_count;
> >  	u64 hv_tsc_page;
> > +	u64 hv_ref_time;
> >  
> >  	#ifdef CONFIG_KVM_MMU_AUDIT
> >  	int audit_point;
> > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> > index 5e4e495a..cb6766a 100644
> > --- a/arch/x86/kvm/x86.c
> > +++ b/arch/x86/kvm/x86.c
> > @@ -1882,14 +1882,19 @@ static int set_msr_hyperv_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data)
> >  			break;
> >  		}
> >  		gfn = data >> HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT;
> > -		addr = gfn_to_hva(kvm, data >>
> > -			HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT);
> > +		addr = gfn_to_hva(kvm, gfn);
> >  		if (kvm_is_error_hva(addr))
> >  			return 1;
> > +		tsc_ref.tsc_sequence =
> > +			boot_cpu_has(X86_FEATURE_CONSTANT_TSC) ? 1 : 0;
> > +		tsc_ref.tsc_scale =
> > +			((10000LL << 32) / vcpu->arch.virtual_tsc_khz) << 32;
> > +		tsc_ref.tsc_offset = 0;
> >  		if (__copy_to_user((void __user *)addr, &tsc_ref, sizeof(tsc_ref)))
> >  			return 1;
> >  		mark_page_dirty(kvm, gfn);
> >  		kvm->arch.hv_tsc_page = data;
> > +		kvm->arch.hv_ref_count = 0;
> >  		break;
> >  	}
> >  	default:
> > @@ -3879,6 +3884,19 @@ long kvm_arch_vm_ioctl(struct file *filp,
> >  		local_irq_enable();
> >  		kvm->arch.kvmclock_offset = delta;
> >  		kvm_gen_update_masterclock(kvm);
> > +
> > +		if (kvm->arch.hv_tsc_page & HV_X64_MSR_TSC_REFERENCE_ENABLE) {
> > +			HV_REFERENCE_TSC_PAGE* tsc_ref;
> > +			u64 curr_time;
> > +			tsc_ref = (HV_REFERENCE_TSC_PAGE*)gfn_to_hva(kvm, 
> > +				kvm->arch.hv_tsc_page >> HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT);
> > +			tsc_ref->tsc_sequence =
> > +				boot_cpu_has(X86_FEATURE_CONSTANT_TSC) ? tsc_ref->tsc_sequence + 1 : 0;
> > +			tsc_ref->tsc_scale = ((10000LL << 32) / __get_cpu_var(cpu_tsc_khz)) << 32;
> > +			curr_time = (((tsc_ref->tsc_scale >> 32) * native_read_tsc()) >> 32) + 
> > +				tsc_ref->tsc_offset;
> > +			tsc_ref->tsc_offset = kvm->arch.hv_ref_time - curr_time;
> > +		}
> >  		break;
> >  	}
> >  	case KVM_GET_CLOCK: {
> > @@ -3896,6 +3914,13 @@ long kvm_arch_vm_ioctl(struct file *filp,
> >  		if (copy_to_user(argp, &user_ns, sizeof(user_ns)))
> >  			goto out;
> >  		r = 0;
> > +		if (kvm->arch.hv_tsc_page & HV_X64_MSR_TSC_REFERENCE_ENABLE) {
> > +			HV_REFERENCE_TSC_PAGE* tsc_ref;
> > +			tsc_ref = (HV_REFERENCE_TSC_PAGE*)gfn_to_hva(kvm,
> > +				kvm->arch.hv_tsc_page >> HV_X64_MSR_TSC_REFERENCE_ADDRESS_SHIFT);
> 
> kvm_read_guest_cached.
> 
> > +			kvm->arch.hv_ref_time = (((tsc_ref->tsc_scale >> 32) * 
> > +				native_read_tsc()) >> 32) + tsc_ref->tsc_offset;
> 
> Why native_read_tsc and not ->read_l1_tsc?
> 
> [VR]
> Is it possible to get pointer to the vcpu instance at this point?

See the suggestion to move this code to kvm_guest_time_update.


> Thanks,
> Vadim. 
> 
> It is easier to trust on the host to check reliability of the TSC: if
> it uses TSC clocksource, then the TSCs are stable. So could condition
> exposing the TSC ref page when ka->use_master_clock=1, see kvm_guest_time_update.
> And hook into pvclock_gtod_notify.
> 
> So in addition to X86_FEATURE_CONSTANT_TSC, check
> ka->use_master_clock=1
>