From mboxrd@z Thu Jan 1 00:00:00 1970 From: Radim =?utf-8?B?S3LEjW3DocWZ?= Subject: Re: KVM-Clock Date: Mon, 22 Feb 2016 21:08:15 +0100 Message-ID: <20160222200815.GA3189@potion.brq.redhat.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: "kvm@vger.kernel.org" To: Avi Cohen Return-path: Received: from mx1.redhat.com ([209.132.183.28]:32937 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751236AbcBVUIS (ORCPT ); Mon, 22 Feb 2016 15:08:18 -0500 Content-Disposition: inline In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: 2016-02-21 16:57+0000, Avi Cohen: > Hello, >=20 > Last week I've sent a mail regarding the kvm-clock accuracy. > Now I try to draw-up my question again, any answer/partial/hint is g= reatly appreciated >=20 > Our application is running in a Tenant's Virtual Machine in a data-c= entre. > We have some OAM functions running in the VMs. > One OAM function is to measure one-way delay between VMa and VMb.=20 > One way delay measurement requires that all machines should be synchr= onized to a common central clock. > Accuracy requirement is in order of 10s nano-seconds, hence only the = 1588v2/PTP is suitable here. > Since we cannot use HW timestamping in a virtual machine (we cannot f= orce using SR-IOV), I thought to run PTP on the physical machines and t= o sync the VMs to the host by the kvm-clock. kvmclock doesn't do synchronization with host clock or UTC. kvmclock bases on host's notion of *passed* time. kvmclock allows the guest to measure a flow of time. It is another layer's job to translate kvmclock result into a timestamp that can be compared. kvmclock was designed like that, because KVM wants to make a guest independent on hosts. > But now I see that the clock in the VM is far away from the host ( ~ = Hundreds of micro-second) , and this before I even run the PTP in the h= ost... What delta do you get after running PTP in guests? (Host and guest seem somehow synchronized, because QEMU stores host tim= e into RTC. The guest reads RTC on boot, but that has nothing to with kvmclock, and RTC's accuracy is poor.) > My test is very simple - I send a packet from host to the VM, I set t= he host time (tx_time) in the packet. When the guest receives the pack= et it reads its time (rx_time) and calculate the delay as : > Delay =3D rx_time - tx_time > I use the clock_gettime(REALTIME) in the host to set tx-time and in t= he guest to read rx_time.=20 > My questions : > 1. Assuming my HW support the paravirtualization clock requirements -= (see below output of cpuinfo) , (kvmclock clock doesn't have any requirements other than presence of TSC, which is why it's the default. The guest can have requirements that aren't met on some HW, though.) > In Theory - Is it possible to achieve 10s ns accuracy between VM clo= ck and the host clock ?=20 It is. (kvmclock on new CPUs has same drift/resolution/jitter/... as TSC. Reading kvmclock is slower that just doing rdtsc, but likely still within tens of nanoseconds.) > or I'm too na=C3=AFve and have to abandon the idea to run this timin= g sensitive application on a VM, and instead run it in Linux containe= r for example? =20 That depends on the reason behind synchronizing clocks, because VM can provide same precision as the host. Running in containers is almost th= e same as running on the host, so you might prefer their trade-offs. > 2. I understand that in the kvm-clock process, the kvm writes (whene= ver it enters the VM) (KVM doesn't update on every entry if you machine has invariable TSC.) > its system_time and the VM_TSC @ current time to the pvclock page , t= hen the guest OS can calculate its current time by: KVM doesn't write its (=3D host's) system_time. KVM writes *guest's* system_time. Guest's system_time at VM_TSC. (system_time is 0 when the VM starts. sytem_time can store ~584 years worth of nanoseconds, but using an arbitrary offset makes everything simpler. This part of kvmclock is pretty confusing, so system_time is likely the source of misunderstanding.) Have you read that kvmclock does synchronization with host time somewhere? Thanks. > Current-time =3D system_time + multiplier (RDTSC() -VM_TSC) (system= _time and VM_TSC as set by the kvmas set by the kvm)) > I understand that there is no VM-exit when the VM calls RDTSC(). Yes. > - is that description correct ? Partly. > I understand that this is supported by the guest OS and this should b= e transparent to my application, correct ? Yes. > My guest and host are Fedora 22. > =20 > 3. Other idea how to achieve this accuracy ? =20 I hope that PTP is enough, because I can't recommend anything that can be done without introducing a new paravirtual device ... an example of potential one-time synchronization with higher accuracy than PTP: A guest asks a host what time it is by issuing a hypercall. The host replies with a (kvmclock timestamp, nanoseconds since some standard time) pair. It's going to be more complex if you want more features.