KVM-Clock

KVM-Clock

[Avi Cohen]

Hello,
Last week I've sent a mail regarding the kvm-clock accuracy.
Now I try to draw-up my question again, any answer/partial/hint  is greatly appreciated

Our application is running in  a Tenant's Virtual Machine in a data-centre.
We have some OAM functions running in  the VMs.
One OAM function is to measure one-way delay between VMa and VMb. 
One way delay measurement requires that all machines should be synchronized to a common central clock.
Accuracy requirement is in order of 10s nano-seconds, hence only the 1588v2/PTP is suitable here.
Since we cannot use HW timestamping in a virtual machine (we cannot force using SR-IOV), I thought to run PTP on the physical machines and to sync the VMs to the host by the kvm-clock.
But now I see that the clock in the VM is far away from the host ( ~ Hundreds of micro-second) , and this before I even run the PTP in the host...
My test is very simple - I send a packet from host to the VM, I set the host time (tx_time) in the packet.  When the guest receives the packet it reads its time (rx_time)  and calculate the delay as :
Delay = rx_time - tx_time
I use the clock_gettime(REALTIME) in the host to set tx-time and in the guest to read rx_time. 
My questions :
1. Assuming my HW support the paravirtualization clock requirements - (see below output of cpuinfo)  , In Theory  - Is it possible to achieve 10s ns accuracy between VM clock and the host clock ? 
or I'm too naïve and have  to abandon the idea to run this timing sensitive application on a VM, and instead run it in  Linux  container for example?  

2. I understand that in the  kvm-clock process, the kvm writes (whenever it enters the VM)  its system_time and the VM_TSC @ current time to the pvclock page , then the guest OS can calculate its current time by:
Current-time = system_time + multiplier (RDTSC() -VM_TSC)   (system_time and VM_TSC as set by the kvmas set by the kvm))
I understand that there is no VM-exit when the VM calls RDTSC().  - is that description correct ?   I understand that this is supported by the guest OS and this should be transparent to my application, correct ?
My guest and host are Fedora 22.
 
3. Other idea how to achieve this accuracy ?   
Thanks in advance
Avi

----------------------------------------------------

Here's the 1st segment  of /proc/cpuinfo:

processor         : 0
vendor_id        : GenuineIntel
cpu family       : 6
model              : 63
model name     : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz stepping           : 2 microcode       : 0x31 cpu MHz                     : 1200.062 cache size        : 30720 KB physical id       : 0 siblings            : 24 core id             : 0 cpu cores         : 12 apicid              : 0 initial apicid    : 0 fpu                   : yes fpu_exception : yes cpuid level       : 15 wp                   : yes flags                : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc bugs                 :
bogomips         : 5199.72
clflush size      : 64
cache_alignment         : 64
address sizes   : 46 bits physical, 48 bits virtual power management:

Re: KVM-Clock

[Radim Krčmář]

2016-02-21 16:57+0000, Avi Cohen:
> Hello,
> 
> Last week I've sent a mail regarding the kvm-clock accuracy.
> Now I try to draw-up my question again, any answer/partial/hint  is greatly appreciated
> 
> Our application is running in  a Tenant's Virtual Machine in a data-centre.
> We have some OAM functions running in  the VMs.
> One OAM function is to measure one-way delay between VMa and VMb. 
> One way delay measurement requires that all machines should be synchronized to a common central clock.
> Accuracy requirement is in order of 10s nano-seconds, hence only the 1588v2/PTP is suitable here.
> Since we cannot use HW timestamping in a virtual machine (we cannot force using SR-IOV), I thought to run PTP on the physical machines and to sync the VMs to the host by the kvm-clock.

kvmclock doesn't do synchronization with host clock or UTC.
kvmclock bases on host's notion of *passed* time.
kvmclock allows the guest to measure a flow of time.

It is another layer's job to translate kvmclock result into a timestamp
that can be compared.  kvmclock was designed like that, because KVM
wants to make a guest independent on hosts.

> But now I see that the clock in the VM is far away from the host ( ~ Hundreds of micro-second) , and this before I even run the PTP in the host...

What delta do you get after running PTP in guests?

(Host and guest seem somehow synchronized, because QEMU stores host time
 into RTC.  The guest reads RTC on boot, but that has nothing to with
 kvmclock, and RTC's accuracy is poor.)

> My test is very simple - I send a packet from host to the VM, I set the host time (tx_time) in the packet.  When the guest receives the packet it reads its time (rx_time)  and calculate the delay as :
> Delay = rx_time - tx_time
> I use the clock_gettime(REALTIME) in the host to set tx-time and in the guest to read rx_time. 
> My questions :
> 1. Assuming my HW support the paravirtualization clock requirements - (see below output of cpuinfo)  ,

(kvmclock clock doesn't have any requirements other than presence of
 TSC, which is why it's the default.  The guest can have requirements
 that aren't met on some HW, though.)

> In Theory  - Is it possible to achieve 10s ns accuracy between VM clock and the host clock ? 

It is.

(kvmclock on new CPUs has same drift/resolution/jitter/... as TSC.
 Reading kvmclock is slower that just doing rdtsc, but likely still
 within tens of nanoseconds.)

> or I'm too naïve and have  to abandon the idea to run this timing sensitive application on a VM, and instead run it in  Linux  container for example?  

That depends on the reason behind synchronizing clocks, because VM can
provide same precision as the host.  Running in containers is almost the
same as running on the host, so you might prefer their trade-offs.

> 2. I understand that in the  kvm-clock process, the kvm writes (whenever it enters the VM)

(KVM doesn't update on every entry if you machine has invariable TSC.)

> its system_time and the VM_TSC @ current time to the pvclock page , then the guest OS can calculate its current time by:

KVM doesn't write its (= host's) system_time.
KVM writes *guest's* system_time.  Guest's system_time at VM_TSC.

(system_time is 0 when the VM starts.  sytem_time can store ~584 years
 worth of nanoseconds, but using an arbitrary offset makes everything
 simpler.  This part of kvmclock is pretty confusing, so system_time is
 likely the source of misunderstanding.)

Have you read that kvmclock does synchronization with host time
somewhere?

Thanks.

> Current-time = system_time + multiplier (RDTSC() -VM_TSC)   (system_time and VM_TSC as set by the kvmas set by the kvm))
> I understand that there is no VM-exit when the VM calls RDTSC().

Yes.

> - is that description correct ?

Partly.

> I understand that this is supported by the guest OS and this should be transparent to my application, correct ?

Yes.

> My guest and host are Fedora 22.
>  
> 3. Other idea how to achieve this accuracy ?   

I hope that PTP is enough, because I can't recommend anything that can
be done without introducing a new paravirtual device ... an example of
potential one-time synchronization with higher accuracy than PTP:

A guest asks a host what time it is by issuing a hypercall.
The host replies with a
  (kvmclock timestamp, nanoseconds since some standard time) pair.

It's going to be more complex if you want more features.

RE: KVM-Clock

[Avi Cohen]

2016-02-21 16:57+0000, Avi Cohen:
> Hello,
> 
> Last week I've sent a mail regarding the kvm-clock accuracy.
> Now I try to draw-up my question again, any answer/partial/hint  is 
> greatly appreciated
> 
> Our application is running in  a Tenant's Virtual Machine in a data-centre.
> We have some OAM functions running in  the VMs.
> One OAM function is to measure one-way delay between VMa and VMb. 
> One way delay measurement requires that all machines should be synchronized to a common central clock.
> Accuracy requirement is in order of 10s nano-seconds, hence only the 1588v2/PTP is suitable here.
> Since we cannot use HW timestamping in a virtual machine (we cannot force using SR-IOV), I thought to run PTP on the physical machines and to sync the VMs to the host by the kvm-clock.

kvmclock doesn't do synchronization with host clock or UTC.
kvmclock bases on host's notion of *passed* time.
kvmclock allows the guest to measure a flow of time.

It is another layer's job to translate kvmclock result into a timestamp that can be compared.  kvmclock was designed like that, because KVM wants to make a guest independent on hosts.

I see the system time written by KVM  whenever the VM is entered  - in  kvm_guest_time_update()
The KVM updates  system_time with it’s monotonic-time (as you say *passed* time)
How can  the guest  (or  another layer's job ) - translate kvmclock result into a timestamp that can be compared ?
This is much required when I'll run the PTP in the host  - in which the CLOCK_REALTIME will be affected (probably not the CLOCK_MONOTONIC)

> But now I see that the clock in the VM is far away from the host ( ~ Hundreds of micro-second) , and this before I even run the PTP in the host...

What delta do you get after running PTP in guests?

I still don’t run the PTP - first I want to sync the guest to the host free-running clock , then If it succeeds I'll run the PTP 
Currently without the PTP , I see about 1 second delta between the 2 clocks.

(Host and guest seem somehow synchronized, because QEMU stores host time  into RTC.  The guest reads RTC on boot, but that has nothing to with  kvmclock, and RTC's accuracy is poor.)

I understand that - but I expect that later the 2 clocks will be in-sync.

> My test is very simple - I send a packet from host to the VM, I set the host time (tx_time) in the packet.  When the guest receives the packet it reads its time (rx_time)  and calculate the delay as :
> Delay = rx_time - tx_time
> I use the clock_gettime(REALTIME) in the host to set tx-time and in the guest to read rx_time. 
> My questions :
> 1. Assuming my HW support the paravirtualization clock requirements - 
> (see below output of cpuinfo)  ,

(kvmclock clock doesn't have any requirements other than presence of  TSC, which is why it's the default.  The guest can have requirements  that aren't met on some HW, though.)

I meant that my HW supports the constant TSC - hence I can rely on the TSC.

> In Theory  - Is it possible to achieve 10s ns accuracy between VM clock and the host clock ? 

It is.

How ? this is the only question


(kvmclock on new CPUs has same drift/resolution/jitter/... as TSC.
 Reading kvmclock is slower that just doing rdtsc, but likely still  within tens of nanoseconds.)

> or I'm too naïve and have  to abandon the idea to run this timing sensitive application on a VM, and instead run it in  Linux  container for example?  

That depends on the reason behind synchronizing clocks, because VM can provide same precision as the host.  Running in containers is almost the same as running on the host, so you might prefer their trade-offs.

> 2. I understand that in the  kvm-clock process, the kvm writes 
> (whenever it enters the VM)

(KVM doesn't update on every entry if you machine has invariable TSC.)

I have constant TSC 
> its system_time and the VM_TSC @ current time to the pvclock page , then the guest OS can calculate its current time by:

KVM doesn't write its (= host's) system_time.
KVM writes *guest's* system_time.  Guest's system_time at VM_TSC.

(system_time is 0 when the VM starts.  sytem_time can store ~584 years  worth of nanoseconds, but using an arbitrary offset makes everything  simpler.  This part of kvmclock is pretty confusing, so system_time is  likely the source of misunderstanding.)

Have you read that kvmclock does synchronization with host time somewhere?

Yes - see - in  vcpu_enter_guest() there is a call to   kvm_guest_time_update()
Which update  the pvclock paget   for the guest - see that it updates the system_time  with the host system-time 

struct pvclock_vcpu_time_info {
 26         u32   version;
 27         u32   pad0;
 28         u64   tsc_timestamp;
 29         u64   system_time;
 30         u32   tsc_to_system_mul;
 31         s8    tsc_shift;
 32         u8    flags;
 33         u8    pad[2];
 34 } __attribute__((__packed__)); /* 32 bytes */

Thanks.

> Current-time = system_time + multiplier (RDTSC() -VM_TSC)   (system_time and VM_TSC as set by the kvmas set by the kvm))
> I understand that there is no VM-exit when the VM calls RDTSC().

Yes.

> - is that description correct ?

Partly.

> I understand that this is supported by the guest OS and this should be transparent to my application, correct ?

Yes.

> My guest and host are Fedora 22.
>  
> 3. Other idea how to achieve this accuracy ?   

I hope that PTP is enough, because I can't recommend anything that can be done without introducing a new paravirtual device ... an example of potential one-time synchronization with higher accuracy than PTP:

A guest asks a host what time it is by issuing a hypercall.
The host replies with a
  (kvmclock timestamp, nanoseconds since some standard time) pair.

It's going to be more complex if you want more features.

Re: KVM-Clock

[Wanpeng Li]

2016-02-23 15:11 GMT+08:00 Avi Cohen <avi.cohen@huawei.com>:
>
> 2016-02-21 16:57+0000, Avi Cohen:
>> Hello,
>>
>> Last week I've sent a mail regarding the kvm-clock accuracy.
>> Now I try to draw-up my question again, any answer/partial/hint  is
>> greatly appreciated
>>
>> Our application is running in  a Tenant's Virtual Machine in a data-centre.
>> We have some OAM functions running in  the VMs.
>> One OAM function is to measure one-way delay between VMa and VMb.
>> One way delay measurement requires that all machines should be synchronized to a common central clock.
>> Accuracy requirement is in order of 10s nano-seconds, hence only the 1588v2/PTP is suitable here.
>> Since we cannot use HW timestamping in a virtual machine (we cannot force using SR-IOV), I thought to run PTP on the physical machines and to sync the VMs to the host by the kvm-clock.
>
> kvmclock doesn't do synchronization with host clock or UTC.
> kvmclock bases on host's notion of *passed* time.
> kvmclock allows the guest to measure a flow of time.
>
> It is another layer's job to translate kvmclock result into a timestamp that can be compared.  kvmclock was designed like that, because KVM wants to make a guest independent on hosts.
>
> I see the system time written by KVM  whenever the VM is entered  - in  kvm_guest_time_update()
> The KVM updates  system_time with it’s monotonic-time (as you say *passed* time)

boot time instead of monotonic time, boot time is equal to monotonic
time + suspend time

> How can  the guest  (or  another layer's job ) - translate kvmclock result into a timestamp that can be compared ?
> This is much required when I'll run the PTP in the host  - in which the CLOCK_REALTIME will be affected (probably not the CLOCK_MONOTONIC)
>
>> But now I see that the clock in the VM is far away from the host ( ~ Hundreds of micro-second) , and this before I even run the PTP in the host...
>
> What delta do you get after running PTP in guests?
>
> I still don’t run the PTP - first I want to sync the guest to the host free-running clock , then If it succeeds I'll run the PTP
> Currently without the PTP , I see about 1 second delta between the 2 clocks.
>
> (Host and guest seem somehow synchronized, because QEMU stores host time  into RTC.  The guest reads RTC on boot, but that has nothing to with  kvmclock, and RTC's accuracy is poor.)
>
> I understand that - but I expect that later the 2 clocks will be in-sync.
>
>> My test is very simple - I send a packet from host to the VM, I set the host time (tx_time) in the packet.  When the guest receives the packet it reads its time (rx_time)  and calculate the delay as :
>> Delay = rx_time - tx_time
>> I use the clock_gettime(REALTIME) in the host to set tx-time and in the guest to read rx_time.
>> My questions :
>> 1. Assuming my HW support the paravirtualization clock requirements -
>> (see below output of cpuinfo)  ,
>
> (kvmclock clock doesn't have any requirements other than presence of  TSC, which is why it's the default.  The guest can have requirements  that aren't met on some HW, though.)
>
> I meant that my HW supports the constant TSC - hence I can rely on the TSC.
>
>> In Theory  - Is it possible to achieve 10s ns accuracy between VM clock and the host clock ?
>
> It is.
>
> How ? this is the only question
>
>
> (kvmclock on new CPUs has same drift/resolution/jitter/... as TSC.
>  Reading kvmclock is slower that just doing rdtsc, but likely still  within tens of nanoseconds.)
>
>> or I'm too naïve and have  to abandon the idea to run this timing sensitive application on a VM, and instead run it in  Linux  container for example?
>
> That depends on the reason behind synchronizing clocks, because VM can provide same precision as the host.  Running in containers is almost the same as running on the host, so you might prefer their trade-offs.
>
>> 2. I understand that in the  kvm-clock process, the kvm writes
>> (whenever it enters the VM)
>
> (KVM doesn't update on every entry if you machine has invariable TSC.)
>
> I have constant TSC
>> its system_time and the VM_TSC @ current time to the pvclock page , then the guest OS can calculate its current time by:
>
> KVM doesn't write its (= host's) system_time.
> KVM writes *guest's* system_time.  Guest's system_time at VM_TSC.
>
> (system_time is 0 when the VM starts.  sytem_time can store ~584 years  worth of nanoseconds, but using an arbitrary offset makes everything  simpler.  This part of kvmclock is pretty confusing, so system_time is  likely the source of misunderstanding.)
>
> Have you read that kvmclock does synchronization with host time somewhere?
>
> Yes - see - in  vcpu_enter_guest() there is a call to   kvm_guest_time_update()
> Which update  the pvclock paget   for the guest - see that it updates the system_time  with the host system-time
>
> struct pvclock_vcpu_time_info {
>  26         u32   version;
>  27         u32   pad0;
>  28         u64   tsc_timestamp;
>  29         u64   system_time;
>  30         u32   tsc_to_system_mul;
>  31         s8    tsc_shift;
>  32         u8    flags;
>  33         u8    pad[2];
>  34 } __attribute__((__packed__)); /* 32 bytes */
>
> Thanks.
>
>> Current-time = system_time + multiplier (RDTSC() -VM_TSC)   (system_time and VM_TSC as set by the kvmas set by the kvm))
>> I understand that there is no VM-exit when the VM calls RDTSC().
>
> Yes.
>
>> - is that description correct ?
>
> Partly.
>
>> I understand that this is supported by the guest OS and this should be transparent to my application, correct ?
>
> Yes.
>
>> My guest and host are Fedora 22.
>>
>> 3. Other idea how to achieve this accuracy ?
>
> I hope that PTP is enough, because I can't recommend anything that can be done without introducing a new paravirtual device ... an example of potential one-time synchronization with higher accuracy than PTP:
>
> A guest asks a host what time it is by issuing a hypercall.
> The host replies with a
>   (kvmclock timestamp, nanoseconds since some standard time) pair.
>
> It's going to be more complex if you want more features.



-- 
Regards,
Wanpeng Li

RE: KVM-Clock

[Avi Cohen]

2016-02-23 15:11 GMT+08:00 Avi Cohen <avi.cohen@huawei.com>:
>
> 2016-02-21 16:57+0000, Avi Cohen:
>> Hello,
>>
>> Last week I've sent a mail regarding the kvm-clock accuracy.
>> Now I try to draw-up my question again, any answer/partial/hint  is 
>> greatly appreciated
>>
>> Our application is running in  a Tenant's Virtual Machine in a data-centre.
>> We have some OAM functions running in  the VMs.
>> One OAM function is to measure one-way delay between VMa and VMb.
>> One way delay measurement requires that all machines should be synchronized to a common central clock.
>> Accuracy requirement is in order of 10s nano-seconds, hence only the 1588v2/PTP is suitable here.
>> Since we cannot use HW timestamping in a virtual machine (we cannot force using SR-IOV), I thought to run PTP on the physical machines and to sync the VMs to the host by the kvm-clock.
>
> kvmclock doesn't do synchronization with host clock or UTC.
> kvmclock bases on host's notion of *passed* time.
> kvmclock allows the guest to measure a flow of time.
>
> It is another layer's job to translate kvmclock result into a timestamp that can be compared.  kvmclock was designed like that, because KVM wants to make a guest independent on hosts.
>
> I see the system time written by KVM  whenever the VM is entered  - in  
> kvm_guest_time_update() The KVM updates  system_time with it’s 
> monotonic-time (as you say *passed* time)

>boot time instead of monotonic time, boot time is equal to monotonic time + suspend time
If I call clock_gettime   with BOOTTIME or MONOTONIC - the 2 clocks are far away -  this is the clock that is counting from boot time.
Only if I call clock_gettime    with REALTIME (both in the host and guest ) - I can see that the clocks are close,  but still about 1 second delta

> How can  the guest  (or  another layer's job ) - translate kvmclock result into a timestamp that can be compared ?
> This is much required when I'll run the PTP in the host  - in which 
> the CLOCK_REALTIME will be affected (probably not the CLOCK_MONOTONIC)
>
>> But now I see that the clock in the VM is far away from the host ( ~ Hundreds of micro-second) , and this before I even run the PTP in the host...
>
> What delta do you get after running PTP in guests?
>
> I still don’t run the PTP - first I want to sync the guest to the host 
> free-running clock , then If it succeeds I'll run the PTP Currently without the PTP , I see about 1 second delta between the 2 clocks.
>
> (Host and guest seem somehow synchronized, because QEMU stores host 
> time  into RTC.  The guest reads RTC on boot, but that has nothing to 
> with  kvmclock, and RTC's accuracy is poor.)
>
> I understand that - but I expect that later the 2 clocks will be in-sync.
>
>> My test is very simple - I send a packet from host to the VM, I set the host time (tx_time) in the packet.  When the guest receives the packet it reads its time (rx_time)  and calculate the delay as :
>> Delay = rx_time - tx_time
>> I use the clock_gettime(REALTIME) in the host to set tx-time and in the guest to read rx_time.
>> My questions :
>> 1. Assuming my HW support the paravirtualization clock requirements - 
>> (see below output of cpuinfo)  ,
>
> (kvmclock clock doesn't have any requirements other than presence of  
> TSC, which is why it's the default.  The guest can have requirements  
> that aren't met on some HW, though.)
>
> I meant that my HW supports the constant TSC - hence I can rely on the TSC.
>
>> In Theory  - Is it possible to achieve 10s ns accuracy between VM clock and the host clock ?
>
> It is.
>
> How ? this is the only question
>
>
> (kvmclock on new CPUs has same drift/resolution/jitter/... as TSC.
>  Reading kvmclock is slower that just doing rdtsc, but likely still  
> within tens of nanoseconds.)
>
>> or I'm too naïve and have  to abandon the idea to run this timing sensitive application on a VM, and instead run it in  Linux  container for example?
>
> That depends on the reason behind synchronizing clocks, because VM can provide same precision as the host.  Running in containers is almost the same as running on the host, so you might prefer their trade-offs.
>
>> 2. I understand that in the  kvm-clock process, the kvm writes 
>> (whenever it enters the VM)
>
> (KVM doesn't update on every entry if you machine has invariable TSC.)
>
> I have constant TSC
>> its system_time and the VM_TSC @ current time to the pvclock page , then the guest OS can calculate its current time by:
>
> KVM doesn't write its (= host's) system_time.
> KVM writes *guest's* system_time.  Guest's system_time at VM_TSC.
>
> (system_time is 0 when the VM starts.  sytem_time can store ~584 years  
> worth of nanoseconds, but using an arbitrary offset makes everything  
> simpler.  This part of kvmclock is pretty confusing, so system_time is  
> likely the source of misunderstanding.)
>
> Have you read that kvmclock does synchronization with host time somewhere?
>
> Yes - see - in  vcpu_enter_guest() there is a call to   kvm_guest_time_update()
> Which update  the pvclock paget   for the guest - see that it updates the system_time  with the host system-time
>
> struct pvclock_vcpu_time_info {
>  26         u32   version;
>  27         u32   pad0;
>  28         u64   tsc_timestamp;
>  29         u64   system_time;
>  30         u32   tsc_to_system_mul;
>  31         s8    tsc_shift;
>  32         u8    flags;
>  33         u8    pad[2];
>  34 } __attribute__((__packed__)); /* 32 bytes */
>
> Thanks.
>
>> Current-time = system_time + multiplier (RDTSC() -VM_TSC)   (system_time and VM_TSC as set by the kvmas set by the kvm))
>> I understand that there is no VM-exit when the VM calls RDTSC().
>
> Yes.
>
>> - is that description correct ?
>
> Partly.
>
>> I understand that this is supported by the guest OS and this should be transparent to my application, correct ?
>
> Yes.
>
>> My guest and host are Fedora 22.
>>
>> 3. Other idea how to achieve this accuracy ?
>
> I hope that PTP is enough, because I can't recommend anything that can be done without introducing a new paravirtual device ... an example of potential one-time synchronization with higher accuracy than PTP:
>
> A guest asks a host what time it is by issuing a hypercall.
> The host replies with a
>   (kvmclock timestamp, nanoseconds since some standard time) pair.
>
> It's going to be more complex if you want more features.



--
Regards,
Wanpeng Li
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: KVM-Clock

[Avi Cohen]

2016-02-23 15:11 GMT+08:00 Avi Cohen <avi.cohen@huawei.com>:
>
> 2016-02-21 16:57+0000, Avi Cohen:
>> Hello,
>>
>> Last week I've sent a mail regarding the kvm-clock accuracy.
>> Now I try to draw-up my question again, any answer/partial/hint  is 
>> greatly appreciated
>>
>> Our application is running in  a Tenant's Virtual Machine in a data-centre.
>> We have some OAM functions running in  the VMs.
>> One OAM function is to measure one-way delay between VMa and VMb.
>> One way delay measurement requires that all machines should be synchronized to a common central clock.
>> Accuracy requirement is in order of 10s nano-seconds, hence only the 1588v2/PTP is suitable here.
>> Since we cannot use HW timestamping in a virtual machine (we cannot force using SR-IOV), I thought to run PTP on the physical machines and to sync the VMs to the host by the kvm-clock.
>
> kvmclock doesn't do synchronization with host clock or UTC.
> kvmclock bases on host's notion of *passed* time.
> kvmclock allows the guest to measure a flow of time.
>
> It is another layer's job to translate kvmclock result into a timestamp that can be compared.  kvmclock was designed like that, because KVM wants to make a guest independent on hosts.
>
> I see the system time written by KVM  whenever the VM is entered  - in
> kvm_guest_time_update() The KVM updates  system_time with it’s 
> monotonic-time (as you say *passed* time)

>boot time instead of monotonic time, boot time is equal to monotonic 
>time + suspend time

When I set  the tx_time in the host to  BOOTTIME, and I read the rx_time in the guest with REALTIME - then the sync is improved. 
But still I see delta time that jitters between 100us to 300us  


> How can  the guest  (or  another layer's job ) - translate kvmclock result into a timestamp that can be compared ?
> This is much required when I'll run the PTP in the host  - in which 
> the CLOCK_REALTIME will be affected (probably not the CLOCK_MONOTONIC)
>
>> But now I see that the clock in the VM is far away from the host ( ~ Hundreds of micro-second) , and this before I even run the PTP in the host...
>
> What delta do you get after running PTP in guests?
>
> I still don’t run the PTP - first I want to sync the guest to the host 
> free-running clock , then If it succeeds I'll run the PTP Currently without the PTP , I see about 1 second delta between the 2 clocks.
>
> (Host and guest seem somehow synchronized, because QEMU stores host 
> time  into RTC.  The guest reads RTC on boot, but that has nothing to 
> with  kvmclock, and RTC's accuracy is poor.)
>
> I understand that - but I expect that later the 2 clocks will be in-sync.
>
>> My test is very simple - I send a packet from host to the VM, I set the host time (tx_time) in the packet.  When the guest receives the packet it reads its time (rx_time)  and calculate the delay as :
>> Delay = rx_time - tx_time
>> I use the clock_gettime(REALTIME) in the host to set tx-time and in the guest to read rx_time.
>> My questions :
>> 1. Assuming my HW support the paravirtualization clock requirements - 
>> (see below output of cpuinfo)  ,
>
> (kvmclock clock doesn't have any requirements other than presence of 
> TSC, which is why it's the default.  The guest can have requirements 
> that aren't met on some HW, though.)
>
> I meant that my HW supports the constant TSC - hence I can rely on the TSC.
>
>> In Theory  - Is it possible to achieve 10s ns accuracy between VM clock and the host clock ?
>
> It is.
>
> How ? this is the only question
>
>
> (kvmclock on new CPUs has same drift/resolution/jitter/... as TSC.
>  Reading kvmclock is slower that just doing rdtsc, but likely still 
> within tens of nanoseconds.)
>
>> or I'm too naïve and have  to abandon the idea to run this timing sensitive application on a VM, and instead run it in  Linux  container for example?
>
> That depends on the reason behind synchronizing clocks, because VM can provide same precision as the host.  Running in containers is almost the same as running on the host, so you might prefer their trade-offs.
>
>> 2. I understand that in the  kvm-clock process, the kvm writes 
>> (whenever it enters the VM)
>
> (KVM doesn't update on every entry if you machine has invariable TSC.)
>
> I have constant TSC
>> its system_time and the VM_TSC @ current time to the pvclock page , then the guest OS can calculate its current time by:
>
> KVM doesn't write its (= host's) system_time.
> KVM writes *guest's* system_time.  Guest's system_time at VM_TSC.
>
> (system_time is 0 when the VM starts.  sytem_time can store ~584 years 
> worth of nanoseconds, but using an arbitrary offset makes everything 
> simpler.  This part of kvmclock is pretty confusing, so system_time is 
> likely the source of misunderstanding.)
>
> Have you read that kvmclock does synchronization with host time somewhere?
>
> Yes - see - in  vcpu_enter_guest() there is a call to   kvm_guest_time_update()
> Which update  the pvclock paget   for the guest - see that it updates the system_time  with the host system-time
>
> struct pvclock_vcpu_time_info {
>  26         u32   version;
>  27         u32   pad0;
>  28         u64   tsc_timestamp;
>  29         u64   system_time;
>  30         u32   tsc_to_system_mul;
>  31         s8    tsc_shift;
>  32         u8    flags;
>  33         u8    pad[2];
>  34 } __attribute__((__packed__)); /* 32 bytes */
>
> Thanks.
>
>> Current-time = system_time + multiplier (RDTSC() -VM_TSC)   (system_time and VM_TSC as set by the kvmas set by the kvm))
>> I understand that there is no VM-exit when the VM calls RDTSC().
>
> Yes.
>
>> - is that description correct ?
>
> Partly.
>
>> I understand that this is supported by the guest OS and this should be transparent to my application, correct ?
>
> Yes.
>
>> My guest and host are Fedora 22.
>>
>> 3. Other idea how to achieve this accuracy ?
>
> I hope that PTP is enough, because I can't recommend anything that can be done without introducing a new paravirtual device ... an example of potential one-time synchronization with higher accuracy than PTP:
>
> A guest asks a host what time it is by issuing a hypercall.
> The host replies with a
>   (kvmclock timestamp, nanoseconds since some standard time) pair.
>
> It's going to be more complex if you want more features.



--
Regards,
Wanpeng Li
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KVM-Clock

[Radim Krčmář]

Please read https://en.wikipedia.org/wiki/Posting_style or related
documents on email quoting.

2016-02-23 07:11+0000, Avi Cohen:
>> 2016-02-21 16:57+0000, Avi Cohen:
>>> Hello,
>>> 
>>> Last week I've sent a mail regarding the kvm-clock accuracy.
>>> Now I try to draw-up my question again, any answer/partial/hint  is 
>>> greatly appreciated
>>> 
>>> Our application is running in  a Tenant's Virtual Machine in a data-centre.
>>> We have some OAM functions running in  the VMs.
>>> One OAM function is to measure one-way delay between VMa and VMb. 
>>> One way delay measurement requires that all machines should be synchronized to a common central clock.
>>> Accuracy requirement is in order of 10s nano-seconds, hence only the 1588v2/PTP is suitable here.
>>> Since we cannot use HW timestamping in a virtual machine (we cannot force using SR-IOV), I thought to run PTP on the physical machines and to sync the VMs to the host by the kvm-clock.
>> 
>> kvmclock doesn't do synchronization with host clock or UTC.
>> kvmclock bases on host's notion of *passed* time.
>> kvmclock allows the guest to measure a flow of time.
>> 
>> It is another layer's job to translate kvmclock result into a timestamp that can be compared.  kvmclock was designed like that, because KVM wants to make a guest independent on hosts.
> 
> I see the system time written by KVM  whenever the VM is entered  - in  kvm_guest_time_update()
> 
> How can  the guest  (or  another layer's job ) - translate kvmclock result into a timestamp that can be compared ?

If the layer has a good idea about the number of ticks the second takes
on both clocks, then the layer is synchronized with a
  (time on one clock, time on second clock) pair.
You get a timestamp comparable with the other clock by providing a
timestamp from one of synchronized clocks.

The hard part is getting the pair.  (A duration of the second is pretty
stable and the pair needs to be renewed on a change anyway.)

>> > In Theory  - Is it possible to achieve 10s ns accuracy between VM clock and the host clock ? 
>> 
>> It is.
> 
> How ? this is the only question

Look at last paragraph of my previous email.
(That one has 1 ns accuracy if KVM is using the masterclock.)

Note that it is impossible to confirm the accuracy with your measurement
because delay between sending and receiving a packet is normally *far*
longer that 100 ns.

>> > its system_time and the VM_TSC @ current time to the pvclock page , then the guest OS can calculate its current time by:
>> 
>> KVM doesn't write its (= host's) system_time.
>> KVM writes *guest's* system_time.  Guest's system_time at VM_TSC.
>> 
>> (system_time is 0 when the VM starts.  sytem_time can store ~584 years  worth of nanoseconds, but using an arbitrary offset makes everything  simpler.  This part of kvmclock is pretty confusing, so system_time is  likely the source of misunderstanding.)
>> 
>> Have you read that kvmclock does synchronization with host time somewhere?
>> 
> Yes - see - in  vcpu_enter_guest() there is a call to   kvm_guest_time_update()
> 
> Which update  the pvclock paget   for the guest - see that it updates the system_time  with the host system-time 

I think you misunderstand the code, see kvmclock_offset.

system_time is incremented by a second if the host thinks that a second
has passed since last update, but that doesn't allow the guest to tell
host time.

RE: KVM-Clock

[Avi Cohen]

> Look at last paragraph of my previous email.
> (That one has 1 ns accuracy if KVM is using the masterclock.)

How to make KVM using the masterclock ? - I don't see any masterclock in traces

Re: KVM-Clock

[Radim Krčmář]

2016-02-24 10:32+0000, Avi Cohen:
>> Look at last paragraph of my previous email.
>> (That one has 1 ns accuracy if KVM is using the masterclock.)
> 
> How to make KVM using the masterclock ? - I don't see any masterclock in traces

KVM should use masterclock whenever it can.  Host dmesg can contain
warning about unstable TSC if masterclock is not being used for good
reasons (grep for "TSC unstable" and "unstable TSC").

Relevant tracepoints are (from arch/x86/kvm/trace.h)
  kvm:kvm_update_master_clock
  kvm:kvm_track_tsc
  kvm:kvm_pvclock_update
  kvm:write_tsc_offset

The host is using TSC clocksource, TSC wasn't marked as unstable, guest
didn't screw with MSR_IA32_TSC or MSR_IA32_TSC_ADJUST, and masterclock
still isn't used?

RE: KVM-Clock

[Avi Cohen]

>KVM should use masterclock whenever it can.  Host dmesg can contain warning about unstable TSC if masterclock is not being used for good reasons (grep for "TSC unstable" and "unstable TSC").

 >Relevant tracepoints are (from arch/x86/kvm/trace.h)
 >kvm:kvm_update_master_clock
 >  kvm:kvm_track_tsc
 > kvm:kvm_pvclock_update
 > kvm:write_tsc_offset

>The host is using TSC clocksource, TSC wasn't marked as unstable, guest didn't screw with MSR_IA32_TSC or MSR_IA32_TSC_ADJUST, and masterclock still isn't used?
I don't see these error messages - so I guess that it use the masterclock , although I don't know how to explicitly verify this.


--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: KVM-Clock

[Marcelo Tosatti]

On Sun, Feb 21, 2016 at 04:57:55PM +0000, Avi Cohen wrote:
> 
> Hello,
> Last week I've sent a mail regarding the kvm-clock accuracy.
> Now I try to draw-up my question again, any answer/partial/hint  is greatly appreciated
> 
> Our application is running in  a Tenant's Virtual Machine in a data-centre.
> We have some OAM functions running in  the VMs.
> One OAM function is to measure one-way delay between VMa and VMb. 
> One way delay measurement requires that all machines should be synchronized to a common central clock.
> Accuracy requirement is in order of 10s nano-seconds, hence only the 1588v2/PTP is suitable here.
> Since we cannot use HW timestamping in a virtual machine (we cannot force using SR-IOV), I thought to run PTP on the physical machines and to sync the VMs to the host by the kvm-clock.
> But now I see that the clock in the VM is far away from the host ( ~ Hundreds of micro-second) , and this before I even run the PTP in the host...
> My test is very simple - I send a packet from host to the VM, I set the host time (tx_time) in the packet.  When the guest receives the packet it reads its time (rx_time)  and calculate the delay as :
> Delay = rx_time - tx_time
> I use the clock_gettime(REALTIME) in the host to set tx-time and in the guest to read rx_time. 
> My questions :
> 1. Assuming my HW support the paravirtualization clock requirements - (see below output of cpuinfo)  , In Theory  - Is it possible to achieve 10s ns accuracy between VM clock and the host clock ? 
> or I'm too naïve and have  to abandon the idea to run this timing sensitive application on a VM, and instead run it in  Linux  container for example?  

You need realtime KVM to run a time sensitive application on a VM.  What 
are your requirements?

RHEL-RT-KVM achieves 20us maximum latency (and that is effectively the
best clock resolution you can give to users, irrespective of
synchronization between guest clock and host clock).

RE: KVM-Clock

[Avi Cohen]

>> 
>> Hello,
>> Last week I've sent a mail regarding the kvm-clock accuracy.
>> Now I try to draw-up my question again, any answer/partial/hint  is 
>> greatly appreciated
>> 
>> Our application is running in  a Tenant's Virtual Machine in a data-centre.
>> We have some OAM functions running in  the VMs.
>> One OAM function is to measure one-way delay between VMa and VMb. 
>> One way delay measurement requires that all machines should be synchronized to a common central clock.
>> Accuracy requirement is in order of 10s nano-seconds, hence only the 1588v2/PTP is suitable here.
>> Since we cannot use HW timestamping in a virtual machine (we cannot force using SR-IOV), I thought to run PTP on the physical machines and to sync the VMs to the host by the kvm-clock.
>> But now I see that the clock in the VM is far away from the host ( ~ Hundreds of micro-second) , and this before I even run the PTP in the host...
>> My test is very simple - I send a packet from host to the VM, I set the host time (tx_time) in the packet.  When the guest receives the packet it reads its time (rx_time)  and calculate the delay as :
>> Delay = rx_time - tx_time
>> I use the clock_gettime(REALTIME) in the host to set tx-time and in the guest to read rx_time. 
>> My questions :
>> 1. Assuming my HW support the paravirtualization clock requirements - (see below output of cpuinfo)  , In Theory  - Is it possible to achieve 10s ns accuracy between VM clock and the host clock ? 
>> or I'm too naïve and have  to abandon the idea to run this timing sensitive application on a VM, and instead run it in  Linux  container for example?  

>You need realtime KVM to run a time sensitive application on a VM.  What are your requirements?

>RHEL-RT-KVM achieves 20us maximum latency (and that is effectively the best clock resolution you can give to users, irrespective of synchronization between guest clock and host clock).

The requirement is that the guest VM will be synced to the host clock - I'm not talking on real-time in the meaning of speed, but I need that the guest clock will measure the exact delay between packet transmission from the host to the packet received in the guest. My problem now is that sometimes when I receive the packet in the guest I calculate a negative delay because my clock in the guest is in delay relative to the host clock.
I expect to get a positive (jittered) delay in the order of 10s of us.  
My test is very simple - I send a packet from host to the VM, I set the host time (tx_time) in the packet.  When the guest receives the packet it reads its time (rx_time)  and calculate the delay as :
Delay = rx_time - tx_time
I use the clock_gettime(REALTIME) in the host to set tx-time and in the guest to read rx_time.

Re: KVM-Clock

[Marcelo Tosatti]

On Wed, Feb 24, 2016 at 08:15:46AM +0000, Avi Cohen wrote:
> >> 
> >> Hello,
> >> Last week I've sent a mail regarding the kvm-clock accuracy.
> >> Now I try to draw-up my question again, any answer/partial/hint  is 
> >> greatly appreciated
> >> 
> >> Our application is running in  a Tenant's Virtual Machine in a data-centre.
> >> We have some OAM functions running in  the VMs.
> >> One OAM function is to measure one-way delay between VMa and VMb. 
> >> One way delay measurement requires that all machines should be synchronized to a common central clock.
> >> Accuracy requirement is in order of 10s nano-seconds, hence only the 1588v2/PTP is suitable here.
> >> Since we cannot use HW timestamping in a virtual machine (we cannot force using SR-IOV), I thought to run PTP on the physical machines and to sync the VMs to the host by the kvm-clock.
> >> But now I see that the clock in the VM is far away from the host ( ~ Hundreds of micro-second) , and this before I even run the PTP in the host...
> >> My test is very simple - I send a packet from host to the VM, I set the host time (tx_time) in the packet.  When the guest receives the packet it reads its time (rx_time)  and calculate the delay as :
> >> Delay = rx_time - tx_time
> >> I use the clock_gettime(REALTIME) in the host to set tx-time and in the guest to read rx_time. 
> >> My questions :
> >> 1. Assuming my HW support the paravirtualization clock requirements - (see below output of cpuinfo)  , In Theory  - Is it possible to achieve 10s ns accuracy between VM clock and the host clock ? 
> >> or I'm too naïve and have  to abandon the idea to run this timing sensitive application on a VM, and instead run it in  Linux  container for example?  
> 
> >You need realtime KVM to run a time sensitive application on a VM.  What are your requirements?
> 
> >RHEL-RT-KVM achieves 20us maximum latency (and that is effectively the best clock resolution you can give to users, irrespective of synchronization between guest clock and host clock).
> 
> The requirement is that the guest VM will be synced to the host clock - I'm not talking on real-time in the meaning of speed, but I need that the guest clock will measure the exact delay between packet transmission from the host to the packet received in the guest. 

This makes no sense - if you want a "reliable clock synchronized up to X
microseconds", then you have to take into account any possible delay
incurred by scheduling (as scheduling determines how reliable a clock
read can be) into the clock reliability calculation.

> My problem now is that sometimes when I receive the packet in the guest I calculate a negative delay because my clock in the guest is in delay relative to the host clock.

Use the guest TSC and host TSC values (rdtsc in both).

Then subtract from host TSC timestamps the tsc_offset value. 

You'll get a "perfect" synchronized RDTSC clock. 

To find out the tsc_offset value, enable kvm_write_tsc_offset tracepoint
in the host (before the particular guest boots).

> I expect to get a positive (jittered) delay in the order of 10s of us.  
> My test is very simple - I send a packet from host to the VM, I set the host time (tx_time) in the packet.  When the guest receives the packet it reads its time (rx_time)  and calculate the delay as :
> Delay = rx_time - tx_time
> I use the clock_gettime(REALTIME) in the host to set tx-time and in the guest to read rx_time.

RE: KVM-Clock

[Avi Cohen]

>> The requirement is that the guest VM will be synced to the host clock - I'm not talking on real-time in the meaning of speed, but I need that the guest clock will measure the exact delay between packet transmission from the host to the packet received in the guest. 

>This makes no sense - if you want a "reliable clock synchronized up to X microseconds", then you have to take into account any possible delay incurred by scheduling (as scheduling determines how reliable a clock read can be) into the clock reliability calculation.

I understand the scheduling latency , anyway my clock is totally not in sync. Actually - I see about 800ms delay in the guest clock relative to the host clock , also call clock_settime (any_time)  in the host and did not see any impact on the guest host.
Also I've looked in the code in kvm_guest_time_update and I think that the clock is not updated because  use_master_clock  is false - how can I enable master_clock in KVM ?


>> My problem now is that sometimes when I receive the packet in the guest I calculate a negative delay because my clock in the guest is in delay relative to the host clock.

>Use the guest TSC and host TSC values (rdtsc in both).

>Then subtract from host TSC timestamps the tsc_offset value. 

>You'll get a "perfect" synchronized RDTSC clock. 

>To find out the tsc_offset value, enable kvm_write_tsc_offset tracepoint in the host (before the particular guest boots).

Do u mean that in the host/guest  I will not call clock_gettime   but:

In host :  tx_time =  tsc_to_ns   *  (rdtsc () - tsc_offset)
In guest : rx_time = tsc_to_ns * rdtsc()
                   Delay = rx_time - tx_time 

How to  enable kvm_write_tsc_offset ?

Re: KVM-Clock

[Marcelo Tosatti]

On Wed, Feb 24, 2016 at 11:51:30AM +0000, Avi Cohen wrote:
> >> The requirement is that the guest VM will be synced to the host clock - I'm not talking on real-time in the meaning of speed, but I need that the guest clock will measure the exact delay between packet transmission from the host to the packet received in the guest. 
> 
> >This makes no sense - if you want a "reliable clock synchronized up to X microseconds", then you have to take into account any possible delay incurred by scheduling (as scheduling determines how reliable a clock read can be) into the clock reliability calculation.
> 
> I understand the scheduling latency , anyway my clock is totally not in sync. Actually - I see about 800ms delay in the guest clock relative to the host clock , also call clock_settime (any_time)  in the host and did not see any impact on the guest host.
> Also I've looked in the code in kvm_guest_time_update and I think that the clock is not updated because  use_master_clock  is false - how can I enable master_clock in KVM ?
> 
> 
> >> My problem now is that sometimes when I receive the packet in the guest I calculate a negative delay because my clock in the guest is in delay relative to the host clock.
> 
> >Use the guest TSC and host TSC values (rdtsc in both).
> 
> >Then subtract from host TSC timestamps the tsc_offset value. 
> 
> >You'll get a "perfect" synchronized RDTSC clock. 
> 
> >To find out the tsc_offset value, enable kvm_write_tsc_offset tracepoint in the host (before the particular guest boots).
> 
> Do u mean that in the host/guest  I will not call clock_gettime   but:
> 
> In host :  tx_time =  tsc_to_ns   *  (rdtsc () - tsc_offset)

Subtracting tsc_offset later is easier.

> In guest : rx_time = tsc_to_ns * rdtsc()
>                    Delay = rx_time - tx_time 
> 
> How to  enable kvm_write_tsc_offset ?

# echo kvm_write_tsc_offset > /sys/kernel/debug/tracing/set_event

Then boot guest

Then cat /sys/kernel/debug/tracing/trace

RE: KVM-Clock

[Avi Cohen]

>Subtracting tsc_offset later is easier.

>> In guest : rx_time = tsc_to_ns * rdtsc()
>>                   Delay = rx_time - tx_time 
>> 
>> How to  enable kvm_write_tsc_offset ?

># echo kvm_write_tsc_offset > /sys/kernel/debug/tracing/set_event

>Then boot guest

>Then cat /sys/kernel/debug/tracing/trace

OK - I did it and it is indeed accurate. But this will not solve my problem, because I need the host realtime clock in my guest and not the host-TSC
In my application VMa in  HOSTa is sending a 1-way delay measurement packet to VMb in HOSTb.
I'm perfectly synching the 2 hosts with PTP.
I need that the  2 guests's clocks will be synched  to their hosts  i.e. VMa/VMb  is sync to HOSTa/HOSTb accordingly   

Re: KVM-Clock

[Marcelo Tosatti]

On Wed, Feb 24, 2016 at 03:27:45PM +0000, Avi Cohen wrote:
> 
> 
> >Subtracting tsc_offset later is easier.
> 
> >> In guest : rx_time = tsc_to_ns * rdtsc()
> >>                   Delay = rx_time - tx_time 
> >> 
> >> How to  enable kvm_write_tsc_offset ?
> 
> ># echo kvm_write_tsc_offset > /sys/kernel/debug/tracing/set_event
> 
> >Then boot guest
> 
> >Then cat /sys/kernel/debug/tracing/trace
> 
> OK - I did it and it is indeed accurate. But this will not solve my problem, because I need the host realtime clock in my guest and not the host-TSC

Configure NTP to synchronize from host NTPd server. How close the host
and guest clocks can be synchronized depends on how reliable is the
transmission of data between guest and host.

Then use the guest/host TSCs to measure how accurate the clocks are.

> In my application VMa in  HOSTa is sending a 1-way delay measurement packet to VMb in HOSTb.
> I'm perfectly synching the 2 hosts with PTP.
> I need that the  2 guests's clocks will be synched  to their hosts  i.e. VMa/VMb  is sync to HOSTa/HOSTb accordingly   

Chronyd in the guest should work, syncing from NTPd server in the host 
(haven't configured that setup myself, about to do so soon).