Guest Time Question

Guest Time Question

[Kay Hayen]

Hello,

I am currently evaluating kvm for using it for an application where correct 
time is absolutely critical. We therefore normally use NTP on these machines 
to synchronize them with GPS time. For a virtual machine in my eyes that 
means, the host should be synchronized and the guests just use that time.

My test setup is on Kubuntu 7.04 with a minimal Kubuntu 7.04 guest. I am using 
its "linux-virtual" kernel package, so I believe I should be correctly 
paravirtualized.

Now what I see with respect to system time, is that the time is lagging behind 
in the guest. Abount 1 minute after 20 minutes, and 1 second only a few 
seconds after a successful ntpdate to the hosting machine. My guess, that 
shows how little the host is used otherwise. 

So in FAQ and Wiki I didn't find how to make the guest use host time. Is that 
possible at all? For VMWare it is said that clock=pit would help, but that 
seemed to be no change.

Can you please point me to what to do? 

Best regards,
Kay Hayen

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: Guest Time Question

[Izik Eidus]

Kay Hayen wrote:
>
>
> So in FAQ and Wiki I didn't find how to make the guest use host time. Is that 
> possible at all? For VMWare it is said that clock=pit would help, but that 
> seemed to be no change.
>   
ok, as for now what you can do is run it with: "-tdf -no-kvm-irqchip 
-no-acpi"

thanks.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: Guest Time Question

[Kay Hayen]

Hello,

tanks you for the reply:

Am Freitag 05 Oktober 2007 08:35:44 schrieb Izik Eidus:
> Kay Hayen wrote:
> > So in FAQ and Wiki I didn't find how to make the guest use host time. Is
> > that possible at all? For VMWare it is said that clock=pit would help,
> > but that seemed to be no change.
>
> ok, as for now what you can do is run it with: "-tdf -no-kvm-irqchip
> -no-acpi"

As I discovered that my version of kvm 28 doesn't have these options, I have 
attempted to compile release 45 on my Ubuntu 7.04, which failed, because 
obviously the patch was not applied. So I downloaded the 2.6.22, the patch 
2.6.23-rc8.

The patches seemed to apply fine, I copied the config from Ubuntu and ran make 
oldconfig, then using make-kpkg buildpackage I ended up with packages that 
refer to rc6. Probably something wrong on my side, with -rc9 that I tried 
first, hoping patches may already be merged, but they are not. I didn't have 
that issue of wrong version number there though. My guess ist that Linus 
still forgets to update kernel versions? :)

Well, but aside from the kernel problem, I was using: 

./configure --prefix=/opt/kvm --with-patched-kernel --kerneldir=/usr/src/linux-headers-2.6.23-rc6/
make 
make install

and the compile worked now. But when I look in /opt/bin I only find qemu 
binaries, but not kvm. Is my expectation that the kvm script at least should 
also be found there wrong?

And given that the kernel wouldn't boot either (I guess I will have to wait 
for you to rebase on rc9, a lot of your patch got rejected there), I also 
have the question if the same effect possibly could be also achieved with the 
outdated version already in Ubuntu 7.04 that doesn't have "-no-kvm-irqchip" 
yet.

Best regards,
Kay Hayen

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: Guest Time Question

[Izik Eidus]

Kay Hayen wrote:
> Hello,
>
> tanks you for the reply:
>
> Am Freitag 05 Oktober 2007 08:35:44 schrieb Izik Eidus:
>   
>
> And given that the kernel wouldn't boot either (I guess I will have to wait 
> for you to rebase on rc9, a lot of your patch got rejected there), I also 
> have the question if the same effect possibly could be also achieved with the 
> outdated version already in Ubuntu 7.04 that doesn't have "-no-kvm-irqchip" 
> yet.
>
>   
yes, if you dont have -no-kvm-irqchip option you dont have to use it.

dont d/l the kernel patchs.
d/l kvm-45.tar.gz from sourceforge and use it ( without --patched-kernel)

then load the modules from kvm-45/kernel/ dir

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: Guest Time Question

[Dong, Eddie]

kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org wrote:
> Hello,
> 
> I am currently evaluating kvm for using it for an application
> where correct
> time is absolutely critical. We therefore normally use NTP on
> these machines
> to synchronize them with GPS time. For a virtual machine in my
> eyes that
> means, the host should be synchronized and the guests just use
> that time.
> 
> My test setup is on Kubuntu 7.04 with a minimal Kubuntu 7.04
> guest. I am using
> its "linux-virtual" kernel package, so I believe I should be
> correctly paravirtualized. 
> 
> Now what I see with respect to system time, is that the time
> is lagging behind
> in the guest. Abount 1 minute after 20 minutes, and 1 second
> only a few
> seconds after a successful ntpdate to the hosting machine. My
> guess, that
> shows how little the host is used otherwise.
> 
> So in FAQ and Wiki I didn't find how to make the guest use
> host time. Is that
> possible at all? For VMWare it is said that clock=pit would
> help, but that
> seemed to be no change.
> 
> Can you please point me to what to do?
> 
> Best regards,
> Kay Hayen
> 
If you can get latest KVM up + clock=pit, I bet you will get a much much
accurate guest time. If you still see > 0.1% shift, please file a bug.
thx,eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: Guest Time Question

[Kay Hayen]

Hello Dong and others,

thanks for the replies. I was trying to get KVM up with latest kernel, but 
didn't immediately succeed. I have it built with the instructions from a 
previous reply and it seemed OK, but I haven't had a chance to try it out 
yet.

As for testing the time shift: To us the virtual machine would be like another 
hardware baseline to verify. So we will have to do that to test the time 
correctness. What to do? 

Sending time stamped broadcast or multicast messages may well do the trick, I 
have yet to look on how the network stack or the guest and host work together 
in paravirtualization with KVM.

Also I would not be sure what a 0.1% shift would be. From what do you mean the 
percentage? Or is that a typo and you would mean 0.1 ms ?

Best regards,
Kay Hayen

> > So in FAQ and Wiki I didn't find how to make the guest use
> > host time. Is that
> > possible at all? For VMWare it is said that clock=pit would
> > help, but that
> > seemed to be no change.
> >
> > Can you please point me to what to do?
> >
> > Best regards,
> > Kay Hayen
>
> If you can get latest KVM up + clock=pit, I bet you will get a much much
> accurate guest time. If you still see > 0.1% shift, please file a bug.
> thx,eddie



-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: Guest Time Question

[Dong, Eddie]

Kay Hayen wrote:
> Hello Dong and others,
> 
> thanks for the replies. I was trying to get KVM up with latest
> kernel, but didn't immediately succeed. I have it built with the
> instructions from a previous reply and it seemed OK, but I haven't
> had a chance to try it out yet.
> 
> As for testing the time shift: To us the virtual machine would be
> like another hardware baseline to verify. So we will have to do that
> to test the time correctness. What to do?
> 
> Sending time stamped broadcast or multicast messages may well do the
> trick, I have yet to look on how the network stack or the guest and
> host work together in paravirtualization with KVM.
> 
> Also I would not be sure what a 0.1% shift would be. From what do you
> mean the percentage? Or is that a typo and you would mean 0.1 ms ?
> 
> Best regards,
> Kay Hayen
> 
>>> So in FAQ and Wiki I didn't find how to make the guest use host
>>> time. Is that possible at all? For VMWare it is said that clock=pit
>>> would help, but that seemed to be no change.
>>> 
>>> Can you please point me to what to do?
>>> 
>>> Best regards,
>>> Kay Hayen
>> 
>> If you can get latest KVM up + clock=pit, I bet you will get a much
>> much accurate guest time. If you still see > 0.1% shift, please file
>> a bug. thx,eddie

I mean wall clock shift will be <0.1%. For network packet, it is
different with
that in native since a VCPU may be descheduled. But it is same even in 
native when you compare a fast box vs. slow box.
thx,eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: Guest Time Question

[Kay Hayen]

Hi,

thanks for your reply:

Am Dienstag 09 Oktober 2007 01:55:16 schrieb Dong, Eddie:
> If you can get latest KVM up + clock=pit, I bet you will get a much much
> accurate guest time. If you still see > 0.1% shift, please file a bug.
> thx,eddie

I have just installed KVM 46 into my Ubuntu 7.04 kernel (2.6.22 based) and it 
seems to work with the old KVM 22 user space. 

Strangely I still get no "kvm" binary installed to my --prefix/bin, only qemu 
stuff, so I tried to run it from the directory, but grub won't boot for some 
reason.

That 0.1% kills me though. do you mean, that every 1 second, 1 will be 1ms 
off? And for every second thereafter, leaving me with 1 second drift after 
only 1000 seconds?

With the KVM 22 userspace and KVM 46 kernelspace I am 20 seconds off, after 
uptime of 33 minutes. That uptime is wrong by 20 seconds I take, but it's 
around 0.1% (relatively exactly even), so it's not what you consider a bug?

I am looking for an absolute value for time difference between guest and host 
that is supposed to be small over longer periods. Is that not achievable for 
a paravirtual guest?! I really need gettimeofday() calls to be near exactly 
the same value for applications in guest and host even after months. 

To give a bit background: We are maintainers of an air traffic management 
system that receives UTC dated input from the outside, and makes estimates 
about flight positions based on that information. In order to make correct 
estimates about current position, the current time and the observation time 
must be compared, and errors in the current time easily will lead to an error 
in the range of miles.

Best regards,
Kay Hayen

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: Guest Time Question

[Dong, Eddie]

>
>That 0.1% kills me though. do you mean, that every 1 second, 1 
>will be 1ms 
>off? And for every second thereafter, leaving me with 1 second 
>drift after 
>only 1000 seconds?
>

Oh, that is not my intension :-(
I just put a rought estimation here. In old Xen
time, I know there are ~10-30 seconds shift after ~10 hours.

>With the KVM 22 userspace and KVM 46 kernelspace I am 20 
>seconds off, after 
>uptime of 33 minutes. That uptime is wrong by 20 seconds I 
>take, but it's 
>around 0.1% (relatively exactly even), so it's not what you 
>consider a bug?

If you see same thing with KVM-46 user space, I think we
should file a bug to track the issue.
But I don't remeber how many changes happen since KVM-22
in user level.

>
>I am looking for an absolute value for time difference between 
>guest and host 
>that is supposed to be small over longer periods. Is that not 
>achievable for 
>a paravirtual guest?! I really need gettimeofday() calls to be 
>near exactly 
>the same value for applications in guest and host even after months. 

Except adding a hypercall to get host time (i.e. pv timer), or 
using network time, I think cetain amount of shift will be always there.

>
>To give a bit background: We are maintainers of an air traffic 
>management 
>system that receives UTC dated input from the outside, and 
>makes estimates 
>about flight positions based on that information. In order to 
>make correct 
>estimates about current position, the current time and the 
>observation time 
>must be compared, and errors in the current time easily will 
>lead to an error 
>in the range of miles.
>
thx,eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: Guest Time Question

[Kay Hayen]

Hello Eddie,

Am Montag 15 Oktober 2007 06:13:44 schrieb Dong, Eddie:
> >That 0.1% kills me though. do you mean, that every 1 second, 1
> >will be 1ms
> >off? And for every second thereafter, leaving me with 1 second
> >drift after
> >only 1000 seconds?
>
> Oh, that is not my intension :-(
> I just put a rought estimation here. In old Xen
> time, I know there are ~10-30 seconds shift after ~10 hours.

That's about the same, isn't it? That's 30 seconds shift after 36000 seconds 
and really a lot.

> >I am looking for an absolute value for time difference between
> >guest and host
> >that is supposed to be small over longer periods. Is that not
> >achievable for
> >a paravirtual guest?! I really need gettimeofday() calls to be
> >near exactly
> >the same value for applications in guest and host even after months.
>
> Except adding a hypercall to get host time (i.e. pv timer), or
> using network time, I think cetain amount of shift will be always there.

Do you consider that NTP between guest and host is acceptable? I found this 
btw: http://support.ntp.org/bin/view/Support/KnownOsIssues

Quote:  "It appears that Xen just passes time-related system calls to the 
underlying master domain, and does not require any additional changes to 
support time sync into the guest domains."

When I today told my manager (technical background) about this, it was met 
with utter surprise, because "that should be really simple". Incidentally, I 
recently claimed in a meeting that with modern Linux and paravirtualization, 
this problem should no longer exist.

Is this a design problem for KVM (the hypercall missing yet) or just something 
that the KVM community didn't yet have/find the time to get around to?

Best regards,
Kay

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

Re: Guest Time Question

[Dong, Eddie]

Kay Hayen wrote:
> Hello Eddie,
> 
> Am Montag 15 Oktober 2007 06:13:44 schrieb Dong, Eddie:
>>> That 0.1% kills me though. do you mean, that every 1 second, 1
>>> will be 1ms
>>> off? And for every second thereafter, leaving me with 1 second
>>> drift after only 1000 seconds?
>> 
>> Oh, that is not my intension :-(
>> I just put a rought estimation here. In old Xen
>> time, I know there are ~10-30 seconds shift after ~10 hours.
> 
> That's about the same, isn't it? That's 30 seconds shift after 36000
> seconds and really a lot.

Yes, but you can simply solve it by sync guest time to network time
per minutes for example. 0.1% shift then means at most 60ms.
If 60ms is still too big, you can sync per second or per 10 second.

Writting a tool in this way is pretty simple. BTW Vmware also
rely on this kind of syncing.

> 
>>> I am looking for an absolute value for time difference between
>>> guest and host that is supposed to be small over longer periods. Is
>>> that not achievable for a paravirtual guest?! I really need
>>> gettimeofday() calls to be 
>>> near exactly
>>> the same value for applications in guest and host even after months.
>> 
>> Except adding a hypercall to get host time (i.e. pv timer), or
>> using network time, I think cetain amount of shift will be always
>> there. 
> 
> Do you consider that NTP between guest and host is acceptable?
> I found this
> btw: http://support.ntp.org/bin/view/Support/KnownOsIssues
> 
> Quote:  "It appears that Xen just passes time-related system
> calls to the
> underlying master domain, and does not require any additional
> changes to
> support time sync into the guest domains."

Yes, but this is for pv domain only. Same issue exist for HVM
domain in Xen. KVM is supporting HVM domain only using 
hardware features. That is why you see something different.

Linux 2.6.24 is coming out with pvtimer inside which will solve this
issue
eventually but you need to wait some time :-)
BTW, using application to sync guest time is not a big issue IMO, like
VMWare did.

> 
> When I today told my manager (technical background) about
> this, it was met
> with utter surprise, because "that should be really simple".
> Incidentally, I recently claimed in a meeting that with modern Linux
> and paravirtualization, this problem should no longer exist.

Yes, you can add this kind of hypercall. I am not sure if KVM
will add this APIs but certainly can. If the usage model is strong
then we may provide APIs for host time etc.

But this only works when you can modify your application
 (to do hypercall), so the best solution is still to 
let application sync the time periodically.

thx,eddie

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/