Clock off in guest

Clock off in guest

[Koen Vermeer]

Hi,

I'm running an Linux AMD64 guest on an AMD64 host. The host is running a
2.6.23 kernel (self compiled), the guest is running a stock
linux-image-2.6.22-3-amd64 Debian kernel.

My problem is that the clock on the guest is off (slow), while the clock
on the host seems to be OK. When doing 'time sleep 10' on the guest, it
takes about 16 'real' seconds for it to finish.

I can change the host kernel, the guest kernel or fiddle around with kvm
command line options, but I would like some guidance where to start.
Which option has the largest probability of solving my problem? Let me
know if you need some dmesg output or some other log data; I didn't want
to flood the list with random logs of both the host and the guest.

Best,
Koen



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Clock off in guest

[Dan Kenigsberg]

On Wed, Feb 13, 2008 at 10:41:44AM +0100, Koen Vermeer wrote:
> Hi,
> 
> I'm running an Linux AMD64 guest on an AMD64 host. The host is running a
> 2.6.23 kernel (self compiled), the guest is running a stock
> linux-image-2.6.22-3-amd64 Debian kernel.
> 
> My problem is that the clock on the guest is off (slow), while the clock
> on the host seems to be OK. When doing 'time sleep 10' on the guest, it
> takes about 16 'real' seconds for it to finish.

What is the clock source of your guest? If it is the traditional PIT,
you describe a known problem in busy hosts. You may try to set
clocksource=tsc in the guest kerenl parameters.

Regards,

Dan.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Clock off in guest

[Uri Lublin]

[-- Attachment #1.1: Type: text/plain, Size: 1191 bytes --]

From: kvm-devel-bounces@lists.sourceforge.net on behalf of Dan Kenigsberg
Sent: Wed 13/02/2008 13:25
To: Koen Vermeer
Cc: kvm-devel@lists.sourceforge.net
Subject: Re: [kvm-devel] Clock off in guest
 
>On Wed, Feb 13, 2008 at 10:41:44AM +0100, Koen Vermeer wrote:
>> Hi,
>> 
>> I'm running an Linux AMD64 guest on an AMD64 host. The host is running a
>> 2.6.23 kernel (self compiled), the guest is running a stock
>> linux-image-2.6.22-3-amd64 Debian kernel.
>> 
>> My problem is that the clock on the guest is off (slow), while the clock
>> on the host seems to be OK. When doing 'time sleep 10' on the guest, it
>> takes about 16 'real' seconds for it to finish.
>
>What is the clock source of your guest? If it is the traditional PIT,
>you describe a known problem in busy hosts. You may try to set
>clocksource=tsc in the guest kerenl parameters.
>
>Regards,
>
>Dan.

Hi,

This would not work if you are using an old version of kvm ( with no in-kernel-apic )
I recommend upgrading to kvm-60 (or latest linux kernel).
Or as an alternative, probably not as good, sometimes (when the guest's clocksource is PIT) adding '-tdf' to the command line helps.

Uri.

[-- Attachment #1.2: Type: text/html, Size: 1774 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: Clock off in guest

[koen]

> This would not work if you are using an old version of kvm ( with no
> in-kernel-apic )
> I recommend upgrading to kvm-60 (or latest linux kernel).

Should I upgrade the guest kernel or the host kernel? My bet is the host
kernel, but the clocksource=tsc applies to the guest, so I'm not really
sure...

> Or as an alternative, probably not as good, sometimes (when the guest's
> clocksource is PIT) adding '-tdf' to the command line helps.

I cannot find this in man kvm or man qemu. Should I add this to the
command line that starts the guest?

Best,
Koen


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Clock off in guest

[koen]

Hi Dan,

> On Wed, Feb 13, 2008 at 10:41:44AM +0100, Koen Vermeer wrote:
>> I'm running an Linux AMD64 guest on an AMD64 host. The host is running a
>> 2.6.23 kernel (self compiled), the guest is running a stock
>> linux-image-2.6.22-3-amd64 Debian kernel.
>> My problem is that the clock on the guest is off (slow), while the clock
>> on the host seems to be OK. When doing 'time sleep 10' on the guest, it
>> takes about 16 'real' seconds for it to finish.
> What is the clock source of your guest? If it is the traditional PIT,
> you describe a known problem in busy hosts. You may try to set
> clocksource=tsc in the guest kerenl parameters.

Thanks for your reply. I tried looking for PIT in dmesg, which didn't give
me anything. Then I tried 'dmesg|grep -i tsc' and I got

Time: tsc clocksource has been installed.

I then ran 'dmesg|grep -i clock' and got

ACPI: PM-Timer IO Port: 0xb008
time.c: Detected 1803.751 MHz processor.
Calibrating delay using timer specific routine.. 14442.19 BogoMIPS
(lpj=28884398)
Using local APIC timer interrupts.
Detected 62.628 MHz APIC timer.
* Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
Time: tsc clocksource has been installed.
Real Time Clock Driver v1.12ac
PCI: Setting latency timer of device 0000:00:01.1 to 64
PCI: Setting latency timer of device 0000:00:01.2 to 64

The kernel command line is rather simple: root=/dev/hda1 ro quiet

Does that answer your question, or should I be looking somewhere else?

Thanks!

Koen


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Clock off in guest

[Dan Kenigsberg]

On Wed, Feb 13, 2008 at 02:00:38PM +0100, koen@vermeer.tv wrote:
> Hi Dan,
> 
> > On Wed, Feb 13, 2008 at 10:41:44AM +0100, Koen Vermeer wrote:
> >> I'm running an Linux AMD64 guest on an AMD64 host. The host is running a
> >> 2.6.23 kernel (self compiled), the guest is running a stock
> >> linux-image-2.6.22-3-amd64 Debian kernel.
> >> My problem is that the clock on the guest is off (slow), while the clock
> >> on the host seems to be OK. When doing 'time sleep 10' on the guest, it
> >> takes about 16 'real' seconds for it to finish.
> > What is the clock source of your guest? If it is the traditional PIT,
> > you describe a known problem in busy hosts. You may try to set
> > clocksource=tsc in the guest kerenl parameters.
> 
> Thanks for your reply. I tried looking for PIT in dmesg, which didn't give
> me anything. Then I tried 'dmesg|grep -i tsc' and I got
> 
> Time: tsc clocksource has been installed.

If this was done in the guest then, yes, your guest's clocksource is
tsc.

However, as Uri mentioned earlier, this is useful only with newer KVMs.
I assume that your host runs the kvm from 2.6.23 which is pretty old in
kvm timescale. Try downloading kvm-60, insmod it to your host and try
running your guest.

regards,
    Dan.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Clock off in guest

[koen]

> On Wed, Feb 13, 2008 at 02:00:38PM +0100, koen@vermeer.tv wrote:
>> > What is the clock source of your guest? If it is the traditional PIT,
>> > you describe a known problem in busy hosts. You may try to set
>> > clocksource=tsc in the guest kerenl parameters.
>> Thanks for your reply. I tried looking for PIT in dmesg, which didn't
>> give
>> me anything. Then I tried 'dmesg|grep -i tsc' and I got
>> Time: tsc clocksource has been installed.
> If this was done in the guest then, yes, your guest's clocksource is
> tsc.

Yes, this was done on the guest. The host shows

Marking TSC unstable due to TSCs unsynchronized
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31
hpet0: 3 32-bit timers, 25000000 Hz
Time: hpet clocksource has been installed.

> However, as Uri mentioned earlier, this is useful only with newer KVMs.
> I assume that your host runs the kvm from 2.6.23 which is pretty old in
> kvm timescale. Try downloading kvm-60, insmod it to your host and try
> running your guest.

I'll try building a kvm-60 module, run the guest with that and report the
results. I cannot do that immediately, though, because people are actually
using the guest system.

Best,
Koen


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Clock off in guest

[Koen Vermeer]

On Wed, 2008-02-13 at 17:55 +0200, Dan Kenigsberg wrote:
> On Wed, Feb 13, 2008 at 02:00:38PM +0100, koen@vermeer.tv wrote:
> However, as Uri mentioned earlier, this is useful only with newer KVMs.
> I assume that your host runs the kvm from 2.6.23 which is pretty old in
> kvm timescale. Try downloading kvm-60, insmod it to your host and try
> running your guest.

OK, I installed both version 60 of both kvm and kvm-source (Debian
Lenny), ran m-a a-i kvm and just to be sure I removed and reloaded the
kvm and kvm-amd modules. modinfo kvm and modinfo kvm-amd shows that
these are indeed version 60.

I then restarted the guest (which is exactly the same as before) and I
tried the 'sleep 10' test again. Same result: It takes about 17 or 18
seconds for the prompt to return again.

I assume that clocksource=tsc isn't useful as it is already using the
tsc. So, what else can I try? Any command line parameter I can add to
the kvm call? Kernel parameters in the guest? Update the guest OS to a
newer kernel (2.6.22 to 2.6.24)? Update the host OS to a newer kernel
(2.6.23 to 2.6.24)?

Thanks for the help!

Koen


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Clock off in guest

[Uri Lublin]

[-- Attachment #1.1: Type: text/plain, Size: 1384 bytes --]

From: koen@vermeer.tv [mailto:koen@vermeer.tv]
Sent: Wed 13/02/2008 14:52
To: Uri Lublin
Cc: kvm-devel@lists.sourceforge.net
Subject: RE: [kvm-devel] Clock off in guest
 
>> This would not work if you are using an old version of kvm ( with no
>> in-kernel-apic )
>> I recommend upgrading to kvm-60 (or latest linux kernel).
>
>Should I upgrade the guest kernel or the host kernel? My bet is the host
>kernel, but the clocksource=tsc applies to the guest, so I'm not really
>sure...

The host kernel or kvm.
If you choose to upgrade your host kernel (and kvm that comes with it), make sure 
you are using recent kvm-userspace too (e.g. kvm-60).

>> Or as an alternative, probably not as good, sometimes (when the guest's
>> clocksource is PIT) adding '-tdf' to the command line helps.
>
>I cannot find this in man kvm or man qemu.

I'm not sure about the man pages, but kvm/qemu's help says:
bash$ /usr/bin/kvm -h | grep tdf
-tdf            inject timer interrupts that got lost

>Should I add this to the command line that starts the guest?

Yes, try adding it to the command line that starts the guest (executable name may vary):
   /usr/bin/kvm [kvm-params] -tdf

Also tdf (time drift fix) only works when using PIT+PIC (no APIC) so sometimes it's helpful
to also add -no-acpi:
   /usr/bin/kvm [kvm-params] -tdf -no-acpi


>
>Best,
>Koen



[-- Attachment #1.2: Type: text/html, Size: 2099 bytes --]

[-- Attachment #2: Type: text/plain, Size: 228 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: Clock off in guest

[Dan Kenigsberg]

On Wed, Feb 13, 2008 at 06:07:36PM +0100, Koen Vermeer wrote:
> On Wed, 2008-02-13 at 17:55 +0200, Dan Kenigsberg wrote:
> > On Wed, Feb 13, 2008 at 02:00:38PM +0100, koen@vermeer.tv wrote:
> > However, as Uri mentioned earlier, this is useful only with newer KVMs.
> > I assume that your host runs the kvm from 2.6.23 which is pretty old in
> > kvm timescale. Try downloading kvm-60, insmod it to your host and try
> > running your guest.
> 
> OK, I installed both version 60 of both kvm and kvm-source (Debian
> Lenny), ran m-a a-i kvm and just to be sure I removed and reloaded the
> kvm and kvm-amd modules. modinfo kvm and modinfo kvm-amd shows that
> these are indeed version 60.
> 
> I then restarted the guest (which is exactly the same as before) and I
> tried the 'sleep 10' test again. Same result: It takes about 17 or 18
> seconds for the prompt to return again.
> 
> I assume that clocksource=tsc isn't useful as it is already using the
> tsc. So, what else can I try? Any command line parameter I can add to
> the kvm call? Kernel parameters in the guest? Update the guest OS to a
> newer kernel (2.6.22 to 2.6.24)? Update the host OS to a newer kernel
> (2.6.23 to 2.6.24)?

Frankly, I'm running out of ideas.
Are you using kvm-60 userspace, too?
You may also try adding -tdf -no-kvm-irqchip options to the command line of
qemu. If the guest is using the old-fashioned pit for some reason, it
should correct its time drift.

Regards,
    Dan.

> 
> Thanks for the help!
> 
> Koen
> 

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Clock off in guest

[Koen Vermeer]

On Wed, 2008-02-13 at 09:58 -0800, Uri Lublin wrote:
> The host kernel or kvm.
> If you choose to upgrade your host kernel (and kvm that comes with
> it), make sure
> you are using recent kvm-userspace too (e.g. kvm-60).

Running 2.6.23 on the host with kvm 60 (userspace) and kvm-source 60
(modules).
> 

> >> Or as an alternative, probably not as good, sometimes (when the
> guest's
> >> clocksource is PIT) adding '-tdf' to the command line helps.
> >I cannot find this in man kvm or man qemu.
> I'm not sure about the man pages, but kvm/qemu's help says:
> bash$ /usr/bin/kvm -h | grep tdf
> -tdf            inject timer interrupts that got lost
> 
It doesn't seem to be mentioned in the man pages, but it is indeed there
when running 'kvm --help'.


> Yes, try adding it to the command line that starts the guest
> (executable name may vary):
>    /usr/bin/kvm [kvm-params] -tdf
> Also tdf (time drift fix) only works when using PIT+PIC (no APIC) so
> sometimes it's helpful
> to also add -no-acpi:
>    /usr/bin/kvm [kvm-params] -tdf -no-acpi
> 
I tried both -tdf and -tdf -no-acpi, but it didn't change anything. I
did notice some lines on the host saying 

time drift is too long, 60 seconds were lost

which weren't there when running without '-tdf'.


You were saying -tdf only works when using PIT+PIC, but, as far as I can
tell, the guest is still using tsc. Should I add a kernel parameter like
clocksource=pit to the guest?

I appreciate the help!

Koen


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Clock off in guest

[Koen Vermeer]

I upgraded the guest kernel to 2.6.24, and the clock now seems to be
fine (at least 'sleep 10' takes about 10 seconds on the wall clock).
Thanks for all the help!

Best,
Koen



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: Clock off in guest

[Steven Mooij]

Koen Vermeer <koen <at> vermeer.tv> writes:

> 
> I upgraded the guest kernel to 2.6.24, and the clock now seems to be
> fine (at least 'sleep 10' takes about 10 seconds on the wall clock).
> Thanks for all the help!
> 

I upgraded to kernel version 2.6.24 as well (and am using kvm version 60) and
time on guests still runs slower than on the host (and real time).

I estimate time on a linux client runs about 1% slower and time on a windows
client about 2%. Though when load is increased on the latter, I've seen this
increase to about 20%.

Anyone any ideas? Or Koen, how's your client running?


Greets,

Steven


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/