[Qemu-devel] Slow clock in guest OS

[Qemu-devel] Slow clock in guest OS

[Sergey Bychkov]

Hi!

I can't understand why clock in guest OS (Windows 2003) goes very slow.

Host OS: Debian GNU/Linux Etch x86_64 (kernel 2.6.18.5-amd64)
Host CPU: something like Intel Core Duo
$ cat /proc/cpuinfo
processor       : 0 # and 1 too
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      :               Intel(R) Pentium(R) D CPU 2.80GHz
stepping        : 4
...

qemu version: snapshot 24.01.2008, compiled with GCC 4 for target 
x86_64-softmmu

"Stable" version in Debian Etch is 8.2 works not very stable - but clock get 
lost only sometimes, when host CPU is busy.
Snapshot works more stable - clock is constantly slow, even M$ NTP server 
(Windows Time) can't help.

I can't find, how to debug this problem.

Sergey Bychkow
ICQ: 21014758
FTN: 2:450/118.55 

Re: [Qemu-devel] Slow clock in guest OS

[Mulyadi Santosa]

Hi...

On Jan 25, 2008 5:08 AM, Sergey Bychkov <sergb@unichrom.com> wrote:
> Hi!
>
> I can't understand why clock in guest OS (Windows 2003) goes very slow.

Are you sure the rtc freq has been made to 1024?
# cat /proc/sys/dev/rtc/max-user-freq
should yield 1024 before you ran qemu.

If yes, then we should hunt into another possibilities...

regards,

Mulyadi.

Re: [Qemu-devel] Slow clock in guest OS

[Sergey Bychkov]

----- Original Message ----- 
From: "Mulyadi Santosa" <mulyadi.santosa@gmail.com>
To: <qemu-devel@nongnu.org>
Sent: 25.01.2008 5:11


>> I can't understand why clock in guest OS (Windows 2003) goes very slow.
>
> Are you sure the rtc freq has been made to 1024?
> # cat /proc/sys/dev/rtc/max-user-freq
> should yield 1024 before you ran qemu.
>
> If yes, then we should hunt into another possibilities...

Yes, it helps, in some extent.
With rtc/max-user-freq set to 1024 and "-clock rtc" option VM now works like 
in 0.8.2 version.
Clock is slower than in host, but Windows Time server could correct this, if 
started.
Busy host or guest CPU slows down clock.

Sometimes VM hangs on start. Htop shows that qemu uses 100% of second CPU in 
user mode. Internal QEMU VNC server listens port but doesn't accepts 
connections, so vncviewer hangs too. If used GUI, is shows monitor prompt, 
that doesn't accept any keyboard input. Could be killed by Ctrl-C in console 
or 'kill -9'. May be it's different problem but they happen both on the same 
host hardware.

Sergey Bychkow
ICQ: 21014758
FTN: 2:450/118.55 

Re: [Qemu-devel] Slow clock in guest OS

[Mulyadi Santosa]

Hi...

On Jan 25, 2008 6:51 PM, Sergey Bychkov <sergb@unichrom.com> wrote:
> Yes, it helps, in some extent.
> With rtc/max-user-freq set to 1024 and "-clock rtc" option VM now works like
> in 0.8.2 version.
> Clock is slower than in host, but Windows Time server could correct this, if
> started.
> Busy host or guest CPU slows down clock.

I remember somebody wrote dyntick patch...but not sure if it is
already merged into qemu CVS.

Maybe you need similar trick like win98 to halt CPU when not doing anything?

> Sometimes VM hangs on start.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this is weird.... can you dump
the generated opcode? pls see qemu-doc/tech for more detail on how to
do it.

>Htop shows that qemu uses 100% of second CPU in
> user mode.

Before I forgot, do you use kqemu? if yes, do you full virtualization mode?

>Internal QEMU VNC server listens port but doesn't accepts
> connections, so vncviewer hangs too. If used GUI, is shows monitor prompt,
> that doesn't accept any keyboard input. Could be killed by Ctrl-C in console
> or 'kill -9'. May be it's different problem but they happen both on the same
> host hardware.
>
very weird....do you use precompiled qemu binary? or do you compile by
yourself? if it is compiled by yourself, what gcc version do you use?

regards,

Mulyadi.

Re: [Qemu-devel] Slow clock in guest OS

[Sergey Bychkov]

----- Original Message ----- 
From: "Mulyadi Santosa" <mulyadi.santosa@gmail.com>
To: <qemu-devel@nongnu.org>
Sent: 25.01.2008 14:02


>> Yes, it helps, in some extent.
>> With rtc/max-user-freq set to 1024 and "-clock rtc" option VM now works 
>> like
>> in 0.8.2 version.
>> Clock is slower than in host, but Windows Time server could correct this, 
>> if
>> started.
>> Busy host or guest CPU slows down clock.
>
> I remember somebody wrote dyntick patch...but not sure if it is
> already merged into qemu CVS.

Yes, it merged (I thought it is the default). Now I tried to test VM with 
it - Win2k3 hangs on boot, on the stage, afaik, switching to protected mode.

> Maybe you need similar trick like win98 to halt CPU when not doing 
> anything?

Win2k3 has NT kernel (like in WinXP/2000) and does't require this. When 
guest OS is idle, qemu uses only few percents of host CPU, and guest clock 
didn't lose too much.

>> Sometimes VM hangs on start.
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this is weird.... can you dump
> the generated opcode? pls see qemu-doc/tech for more detail on how to
> do it.

May be I will try to work with this problem later and report it in different 
thread.

>>Htop shows that qemu uses 100% of second CPU in
>> user mode.
>
> Before I forgot, do you use kqemu? if yes, do you full virtualization 
> mode?

Yes. No.
But I have got this problem sometimes with -kernel-kqemu and with -no-kqemu 
options.

>>Internal QEMU VNC server listens port but doesn't accepts
>> connections, so vncviewer hangs too. If used GUI, is shows monitor 
>> prompt,
>> that doesn't accept any keyboard input. Could be killed by Ctrl-C in 
>> console
>> or 'kill -9'. May be it's different problem but they happen both on the 
>> same
>> host hardware.
>>
> very weird....do you use precompiled qemu binary? or do you compile by
> yourself? if it is compiled by yourself, what gcc version do you use?

Previous (stable) version of qemu 0.8.2 was from Debian Etch amd64 
(x86_64) - precompiled.
Yesterday's snapshot (qemu-snapshot-2008-01-24_05) compiled manually with 
GCC-4, for targets x86_64-softmmu, i386-softmmu.
==
$ 
./configure --disable-gcc-check --disable-gfx-check --target-list="i386-softmmu 
x86_64-softmmu" --prefix=$HOME
make
make install
==
qemu-system-x86_64 only used, because host OS is x86_64 and I want kqemu to 
be used.
Win2k3 is 32bit version.

PS There is no documentation about "-clock" option in latest qemu documents, 
exept source code, of cause :)

Yours sincerely,
Sergey Bychkow
ICQ: 21014758
FTN: 2:450/118.55 

Re: [Qemu-devel] Slow clock in guest OS

[Sergey Bychkov]

>> Sometimes VM hangs on start.
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this is weird.... can you dump
> the generated opcode? pls see qemu-doc/tech for more detail on how to
> do it.

Sorry, did You mean "http://fabrice.bellard.free.fr/qemu/qemu-tech.html" or 
what?

Sergey Bychkow
ICQ: 21014758
FTN: 2:450/118.55 

Re: [Qemu-devel] Slow clock in guest OS

[Mulyadi Santosa]

Hi..

On Jan 26, 2008 2:05 AM, Sergey Bychkov
> Sorry, did You mean "http://fabrice.bellard.free.fr/qemu/qemu-tech.html" or
> what?

yep...that's the one...

regards,

Mulyadi

[Qemu-devel] More about slow clock in guest OS

[Sergey Bychkov]

----- Original Message ----- 
From: "Mulyadi Santosa" <mulyadi.santosa@gmail.com>
To: <qemu-devel@nongnu.org>
Sent: 26.01.2008 16:25
Subject: Re: [Qemu-devel] Slow clock in guest OS


>> Sorry, did You mean "http://fabrice.bellard.free.fr/qemu/qemu-tech.html" 
>> or
>> what?
>
> yep...that's the one...

As I can see, this document anything about what to do if qemu hung :)

After some investigations I can say that with the latest (2008/01/30) qemu 
from cvs, compiled with gcc-3.4 on linux x86_64 host, guest OS win2k3 works 
not too good.
With "-clock dynticks" clock in OS is very slow - and windows time server 
can't adjust.
With "-clock rtc" hung periodically - for up to 5 minutes, 300 seconds. This 
could happen at bootstrap - when no OS, only BIOS. Then it resumes and works 
for some random period of time, then hangs again, and so on. This behaviour 
doesn't depend on guest OS, and was reproduced with Knoppix live CD. Note 
that host should be periodically busy executing something like torrent 
client.

Details and startup scripts:

$ cat knoppix-qemu-nographic.sh
==
QEMU=$HOME/bin/qemu-system-x86_64
[ -x "$QEMU" ] || QEMU=/usr/bin/qemu-system-x86_64
sudo modprobe kqemu
echo will use executable $QEMU
DIR=`dirname $0`
[ -z "$DIR" -o "$DIR" == "." ] && DIR=`pwd`
echo DIR is $DIR
PREV_RTC_FREQ=`cat /proc/sys/dev/rtc/max-user-freq`
[ "$PREV_RTC_FREQ" -ge "1024" ] || echo 'WARN: RTC FREQ too slow:' 
$PREV_RTC_FREQ
sudo $QEMU -net nic,model=rtl8139,macaddr=52:54:00:80:80:01 -net 
tap,script=$DIR/qemu-ifup-br0,downscript=$DIR/qemu-ifdown-br0 -localtime -cdrom 
/distrib/knoppix/KNOPPIX_V5.1.0CD-2006-12-30-EN.iso -m 384 -pidfile 
$DIR/virt1-knoppix.pid -smp 1 -no-kqemu -clock rtc -vnc :9
==
$ cat qemu-ifup-br0
==
#!/bin/sh

echo Configuring virtual interface $1
if [ "$UID" -eq "0" ]
then
    BRIF=br0
    if brctl addif $BRIF $1
    then
        echo $1 added to $BRIF
        ifconfig $1 0.0.0.0 up
        echo $1 configured for bridge $BRIF
    else
        IP_LOCAL=169.254.1.1
        echo $1 not added to $BRIF
        ifconfig $1 $IP_LOCAL up
        echo $1 configured to $IP_LOCAL \(like autoip\)
        # TODO: try to make real autoip
    fi
else
    echo Will sudo $0
    exec sudo -p "Password for $0:" $0 $*
fi
==
$ cat qemu-ifdown-br0
==
#!/bin/sh
BRIF=br0
ifconfig $1
brctl show $BRIF
brctl showmacs $BRIF
echo Deconfiguring virtual interface $1 will be done automatically
==
Bridge interface should be configured in system
$ cat /etc/network/interfaces
==
#auto eth0 - this interface should not be configured
iface eth0 inet static
...
# The primary network interface - now in bridge mode
auto br0
iface br0 inet static
        bridge_ports eth0
...
==
$ cat /proc/sys/dev/rtc/max-user-freq
==
4096
==
$ uname -a
==
Linux *** 2.6.18-5-amd64 #1 SMP Sat Dec 22 20:43:59 UTC 2007 x86_64 
GNU/Linux
==
$ cat /proc/cpuinfo
==
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      :               Intel(R) Pentium(R) D CPU 2.80GHz
stepping        : 4
cpu MHz         : 2800.000
cache size      : 2048 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm 
constant_tsc pni monitor ds_cpl est cid cx16 xtpr lahf_lm
bogomips        : 5633.18
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
...
==
$ ~/bin/qemu-system-x86_64
QEMU PC emulator version 0.9.1, Copyright (c) 2003-2008 Fabrice Bellard
...
==

May be somebody will understand what's happen :)

Sergey Bychkow
ICQ: 21014758
FTN: 2:450/118.55 

Re: [Qemu-devel] More about slow clock in guest OS

[Mulyadi Santosa]

hello Sergey...

On 1/31/08, Sergey Bychkov <sergb@unichrom.com> wrote:
> As I can see, this document anything about what to do if qemu hung :)

reading always helps in some degree :)

> After some investigations I can say that with the latest (2008/01/30) qemu
> from cvs, compiled with gcc-3.4 on linux x86_64 host, guest OS win2k3 works
> not too good.
So, in other words "it works but not so well". may I interpret it as a progress?

> With "-clock dynticks" clock in OS is very slow - and windows time server
> can't adjust.

Probably due to cost of Qemu timer rearming. And maybe Windows 2003
does certain dyntick by its own...thus enlarging the timer rearming
cost.

> With "-clock rtc" hung periodically - for up to 5 minutes, 300 seconds.

pffff... too much timers get expired? lock contention somewhere...anybody?

>This
> could happen at bootstrap - when no OS, only BIOS. Then it resumes and works
> for some random period of time, then hangs again, and so on. This behaviour
> doesn't depend on guest OS, and was reproduced with Knoppix live CD. Note
> that host should be periodically busy executing something like torrent
> client.
Thus..maybe it's Qemu's fault...

> Details and startup scripts:
>
> $ cat knoppix-qemu-nographic.sh
> ==
> QEMU=$HOME/bin/qemu-system-x86_64
> [ -x "$QEMU" ] || QEMU=/usr/bin/qemu-system-x86_64
> sudo modprobe kqemu
> echo will use executable $QEMU
> DIR=`dirname $0`
> [ -z "$DIR" -o "$DIR" == "." ] && DIR=`pwd`
> echo DIR is $DIR
> PREV_RTC_FREQ=`cat /proc/sys/dev/rtc/max-user-freq`
> [ "$PREV_RTC_FREQ" -ge "1024" ] || echo 'WARN: RTC FREQ too slow:'
> $PREV_RTC_FREQ
> sudo $QEMU -net nic,model=rtl8139,macaddr=52:54:00:80:80:01 -net
> tap,script=$DIR/qemu-ifup-br0,downscript=$DIR/qemu-ifdown-br0 -localtime -cdrom
> /distrib/knoppix/KNOPPIX_V5.1.0CD-2006-12-30-EN.iso -m 384 -pidfile
> $DIR/virt1-knoppix.pid -smp 1 -no-kqemu -clock rtc -vnc :9

Ehm:
1. can you simply use SDL output instead of VNC?
2. what if you don't use -localtime? just courious...

> ==
> $ cat /proc/sys/dev/rtc/max-user-freq
> ==
> 4096
> ==

ehm...I think 1024 is enough for most cases..

> $ uname -a
> ==
> Linux *** 2.6.18-5-amd64 #1 SMP Sat Dec 22 20:43:59 UTC 2007 x86_64
> GNU/Linux
> ==

is that kernel version of the host? can you upgrade it ? let's say to
the latest 2.6.24?

PS: could you try KVM too? but ehm...well, sounds like you don't have
VT enabled Intel processor or SVM enabled AMD processor. So, if you
indeed have one...try KVM..

regards,

Mulyadi.

Re: [Qemu-devel] More about slow clock in guest OS

[Sergey Bychkov]

----- Original Message ----- 
From: "Mulyadi Santosa" <mulyadi.santosa@gmail.com>
To: <qemu-devel@nongnu.org>
Cc: <sergb@unichrom.com>
Sent: 1.02.2008 4:41
Subject: Re: [Qemu-devel] More about slow clock in guest OS


>> After some investigations I can say that with the latest (2008/01/30) 
>> qemu
>> from cvs, compiled with gcc-3.4 on linux x86_64 host, guest OS win2k3 
>> works
>> not too good.
> So, in other words "it works but not so well". may I interpret it as a 
> progress?

Yes, now there is a choice: stable but slow (dynticks) and unstable but 
faster (rtc)

Unfortunately, there is no option "stable and correct" yet ;)

>> With "-clock dynticks" clock in OS is very slow - and windows time server
>> can't adjust.
>
> Probably due to cost of Qemu timer rearming. And maybe Windows 2003
> does certain dyntick by its own...thus enlarging the timer rearming
> cost.

Tests say that in dynticks mode clock in VM is 4-5 times slower than in real 
world.

>
>> With "-clock rtc" hung periodically - for up to 5 minutes, 300 seconds.
>
> pffff... too much timers get expired? lock contention somewhere...anybody?

I even stop experiments with "rtc" because of instability. Maybe somebody 
else will find this problem in the future, I'm focusing in testing 
"dynticks" mode now.

> Thus..maybe it's Qemu's fault...

>> sudo $QEMU -net nic,model=rtl8139,macaddr=52:54:00:80:80:01 -net
>> tap,script=$DIR/qemu-ifup-br0,downscript=$DIR/qemu-ifdown-br0 -localtime  
>> -cdrom
>> /distrib/knoppix/KNOPPIX_V5.1.0CD-2006-12-30-EN.iso -m 384 -pidfile
>> $DIR/virt1-knoppix.pid -smp 1 -no-kqemu -clock rtc -vnc :9
>
> Ehm:
> 1. can you simply use SDL output instead of VNC?
> 2. what if you don't use -localtime? just courious...
1. Yes, but main purpose is to run VM on linux server, without X.
2. The only difference is that initial clock is two hours out. Windows 
always assumes that hardware clock is local.
>
>> ==
>> $ cat /proc/sys/dev/rtc/max-user-freq
>> ==
>> 4096
>> ==
>
> ehm...I think 1024 is enough for most cases..
There were tests, and this parameter doesn't affect dynticks mode, so now it 
is in default value, 64.
>
>> $ uname -a
>> ==
>> Linux *** 2.6.18-5-amd64 #1 SMP Sat Dec 22 20:43:59 UTC 2007 x86_64
>> GNU/Linux
>> ==
>
> is that kernel version of the host? can you upgrade it ? let's say to
> the latest 2.6.24?
I will upgrade it when Debian releases next stable release :)
>
> PS: could you try KVM too? but ehm...well, sounds like you don't have
> VT enabled Intel processor or SVM enabled AMD processor. So, if you
> indeed have one...try KVM..

At this time I apply some weird combination of standard and self-made tools 
to keep clock in VM in sync with outer world.
May be I will try to compare behaviour of different OSes in qemu and/or KVM.

Sergey Bychkow
ICQ: 21014758
FTN: 2:450/118.55 

Re: [Qemu-devel] More about slow clock in guest OS

[Mulyadi Santosa]

Hi..

Just want to reply shortly....I guess you can lead your own research
from here since I almost reach my knowledge limit especially dealing
with Qemu internals. However, I greatly appreciate your effort and
time sharing your discoveries to me and the rest of Qemu's community.

keep up the good work!

regards,

Mulyadi.

Re: [Qemu-devel] Slow clock in guest OS

[Sergey Bychkov]

----- Original Message ----- 
From: "Sergey Bychkov" <sergb@unichrom.com>
To: <qemu-devel@nongnu.org>
Sent: 25.01.2008 0:08
Subject: [Qemu-devel] Slow clock in guest OS


> I can't understand why clock in guest OS (Windows 2003) goes very slow.

I have found that slow clock was inspired by working UltraVNC server 
installed in guest OS.
Possibly, often queries to video driver force qemu to "forget" to send clock 
IRQs to guest.
At this time I didn't find more details about this problem, but stopping 
UltraVNC service completely remove it.

Sergey Bychkow
ICQ: 21014758
FTN: 2:450/118.55 

Re: [Qemu-devel] Slow clock in guest OS

[Mulyadi Santosa]

Hi...

On Fri, Mar 7, 2008 at 8:39 PM, Sergey Bychkov <sergb@unichrom.com> wrote:
>  I have found that slow clock was inspired by working UltraVNC server
>  installed in guest OS.
>  Possibly, often queries to video driver force qemu to "forget" to send clock
>  IRQs to guest.
>  At this time I didn't find more details about this problem, but stopping
>  UltraVNC service completely remove it.

Hm...or somehow doing that make Qemu so busy and "forget" to process
pending interrupts.

Good detective work, IMHO. I suggest to write about it in Qemu FAQ.
Care to submit a patch for qemu-doc?

regards,

Mulyadi.