[Qemu-devel] Qemu / KQemu on 64-bit (x86_64) host systems

[Qemu-devel] Qemu / KQemu on 64-bit (x86_64) host systems

[Werner Dittmann]

During several tests with Qemu / Kqemu it seems that Qemu
has problems with x86_64 host systems. My system is an
AMD 64 X2 (Dual Core), running openSUSE 10.2, 2GB memory.

Various versions of Qemu/Kqemu available and under test:
0.8.2, 0.9.0, and CVS. Kqemu 1.3.0pre9, 1.3.0pre11

When building Qemu I use the following configure setup,
using a gcc 3.4:
./configure --prefix=/usr/local/ \
 --cc=/opt/gcc34/bin/gcc-3.4 --host-cc=/opt/gcc34/bin/gcc-3.4 \
 --enable-alsa  --enable-adlib \
 --target-list="i386-softmmu x86_64-softmmu"

Kqemu built with standard (system) gcc.

I always use qemu-system-x86_64 to start Qemu.

Here the problems:

Installing a 32bit Linux system (Debian, Kernel 2.6.18):
- works with pure Qemu (-no-kqemu)
- fails with Kqemu support enabled. The failure is a loop
  before or during the kernel hands over control to INIT

I used gdb to get some more information about the problems
using the following command:
 gdb qemu-system-x86_64

using a .gdbinit that sets the args, etc.

When the kernel goes into the loop I interrupt with ^C
several times, most of the time it was in code_gen_buffer,
here in the function "compute_c_subl".

Because I'm _not_ sure this is the correct way to debug Qemu
I cannot say if this is normal or not. At least the function
always returns  1 (it seems that it is called over and over
again with). The last relevant statement in this function is:

cmp  %eax,0x90(%r14)
seta %al

where the conetent of %eax is zero, the content of the memory
is 0x00000000ffffeb3e. The return says: the memory content is
bigger than 0x0 (which is true for 64bit, but also true for
32bit unsigned, "compute_c_subl" compares two unsigned 32bit
integers). As said, take these findings with a grain of
salt.

My general thought about the problem: running 32bit code
on a 64bit host with similar architecture as this is the case
of x86 / x86_64 could easily result in problems with signedness,
sign bit extension, different pointer/word/interger sizes...

BTW: is there a Howto or other information how to debug
Qemu when the loaded kernel loops or crashes? That would be
great and would make it easier to step in here and provide some
help (or is this a somewhat good kept secret :-) ? ).

The next problems are fairly old, they are also reported in the
Qemu user's wiki - but without an answer o solution.

Installing a 64bit Linux system (openSuse 10.1, 10.2):
- fails with Qemu (-no-kqemu), loops when Grub shall install
  the bootloader.
- fails with Kqemu enabled, crashes at various addresses and
  prints register contents.

Any hints what this could be? Solutions?

Regards,
Werner

Re: [Qemu-devel] Qemu / KQemu on 64-bit (x86_64) host systems

[andrzej zaborowski]

Hi,

On 16/04/07, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:
> During several tests with Qemu / Kqemu it seems that Qemu
> has problems with x86_64 host systems. My system is an
> AMD 64 X2 (Dual Core), running openSUSE 10.2, 2GB memory.
>
> Various versions of Qemu/Kqemu available and under test:
> 0.8.2, 0.9.0, and CVS. Kqemu 1.3.0pre9, 1.3.0pre11
>
> When building Qemu I use the following configure setup,
> using a gcc 3.4:
> ./configure --prefix=/usr/local/ \
>  --cc=/opt/gcc34/bin/gcc-3.4 --host-cc=/opt/gcc34/bin/gcc-3.4 \
>  --enable-alsa  --enable-adlib \
>  --target-list="i386-softmmu x86_64-softmmu"
>
> Kqemu built with standard (system) gcc.
>
> I always use qemu-system-x86_64 to start Qemu.
>
> Here the problems:
>
> Installing a 32bit Linux system (Debian, Kernel 2.6.18):
> - works with pure Qemu (-no-kqemu)
> - fails with Kqemu support enabled. The failure is a loop
>   before or during the kernel hands over control to INIT

Does your host happen to be dual-core? If so, please try adding
"notsc" to the guest kernel commandline and report if it makes a
difference.

>
> I used gdb to get some more information about the problems
> using the following command:
>  gdb qemu-system-x86_64
>
> using a .gdbinit that sets the args, etc.
>
> When the kernel goes into the loop I interrupt with ^C
> several times, most of the time it was in code_gen_buffer,
> here in the function "compute_c_subl".
>
> Because I'm _not_ sure this is the correct way to debug Qemu
> I cannot say if this is normal or not. At least the function
> always returns  1 (it seems that it is called over and over
> again with). The last relevant statement in this function is:
>
> cmp  %eax,0x90(%r14)
> seta %al
>
> where the conetent of %eax is zero, the content of the memory
> is 0x00000000ffffeb3e. The return says: the memory content is
> bigger than 0x0 (which is true for 64bit, but also true for
> 32bit unsigned, "compute_c_subl" compares two unsigned 32bit
> integers). As said, take these findings with a grain of
> salt.
>
> My general thought about the problem: running 32bit code
> on a 64bit host with similar architecture as this is the case
> of x86 / x86_64 could easily result in problems with signedness,
> sign bit extension, different pointer/word/interger sizes...
>
> BTW: is there a Howto or other information how to debug
> Qemu when the loaded kernel loops or crashes? That would be
> great and would make it easier to step in here and provide some
> help (or is this a somewhat good kept secret :-) ? ).

Use qemu's gdb server, it's documented.

>
> The next problems are fairly old, they are also reported in the
> Qemu user's wiki - but without an answer o solution.
>
> Installing a 64bit Linux system (openSuse 10.1, 10.2):
> - fails with Qemu (-no-kqemu), loops when Grub shall install
>   the bootloader.
> - fails with Kqemu enabled, crashes at various addresses and
>   prints register contents.
>
> Any hints what this could be? Solutions?
>
> Regards,
> Werner
>
>
>
>

Regards,
Andrzej

Re: [Qemu-devel] Qemu / KQemu on 64-bit (x86_64) host systems

[Bernhard Kauer]

On Mon, Apr 16, 2007 at 08:47:07AM +0200, Werner Dittmann wrote:
> My general thought about the problem: running 32bit code
> on a 64bit host with similar architecture as this is the case
> of x86 / x86_64 could easily result in problems with signedness,
> sign bit extension, different pointer/word/interger sizes...


qemu-system-x86_64 has indeed some signedness problem. I posted a
patch <20070410133045.GC6046@chrom.inf.tu-dresden.de> a couple of
days ago for the problem that the pagefault address gets incorrectly
sign extended when running 32-bit code. Nevertheless its just a
workaround, the place where this wrong sign extension happend needs
still to be found.


	Bernhard Kauer

Re: [Qemu-devel] Qemu / KQemu on 64-bit (x86_64) host systems

[Werner Dittmann]

andrzej zaborowski wrote:
> Hi,
> 
> On 16/04/07, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:
>> During several tests with Qemu / Kqemu it seems that Qemu
>> has problems with x86_64 host systems. My system is an
>> AMD 64 X2 (Dual Core), running openSUSE 10.2, 2GB memory.
>>
Indeed it is a dual CPU. Adding notsc as an additional parameter to
the kernel commandline (the guest system uses Lilo). Using this
parameter the kernel was able to start INIT and the init.d startup
scripts (not always, but most of the time). After a short time
the kernel starts to loop again, e.g. during hotplug handling.

Using th same technique as before (gdb) it's the same picture:
mainly in compute_c_subl or compute_c_addl.

> 
> Does your host happen to be dual-core? If so, please try adding
> "notsc" to the guest kernel commandline and report if it makes a
> difference.
> 
>>
I thought qemu's gdb server is used to debug kernels running
inside Qemu but not to debug Qemu itself. IMHO the problem is not
in the kernel (the kernel works perfectly on a real HW processor)
but in Qemu.
> 
> Use qemu's gdb server, it's documented.
> 
>>
>>
>>
> 
> Regards,
> Andrzej
> 

Regards,
Werner

Re: [Qemu-devel] Qemu / KQemu on 64-bit (x86_64) host systems

[andrzej zaborowski]

Hi,

On 17/04/07, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:
> andrzej zaborowski wrote:
> > Hi,
> >
> > On 16/04/07, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:
> >> During several tests with Qemu / Kqemu it seems that Qemu
> >> has problems with x86_64 host systems. My system is an
> >> AMD 64 X2 (Dual Core), running openSUSE 10.2, 2GB memory.
> >>
> Indeed it is a dual CPU. Adding notsc as an additional parameter to
> the kernel commandline (the guest system uses Lilo). Using this
> parameter the kernel was able to start INIT and the init.d startup
> scripts (not always, but most of the time). After a short time
> the kernel starts to loop again, e.g. during hotplug handling.

Hmm, I was thinking that it may be the same problem as I described in
http://lists.gnu.org/archive/html/qemu-devel/2007-03/msg00652.html but
if the lockup is happening anyway then I'm not sure. On the other hand
"notsc" does make a difference, right?

>
> Using th same technique as before (gdb) it's the same picture:
> mainly in compute_c_subl or compute_c_addl.
>
> >
> > Does your host happen to be dual-core? If so, please try adding
> > "notsc" to the guest kernel commandline and report if it makes a
> > difference.
> >
> >>
> I thought qemu's gdb server is used to debug kernels running
> inside Qemu but not to debug Qemu itself. IMHO the problem is not
> in the kernel (the kernel works perfectly on a real HW processor)
> but in Qemu.

That's right, qemu's gdb server is for debugging the guest. However,
it's often much easier to debug qemu knowing what guest code is
causing the qemu bug, especially C code in case of opensource guests,
and especially a guest lockup or guest crash.

Other than that I think there's only the -d.

Regards,
Andrzej

Re: [Qemu-devel] Qemu / KQemu on 64-bit (x86_64) host systems

[Werner Dittmann]

Andrzej,

setting "notsc" makes the difference (yesterday I forgot to start
lilo after modifying /etc/lilo.conf to include notsc"). Now it
work even with -kernel-kqemu. Not fully tested though, but much
better than before.

Thanks,
Werner


andrzej zaborowski wrote:
> Hi,
> 
> On 17/04/07, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:
>> andrzej zaborowski wrote:
>> > Hi,
>> >
>> > On 16/04/07, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:
>> >> During several tests with Qemu / Kqemu it seems that Qemu
>> >> has problems with x86_64 host systems. My system is an
>> >> AMD 64 X2 (Dual Core), running openSUSE 10.2, 2GB memory.
>> >>
>> Indeed it is a dual CPU. Adding notsc as an additional parameter to
>> the kernel commandline (the guest system uses Lilo). Using this
>> parameter the kernel was able to start INIT and the init.d startup
>> scripts (not always, but most of the time). After a short time
>> the kernel starts to loop again, e.g. during hotplug handling.
> 
> Hmm, I was thinking that it may be the same problem as I described in
> http://lists.gnu.org/archive/html/qemu-devel/2007-03/msg00652.html but
> if the lockup is happening anyway then I'm not sure. On the other hand
> "notsc" does make a difference, right?
> 
>>
>> Using th same technique as before (gdb) it's the same picture:
>> mainly in compute_c_subl or compute_c_addl.
>>
>> >
>> > Does your host happen to be dual-core? If so, please try adding
>> > "notsc" to the guest kernel commandline and report if it makes a
>> > difference.
>> >
>> >>
>> I thought qemu's gdb server is used to debug kernels running
>> inside Qemu but not to debug Qemu itself. IMHO the problem is not
>> in the kernel (the kernel works perfectly on a real HW processor)
>> but in Qemu.
> 
> That's right, qemu's gdb server is for debugging the guest. However,
> it's often much easier to debug qemu knowing what guest code is
> causing the qemu bug, especially C code in case of opensource guests,
> and especially a guest lockup or guest crash.
> 
> Other than that I think there's only the -d.
> 
> Regards,
> Andrzej
> 
> 
> 

Re: [Qemu-devel] Qemu / KQemu on 64-bit (x86_64) host systems

[Werner Dittmann]

Andrzej,

just another remark: after setting the kernel parameter to "notsc"
the kernel now detects the CPU correctly. Without this setting the
CPU detetion was wrong (displays the wrong CPU type, frequency, etc).
Are there any know side-effects if notsc (no time stamp counter) is
set?

Regards,
Werner

andrzej zaborowski wrote:
> Hi,
> 
> On 17/04/07, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:
>> andrzej zaborowski wrote:
>> > Hi,
>> >
>> > On 16/04/07, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:
>> >> During several tests with Qemu / Kqemu it seems that Qemu
>> >> has problems with x86_64 host systems. My system is an
>> >> AMD 64 X2 (Dual Core), running openSUSE 10.2, 2GB memory.
>> >>
>> Indeed it is a dual CPU. Adding notsc as an additional parameter to
>> the kernel commandline (the guest system uses Lilo). Using this
>> parameter the kernel was able to start INIT and the init.d startup
>> scripts (not always, but most of the time). After a short time
>> the kernel starts to loop again, e.g. during hotplug handling.
> 
> Hmm, I was thinking that it may be the same problem as I described in
> http://lists.gnu.org/archive/html/qemu-devel/2007-03/msg00652.html but
> if the lockup is happening anyway then I'm not sure. On the other hand
> "notsc" does make a difference, right?
> 
>>
>> Using th same technique as before (gdb) it's the same picture:
>> mainly in compute_c_subl or compute_c_addl.
>>
>> >
>> > Does your host happen to be dual-core? If so, please try adding
>> > "notsc" to the guest kernel commandline and report if it makes a
>> > difference.
>> >
>> >>
>> I thought qemu's gdb server is used to debug kernels running
>> inside Qemu but not to debug Qemu itself. IMHO the problem is not
>> in the kernel (the kernel works perfectly on a real HW processor)
>> but in Qemu.
> 
> That's right, qemu's gdb server is for debugging the guest. However,
> it's often much easier to debug qemu knowing what guest code is
> causing the qemu bug, especially C code in case of opensource guests,
> and especially a guest lockup or guest crash.
> 
> Other than that I think there's only the -d.
> 
> Regards,
> Andrzej
> 
> 
> 

Re: [Qemu-devel] Qemu / KQemu on 64-bit (x86_64) host systems

[andrzej zaborowski]

Hi,

On 18/04/07, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:
> Andrzej,
>
> just another remark: after setting the kernel parameter to "notsc"
> the kernel now detects the CPU correctly. Without this setting the
> CPU detetion was wrong (displays the wrong CPU type, frequency, etc).
> Are there any know side-effects if notsc (no time stamp counter) is
> set?

I don't know what clocksource is chosen with "notsc", it may be
forgettably slower one (but not necessarily). What do you mean by
wrong CPU?

Apparently this is the same SMP issue I described in
http://lists.gnu.org/archive/html/qemu-devel/2007-03/msg00652.html so
it's not x86_64 related. If you apply the patch from that post, you
can skip "notsc".

Regards,
Andrzej

Re: [Qemu-devel] Qemu / KQemu on 64-bit (x86_64) host systems

[Werner Dittmann]

Andrzej,

the guest Linux system reported some AMD CPU type (can't remember
which one) which is not in my system. Now when the guest Linux starts
is correctly reports: CPU 0 AMD X2 4200+ ....

Regards,
Werner

andrzej zaborowski wrote:
> Hi,
> 
> On 18/04/07, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:
>> Andrzej,
>>
>> just another remark: after setting the kernel parameter to "notsc"
>> the kernel now detects the CPU correctly. Without this setting the
>> CPU detetion was wrong (displays the wrong CPU type, frequency, etc).
>> Are there any know side-effects if notsc (no time stamp counter) is
>> set?
> 
> I don't know what clocksource is chosen with "notsc", it may be
> forgettably slower one (but not necessarily). What do you mean by
> wrong CPU?
> 
> Apparently this is the same SMP issue I described in
> http://lists.gnu.org/archive/html/qemu-devel/2007-03/msg00652.html so
> it's not x86_64 related. If you apply the patch from that post, you
> can skip "notsc".
> 
> Regards,
> Andrzej
> 
> 
> 

Re: [Qemu-devel] Qemu / KQemu on 64-bit (x86_64) host systems

[andrzej zaborowski]

On 19/04/07, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:
> Andrzej,
>
> the guest Linux system reported some AMD CPU type (can't remember
> which one) which is not in my system. Now when the guest Linux starts
> is correctly reports: CPU 0 AMD X2 4200+ ....

That's a deficiency of the kqemu approach and it's not correct because
an AMD X2 4200+ is not emulated. Your virtual machine is *not* your
PC.

Have you tried the patch btw?

Regards,
Andrzej

Re: [Qemu-devel] Qemu / KQemu on 64-bit (x86_64) host systems

[Werner Dittmann]

andrzej zaborowski wrote:
> On 19/04/07, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:
>> Andrzej,
>>
>> the guest Linux system reported some AMD CPU type (can't remember
>> which one) which is not in my system. Now when the guest Linux starts
>> is correctly reports: CPU 0 AMD X2 4200+ ....
> 
> That's a deficiency of the kqemu approach and it's not correct because
> an AMD X2 4200+ is not emulated. Your virtual machine is *not* your
> PC.
I thought so :-) . IMHO it just perfoms a CPUID check here.
> 
Not yet, it's on my list during the tests. I'll give you feedback after
I tested it.

> Have you tried the patch btw?
> 
> Regards,
> Andrzej

Regards,
Werner

Re: [Qemu-devel] Qemu / KQemu on 64-bit (x86_64) host systems

[Werner Dittmann]

Andrzej,

implemented the patch, works ok even if I removed the notsc parameter
from the kernel parameter line. I did some tests with the systems,
compiled quite some code - works great. I didn't make any performance
tests to compare the solutins - but it looks ok preformace wise.

(This is a 64bit host and a 32bit guest).

Regards,
Werner

andrzej zaborowski wrote:
> On 19/04/07, Werner Dittmann <Werner.Dittmann@t-online.de> wrote:
>> Andrzej,
>>
> 
> That's a deficiency of the kqemu approach and it's not correct because
> an AMD X2 4200+ is not emulated. Your virtual machine is *not* your
> PC.
> 
> Have you tried the patch btw?
> 
> Regards,
> Andrzej
> 
> 
>