[Qemu-devel] (i386) TCG is broken in 1.5-rc0 with guest kernel 3.2?

[Qemu-devel] (i386) TCG is broken in 1.5-rc0 with guest kernel 3.2?

[Michael Tokarev]

Something strange is going on here.

I just tried 1.5-rc0 (32bit x86 userspace), and guest kernel
crashes in an interesting way right at init time.

 qemu-system-x86_64 -net none -kernel /boot/vmlinuz-3.2-amd64 -append console=ttyS0 -serial file:serial

This results in:

[    0.071315] mce: CPU supports 10 MCE banks
[    0.082316] SMP alternatives: switching to UP code
[    0.658457] Freeing SMP alternatives: 16k freed
[    0.660716] ACPI: Core revision 20110623
[    0.744749] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.778545] CPU0: AMD QEMU Virtual CPU version 1.4.90 stepping 03
[    0.887450] Performance Events: Broken PMU hardware detected, using software events only.
[    0.889372] NMI watchdog disabled (cpu0): hardware events not enabled
[    0.889372] Brought up 1 CPUs
[    0.889372] Total of 1 processors activated (6432.01 BogoMIPS).
[    0.889372] devtmpfs: initialized
[    0.889372] ------------[ cut here ]------------
[    0.889372] WARNING: at fs/sysfs/group.c:138 dpm_sysfs_add+0xb4/0xc0()
[    0.889372] Hardware name: Bochs
[    0.889372] sysfs group ffffffff814bece0 not found for kobject 'platform'
[    0.889372] Modules linked in:
[    0.889372] Pid: 1, comm: swapper/0 Not tainted 3.2.0-amd64 #3.2.42
[    0.889372] Call Trace:
[    0.889372]  [<ffffffff812791c4>] ? dpm_sysfs_add+0xb4/0xc0
...

[    0.889372] ---[ end trace ef00d87ab31a38ca ]---
[    0.889372] NET: Registered protocol family 16
[    0.889372] ------------[ cut here ]------------
[    0.889372] WARNING: at fs/sysfs/group.c:138 dpm_sysfs_add+0xb4/0xc0()
[    0.889372] Hardware name: Bochs
[    0.889372] sysfs group ffffffff814bece0 not found for kobject 'vtcon0'
[    0.889372] Modules linked in:
[    0.889372] Pid: 1, comm: swapper/0 Tainted: G        W    3.2.0-amd64 #3.2.42
[    0.889372] Call Trace:
[    0.889372]  [<ffffffff812791c4>] ? dpm_sysfs_add+0xb4/0xc0
...

and finally
[    0.889372] kernel BUG at kernel/params.c:780!
[    0.889372] invalid opcode: 0000 [#1] SMP
...
[    0.893968] Kernel panic - not syncing: Attempted to kill init!



The same happens with other 3.2.x kernels, for example, ubuntu precise:
http://archive.ubuntu.com/ubuntu/dists/precise/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/linux


qemu-system-x86_64 from qemu-1.4.x lets these kernels to boot fine,
it all boots fine when adding -enable-kvm.  So it is some change(s)
in TCG between 1.4 and 1.5, maybe triggering a bug in guest kernel.

For added fun, for example, 3.8 guest kernel does NOT have problems
with this config.

I'm trying to bisect the issue now.

This happened when CarlFK (Cc'd) mentioned that he has issues with
installing ubuntu precise in a 1.5 virtual machine.  But in his
case, the prob was different, -- his installer does not configure
NIC properly due to dhclient segfaulting, which, again, does not
happen with previous versions of qemu.  This may be related or
not, I don't know yet.

That's just a heads-up for now, will post more details as something
will become available.

Thanks,

/mjt

Re: [Qemu-devel] (i386) TCG is broken in 1.5-rc0 with guest kernel 3.2?

[Michael Tokarev]

08.05.2013 12:22, Michael Tokarev wrote:
> Something strange is going on here.
> 
> I just tried 1.5-rc0 (32bit x86 userspace), and guest kernel
> crashes in an interesting way right at init time.
> 
>  qemu-system-x86_64 -net none -kernel /boot/vmlinuz-3.2-amd64 -append console=ttyS0 -serial file:serial
> 
> This results in:
> 
> [    0.071315] mce: CPU supports 10 MCE banks
> [    0.082316] SMP alternatives: switching to UP code
> [    0.658457] Freeing SMP alternatives: 16k freed
> [    0.660716] ACPI: Core revision 20110623
> [    0.744749] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
> [    0.778545] CPU0: AMD QEMU Virtual CPU version 1.4.90 stepping 03
> [    0.887450] Performance Events: Broken PMU hardware detected, using software events only.
> [    0.889372] NMI watchdog disabled (cpu0): hardware events not enabled
> [    0.889372] Brought up 1 CPUs
> [    0.889372] Total of 1 processors activated (6432.01 BogoMIPS).
> [    0.889372] devtmpfs: initialized
> [    0.889372] ------------[ cut here ]------------
> [    0.889372] WARNING: at fs/sysfs/group.c:138 dpm_sysfs_add+0xb4/0xc0()
> [    0.889372] Hardware name: Bochs
> [    0.889372] sysfs group ffffffff814bece0 not found for kobject 'platform'
...


This is broken by the following:

commit 57eb0cc85469a8948d1036ab830951e63aa32f66
Author: Richard Henderson <rth@twiddle.net>
Date:   Wed Jan 16 11:00:14 2013 -0800

    target-i386: expand cmov via movcond

    Signed-off-by: Richard Henderson <rth@twiddle.net>


After this change, (at least) 3.2 kernel starts behaving badly,
being unable to boot in qemu TCG mode:

> The same happens with other 3.2.x kernels, for example, ubuntu precise:
> http://archive.ubuntu.com/ubuntu/dists/precise/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/linux
> 
> 
> qemu-system-x86_64 from qemu-1.4.x lets these kernels to boot fine,
> it all boots fine when adding -enable-kvm.  So it is some change(s)
> in TCG between 1.4 and 1.5, maybe triggering a bug in guest kernel.
> 
> For added fun, for example, 3.8 guest kernel does NOT have problems
> with this config.


I'm trying 64bits now too...

/mjt

Re: [Qemu-devel] (i386) TCG is broken in 1.5-rc0 with guest kernel 3.2?

[Michael Tokarev]

08.05.2013 13:03, Michael Tokarev wrote:
> This is broken by the following:
> 
> commit 57eb0cc85469a8948d1036ab830951e63aa32f66
> Author: Richard Henderson <rth@twiddle.net>
> Date:   Wed Jan 16 11:00:14 2013 -0800
> 
>     target-i386: expand cmov via movcond
> 
>     Signed-off-by: Richard Henderson <rth@twiddle.net>

This change works fine on amd64.  Only i386 (32bit x86)
userspace is broken.  Running amd64 guest kernel inside
in all cases.

Thanks,

/mjt

Re: [Qemu-devel] (i386) TCG is broken in 1.5-rc0 with guest kernel 3.2?

[Richard Henderson]

On 05/08/2013 02:03 AM, Michael Tokarev wrote:
> I just tried 1.5-rc0 (32bit x86 userspace), and guest kernel
>> crashes in an interesting way right at init time.
>> 
>>  qemu-system-x86_64 -net none -kernel /boot/vmlinuz-3.2-amd64 -append console=ttyS0 -serial file:serial
>> 

Irritatingly, this works for me, with the current fedora 18 kernel,

x86_64-softmmu/qemu-system-x86_64: ELF 32-bit LSB shared object, Intel 80386,
version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32,
BuildID[sha1]=0x3286b0bfb3272b9b3004abe8d0c306f12470d77d, not stripped

./x86_64-softmmu/qemu-system-x86_64 -net none -kernel /boot/vmlinuz
3.8.11-200.fc18.x86_64 -append console=ttyS0 -serial stdio

Which makes it all the way until it panics for lack of root filesystem.

Hopefully this is the problem fixed by Aurelien's tcg/optimize.c patch today
(not yet applied to qemu head)?



r~

Re: [Qemu-devel] (i386) TCG is broken in 1.5-rc0 with guest kernel 3.2?

[Aurelien Jarno]

On Thu, May 09, 2013 at 09:43:20AM -0700, Richard Henderson wrote:
> On 05/08/2013 02:03 AM, Michael Tokarev wrote:
> > I just tried 1.5-rc0 (32bit x86 userspace), and guest kernel
> >> crashes in an interesting way right at init time.
> >> 
> >>  qemu-system-x86_64 -net none -kernel /boot/vmlinuz-3.2-amd64 -append console=ttyS0 -serial file:serial
> >> 
> 
> Irritatingly, this works for me, with the current fedora 18 kernel,
> 
> x86_64-softmmu/qemu-system-x86_64: ELF 32-bit LSB shared object, Intel 80386,
> version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32,
> BuildID[sha1]=0x3286b0bfb3272b9b3004abe8d0c306f12470d77d, not stripped
> 
> ./x86_64-softmmu/qemu-system-x86_64 -net none -kernel /boot/vmlinuz
> 3.8.11-200.fc18.x86_64 -append console=ttyS0 -serial stdio
> 
> Which makes it all the way until it panics for lack of root filesystem.
> 
> Hopefully this is the problem fixed by Aurelien's tcg/optimize.c patch today
> (not yet applied to qemu head)?
> 

Yes, it is.

-- 
Aurelien Jarno                          GPG: 1024D/F1BCDB73
aurelien@aurel32.net                 http://www.aurel32.net

Re: [Qemu-devel] (i386) TCG is broken in 1.5-rc0 with guest kernel 3.2?

[Michael Tokarev]

09.05.2013 20:52, Aurelien Jarno wrote:
> On Thu, May 09, 2013 at 09:43:20AM -0700, Richard Henderson wrote:
>> On 05/08/2013 02:03 AM, Michael Tokarev wrote:
>>> I just tried 1.5-rc0 (32bit x86 userspace), and guest kernel
>>>> crashes in an interesting way right at init time.
>>>>
>>>>  qemu-system-x86_64 -net none -kernel /boot/vmlinuz-3.2-amd64 -append console=ttyS0 -serial file:serial
>>>>
>>
>> Irritatingly, this works for me, with the current fedora 18 kernel,
>>
>> x86_64-softmmu/qemu-system-x86_64: ELF 32-bit LSB shared object, Intel 80386,
>> version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32,
>> BuildID[sha1]=0x3286b0bfb3272b9b3004abe8d0c306f12470d77d, not stripped
>>
>> ./x86_64-softmmu/qemu-system-x86_64 -net none -kernel /boot/vmlinuz
>> 3.8.11-200.fc18.x86_64 -append console=ttyS0 -serial stdio
>>
>> Which makes it all the way until it panics for lack of root filesystem.
>>
>> Hopefully this is the problem fixed by Aurelien's tcg/optimize.c patch today
>> (not yet applied to qemu head)?
> 
> Yes, it is.

Yes, the patch by Aurelien fixes the reported problem for me.

Thanks!

/mjt

Re: [Qemu-devel] (i386) TCG is broken in 1.5-rc0 with guest kernel 3.2?

[Richard W.M. Jones]

On Wed, May 08, 2013 at 12:22:01PM +0400, Michael Tokarev wrote:
[...]

There is a long term (since 1.4), but different, bug in i386 / tcg
which causes processes to segfault very rarely and randomly.  In fact
it sounds a bit like what you describe here:

> This happened when CarlFK (Cc'd) mentioned that he has issues with
> installing ubuntu precise in a 1.5 virtual machine.  But in his
> case, the prob was different, -- his installer does not configure
> NIC properly due to dhclient segfaulting, which, again, does not
> happen with previous versions of qemu.  This may be related or
> not, I don't know yet.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)