[Qemu-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Qemu-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Fabio Fantoni]

Today I tried latest qemu 2.0 compiled from git (commit 
63678e17cf399ff81b93417fe7bee8d6ef6b6b1b) on this dom0:
Debian 7 (Wheezy) 64 bit with kernel from package 
linux-image-3.2.0-4-amd64 version 3.2.54-2 and all dependency packages 
for xen, spice and usb redirection.
Seabios 1.7.3-3, spice 0.12.4-0nocelt2 and usbredir 0.6-2 compiled from 
debian unstable sources.
The xen-unstable upstream commit is 
4787f667bcee205c56a27da59b766a53e1e929eb, plus these patches not upstream:
tools: various things just to build test
tools: Improve make debball
libxl: Add qxl vga interface support for upstream qemu
libxl: add basic spice support for pv domUs

Qemu crashes always on domU S.O. start, on both pv and hvm domUs.

Same dom0 with qemu 1.6 from xen-unstable repository used for some tests 
yesterday and was full working.
I also update seabios to 1.7.4-4 compiled from debian unstable sources 
but the problem persists.

I looked on dom0 logs, qemu logs and xl dmesg and I found only a qemu 
segfault related on each domU in dom0 syslog, for example the latest:
[  844.273170] qemu-system-i38[3545]: segfault at 8 ip 00007fa905dcc4c1 
sp 00007fff41220810 error 4 in qemu-system-i386[7fa905ad5000+598000]

If you need more informations, tests and/or logs tell me and I'll post them.

Thanks for any reply.

Re: [Qemu-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Laszlo Ersek]

On 04/01/14 17:01, Fabio Fantoni wrote:
> Today I tried latest qemu 2.0 compiled from git (commit
> 63678e17cf399ff81b93417fe7bee8d6ef6b6b1b) on this dom0:
> Debian 7 (Wheezy) 64 bit with kernel from package
> linux-image-3.2.0-4-amd64 version 3.2.54-2 and all dependency packages
> for xen, spice and usb redirection.
> Seabios 1.7.3-3, spice 0.12.4-0nocelt2 and usbredir 0.6-2 compiled from
> debian unstable sources.
> The xen-unstable upstream commit is
> 4787f667bcee205c56a27da59b766a53e1e929eb, plus these patches not upstream:
> tools: various things just to build test
> tools: Improve make debball
> libxl: Add qxl vga interface support for upstream qemu
> libxl: add basic spice support for pv domUs
> 
> Qemu crashes always on domU S.O. start, on both pv and hvm domUs.
> 
> Same dom0 with qemu 1.6 from xen-unstable repository used for some tests
> yesterday and was full working.
> I also update seabios to 1.7.4-4 compiled from debian unstable sources
> but the problem persists.
> 
> I looked on dom0 logs, qemu logs and xl dmesg and I found only a qemu
> segfault related on each domU in dom0 syslog, for example the latest:
> [  844.273170] qemu-system-i38[3545]: segfault at 8 ip 00007fa905dcc4c1
> sp 00007fff41220810 error 4 in qemu-system-i386[7fa905ad5000+598000]
> 
> If you need more informations, tests and/or logs tell me and I'll post
> them.

Whoever looks into this would be greatly helped:
- if you bisected the issue (between 1.6 and 2.0-rcX),
- if you posted qemu's backtrace at the sigsegv.

Laszlo

Re: [Qemu-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Fabio Fantoni]

Il 01/04/2014 18:24, Laszlo Ersek ha scritto:
> On 04/01/14 17:01, Fabio Fantoni wrote:
>> Today I tried latest qemu 2.0 compiled from git (commit
>> 63678e17cf399ff81b93417fe7bee8d6ef6b6b1b) on this dom0:
>> Debian 7 (Wheezy) 64 bit with kernel from package
>> linux-image-3.2.0-4-amd64 version 3.2.54-2 and all dependency packages
>> for xen, spice and usb redirection.
>> Seabios 1.7.3-3, spice 0.12.4-0nocelt2 and usbredir 0.6-2 compiled from
>> debian unstable sources.
>> The xen-unstable upstream commit is
>> 4787f667bcee205c56a27da59b766a53e1e929eb, plus these patches not upstream:
>> tools: various things just to build test
>> tools: Improve make debball
>> libxl: Add qxl vga interface support for upstream qemu
>> libxl: add basic spice support for pv domUs
>>
>> Qemu crashes always on domU S.O. start, on both pv and hvm domUs.
>>
>> Same dom0 with qemu 1.6 from xen-unstable repository used for some tests
>> yesterday and was full working.
>> I also update seabios to 1.7.4-4 compiled from debian unstable sources
>> but the problem persists.
>>
>> I looked on dom0 logs, qemu logs and xl dmesg and I found only a qemu
>> segfault related on each domU in dom0 syslog, for example the latest:
>> [  844.273170] qemu-system-i38[3545]: segfault at 8 ip 00007fa905dcc4c1
>> sp 00007fff41220810 error 4 in qemu-system-i386[7fa905ad5000+598000]
>>
>> If you need more informations, tests and/or logs tell me and I'll post
>> them.
> Whoever looks into this would be greatly helped:
> - if you bisected the issue (between 1.6 and 2.0-rcX),

I tried time ago qemu 1.7 and qemu 2.0 on start of development without 
problem on domUs start but I'll retry.

> - if you posted qemu's backtrace at the sigsegv.

I tried to use gdb following this old post:
https://lists.gnu.org/archive/html/qemu-devel/2011-12/msg02575.html
but with same changes:

/usr/lib/xen/bin# vi qemu-system-i386
#!/bin/sh
exec gdbserver 0.0.0.0:1234 /usr/lib/xen/bin/qemu-system-i386.bak "$@"

gdb /usr/lib/xen/bin/qemu-system-i386.bak
target remote localhost:1234

This command with gdb on qemu fails:
xl -vvv create /etc/xen/wheezy.cfg
...
libxl: error: libxl_dm.c:1378:device_model_spawn_outcome: domain 13 
device model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1207:domcreate_devmodel_started: device 
model did not start: -3
libxl: debug: libxl_dm.c:1485:kill_device_model: Device Model signaled
...

the dom0 syslog show segfault also in this case and the qemu log is 
different on first lines (probably for gdbserver):
less /var/log/xen/qemu-dm-wheezy.log
Process /usr/lib/xen/bin/qemu-system-i386.bak created; pid = 8238
Listening on port 1234
Remote debugging from host 127.0.0.1
xc: error: linux_gnttab_set_max_grants: ioctl SET_MAX_GRANTS failed (22 
= Invalid argument): Internal error
xen be: qdisk-51712: xc_gnttab_set_max_grants failed: Invalid argument


gdb on xl create show:
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007ffff7dddaf0 in ?? () from /lib64/ld-linux-x86-64.so.2
(gdb)

(gdb) bt full
#0  0x00007ffff7dddaf0 in ?? () from /lib64/ld-linux-x86-64.so.2
No symbol table info available.
#1  0x0000000000000013 in ?? ()
No symbol table info available.
#2  0x00007fffffffe871 in ?? ()
No symbol table info available.
#3  0x00007fffffffe897 in ?? ()
No symbol table info available.
#4  0x00007fffffffe8a2 in ?? ()
No symbol table info available.
#5  0x00007fffffffe8a5 in ?? ()
No symbol table info available.
#6  0x00007fffffffe8ae in ?? ()
No symbol table info available.
#7  0x00007fffffffe8ef in ?? ()
No symbol table info available.
#8  0x00007fffffffe8f4 in ?? ()
No symbol table info available.
#9  0x00007fffffffe913 in ?? ()
No symbol table info available.
#10 0x00007fffffffe91f in ?? ()
No symbol table info available.
#11 0x00007fffffffe92b in ?? ()
No symbol table info available.
#12 0x00007fffffffe931 in ?? ()
---Type <return> to continue, or q <return> to quit---

the qemu include debug and is not stripped:
file /usr/lib/xen/bin/qemu-system-i386.bak
/usr/lib/xen/bin/qemu-system-i386.bak: ELF 64-bit LSB shared object, 
x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for 
GNU/Linux 2.6.26, 
BuildID[sha1]=0x5aa043b5524d74d166ead62527343080384d586b, not stripped
and I also tried:
aptitude install libc6-dbg
but same result.

I not understand what I missed for correct xl create and/or gdb 
informations.
Can someone help me please?

Thanks for any reply

>
> Laszlo
>
>

Re: [Qemu-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Laszlo Ersek]

On 04/02/14 13:13, Fabio Fantoni wrote:
> Il 01/04/2014 18:24, Laszlo Ersek ha scritto:
>> On 04/01/14 17:01, Fabio Fantoni wrote:
>>> Today I tried latest qemu 2.0 compiled from git (commit
>>> 63678e17cf399ff81b93417fe7bee8d6ef6b6b1b) on this dom0:
>>> Debian 7 (Wheezy) 64 bit with kernel from package
>>> linux-image-3.2.0-4-amd64 version 3.2.54-2 and all dependency packages
>>> for xen, spice and usb redirection.
>>> Seabios 1.7.3-3, spice 0.12.4-0nocelt2 and usbredir 0.6-2 compiled from
>>> debian unstable sources.
>>> The xen-unstable upstream commit is
>>> 4787f667bcee205c56a27da59b766a53e1e929eb, plus these patches not
>>> upstream:
>>> tools: various things just to build test
>>> tools: Improve make debball
>>> libxl: Add qxl vga interface support for upstream qemu
>>> libxl: add basic spice support for pv domUs
>>>
>>> Qemu crashes always on domU S.O. start, on both pv and hvm domUs.

I may have misunderstood you (hence my gdb suggestion may not have been
appropriate) -- does the guest kernel crash *in* qemu, or does the qemu
host-side process crash? I understood your message to imply the latter.

>>>
>>> Same dom0 with qemu 1.6 from xen-unstable repository used for some tests
>>> yesterday and was full working.
>>> I also update seabios to 1.7.4-4 compiled from debian unstable sources
>>> but the problem persists.
>>>
>>> I looked on dom0 logs, qemu logs and xl dmesg and I found only a qemu
>>> segfault related on each domU in dom0 syslog, for example the latest:
>>> [  844.273170] qemu-system-i38[3545]: segfault at 8 ip 00007fa905dcc4c1
>>> sp 00007fff41220810 error 4 in qemu-system-i386[7fa905ad5000+598000]

Can you reproduce this qemu process SIGSEGV while running qemu in
(host-)gdb? Or else, can you save a coredump and look into it with gdb?

The steps you describe with gdbserver target the guest OS as debuggee. I
suggested that the host side qemu process be debugged (because that's
what crashes).

Laszlo

>>>
>>> If you need more informations, tests and/or logs tell me and I'll post
>>> them.
>> Whoever looks into this would be greatly helped:
>> - if you bisected the issue (between 1.6 and 2.0-rcX),
> 
> I tried time ago qemu 1.7 and qemu 2.0 on start of development without
> problem on domUs start but I'll retry.
> 
>> - if you posted qemu's backtrace at the sigsegv.
> 
> I tried to use gdb following this old post:
> https://lists.gnu.org/archive/html/qemu-devel/2011-12/msg02575.html
> but with same changes:
> 
> /usr/lib/xen/bin# vi qemu-system-i386
> #!/bin/sh
> exec gdbserver 0.0.0.0:1234 /usr/lib/xen/bin/qemu-system-i386.bak "$@"
> 
> gdb /usr/lib/xen/bin/qemu-system-i386.bak
> target remote localhost:1234
> 
> This command with gdb on qemu fails:
> xl -vvv create /etc/xen/wheezy.cfg
> ...
> libxl: error: libxl_dm.c:1378:device_model_spawn_outcome: domain 13
> device model: spawn failed (rc=-3)
> libxl: error: libxl_create.c:1207:domcreate_devmodel_started: device
> model did not start: -3
> libxl: debug: libxl_dm.c:1485:kill_device_model: Device Model signaled
> ...
> 
> the dom0 syslog show segfault also in this case and the qemu log is
> different on first lines (probably for gdbserver):
> less /var/log/xen/qemu-dm-wheezy.log
> Process /usr/lib/xen/bin/qemu-system-i386.bak created; pid = 8238
> Listening on port 1234
> Remote debugging from host 127.0.0.1
> xc: error: linux_gnttab_set_max_grants: ioctl SET_MAX_GRANTS failed (22
> = Invalid argument): Internal error
> xen be: qdisk-51712: xc_gnttab_set_max_grants failed: Invalid argument
> 
> 
> gdb on xl create show:
> (gdb) target remote localhost:1234
> Remote debugging using localhost:1234
> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> 0x00007ffff7dddaf0 in ?? () from /lib64/ld-linux-x86-64.so.2
> (gdb)
> 
> (gdb) bt full
> #0  0x00007ffff7dddaf0 in ?? () from /lib64/ld-linux-x86-64.so.2
> No symbol table info available.
> #1  0x0000000000000013 in ?? ()
> No symbol table info available.
> #2  0x00007fffffffe871 in ?? ()
> No symbol table info available.
> #3  0x00007fffffffe897 in ?? ()
> No symbol table info available.
> #4  0x00007fffffffe8a2 in ?? ()
> No symbol table info available.
> #5  0x00007fffffffe8a5 in ?? ()
> No symbol table info available.
> #6  0x00007fffffffe8ae in ?? ()
> No symbol table info available.
> #7  0x00007fffffffe8ef in ?? ()
> No symbol table info available.
> #8  0x00007fffffffe8f4 in ?? ()
> No symbol table info available.
> #9  0x00007fffffffe913 in ?? ()
> No symbol table info available.
> #10 0x00007fffffffe91f in ?? ()
> No symbol table info available.
> #11 0x00007fffffffe92b in ?? ()
> No symbol table info available.
> #12 0x00007fffffffe931 in ?? ()
> ---Type <return> to continue, or q <return> to quit---
> 
> the qemu include debug and is not stripped:
> file /usr/lib/xen/bin/qemu-system-i386.bak
> /usr/lib/xen/bin/qemu-system-i386.bak: ELF 64-bit LSB shared object,
> x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for
> GNU/Linux 2.6.26,
> BuildID[sha1]=0x5aa043b5524d74d166ead62527343080384d586b, not stripped
> and I also tried:
> aptitude install libc6-dbg
> but same result.
> 
> I not understand what I missed for correct xl create and/or gdb
> informations.
> Can someone help me please?
> 
> Thanks for any reply
> 
>>
>> Laszlo
>>
>>
> 

Re: [Qemu-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Fabio Fantoni]

Il 02/04/2014 15:31, Laszlo Ersek ha scritto:
> On 04/02/14 13:13, Fabio Fantoni wrote:
>> Il 01/04/2014 18:24, Laszlo Ersek ha scritto:
>>> On 04/01/14 17:01, Fabio Fantoni wrote:
>>>> Today I tried latest qemu 2.0 compiled from git (commit
>>>> 63678e17cf399ff81b93417fe7bee8d6ef6b6b1b) on this dom0:
>>>> Debian 7 (Wheezy) 64 bit with kernel from package
>>>> linux-image-3.2.0-4-amd64 version 3.2.54-2 and all dependency packages
>>>> for xen, spice and usb redirection.
>>>> Seabios 1.7.3-3, spice 0.12.4-0nocelt2 and usbredir 0.6-2 compiled from
>>>> debian unstable sources.
>>>> The xen-unstable upstream commit is
>>>> 4787f667bcee205c56a27da59b766a53e1e929eb, plus these patches not
>>>> upstream:
>>>> tools: various things just to build test
>>>> tools: Improve make debball
>>>> libxl: Add qxl vga interface support for upstream qemu
>>>> libxl: add basic spice support for pv domUs
>>>>
>>>> Qemu crashes always on domU S.O. start, on both pv and hvm domUs.
> I may have misunderstood you (hence my gdb suggestion may not have been
> appropriate) -- does the guest kernel crash *in* qemu, or does the qemu
> host-side process crash? I understood your message to imply the latter.
>
>>>> Same dom0 with qemu 1.6 from xen-unstable repository used for some tests
>>>> yesterday and was full working.
>>>> I also update seabios to 1.7.4-4 compiled from debian unstable sources
>>>> but the problem persists.
>>>>
>>>> I looked on dom0 logs, qemu logs and xl dmesg and I found only a qemu
>>>> segfault related on each domU in dom0 syslog, for example the latest:
>>>> [  844.273170] qemu-system-i38[3545]: segfault at 8 ip 00007fa905dcc4c1
>>>> sp 00007fff41220810 error 4 in qemu-system-i386[7fa905ad5000+598000]
> Can you reproduce this qemu process SIGSEGV while running qemu in
> (host-)gdb? Or else, can you save a coredump and look into it with gdb?
>
> The steps you describe with gdbserver target the guest OS as debuggee. I
> suggested that the host side qemu process be debugged (because that's
> what crashes).
>
> Laszlo

The gdbserver target in my previous test was 
/usr/lib/xen/bin/qemu-system-i386.bak on dom0 which is called by xl 
create and crashes with segfault. I don't understand how doing that 
would target the guest OS as debuggee.
Can you describe the steps to target the right process?
Thanks for any reply.

>
>>>> If you need more informations, tests and/or logs tell me and I'll post
>>>> them.
>>> Whoever looks into this would be greatly helped:
>>> - if you bisected the issue (between 1.6 and 2.0-rcX),
>> I tried time ago qemu 1.7 and qemu 2.0 on start of development without
>> problem on domUs start but I'll retry.
>>
>>> - if you posted qemu's backtrace at the sigsegv.
>> I tried to use gdb following this old post:
>> https://lists.gnu.org/archive/html/qemu-devel/2011-12/msg02575.html
>> but with same changes:
>>
>> /usr/lib/xen/bin# vi qemu-system-i386
>> #!/bin/sh
>> exec gdbserver 0.0.0.0:1234 /usr/lib/xen/bin/qemu-system-i386.bak "$@"
>>
>> gdb /usr/lib/xen/bin/qemu-system-i386.bak
>> target remote localhost:1234
>>
>> This command with gdb on qemu fails:
>> xl -vvv create /etc/xen/wheezy.cfg
>> ...
>> libxl: error: libxl_dm.c:1378:device_model_spawn_outcome: domain 13
>> device model: spawn failed (rc=-3)
>> libxl: error: libxl_create.c:1207:domcreate_devmodel_started: device
>> model did not start: -3
>> libxl: debug: libxl_dm.c:1485:kill_device_model: Device Model signaled
>> ...
>>
>> the dom0 syslog show segfault also in this case and the qemu log is
>> different on first lines (probably for gdbserver):
>> less /var/log/xen/qemu-dm-wheezy.log
>> Process /usr/lib/xen/bin/qemu-system-i386.bak created; pid = 8238
>> Listening on port 1234
>> Remote debugging from host 127.0.0.1
>> xc: error: linux_gnttab_set_max_grants: ioctl SET_MAX_GRANTS failed (22
>> = Invalid argument): Internal error
>> xen be: qdisk-51712: xc_gnttab_set_max_grants failed: Invalid argument
>>
>>
>> gdb on xl create show:
>> (gdb) target remote localhost:1234
>> Remote debugging using localhost:1234
>> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/ld-linux-x86-64.so.2
>> 0x00007ffff7dddaf0 in ?? () from /lib64/ld-linux-x86-64.so.2
>> (gdb)
>>
>> (gdb) bt full
>> #0  0x00007ffff7dddaf0 in ?? () from /lib64/ld-linux-x86-64.so.2
>> No symbol table info available.
>> #1  0x0000000000000013 in ?? ()
>> No symbol table info available.
>> #2  0x00007fffffffe871 in ?? ()
>> No symbol table info available.
>> #3  0x00007fffffffe897 in ?? ()
>> No symbol table info available.
>> #4  0x00007fffffffe8a2 in ?? ()
>> No symbol table info available.
>> #5  0x00007fffffffe8a5 in ?? ()
>> No symbol table info available.
>> #6  0x00007fffffffe8ae in ?? ()
>> No symbol table info available.
>> #7  0x00007fffffffe8ef in ?? ()
>> No symbol table info available.
>> #8  0x00007fffffffe8f4 in ?? ()
>> No symbol table info available.
>> #9  0x00007fffffffe913 in ?? ()
>> No symbol table info available.
>> #10 0x00007fffffffe91f in ?? ()
>> No symbol table info available.
>> #11 0x00007fffffffe92b in ?? ()
>> No symbol table info available.
>> #12 0x00007fffffffe931 in ?? ()
>> ---Type <return> to continue, or q <return> to quit---
>>
>> the qemu include debug and is not stripped:
>> file /usr/lib/xen/bin/qemu-system-i386.bak
>> /usr/lib/xen/bin/qemu-system-i386.bak: ELF 64-bit LSB shared object,
>> x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for
>> GNU/Linux 2.6.26,
>> BuildID[sha1]=0x5aa043b5524d74d166ead62527343080384d586b, not stripped
>> and I also tried:
>> aptitude install libc6-dbg
>> but same result.
>>
>> I not understand what I missed for correct xl create and/or gdb
>> informations.
>> Can someone help me please?
>>
>> Thanks for any reply
>>
>>> Laszlo
>>>
>>>

Re: [Qemu-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Anthony PERARD]

On Wed, Apr 02, 2014 at 01:13:31PM +0200, Fabio Fantoni wrote:
> >- if you posted qemu's backtrace at the sigsegv.
> 
> I tried to use gdb following this old post:
> https://lists.gnu.org/archive/html/qemu-devel/2011-12/msg02575.html
> but with same changes:
> 
> /usr/lib/xen/bin# vi qemu-system-i386
> #!/bin/sh
> exec gdbserver 0.0.0.0:1234 /usr/lib/xen/bin/qemu-system-i386.bak "$@"
> 
> gdb /usr/lib/xen/bin/qemu-system-i386.bak
> target remote localhost:1234
> 
> This command with gdb on qemu fails:
> xl -vvv create /etc/xen/wheezy.cfg
> ...
> libxl: error: libxl_dm.c:1378:device_model_spawn_outcome: domain 13 device
> model: spawn failed (rc=-3)
> libxl: error: libxl_create.c:1207:domcreate_devmodel_started: device model
> did not start: -3
> libxl: debug: libxl_dm.c:1485:kill_device_model: Device Model signaled
> ...
> 
> the dom0 syslog show segfault also in this case and the qemu log is
> different on first lines (probably for gdbserver):
> less /var/log/xen/qemu-dm-wheezy.log
> Process /usr/lib/xen/bin/qemu-system-i386.bak created; pid = 8238
> Listening on port 1234
> Remote debugging from host 127.0.0.1
> xc: error: linux_gnttab_set_max_grants: ioctl SET_MAX_GRANTS failed (22 =
> Invalid argument): Internal error
> xen be: qdisk-51712: xc_gnttab_set_max_grants failed: Invalid argument
> 
> 
> gdb on xl create show:
> (gdb) target remote localhost:1234
> Remote debugging using localhost:1234
> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> 0x00007ffff7dddaf0 in ?? () from /lib64/ld-linux-x86-64.so.2
> (gdb)
> 
> (gdb) bt full
> #0  0x00007ffff7dddaf0 in ?? () from /lib64/ld-linux-x86-64.so.2
> No symbol table info available.
> #1  0x0000000000000013 in ?? ()
> No symbol table info available.
> #2  0x00007fffffffe871 in ?? ()
> No symbol table info available.
> #3  0x00007fffffffe897 in ?? ()
> No symbol table info available.
> #4  0x00007fffffffe8a2 in ?? ()
> No symbol table info available.
> #5  0x00007fffffffe8a5 in ?? ()
> No symbol table info available.
> #6  0x00007fffffffe8ae in ?? ()
> No symbol table info available.
> #7  0x00007fffffffe8ef in ?? ()
> No symbol table info available.
> #8  0x00007fffffffe8f4 in ?? ()
> No symbol table info available.
> #9  0x00007fffffffe913 in ?? ()
> No symbol table info available.
> #10 0x00007fffffffe91f in ?? ()
> No symbol table info available.
> #11 0x00007fffffffe92b in ?? ()
> No symbol table info available.
> #12 0x00007fffffffe931 in ?? ()
> ---Type <return> to continue, or q <return> to quit---
> 
> the qemu include debug and is not stripped:
> file /usr/lib/xen/bin/qemu-system-i386.bak
> /usr/lib/xen/bin/qemu-system-i386.bak: ELF 64-bit LSB shared object, x86-64,
> version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux
> 2.6.26, BuildID[sha1]=0x5aa043b5524d74d166ead62527343080384d586b, not
> stripped
> and I also tried:
> aptitude install libc6-dbg
> but same result.
> 
> I not understand what I missed for correct xl create and/or gdb
> informations.
> Can someone help me please?

Using gdb on qemu is not easy, you need to be quick.

When you "xl create", you have about 10 second to start gdb on qemu,
otherwise, xl will fail to create a guest.

So I advise you to start "gdb /usr/lib/xen/bin/qemu-system-i386.bak" in
a second terminal, write "target remote localhost:1234" BUT not Enter,
to keep the command ready to run.
Then, start "xl create" and imediatly, run the "target" command in gdb
and "c" (for continue) which will start qemu.

That should help you reach the point where you can get the backtrace,
after the segv.

-- 
Anthony PERARD

Re: [Qemu-devel] [Xen-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Ian Campbell]

On Wed, 2014-04-02 at 17:03 +0100, Anthony PERARD wrote:
> When you "xl create", you have about 10 second to start gdb on qemu,
> otherwise, xl will fail to create a guest.

I'd be quite happy with adding a backdoor to libxl for this case. i.e.
looking for the existence LIBXL_DEBUG_QEMU in the environment and make
that timeout (practically) infinite.

Alternatively it could just print the qemu command line and wait for you
to run it yourself (assuming that fits into the control flow...)

Ian.

Re: [Qemu-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Fabio Fantoni]

Il 02/04/2014 18:03, Anthony PERARD ha scritto:
> On Wed, Apr 02, 2014 at 01:13:31PM +0200, Fabio Fantoni wrote:
>>> - if you posted qemu's backtrace at the sigsegv.
>> I tried to use gdb following this old post:
>> https://lists.gnu.org/archive/html/qemu-devel/2011-12/msg02575.html
>> but with same changes:
>>
>> /usr/lib/xen/bin# vi qemu-system-i386
>> #!/bin/sh
>> exec gdbserver 0.0.0.0:1234 /usr/lib/xen/bin/qemu-system-i386.bak "$@"
>>
>> gdb /usr/lib/xen/bin/qemu-system-i386.bak
>> target remote localhost:1234
>>
>> This command with gdb on qemu fails:
>> xl -vvv create /etc/xen/wheezy.cfg
>> ...
>> libxl: error: libxl_dm.c:1378:device_model_spawn_outcome: domain 13 device
>> model: spawn failed (rc=-3)
>> libxl: error: libxl_create.c:1207:domcreate_devmodel_started: device model
>> did not start: -3
>> libxl: debug: libxl_dm.c:1485:kill_device_model: Device Model signaled
>> ...
>>
>> the dom0 syslog show segfault also in this case and the qemu log is
>> different on first lines (probably for gdbserver):
>> less /var/log/xen/qemu-dm-wheezy.log
>> Process /usr/lib/xen/bin/qemu-system-i386.bak created; pid = 8238
>> Listening on port 1234
>> Remote debugging from host 127.0.0.1
>> xc: error: linux_gnttab_set_max_grants: ioctl SET_MAX_GRANTS failed (22 =
>> Invalid argument): Internal error
>> xen be: qdisk-51712: xc_gnttab_set_max_grants failed: Invalid argument
>>
>>
>> gdb on xl create show:
>> (gdb) target remote localhost:1234
>> Remote debugging using localhost:1234
>> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/ld-linux-x86-64.so.2
>> 0x00007ffff7dddaf0 in ?? () from /lib64/ld-linux-x86-64.so.2
>> (gdb)
>>
>> (gdb) bt full
>> #0  0x00007ffff7dddaf0 in ?? () from /lib64/ld-linux-x86-64.so.2
>> No symbol table info available.
>> #1  0x0000000000000013 in ?? ()
>> No symbol table info available.
>> #2  0x00007fffffffe871 in ?? ()
>> No symbol table info available.
>> #3  0x00007fffffffe897 in ?? ()
>> No symbol table info available.
>> #4  0x00007fffffffe8a2 in ?? ()
>> No symbol table info available.
>> #5  0x00007fffffffe8a5 in ?? ()
>> No symbol table info available.
>> #6  0x00007fffffffe8ae in ?? ()
>> No symbol table info available.
>> #7  0x00007fffffffe8ef in ?? ()
>> No symbol table info available.
>> #8  0x00007fffffffe8f4 in ?? ()
>> No symbol table info available.
>> #9  0x00007fffffffe913 in ?? ()
>> No symbol table info available.
>> #10 0x00007fffffffe91f in ?? ()
>> No symbol table info available.
>> #11 0x00007fffffffe92b in ?? ()
>> No symbol table info available.
>> #12 0x00007fffffffe931 in ?? ()
>> ---Type <return> to continue, or q <return> to quit---
>>
>> the qemu include debug and is not stripped:
>> file /usr/lib/xen/bin/qemu-system-i386.bak
>> /usr/lib/xen/bin/qemu-system-i386.bak: ELF 64-bit LSB shared object, x86-64,
>> version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux
>> 2.6.26, BuildID[sha1]=0x5aa043b5524d74d166ead62527343080384d586b, not
>> stripped
>> and I also tried:
>> aptitude install libc6-dbg
>> but same result.
>>
>> I not understand what I missed for correct xl create and/or gdb
>> informations.
>> Can someone help me please?
> Using gdb on qemu is not easy, you need to be quick.
>
> When you "xl create", you have about 10 second to start gdb on qemu,
> otherwise, xl will fail to create a guest.
>
> So I advise you to start "gdb /usr/lib/xen/bin/qemu-system-i386.bak" in
> a second terminal, write "target remote localhost:1234" BUT not Enter,
> to keep the command ready to run.
> Then, start "xl create" and imediatly, run the "target" command in gdb
> and "c" (for continue) which will start qemu.
>
> That should help you reach the point where you can get the backtrace,
> after the segv.
>

Thanks for your reply.
I following your istructions, after "c" xl create finish.
gdb show:
> (gdb) C
> Continuing.
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x000055555584b4c1 in qemu_input_event_send (src=0x0, evt=0x55555630a420)
>     at ui/input.c:146
> 146         s->handler->event(s->dev, src, evt);

> (gdb) bt full
> #0  0x000055555584b4c1 in qemu_input_event_send (src=0x0, 
> evt=0x55555630a420)
>     at ui/input.c:146
>         s = 0x0
> #1  0x000055555584baef in qemu_input_queue_rel (src=0x0, 
> axis=INPUT_AXIS_X,
>     value=1) at ui/input.c:272
>         evt = 0x55555630a420
> #2  0x0000555555871043 in pointer_event (vs=0x55555630a4c0, 
> button_mask=0,
>     x=478, y=186) at ui/vnc.c:1531
>         bmap = {1, 2, 4, 8, 16}
>         con = 0x0
>         width = 640
>         height = 480
> #3  0x0000555555872c00 in protocol_client_msg (vs=0x55555630a4c0,
>     data=0x5555562fe420 "\005", len=6) at ui/vnc.c:2139
>         i = 1446028480
>         limit = 22063
>         vd = 0x7ffff7eb6010
> #4  0x0000555555870861 in vnc_client_read (opaque=0x55555630a4c0)
>     at ui/vnc.c:1389
>         len = 6
>         ret = 6
>         vs = 0x55555630a4c0
>         ret = 6
> #5  0x00005555557cd5cc in qemu_iohandler_poll (pollfds=0x5555562e6a00, 
> ret=1)
>     at iohandler.c:143
> ---Type <return> to continue, or q <return> to quit---
>         revents = 1
>         pioh = 0x5555562fa780
>         ioh = 0x5555562f5560
> #6  0x00005555557ce678 in main_loop_wait (nonblocking=0) at 
> main-loop.c:485
>         ret = 1
>         timeout = 4294967295
>         timeout_ns = 79570735
> #7  0x000055555587a070 in main_loop () at vl.c:2051
>         nonblocking = false
>         last_io = 1
> #8  0x0000555555881a4a in main (argc=19, argv=0x7fffffffe5f8,
>     envp=0x7fffffffe698) at vl.c:4507
>         i = 64
>         snapshot = 0
>         linux_boot = 0
>         icount_option = 0x0
>         initrd_filename = 0x0
>         kernel_filename = 0x0
>         kernel_cmdline = 0x555555a1a464 ""
>         boot_order = 0x0
>         ds = 0x5555562f34a0
>         cyls = 0
>         heads = 0
>         secs = 0
>         translation = 0
> ---Type <return> to continue, or q <return> to quit---
>         hda_opts = 0x0
>         opts = 0x0
>         machine_opts = 0x5555562e6600
>         olist = 0x555555dffe00
>         optind = 19
>         optarg = 0x7fffffffe969 "1025"
>         loadvm = 0x0
>         machine_class = 0x5555562d6d50
>         machine = 0x555555e0d2c0
>         cpu_model = 0x0
>         vga_model = 0x0
>         qtest_chrdev = 0x0
>         qtest_log = 0x0
>         pid_file = 0x0
>         incoming = 0x0
>         show_vnc_port = 0
>         defconfig = true
>         userconfig = true
>         log_mask = 0x0
>         log_file = 0x0
>         mem_trace = {malloc = 0x55555587d902 <malloc_and_trace>,
>           realloc = 0x55555587d95a <realloc_and_trace>,
>           free = 0x55555587d9c1 <free_and_trace>, calloc = 0, 
> try_malloc = 0,
>           try_realloc = 0}
>         trace_events = 0x0
> ---Type <return> to continue, or q <return> to quit---
>         trace_file = 0x0
>         __func__ = "main"
>         args = {machine = 0x555555e0d2c0, ram_size = 1074790400,
>           boot_order = 0x0, kernel_filename = 0x0,
>           kernel_cmdline = 0x555555a1a464 "", initrd_filename = 0x0,
>           cpu_model = 0x0}
> (gdb) 

Seems that do segfault when I connect to vnc or spice, in the test of 
this backtrace after connect to vnc, spice and other things of my 
patches are disabled, so do not think it is a problem caused by my patches.

Re: [Qemu-devel] [Xen-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Ian Campbell]

On Thu, 2014-04-03 at 10:15 +0200, Fabio Fantoni wrote:
> Seems that do segfault when I connect to vnc or spice, in the test of 
> this backtrace after connect to vnc, spice and other things of my 
> patches are disabled, so do not think it is a problem caused by my patches.

The last spice patch of yours I saw was incorrectly accessing the wrong
half of various unions which is liable to cause all sorts of corruption
or strange behaviour. Please can you reproduce this issue without any
patches applied.

Ian.

Re: [Qemu-devel] [Xen-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Fabio Fantoni]

Il 03/04/2014 10:45, Ian Campbell ha scritto:
> On Thu, 2014-04-03 at 10:15 +0200, Fabio Fantoni wrote:
>> Seems that do segfault when I connect to vnc or spice, in the test of
>> this backtrace after connect to vnc, spice and other things of my
>> patches are disabled, so do not think it is a problem caused by my patches.
> The last spice patch of yours I saw was incorrectly accessing the wrong
> half of various unions which is liable to cause all sorts of corruption
> or strange behaviour. Please can you reproduce this issue without any
> patches applied.
>
> Ian.
>

After saw the full backtrace I saw on qemu git recent patches with fix 
on input, than I tried to update qemu to latest commit 
(82c6f513735297ad76acaaf2e87f0c5a0b3647a7) and now the segfault seems 
solve, I did some fast test with vnc and spice on same pv domUs without 
qemu crashes.
About libxl patch of spice support for pv domUs I'll improve it 
following your reply and also try to find more details about pointer not 
visible but working with spice on pv domUs.
Thanks to all for your help.

Re: [Qemu-devel] [Xen-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Fabio Fantoni]

[-- Attachment #1: Type: text/plain, Size: 5269 bytes --]

Il 03/04/2014 12:13, Fabio Fantoni ha scritto:
> Il 03/04/2014 10:45, Ian Campbell ha scritto:
>> On Thu, 2014-04-03 at 10:15 +0200, Fabio Fantoni wrote:
>>> Seems that do segfault when I connect to vnc or spice, in the test of
>>> this backtrace after connect to vnc, spice and other things of my
>>> patches are disabled, so do not think it is a problem caused by my 
>>> patches.
>> The last spice patch of yours I saw was incorrectly accessing the wrong
>> half of various unions which is liable to cause all sorts of corruption
>> or strange behaviour. Please can you reproduce this issue without any
>> patches applied.
>>
>> Ian.
>>
>
> After saw the full backtrace I saw on qemu git recent patches with fix 
> on input, than I tried to update qemu to latest commit 
> (82c6f513735297ad76acaaf2e87f0c5a0b3647a7) and now the segfault seems 
> solve, I did some fast test with vnc and spice on same pv domUs 
> without qemu crashes.
> About libxl patch of spice support for pv domUs I'll improve it 
> following your reply and also try to find more details about pointer 
> not visible but working with spice on pv domUs.
> Thanks to all for your help.


Today I did some tests also with hvm and spice and I found another 
segfault with different backtrace to solve:
> (gdb) c
> Continuing.
>
> *Program received signal SIGSEGV, Segmentation fault.**
> **0x0000555555855d30 in interface_client_monitors_config 
> (sin=0x5555563b0260, **
> **    mc=0x0) at ui/spice-display.c:557**
> **557         if (mc->num_of_monitors > 0) {*

> (gdb) bt full
> #0  0x0000555555855d30 in interface_client_monitors_config (
>     sin=0x5555563b0260, mc=0x0) at ui/spice-display.c:557
>         ssd = 0x5555563b0210
>         info = {xoff = 0, yoff = 0, width = 0, height = 0}
>         rc = 32767
>         __func__ = "interface_client_monitors_config"
> #1  0x00007ffff4af5113 in ?? ()
>    from /usr/lib/x86_64-linux-gnu/libspice-server.so.1
> No symbol table info available.
> #2  0x00007ffff4ad87f5 in ?? ()
>    from /usr/lib/x86_64-linux-gnu/libspice-server.so.1
> No symbol table info available.
> #3  0x00007ffff4b1af76 in ?? ()
>    from /usr/lib/x86_64-linux-gnu/libspice-server.so.1
> No symbol table info available.
> #4  0x00007ffff4ae989a in ?? ()
>    from /usr/lib/x86_64-linux-gnu/libspice-server.so.1
> No symbol table info available.
> #5  0x00007ffff4aee470 in ?? ()
>    from /usr/lib/x86_64-linux-gnu/libspice-server.so.1
> No symbol table info available.
> #6  0x00007ffff4af0d8c in ?? ()
>    from /usr/lib/x86_64-linux-gnu/libspice-server.so.1
> No symbol table info available.
> #7  0x0000555555851f82 in watch_read (opaque=0x55555666a8d0)
> ---Type <return> to continue, or q <return> to quit---
>     at ui/spice-core.c:101
>         watch = 0x55555666a8d0
> #8  0x00005555557ce1f8 in qemu_iohandler_poll (pollfds=0x5555562e8e00, 
> ret=2)
>     at iohandler.c:143
>         revents = 1
>         pioh = 0x55555634e080
>         ioh = 0x55555666adb0
> #9  0x00005555557cf2a4 in main_loop_wait (nonblocking=0) at 
> main-loop.c:485
>         ret = 2
>         timeout = 4294967295
>         timeout_ns = 25664603
> #10 0x000055555587acd8 in main_loop () at vl.c:2051
>         nonblocking = false
>         last_io = 3
> #11 0x00005555558826b2 in main (argc=36, argv=0x7fffffffe368,
>     envp=0x7fffffffe490) at vl.c:4507
>         i = 64
>         snapshot = 0
>         linux_boot = 0
>         icount_option = 0x0
>         initrd_filename = 0x0
>         kernel_filename = 0x0
>         kernel_cmdline = 0x555555a1b5c4 ""
>         boot_order = 0x5555562e7ee0 "dc"
>         ds = 0x5555563d8fd0
> ---Type <return> to continue, or q <return> to quit---
>         cyls = 0
>         heads = 0
>         secs = 0
>         translation = 0
>         hda_opts = 0x0
>         opts = 0x5555562e7e30
>         machine_opts = 0x5555562e84b0
>         olist = 0x555555e00e00
>         optind = 36
>         optarg = 0x7fffffffe923 
> "if=ide,index=1,media=cdrom,cache=writeback,id=ide-832"
>         loadvm = 0x0
>         machine_class = 0x5555562e02a0
>         machine = 0x555555e067e0
>         cpu_model = 0x0
>         vga_model = 0x0
>         qtest_chrdev = 0x0
>         qtest_log = 0x0
>         pid_file = 0x0
>         incoming = 0x0
>         show_vnc_port = 0
>         defconfig = true
>         userconfig = true
>         log_mask = 0x0
>         log_file = 0x0
> ---Type <return> to continue, or q <return> to quit---
>         mem_trace = {malloc = 0x55555587e56a <malloc_and_trace>,
>           realloc = 0x55555587e5c2 <realloc_and_trace>,
>           free = 0x55555587e629 <free_and_trace>, calloc = 0, 
> try_malloc = 0,
>           try_realloc = 0}
>         trace_events = 0x0
>         trace_file = 0x0
>         __func__ = "main"
>         args = {machine = 0x555555e067e0, ram_size = 2130706432,
>           boot_order = 0x5555562e7ee0 "dc", kernel_filename = 0x0,
>           kernel_cmdline = 0x555555a1b5c4 "", initrd_filename = 0x0,
>           cpu_model = 0x0}
> (gdb)

qemu from source git/master commit 82c6f513735297ad76acaaf2e87f0c5a0b3647a7
spice server packages is version 0.12.4-0nocelt2 recompiled from debian 
unstable source.

If you need more informations/tests tell me and I'll post them.

Thanks for any reply.

[-- Attachment #2: Type: text/html, Size: 7514 bytes --]

Re: [Qemu-devel] [Spice-devel] [Xen-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Christophe Fergeau]

[-- Attachment #1: Type: text/plain, Size: 1013 bytes --]

On Mon, Apr 07, 2014 at 11:59:06AM +0200, Fabio Fantoni wrote:
> 
> Today I did some tests also with hvm and spice and I found another
> segfault with different backtrace to solve:
> >(gdb) c
> >Continuing.
> >
> >*Program received signal SIGSEGV, Segmentation fault.**
> >**0x0000555555855d30 in interface_client_monitors_config
> >(sin=0x5555563b0260, **
> >**    mc=0x0) at ui/spice-display.c:557**
> >**557         if (mc->num_of_monitors > 0) {*
> 
> >(gdb) bt full
> >#0  0x0000555555855d30 in interface_client_monitors_config (
> >    sin=0x5555563b0260, mc=0x0) at ui/spice-display.c:557
> >        ssd = 0x5555563b0210
> >        info = {xoff = 0, yoff = 0, width = 0, height = 0}
> >        rc = 32767
> >        __func__ = "interface_client_monitors_config"
> >#1  0x00007ffff4af5113 in ?? ()
> >   from /usr/lib/x86_64-linux-gnu/libspice-server.so.1
> >No symbol table info available.

A backtrace with spice-server debugging symbols installed would be helpful.

Christophe

[-- Attachment #2: Type: application/pgp-signature, Size: 181 bytes --]

Re: [Qemu-devel] [Spice-devel] [Xen-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Fabio Fantoni]

Il 07/04/2014 12:20, Christophe Fergeau ha scritto:
> On Mon, Apr 07, 2014 at 11:59:06AM +0200, Fabio Fantoni wrote:
>> Today I did some tests also with hvm and spice and I found another
>> segfault with different backtrace to solve:
>>> (gdb) c
>>> Continuing.
>>>
>>> *Program received signal SIGSEGV, Segmentation fault.**
>>> **0x0000555555855d30 in interface_client_monitors_config
>>> (sin=0x5555563b0260, **
>>> **    mc=0x0) at ui/spice-display.c:557**
>>> **557         if (mc->num_of_monitors > 0) {*
>>> (gdb) bt full
>>> #0  0x0000555555855d30 in interface_client_monitors_config (
>>>     sin=0x5555563b0260, mc=0x0) at ui/spice-display.c:557
>>>         ssd = 0x5555563b0210
>>>         info = {xoff = 0, yoff = 0, width = 0, height = 0}
>>>         rc = 32767
>>>         __func__ = "interface_client_monitors_config"
>>> #1  0x00007ffff4af5113 in ?? ()
>>>    from /usr/lib/x86_64-linux-gnu/libspice-server.so.1
>>> No symbol table info available.
> A backtrace with spice-server debugging symbols installed would be helpful.
>
> Christophe

Sorry, the -dbg for spice-server on official debian packages is missing, 
now I created and installed also the -dbg package and this is the new 
backtrace:

> (gdb) c
> Continuing.
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000555555855d30 in interface_client_monitors_config 
> (sin=0x5555563b0260,
>     mc=0x0) at ui/spice-display.c:557
> 557         if (mc->num_of_monitors > 0) {
> (gdb) bt full
> #0  0x0000555555855d30 in interface_client_monitors_config (
>     sin=0x5555563b0260, mc=0x0) at ui/spice-display.c:557
>         ssd = 0x5555563b0210
>         info = {xoff = 0, yoff = 0, width = 0, height = 0}
>         rc = 32767
>         __func__ = "interface_client_monitors_config"
> #1  0x00007ffff4af5113 in red_dispatcher_use_client_monitors_config ()
>     at red_dispatcher.c:318
>         now = 0x5555563b0300
> #2  0x00007ffff4ad87f5 in agent_msg_filter_process_data (
>     filter=filter@entry=0x5555562eb0c4,
>     data=data@entry=0x7fffe0280128 "\001", len=328, len@entry=348)
>     at agent-msg-filter.c:95
>         msg_header = {protocol = <optimized out>, type = <optimized out>,
>           opaque = <optimized out>, size = 328,
>           data = 0x831fd4 <Address 0x831fd4 out of bounds>}
>         __FUNCTION__ = "agent_msg_filter_process_data"
> #3  0x00007ffff4b1af76 in reds_on_main_agent_data (mcc=0x555556326e70,
>     message=0x7fffe0280128, size=348) at reds.c:1117
>         dev_state = 0x5555562eb0a8
>         header = <optimized out>
>         res = <optimized out>
>         __FUNCTION__ = "reds_on_main_agent_data"
> #4  0x00007ffff4ae989a in main_channel_handle_parsed (rcc=0x555556326e70,
>     size=<optimized out>, type=<optimized out>, message=0x7fffe0280128)
> ---Type <return> to continue, or q <return> to quit---
>     at main_channel.c:911
>         main_chan = 0x5555562ef2b0
>         mcc = 0x555556326e70
>         __FUNCTION__ = "main_channel_handle_parsed"
> #5  0x00007ffff4aee470 in red_peer_handle_incoming 
> (handler=0x55555632af80,
>     stream=0x5555565adba0) at red_channel.c:287
>         ret_handle = <optimized out>
>         bytes_read = <optimized out>
>         msg_type = 107
>         parsed = <optimized out>
>         parsed_free = 0x7ffff4ba8620 <nofree>
>         msg_size = 348
> #6  red_channel_client_receive (rcc=rcc@entry=0x555556326e70)
>     at red_channel.c:309
> No locals.
> #7  0x00007ffff4af0d8c in red_channel_client_event (fd=<optimized out>,
>     event=<optimized out>, data=0x555556326e70) at red_channel.c:1435
>         rcc = 0x555556326e70
> #8  0x0000555555851f82 in watch_read (opaque=0x55555666e0a0)
>     at ui/spice-core.c:101
>         watch = 0x55555666e0a0
> #9  0x00005555557ce1f8 in qemu_iohandler_poll (pollfds=0x5555562e8e00, 
> ret=1)
>     at iohandler.c:143
>         revents = 1
>         pioh = 0x55555634e080
> ---Type <return> to continue, or q <return> to quit---
>         ioh = 0x55555632fa30
> #10 0x00005555557cf2a4 in main_loop_wait (nonblocking=0) at 
> main-loop.c:485
>         ret = 1
>         timeout = 4294967295
>         timeout_ns = 4237075
> #11 0x000055555587acd8 in main_loop () at vl.c:2051
>         nonblocking = false
>         last_io = 1
> #12 0x00005555558826b2 in main (argc=36, argv=0x7fffffffe358,
>     envp=0x7fffffffe480) at vl.c:4507
>         i = 64
>         snapshot = 0
>         linux_boot = 0
>         icount_option = 0x0
>         initrd_filename = 0x0
>         kernel_filename = 0x0
>         kernel_cmdline = 0x555555a1b5c4 ""
>         boot_order = 0x5555562e7ee0 "dc"
>         ds = 0x5555563d8fd0
>         cyls = 0
>         heads = 0
>         secs = 0
>         translation = 0
>         hda_opts = 0x0
>         opts = 0x5555562e7e30
> ---Type <return> to continue, or q <return> to quit---
>         machine_opts = 0x5555562e84b0
>         olist = 0x555555e00e00
>         optind = 36
>         optarg = 0x7fffffffe915 
> "if=ide,index=1,media=cdrom,cache=writeback,id=ide-832"
>         loadvm = 0x0
>         machine_class = 0x5555562e02a0
>         machine = 0x555555e067e0
>         cpu_model = 0x0
>         vga_model = 0x0
>         qtest_chrdev = 0x0
>         qtest_log = 0x0
>         pid_file = 0x0
>         incoming = 0x0
>         show_vnc_port = 0
>         defconfig = true
>         userconfig = true
>         log_mask = 0x0
>         log_file = 0x0
>         mem_trace = {malloc = 0x55555587e56a <malloc_and_trace>,
>           realloc = 0x55555587e5c2 <realloc_and_trace>,
>           free = 0x55555587e629 <free_and_trace>, calloc = 0, 
> try_malloc = 0,
>           try_realloc = 0}
>         trace_events = 0x0
>         trace_file = 0x0
> ---Type <return> to continue, or q <return> to quit---
>         __func__ = "main"
>         args = {machine = 0x555555e067e0, ram_size = 2130706432,
>           boot_order = 0x5555562e7ee0 "dc", kernel_filename = 0x0,
>           kernel_cmdline = 0x555555a1b5c4 "", initrd_filename = 0x0,
>           cpu_model = 0x0}
> (gdb)

If you need more informations/tests tell me and I'll post them.

Thanks for any reply.

Re: [Qemu-devel] [Spice-devel] [Xen-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Fabio Fantoni]

Il 07/04/2014 15:19, Fabio Fantoni ha scritto:
> Il 07/04/2014 12:20, Christophe Fergeau ha scritto:
>> On Mon, Apr 07, 2014 at 11:59:06AM +0200, Fabio Fantoni wrote:
>>> Today I did some tests also with hvm and spice and I found another
>>> segfault with different backtrace to solve:
>>>> (gdb) c
>>>> Continuing.
>>>>
>>>> *Program received signal SIGSEGV, Segmentation fault.**
>>>> **0x0000555555855d30 in interface_client_monitors_config
>>>> (sin=0x5555563b0260, **
>>>> **    mc=0x0) at ui/spice-display.c:557**
>>>> **557         if (mc->num_of_monitors > 0) {*
>>>> (gdb) bt full
>>>> #0  0x0000555555855d30 in interface_client_monitors_config (
>>>>     sin=0x5555563b0260, mc=0x0) at ui/spice-display.c:557
>>>>         ssd = 0x5555563b0210
>>>>         info = {xoff = 0, yoff = 0, width = 0, height = 0}
>>>>         rc = 32767
>>>>         __func__ = "interface_client_monitors_config"
>>>> #1  0x00007ffff4af5113 in ?? ()
>>>>    from /usr/lib/x86_64-linux-gnu/libspice-server.so.1
>>>> No symbol table info available.
>> A backtrace with spice-server debugging symbols installed would be 
>> helpful.
>>
>> Christophe
>
> Sorry, the -dbg for spice-server on official debian packages is 
> missing, now I created and installed also the -dbg package and this is 
> the new backtrace:
>
>> (gdb) c
>> Continuing.
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x0000555555855d30 in interface_client_monitors_config 
>> (sin=0x5555563b0260,
>>     mc=0x0) at ui/spice-display.c:557
>> 557         if (mc->num_of_monitors > 0) {
>> (gdb) bt full
>> #0  0x0000555555855d30 in interface_client_monitors_config (
>>     sin=0x5555563b0260, mc=0x0) at ui/spice-display.c:557
>>         ssd = 0x5555563b0210
>>         info = {xoff = 0, yoff = 0, width = 0, height = 0}
>>         rc = 32767
>>         __func__ = "interface_client_monitors_config"
>> #1  0x00007ffff4af5113 in red_dispatcher_use_client_monitors_config ()
>>     at red_dispatcher.c:318
>>         now = 0x5555563b0300
>> #2  0x00007ffff4ad87f5 in agent_msg_filter_process_data (
>>     filter=filter@entry=0x5555562eb0c4,
>>     data=data@entry=0x7fffe0280128 "\001", len=328, len@entry=348)
>>     at agent-msg-filter.c:95
>>         msg_header = {protocol = <optimized out>, type = <optimized 
>> out>,
>>           opaque = <optimized out>, size = 328,
>>           data = 0x831fd4 <Address 0x831fd4 out of bounds>}
>>         __FUNCTION__ = "agent_msg_filter_process_data"
>> #3  0x00007ffff4b1af76 in reds_on_main_agent_data (mcc=0x555556326e70,
>>     message=0x7fffe0280128, size=348) at reds.c:1117
>>         dev_state = 0x5555562eb0a8
>>         header = <optimized out>
>>         res = <optimized out>
>>         __FUNCTION__ = "reds_on_main_agent_data"
>> #4  0x00007ffff4ae989a in main_channel_handle_parsed 
>> (rcc=0x555556326e70,
>>     size=<optimized out>, type=<optimized out>, message=0x7fffe0280128)
>> ---Type <return> to continue, or q <return> to quit---
>>     at main_channel.c:911
>>         main_chan = 0x5555562ef2b0
>>         mcc = 0x555556326e70
>>         __FUNCTION__ = "main_channel_handle_parsed"
>> #5  0x00007ffff4aee470 in red_peer_handle_incoming 
>> (handler=0x55555632af80,
>>     stream=0x5555565adba0) at red_channel.c:287
>>         ret_handle = <optimized out>
>>         bytes_read = <optimized out>
>>         msg_type = 107
>>         parsed = <optimized out>
>>         parsed_free = 0x7ffff4ba8620 <nofree>
>>         msg_size = 348
>> #6  red_channel_client_receive (rcc=rcc@entry=0x555556326e70)
>>     at red_channel.c:309
>> No locals.
>> #7  0x00007ffff4af0d8c in red_channel_client_event (fd=<optimized out>,
>>     event=<optimized out>, data=0x555556326e70) at red_channel.c:1435
>>         rcc = 0x555556326e70
>> #8  0x0000555555851f82 in watch_read (opaque=0x55555666e0a0)
>>     at ui/spice-core.c:101
>>         watch = 0x55555666e0a0
>> #9  0x00005555557ce1f8 in qemu_iohandler_poll 
>> (pollfds=0x5555562e8e00, ret=1)
>>     at iohandler.c:143
>>         revents = 1
>>         pioh = 0x55555634e080
>> ---Type <return> to continue, or q <return> to quit---
>>         ioh = 0x55555632fa30
>> #10 0x00005555557cf2a4 in main_loop_wait (nonblocking=0) at 
>> main-loop.c:485
>>         ret = 1
>>         timeout = 4294967295
>>         timeout_ns = 4237075
>> #11 0x000055555587acd8 in main_loop () at vl.c:2051
>>         nonblocking = false
>>         last_io = 1
>> #12 0x00005555558826b2 in main (argc=36, argv=0x7fffffffe358,
>>     envp=0x7fffffffe480) at vl.c:4507
>>         i = 64
>>         snapshot = 0
>>         linux_boot = 0
>>         icount_option = 0x0
>>         initrd_filename = 0x0
>>         kernel_filename = 0x0
>>         kernel_cmdline = 0x555555a1b5c4 ""
>>         boot_order = 0x5555562e7ee0 "dc"
>>         ds = 0x5555563d8fd0
>>         cyls = 0
>>         heads = 0
>>         secs = 0
>>         translation = 0
>>         hda_opts = 0x0
>>         opts = 0x5555562e7e30
>> ---Type <return> to continue, or q <return> to quit---
>>         machine_opts = 0x5555562e84b0
>>         olist = 0x555555e00e00
>>         optind = 36
>>         optarg = 0x7fffffffe915 
>> "if=ide,index=1,media=cdrom,cache=writeback,id=ide-832"
>>         loadvm = 0x0
>>         machine_class = 0x5555562e02a0
>>         machine = 0x555555e067e0
>>         cpu_model = 0x0
>>         vga_model = 0x0
>>         qtest_chrdev = 0x0
>>         qtest_log = 0x0
>>         pid_file = 0x0
>>         incoming = 0x0
>>         show_vnc_port = 0
>>         defconfig = true
>>         userconfig = true
>>         log_mask = 0x0
>>         log_file = 0x0
>>         mem_trace = {malloc = 0x55555587e56a <malloc_and_trace>,
>>           realloc = 0x55555587e5c2 <realloc_and_trace>,
>>           free = 0x55555587e629 <free_and_trace>, calloc = 0, 
>> try_malloc = 0,
>>           try_realloc = 0}
>>         trace_events = 0x0
>>         trace_file = 0x0
>> ---Type <return> to continue, or q <return> to quit---
>>         __func__ = "main"
>>         args = {machine = 0x555555e067e0, ram_size = 2130706432,
>>           boot_order = 0x5555562e7ee0 "dc", kernel_filename = 0x0,
>>           kernel_cmdline = 0x555555a1b5c4 "", initrd_filename = 0x0,
>>           cpu_model = 0x0}
>> (gdb)
>
> If you need more informations/tests tell me and I'll post them.
>
> Thanks for any reply.

I see this patch:
http://git.qemu.org/?p=qemu.git;a=commit;h=dc491cfc14074064ed54a872b62cce6ca1330644

I tested it and segfault didn't happen anymore.
Thanks to Gerd Hoffmann for the fast fix.

Re: [Qemu-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[John Baboval]

Can you post your options to configure? The tip seems to be working here...

On 04/01/2014 11:01 AM, Fabio Fantoni wrote:
> Today I tried latest qemu 2.0 compiled from git (commit 
> 63678e17cf399ff81b93417fe7bee8d6ef6b6b1b) on this dom0:
> Debian 7 (Wheezy) 64 bit with kernel from package 
> linux-image-3.2.0-4-amd64 version 3.2.54-2 and all dependency packages 
> for xen, spice and usb redirection.
> Seabios 1.7.3-3, spice 0.12.4-0nocelt2 and usbredir 0.6-2 compiled 
> from debian unstable sources.
> The xen-unstable upstream commit is 
> 4787f667bcee205c56a27da59b766a53e1e929eb, plus these patches not 
> upstream:
> tools: various things just to build test
> tools: Improve make debball
> libxl: Add qxl vga interface support for upstream qemu
> libxl: add basic spice support for pv domUs
>
> Qemu crashes always on domU S.O. start, on both pv and hvm domUs.
>
> Same dom0 with qemu 1.6 from xen-unstable repository used for some 
> tests yesterday and was full working.
> I also update seabios to 1.7.4-4 compiled from debian unstable sources 
> but the problem persists.
>
> I looked on dom0 logs, qemu logs and xl dmesg and I found only a qemu 
> segfault related on each domU in dom0 syslog, for example the latest:
> [  844.273170] qemu-system-i38[3545]: segfault at 8 ip 
> 00007fa905dcc4c1 sp 00007fff41220810 error 4 in 
> qemu-system-i386[7fa905ad5000+598000]
>
> If you need more informations, tests and/or logs tell me and I'll post 
> them.
>
> Thanks for any reply.
>
>
>

Re: [Qemu-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Anthony PERARD]

On Tue, Apr 01, 2014 at 05:01:18PM +0200, Fabio Fantoni wrote:
> Today I tried latest qemu 2.0 compiled from git (commit
> 63678e17cf399ff81b93417fe7bee8d6ef6b6b1b) on this dom0:
> Debian 7 (Wheezy) 64 bit with kernel from package linux-image-3.2.0-4-amd64
> version 3.2.54-2 and all dependency packages for xen, spice and usb
> redirection.
> Seabios 1.7.3-3, spice 0.12.4-0nocelt2 and usbredir 0.6-2 compiled from
> debian unstable sources.
> The xen-unstable upstream commit is
> 4787f667bcee205c56a27da59b766a53e1e929eb, plus these patches not upstream:
> tools: various things just to build test
> tools: Improve make debball
> libxl: Add qxl vga interface support for upstream qemu
> libxl: add basic spice support for pv domUs

Have you try without your last two patches? Or even any patch at all?

> Qemu crashes always on domU S.O. start, on both pv and hvm domUs.
> 
> Same dom0 with qemu 1.6 from xen-unstable repository used for some tests
> yesterday and was full working.
> I also update seabios to 1.7.4-4 compiled from debian unstable sources but
> the problem persists.
> 
> I looked on dom0 logs, qemu logs and xl dmesg and I found only a qemu
> segfault related on each domU in dom0 syslog, for example the latest:
> [  844.273170] qemu-system-i38[3545]: segfault at 8 ip 00007fa905dcc4c1 sp
> 00007fff41220810 error 4 in qemu-system-i386[7fa905ad5000+598000]
> 
> If you need more informations, tests and/or logs tell me and I'll post them.

What is in your guest config files?

-- 
Anthony PERARD

Re: [Qemu-devel] Qemu 2.0 regression with xen: qemu crash on any domUs S.O. start

[Fabio Fantoni]

Il 02/04/2014 18:05, Anthony PERARD ha scritto:
> On Tue, Apr 01, 2014 at 05:01:18PM +0200, Fabio Fantoni wrote:
>> Today I tried latest qemu 2.0 compiled from git (commit
>> 63678e17cf399ff81b93417fe7bee8d6ef6b6b1b) on this dom0:
>> Debian 7 (Wheezy) 64 bit with kernel from package linux-image-3.2.0-4-amd64
>> version 3.2.54-2 and all dependency packages for xen, spice and usb
>> redirection.
>> Seabios 1.7.3-3, spice 0.12.4-0nocelt2 and usbredir 0.6-2 compiled from
>> debian unstable sources.
>> The xen-unstable upstream commit is
>> 4787f667bcee205c56a27da59b766a53e1e929eb, plus these patches not upstream:
>> tools: various things just to build test
>> tools: Improve make debball
>> libxl: Add qxl vga interface support for upstream qemu
>> libxl: add basic spice support for pv domUs
> Have you try without your last two patches? Or even any patch at all?

I did some thing with same build and qemu 1.6 the day before and all was 
working, the problem start with change of qemu with 2.0.
I also tried disabling all things of my patches.

>
>> Qemu crashes always on domU S.O. start, on both pv and hvm domUs.
>>
>> Same dom0 with qemu 1.6 from xen-unstable repository used for some tests
>> yesterday and was full working.
>> I also update seabios to 1.7.4-4 compiled from debian unstable sources but
>> the problem persists.
>>
>> I looked on dom0 logs, qemu logs and xl dmesg and I found only a qemu
>> segfault related on each domU in dom0 syslog, for example the latest:
>> [  844.273170] qemu-system-i38[3545]: segfault at 8 ip 00007fa905dcc4c1 sp
>> 00007fff41220810 error 4 in qemu-system-i386[7fa905ad5000+598000]
>>
>> If you need more informations, tests and/or logs tell me and I'll post them.
> What is in your guest config files?
>

The cfg of test with full backtrace on other my mail:
> name='wheezy'
> memory=1024
> vcpus=2
> device_model_override="/usr/lib/xen/bin/qemu-gdb"
> disk=['/mnt/vm/disks/wheezy.disk1.xm,raw,xvda,rw']
> vif=['bridge=xenbr0']
> vfb=['vnc=1,vncunused=1,vnclisten=0.0.0.0,keymap=it']
> #vfb=['vnc=0']
> #vnc=0
> #vncunused=1
> #keymap="it"
> #vnclisten=0.0.0.0
> #spice=1
> #spicehost="0.0.0.0"
> #spiceport=6002
> #spicepasswd="test"
> #bootloader='pygrub'
> # Only for installation
> #kernel = "/boot/domu/wheezy/vmlinuz"
> kernel = "/mnt/vm/pvgrub2/grub/pvgrub2.xen"
> #ramdisk = "/boot/domu/wheezy/initrd.gz"
> #extra = "debian-installer/exit/always_halt=true -- quiet console=tty0"