[Qemu-devel] [qom-next] Bisecting virtio-scsi issue

[Qemu-devel] [qom-next] Bisecting virtio-scsi issue

[Andreas Färber]

Hi,

After multiple runs of not-so-successful bisecting, it appears as if the
qom-next commit "qbus: Make child devices links" [1] is causing
assertions for both virtio-scsi and ahci but not for qemu-system-x86_64
without parameters:

$ x86_64-softmmu/qemu-system-x86_64 -enable-kvm -device
virtio-scsi-pci,id=mcbus
qemu-system-x86_64: /home/andreas/QEMU/qemu-rcar/memory.c:1259:
memory_region_add_subregion_common: Assertion `!subregion->parent' failed.

$ x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 1G -boot menu=on
-drive file=../openSUSE-12.1-DVD-x86_64.iso,if=none,id=idecd,readonly=on
-device ide-cd,bus=ide.0,drive=idecd -device ahci,id=ahci -device
ide-cd,bus=ahci.0,drive=ahcicd -drive
file=../openSUSE-12.1-GNOME-LiveCD-x86_64.iso,if=none,id=ahcicd,readonly=on
qemu-system-x86_64: /home/andreas/QEMU/qemu-rcar/memory.c:1259:
memory_region_add_subregion_common: Assertion `!subregion->parent' failed.

qemu.git appears to be okay. But I also got quite a few segfaults during
the bisects that I had to skip (I believe mengcong was already tracking
down some virtio write error), so I'll recheck without git-bisect and
doing fully clean rebuilds tomorrow if nobody has any insights.

Cheers,
Andreas

[1] http://repo.or.cz/w/qemu/afaerber.git/shortlog/refs/heads/qom-next-2

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [qom-next] Bisecting virtio-scsi issue

[Avi Kivity]

On 06/16/2012 03:41 AM, Andreas Färber wrote:
> Hi,
> 
> After multiple runs of not-so-successful bisecting, it appears as if the
> qom-next commit "qbus: Make child devices links" [1] is causing
> assertions for both virtio-scsi and ahci but not for qemu-system-x86_64
> without parameters:
> 
> $ x86_64-softmmu/qemu-system-x86_64 -enable-kvm -device
> virtio-scsi-pci,id=mcbus
> qemu-system-x86_64: /home/andreas/QEMU/qemu-rcar/memory.c:1259:
> memory_region_add_subregion_common: Assertion `!subregion->parent' failed.
> 
> $ x86_64-softmmu/qemu-system-x86_64 -enable-kvm -m 1G -boot menu=on
> -drive file=../openSUSE-12.1-DVD-x86_64.iso,if=none,id=idecd,readonly=on
> -device ide-cd,bus=ide.0,drive=idecd -device ahci,id=ahci -device
> ide-cd,bus=ahci.0,drive=ahcicd -drive
> file=../openSUSE-12.1-GNOME-LiveCD-x86_64.iso,if=none,id=ahcicd,readonly=on
> qemu-system-x86_64: /home/andreas/QEMU/qemu-rcar/memory.c:1259:
> memory_region_add_subregion_common: Assertion `!subregion->parent' failed.
> 

A stack trace would be helpful, as well as a printout of what
subregion->parent actually is.

You can also run 'qemu mtree' from gdb after including scripts/qemu-gdb.py.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [qom-next] Bisecting virtio-scsi issue

[Andreas Färber]

Am 17.06.2012 10:37, schrieb Avi Kivity:
> On 06/16/2012 03:41 AM, Andreas Färber wrote:
>> Hi,
>>
>> After multiple runs of not-so-successful bisecting, it appears as if the
>> qom-next commit "qbus: Make child devices links" [1] is causing
>> assertions for both virtio-scsi and ahci but not for qemu-system-x86_64
>> without parameters:
>>
>> $ x86_64-softmmu/qemu-system-x86_64 -enable-kvm -device
>> virtio-scsi-pci,id=mcbus
>> qemu-system-x86_64: /home/andreas/QEMU/qemu-rcar/memory.c:1259:
>> memory_region_add_subregion_common: Assertion `!subregion->parent' failed.
> 
> A stack trace would be helpful,

$ gdb --ex run --args x86_64-softmmu/qemu-system-x86_64 -enable-kvm
-device virtio-scsi-pci,id=mcbus
[...]
qemu-system-x86_64: /home/andreas/QEMU/qemu-rcar/memory.c:1259:
memory_region_add_subregion_common: Assertion `!subregion->parent' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff5769d95 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff5769d95 in raise () from /lib64/libc.so.6
#1  0x00007ffff576b2ab in abort () from /lib64/libc.so.6
#2  0x00007ffff57628fe in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff57629a2 in __assert_fail () from /lib64/libc.so.6
#4  0x000055555576764c in memory_region_add_subregion_common (
    mr=<optimized out>, offset=<optimized out>, subregion=<optimized out>)
    at /home/andreas/QEMU/qemu-rcar/memory.c:1259
#5  memory_region_add_subregion_common (mr=<optimized out>,
    offset=<optimized out>, subregion=<optimized out>)
    at /home/andreas/QEMU/qemu-rcar/memory.c:1253
#6  0x000055555574203f in apic_init (apic_id=0 '\000', env=0x555556468d90)
    at /home/andreas/QEMU/qemu-rcar/hw/i386/../pc.c:911
#7  pc_new_cpu (cpu_model=0x55555583aec3 "qemu64")
    at /home/andreas/QEMU/qemu-rcar/hw/i386/../pc.c:948
#8  pc_cpus_init (cpu_model=0x55555583aec3 "qemu64")
    at /home/andreas/QEMU/qemu-rcar/hw/i386/../pc.c:969
#9  0x0000555555742d6c in pc_init1 (system_memory=0x5555564627b0, system_io=
    0x555556462880, ram_size=134217728, boot_device=0x7fffffffdd50 "cad",
    kernel_filename=0x0, kernel_cmdline=0x555555813c57 "", initrd_filename=
    0x0, cpu_model=0x0, pci_enabled=1, kvmclock_enabled=1)
    at /home/andreas/QEMU/qemu-rcar/hw/i386/../pc_piix.c:151
#10 0x00005555557437c8 in pc_init_pci (ram_size=134217728, boot_device=
    0x7fffffffdd50 "cad", kernel_filename=0x0, kernel_cmdline=
---Type <return> to continue, or q <return> to quit---
    0x555555813c57 "", initrd_filename=0x0, cpu_model=<optimized out>)
    at /home/andreas/QEMU/qemu-rcar/hw/i386/../pc_piix.c:296
#11 0x00005555555c3429 in main (argc=<optimized out>, argv=<optimized out>,
    envp=<optimized out>) at /home/andreas/QEMU/qemu-rcar/vl.c:3517

> as well as a printout of what
> subregion->parent actually is.

(gdb) select-frame 4
(gdb) print subregion->parent
value has been optimized out

Any suggestion?

> You can also run 'qemu mtree' from gdb after including scripts/qemu-gdb.py.

Unfortunately neither the script nor the commit introducing it nor
Google offer any usage instructions...

(gdb) include
Undefined command: "include".  Try "help".

(gdb) shell sh ../qemu-rcar/scripts/qemu-gdb.py
^C(gdb) Quit

BTW if some write is going wrong somewhere then this symptom here could
be just accidental. After all we don't seem to be changing any
MemoryRegion in this commit.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [qom-next] Bisecting virtio-scsi issue

[Avi Kivity]

On 06/17/2012 05:08 PM, Andreas Färber wrote:
> Am 17.06.2012 10:37, schrieb Avi Kivity:
>> On 06/16/2012 03:41 AM, Andreas Färber wrote:
>>> Hi,
>>>
>>> After multiple runs of not-so-successful bisecting, it appears as if the
>>> qom-next commit "qbus: Make child devices links" [1] is causing
>>> assertions for both virtio-scsi and ahci but not for qemu-system-x86_64
>>> without parameters:
>>>
>>> $ x86_64-softmmu/qemu-system-x86_64 -enable-kvm -device
>>> virtio-scsi-pci,id=mcbus
>>> qemu-system-x86_64: /home/andreas/QEMU/qemu-rcar/memory.c:1259:
>>> memory_region_add_subregion_common: Assertion `!subregion->parent' failed.
>> 
>> A stack trace would be helpful,
> 
> $ gdb --ex run --args x86_64-softmmu/qemu-system-x86_64 -enable-kvm
> -device virtio-scsi-pci,id=mcbus
> [...]
> qemu-system-x86_64: /home/andreas/QEMU/qemu-rcar/memory.c:1259:
> memory_region_add_subregion_common: Assertion `!subregion->parent' failed.
> 
> Program received signal SIGABRT, Aborted.
> 0x00007ffff5769d95 in raise () from /lib64/libc.so.6
> (gdb) bt
> #0  0x00007ffff5769d95 in raise () from /lib64/libc.so.6
> #1  0x00007ffff576b2ab in abort () from /lib64/libc.so.6
> #2  0x00007ffff57628fe in __assert_fail_base () from /lib64/libc.so.6
> #3  0x00007ffff57629a2 in __assert_fail () from /lib64/libc.so.6
> #4  0x000055555576764c in memory_region_add_subregion_common (
>     mr=<optimized out>, offset=<optimized out>, subregion=<optimized out>)
>     at /home/andreas/QEMU/qemu-rcar/memory.c:1259
> #5  memory_region_add_subregion_common (mr=<optimized out>,
>     offset=<optimized out>, subregion=<optimized out>)
>     at /home/andreas/QEMU/qemu-rcar/memory.c:1253
> #6  0x000055555574203f in apic_init (apic_id=0 '\000', env=0x555556468d90)
>     at /home/andreas/QEMU/qemu-rcar/hw/i386/../pc.c:911
> #7  pc_new_cpu (cpu_model=0x55555583aec3 "qemu64")
>     at /home/andreas/QEMU/qemu-rcar/hw/i386/../pc.c:948
> #8  pc_cpus_init (cpu_model=0x55555583aec3 "qemu64")
>     at /home/andreas/QEMU/qemu-rcar/hw/i386/../pc.c:969
> #9  0x0000555555742d6c in pc_init1 (system_memory=0x5555564627b0, system_io=
>     0x555556462880, ram_size=134217728, boot_device=0x7fffffffdd50 "cad",
>     kernel_filename=0x0, kernel_cmdline=0x555555813c57 "", initrd_filename=
>     0x0, cpu_model=0x0, pci_enabled=1, kvmclock_enabled=1)
>     at /home/andreas/QEMU/qemu-rcar/hw/i386/../pc_piix.c:151
> #10 0x00005555557437c8 in pc_init_pci (ram_size=134217728, boot_device=
>     0x7fffffffdd50 "cad", kernel_filename=0x0, kernel_cmdline=
> ---Type <return> to continue, or q <return> to quit---
>     0x555555813c57 "", initrd_filename=0x0, cpu_model=<optimized out>)
>     at /home/andreas/QEMU/qemu-rcar/hw/i386/../pc_piix.c:296
> #11 0x00005555555c3429 in main (argc=<optimized out>, argv=<optimized out>,
>     envp=<optimized out>) at /home/andreas/QEMU/qemu-rcar/vl.c:3517
> 
>> as well as a printout of what
>> subregion->parent actually is.
> 
> (gdb) select-frame 4
> (gdb) print subregion->parent
> value has been optimized out
> 
> Any suggestion?

--enable-debug
or look for it in another frame
or both

> 
>> You can also run 'qemu mtree' from gdb after including scripts/qemu-gdb.py.
> 
> Unfortunately neither the script nor the commit introducing it nor
> Google offer any usage instructions...
> 
> (gdb) include
> Undefined command: "include".  Try "help".
> 
> (gdb) shell sh ../qemu-rcar/scripts/qemu-gdb.py
> ^C(gdb) Quit


(gdb) source /path/to/qemu/scripts/qemu-gdb.py
(gdb) qemu mtree

There is a facility to autoload those scripts, we should use it.

> 
> BTW if some write is going wrong somewhere then this symptom here could
> be just accidental. After all we don't seem to be changing any
> MemoryRegion in this commit.
> 

Right.  If subregion->parent points at garbage then this would prove it.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [qom-next] Bisecting virtio-scsi issue

[Andreas Färber]

Am 17.06.2012 16:16, schrieb Avi Kivity:
> On 06/17/2012 05:08 PM, Andreas Färber wrote:
>> Am 17.06.2012 10:37, schrieb Avi Kivity:
>>> On 06/16/2012 03:41 AM, Andreas Färber wrote:
>>>> After multiple runs of not-so-successful bisecting, it appears as if the
>>>> qom-next commit "qbus: Make child devices links" [1] is causing
>>>> assertions for both virtio-scsi and ahci but not for qemu-system-x86_64
>>>> without parameters:
>>>>
>>>> $ x86_64-softmmu/qemu-system-x86_64 -enable-kvm -device
>>>> virtio-scsi-pci,id=mcbus
>>>> qemu-system-x86_64: /home/andreas/QEMU/qemu-rcar/memory.c:1259:
>>>> memory_region_add_subregion_common: Assertion `!subregion->parent' failed.
[...]
>>> You can also run 'qemu mtree' from gdb after including scripts/qemu-gdb.py.
> 
> (gdb) source /path/to/qemu/scripts/qemu-gdb.py
> (gdb) qemu mtree

(gdb) qemu mtree
0000000000000000-7ffffffffffffffe system (@ 0x5555564627b0)
0000000000000000-000000000000ffff io (@ 0x555556462880)
  000000000000007e-000000000000007f kvmvapic (I/O) (@ 0x55555647c838)

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [qom-next] Bisecting virtio-scsi issue

[Anthony Liguori]

On 06/17/2012 09:28 AM, Andreas Färber wrote:
> Am 17.06.2012 16:16, schrieb Avi Kivity:
>> On 06/17/2012 05:08 PM, Andreas Färber wrote:
>>> Am 17.06.2012 10:37, schrieb Avi Kivity:
>>>> On 06/16/2012 03:41 AM, Andreas Färber wrote:
>>>>> After multiple runs of not-so-successful bisecting, it appears as if the
>>>>> qom-next commit "qbus: Make child devices links" [1] is causing
>>>>> assertions for both virtio-scsi and ahci but not for qemu-system-x86_64
>>>>> without parameters:
>>>>>
>>>>> $ x86_64-softmmu/qemu-system-x86_64 -enable-kvm -device
>>>>> virtio-scsi-pci,id=mcbus
>>>>> qemu-system-x86_64: /home/andreas/QEMU/qemu-rcar/memory.c:1259:
>>>>> memory_region_add_subregion_common: Assertion `!subregion->parent' failed.
> [...]
>>>> You can also run 'qemu mtree' from gdb after including scripts/qemu-gdb.py.
>>
>> (gdb) source /path/to/qemu/scripts/qemu-gdb.py
>> (gdb) qemu mtree
>
> (gdb) qemu mtree
> 0000000000000000-7ffffffffffffffe system (@ 0x5555564627b0)
> 0000000000000000-000000000000ffff io (@ 0x555556462880)
>    000000000000007e-000000000000007f kvmvapic (I/O) (@ 0x55555647c838)

FWIW:


commit 55016e6ab178f0d27f85fc6b09d235817ccd4b4c
Author: Paolo Bonzini <pbonzini@redhat.com>
Date:   Thu Apr 12 18:00:18 2012 +0200


And I can't reproduce with virtio-scsi or AHCI.

Regards,

Anthony Liguori

>
> Andreas
>

Re: [Qemu-devel] [qom-next] Bisecting virtio-scsi issue

[Avi Kivity]

On 06/17/2012 05:33 PM, Anthony Liguori wrote:

>> (gdb) qemu mtree
>> 0000000000000000-7ffffffffffffffe system (@ 0x5555564627b0)
>> 0000000000000000-000000000000ffff io (@ 0x555556462880)
>>    000000000000007e-000000000000007f kvmvapic (I/O) (@ 0x55555647c838)
> 
> FWIW:
> 
> 
> commit 55016e6ab178f0d27f85fc6b09d235817ccd4b4c
> Author: Paolo Bonzini <pbonzini@redhat.com>
> Date:   Thu Apr 12 18:00:18 2012 +0200
> 
> 

Is this the fix?  Where is it?

> 


-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [qom-next] Bisecting virtio-scsi issue

[Andreas Färber]

Am 17.06.2012 16:57, schrieb Avi Kivity:
> On 06/17/2012 05:33 PM, Anthony Liguori wrote:
> 
>>> (gdb) qemu mtree
>>> 0000000000000000-7ffffffffffffffe system (@ 0x5555564627b0)
>>> 0000000000000000-000000000000ffff io (@ 0x555556462880)
>>>    000000000000007e-000000000000007f kvmvapic (I/O) (@ 0x55555647c838)
>>
>> FWIW:
>>
>>
>> commit 55016e6ab178f0d27f85fc6b09d235817ccd4b4c
>> Author: Paolo Bonzini <pbonzini@redhat.com>
>> Date:   Thu Apr 12 18:00:18 2012 +0200
> 
> Is this the fix?  Where is it?

Nah, it's on qom-next-2 branch (for upcoming PULL).

It turned out it's really got nothing to do with this fix, it's a
dependency problem due to the added level of subdirs in the target-*/
directories: While bisecting, hw/kvm/apic.o apparently didn't get
rebuilt when object.h or qdev.h changed.

Anthony posted a patch that I'm testing now.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg