From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46352) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UnQwb-0007OA-I7 for qemu-devel@nongnu.org; Fri, 14 Jun 2013 06:09:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UnQwY-0007UP-Tt for qemu-devel@nongnu.org; Fri, 14 Jun 2013 06:09:21 -0400 Received: from mail-pa0-x22f.google.com ([2607:f8b0:400e:c03::22f]:57269) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UnQwY-0007TZ-JK for qemu-devel@nongnu.org; Fri, 14 Jun 2013 06:09:18 -0400 Received: by mail-pa0-f47.google.com with SMTP id kl14so508991pab.6 for ; Fri, 14 Jun 2013 03:09:16 -0700 (PDT) Message-ID: <51BAEBC4.4030607@ozlabs.ru> Date: Fri, 14 Jun 2013 20:09:08 +1000 From: Alexey Kardashevskiy MIME-Version: 1.0 References: <1370348041-6768-1-git-send-email-pbonzini@redhat.com> <1370348041-6768-4-git-send-email-pbonzini@redhat.com> <51B96676.9010707@ozlabs.ru> <51B98A8A.5020200@ozlabs.ru> In-Reply-To: <51B98A8A.5020200@ozlabs.ru> Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH v2 03/17] memory: add ref/unref calls List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paolo Bonzini Cc: peter.maydell@linaro.org, qemu-devel@nongnu.org, Paul Mackerras , Avi Kivity Hi. Ok. Back to the bug with this patch. The initial problem with this patch is that "make check" fails. Please help with subpages. It turned out that tests use MALLOC_PERTURB_ which is normally off. Who does not know - this is a way to tell glibc to fill released memory with some value and then debug accesses to released memory. Some bright mind made it random what confuses a lot (and btw valgrind found nothing :-/ ). So I spend some time before figured out how to reproduce it outside of the qtest thingy. The tree is qemu.org/master "bd5c51e Michael Roth qemu-char: don't issue CHR_EVENT_OPEN in a BH" + replayed patches till the one from $subj on top of it. QEMU is configured as "configure --target-list=x86_64-softmmu". The magic is: export MALLOC_PERTURB_=123 nc -l -U ~/qtest-16318.sock & nc -l -U ~/qtest-16318.qmp & x86_64-softmmu/qemu-system-x86_64 \ -qtest unix:/home/alexey/qtest-16318.sock,nowait \ -qtest-log /dev/null \ -qmp unix:/home/alexey/qtest-16318.qmp,nowait \ -pidfile ~/qtest-16318.pid -machine accel=qtest -vnc none Immediate crash at (the very last backtrace in this mail is that crash). x86_cpu_apic_realize() creates a subpage for IO: #0 aik_dbg_start (f=f@entry=0x5555558c4b41 "subpage_init", l=l@entry=0x6a0, mr=mr@entry=0x555556556d30) at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc.c:1297 #1 0x0000555555774299 in subpage_init (base=0x0, as=0x5555564a9260) at /home/alexey/pcipassthru/qemu-impreza/exec.c:1696 #2 register_subpage (d=d@entry=0x555556523d00, section=section@entry=0x7fffffffd620) at /home/alexey/pcipassthru/qemu-impreza/exec.c:845 #3 0x000055555577447d in mem_add (listener=0x555556523d08, section=) at /home/alexey/pcipassthru/qemu-impreza/exec.c:881 #4 0x00005555557c2d69 in address_space_update_topology_pass (as=as@entry=0x5555564a9260, adding=adding@entry=0x1, old_view=..., new_view=...) at /home/alexey/pcipassthru/qemu-impreza/memory.c:751 #5 0x00005555557c64b8 in address_space_update_topology (as=0x5555564a9260) at /home/alexey/pcipassthru/qemu-impreza/memory.c:766 #6 memory_region_transaction_commit () at /home/alexey/pcipassthru/qemu-impreza/memory.c:790 #7 0x00005555557c79cd in memory_region_add_subregion_common (mr=0x555556523c30, offset=offset@entry=0x7e, subregion=subregion@entry=0x555556550a28) at /home/alexey/pcipassthru/qemu-impreza/memory.c:1518 #8 0x00005555557c7ae8 in memory_region_add_subregion (mr=, offset=offset@entry=0x7e, subregion=subregion@entry=0x555556550a28) at /home/alexey/pcipassthru/qemu-impreza/memory.c:1527 #9 0x0000555555663995 in sysbus_add_io (dev=dev@entry=0x55555654e700, addr=addr@entry=0x7e, mem=mem@entry=0x555556550a28) at /home/alexey/pcipassthru/qemu-impreza/hw/core/sysbus.c:242 #10 0x000055555579cfce in vapic_init (dev=0x55555654e700) at /home/alexey/pcipassthru/qemu-impreza/hw/i386/kvmvapic.c:707 #11 0x0000555555661651 in device_realize (dev=0x55555654e700, err=0x7fffffffda40) at /home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:178 #12 0x0000555555662cf3 in device_set_realized (obj=0x55555654e700, value=0x1, err=0x7fffffffdb50) at /home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:699 #13 0x000055555573358e in property_set_bool (obj=0x55555654e700, v=, opaque=0x55555653c1f0, name=, errp=0x7fffffffdb50) at /home/alexey/pcipassthru/qemu-impreza/qom/object.c:1301 #14 0x0000555555736445 in object_property_set_qobject (obj=0x55555654e700, value=, name=0x555555896553 "realized", errp=0x7fffffffdb50) at /home/alexey/pcipassthru/qemu-impreza/qom/qom-qobject.c:24 #15 0x000055555573525e in object_property_set_bool (obj=obj@entry=0x55555654e700, value=value@entry=0x1, name=name@entry=0x555555896553 "realized", errp=errp@entry=0x7fffffffdb50) at /home/alexey/pcipassthru/qemu-impreza/qom/object.c:852 #16 0x0000555555661c3a in qdev_init (dev=dev@entry=0x55555654e700) at /home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:163 #17 0x0000555555661e91 in qdev_init_nofail (dev=dev@entry=0x55555654e700) at /home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:277 #18 0x0000555555663789 in sysbus_create_varargs (name=name@entry=0x5555558c73a1 "kvmvapic", addr=addr@entry=0xffffffffffffffff) at /home/alexey/pcipassthru/qemu-impreza/hw/core/sysbus.c:157 #19 0x00005555557a4ead in sysbus_create_simple (irq=0x0, addr=0xffffffffffffffff, name=0x5555558c73a1 "kvmvapic") at /home/alexey/pcipassthru/qemu-impreza/include/hw/sysbus.h:75 #20 apic_init_common (dev=0x555556535350) at /home/alexey/pcipassthru/qemu-impreza/hw/intc/apic_common.c:311 #21 0x0000555555790fb6 in icc_device_realize (dev=0x555556535350, errp=0x7fffffffdc80) at /home/alexey/pcipassthru/qemu-impreza/hw/cpu/icc_bus.c:50 #22 0x0000555555662cf3 in device_set_realized (obj=0x555556535350, value=0x1, err=0x7fffffffdd90) at /home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:699 #23 0x000055555573358e in property_set_bool (obj=0x555556535350, v=, opaque=0x555556535610, name=, errp=0x7fffffffdd90) at /home/alexey/pcipassthru/qemu-impreza/qom/object.c:1301 #24 0x0000555555736445 in object_property_set_qobject (obj=0x555556535350, value=, name=0x555555896553 "realized", errp=0x7fffffffdd90) at /home/alexey/pcipassthru/qemu-impreza/qom/qom-qobject.c:24 #25 0x000055555573525e in object_property_set_bool (obj=obj@entry=0x555556535350, value=value@entry=0x1, name=name@entry=0x555555896553 "realized", errp=errp@entry=0x7fffffffdd90) at /home/alexey/pcipassthru/qemu-impreza/qom/object.c:852 #26 0x0000555555661c3a in qdev_init (dev=0x555556535350) at /home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:163 #27 0x00005555557d9a7c in x86_cpu_apic_realize (errp=0x7fffffffddd0, cpu=0x55555653df50) at /home/alexey/pcipassthru/qemu-impreza/target-i386/cpu.c:2327 #28 x86_cpu_realizefn (dev=0x55555653df50, errp=0x7fffffffde20) at /home/alexey/pcipassthru/qemu-impreza/target-i386/cpu.c:2397 #29 0x0000555555662cf3 in device_set_realized (obj=0x55555653df50, value=0x1, err=0x7fffffffdf30) at /home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:699 #30 0x000055555573358e in property_set_bool (obj=0x55555653df50, v=, opaque=0x55555652e390, name=, errp=0x7fffffffdf30) at /home/alexey/pcipassthru/qemu-impreza/qom/object.c:1301 ---Type to continue, or q to quit--- #31 0x0000555555736445 in object_property_set_qobject (obj=0x55555653df50, value=, name=0x555555896553 "realized", errp=0x7fffffffdf30) at /home/alexey/pcipassthru/qemu-impreza/qom/qom-qobject.c:24 #32 0x000055555573525e in object_property_set_bool (obj=0x55555653df50, value=, name=0x555555896553 "realized", errp=0x7fffffffdf30) at /home/alexey/pcipassthru/qemu-impreza/qom/object.c:852 #33 0x000055555579f3b0 in pc_new_cpu (cpu_model=, apic_id=0x0, icc_bridge=, errp=0x7fffffffdf70) at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc.c:911 #34 0x00005555557a0fc1 in pc_cpus_init (cpu_model=0x5555558c7a2d "qemu64", cpu_model@entry=0x0, icc_bridge=icc_bridge@entry=0x55555652b420) at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc.c:964 #35 0x00005555557a129f in pc_init1 (system_memory=0x555556522e60, system_io=0x555556523c30, ram_size=ram_size@entry=0x8000000, boot_device=boot_device@entry=0x555555891aaa "cad", kernel_filename=kernel_filename@entry=0x0, kernel_cmdline=kernel_cmdline@entry=0x5555558d8fb6 "", initrd_filename=initrd_filename@entry=0x0, cpu_model=cpu_model@entry=0x0, pci_enabled=pci_enabled@entry=0x1, kvmclock_enabled=kvmclock_enabled@entry=0x1) at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc_piix.c:98 #36 0x00005555557a1aea in pc_init_pci (args=) at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc_piix.c:242 #37 0x00005555555dcea0 in main (argc=, argv=, envp=) at /home/alexey/pcipassthru/qemu-impreza/vl.c:4307 This subpage is released later due to some magic which I do not understand: (gdb) bt #0 aik_dbg (f=f@entry=0x5555558c4c20 "destroy_page_desc", l=l@entry=0x305) at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc.c:1284 #1 0x0000555555773d48 in destroy_page_desc (section_index=) at /home/alexey/pcipassthru/qemu-impreza/exec.c:773 #2 destroy_l2_mapping (level=0x0, lp=0x555556559e10) at /home/alexey/pcipassthru/qemu-impreza/exec.c:791 #3 destroy_l2_mapping (lp=0x555556559e10, level=0x0) at /home/alexey/pcipassthru/qemu-impreza/exec.c:777 #4 0x0000555555773c88 in destroy_l2_mapping (level=0x1, lp=0x555556559610) at /home/alexey/pcipassthru/qemu-impreza/exec.c:789 #5 destroy_l2_mapping (lp=0x555556559610, level=0x1) at /home/alexey/pcipassthru/qemu-impreza/exec.c:777 #6 0x0000555555773c88 in destroy_l2_mapping (level=0x2, lp=0x555556558e10) at /home/alexey/pcipassthru/qemu-impreza/exec.c:789 #7 destroy_l2_mapping (lp=0x555556558e10, level=0x2) at /home/alexey/pcipassthru/qemu-impreza/exec.c:777 #8 0x0000555555773c88 in destroy_l2_mapping (level=0x3, lp=0x555556523d00) at /home/alexey/pcipassthru/qemu-impreza/exec.c:789 #9 destroy_l2_mapping (lp=0x555556523d00, level=0x3) at /home/alexey/pcipassthru/qemu-impreza/exec.c:777 #10 0x0000555555773df8 in destroy_all_mappings (d=0x555556523d00) at /home/alexey/pcipassthru/qemu-impreza/exec.c:800 #11 mem_begin (listener=0x555556523d08) at /home/alexey/pcipassthru/qemu-impreza/exec.c:1732 #12 0x00005555557c6168 in memory_region_transaction_commit () at /home/alexey/pcipassthru/qemu-impreza/memory.c:787 #13 0x00005555557c79cd in memory_region_add_subregion_common (mr=mr@entry=0x555556522e60, offset=offset@entry=0xfee00000, subregion=subregion@entry=0x55555652d7b8) at /home/alexey/pcipassthru/qemu-impreza/memory.c:1518 #14 0x00005555557c7a72 in memory_region_add_subregion_overlap (mr=0x555556522e60, offset=0xfee00000, subregion=0x55555652d7b8, priority=) at /home/alexey/pcipassthru/qemu-impreza/memory.c:1537 #15 0x00005555557a1038 in pc_cpus_init (cpu_model=0x5555558c7a2d "qemu64", cpu_model@entry=0x0, icc_bridge=icc_bridge@entry=0x55555652b420) at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc.c:976 #16 0x00005555557a129f in pc_init1 (system_memory=0x555556522e60, system_io=0x555556523c30, ram_size=ram_size@entry=0x8000000, boot_device=boot_device@entry=0x555555891aaa "cad", kernel_filename=kernel_filename@entry=0x0, kernel_cmdline=kernel_cmdline@entry=0x5555558d8fb6 "", initrd_filename=initrd_filename@entry=0x0, cpu_model=cpu_model@entry=0x0, pci_enabled=pci_enabled@entry=0x1, kvmclock_enabled=kvmclock_enabled@entry=0x1) at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc_piix.c:98 #17 0x00005555557a1aea in pc_init_pci (args=) at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc_piix.c:242 #18 0x00005555555dcea0 in main (argc=, argv=, envp=) at /home/alexey/pcipassthru/qemu-impreza/vl.c:4307 (gdb) And - crash: #0 object_unref (obj=0xa7a7a7a7a7a7a7a7) at /home/alexey/pcipassthru/qemu-impreza/qom/object.c:691 #1 0x00005555557c505c in memory_region_unref (mr=) at /home/alexey/pcipassthru/qemu-impreza/memory.c:1172 #2 0x0000555555775953 in phys_sections_clear () at /home/alexey/pcipassthru/qemu-impreza/exec.c:826 #3 0x0000555555775999 in core_begin (listener=) at /home/alexey/pcipassthru/qemu-impreza/exec.c:1738 #4 0x00005555557c6168 in memory_region_transaction_commit () at /home/alexey/pcipassthru/qemu-impreza/memory.c:787 #5 0x00005555557c79cd in memory_region_add_subregion_common (mr=mr@entry=0x555556522e60, offset=offset@entry=0xfee00000, subregion=subregion@entry=0x55555652d7b8) at /home/alexey/pcipassthru/qemu-impreza/memory.c:1518 #6 0x00005555557c7a72 in memory_region_add_subregion_overlap (mr=0x555556522e60, offset=0xfee00000, subregion=0x55555652d7b8, priority=) at /home/alexey/pcipassthru/qemu-impreza/memory.c:1537 #7 0x00005555557a1038 in pc_cpus_init (cpu_model=0x5555558c7a2d "qemu64", cpu_model@entry=0x0, icc_bridge=icc_bridge@entry=0x55555652b420) at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc.c:976 #8 0x00005555557a129f in pc_init1 (system_memory=0x555556522e60, system_io=0x555556523c30, ram_size=ram_size@entry=0x8000000, boot_device=boot_device@entry=0x555555891aaa "cad", kernel_filename=kernel_filename@entry=0x0, kernel_cmdline=kernel_cmdline@entry=0x5555558d8fb6 "", initrd_filename=initrd_filename@entry=0x0, cpu_model=cpu_model@entry=0x0, pci_enabled=pci_enabled@entry=0x1, kvmclock_enabled=kvmclock_enabled@entry=0x1) at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc_piix.c:98 #9 0x00005555557a1aea in pc_init_pci (args=) at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc_piix.c:242 #10 0x00005555555dcea0 in main (argc=, argv=, envp=) at /home/alexey/pcipassthru/qemu-impreza/vl.c:4307 (gdb) On 06/13/2013 07:02 PM, Alexey Kardashevskiy wrote: > Fails on qtest_init() in tests/libqtest.c, "Broken pipe". I cannot easily > see what is wrong here with this patch but it is 100% reproducible on x86_64 > :( > > > On 06/13/2013 04:28 PM, Alexey Kardashevskiy wrote: >> Hi! >> >> I do not know how (yet) but this patch breaks qtest on x86 (I bisected it): >> >> >> make check-qtest V=1 >> QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 >> MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k >> --verbose -m=quick tests/fdc-test tests/ide-test tests/hd-geo-test >> tests/rtc-test tests/i440fx-test tests/fw_cfg-test >> TEST: tests/fdc-test... (pid=13049) >> Broken pipe >> FAIL: tests/fdc-test >> TEST: tests/ide-test... (pid=13053) >> /x86_64/ide/identify: >> Broken pipe >> FAIL >> GTester: last random seed: R02S2f8a8fd53ff256765db44cefb0a920ce >> (pid=13057) >> /x86_64/ide/bmdma/setup: >> Broken pipe >> FAIL >> GTester: last random seed: R02S0cec5d222cfd196e6e839e06d7ddde89 >> (pid=13061) >> /x86_64/ide/bmdma/simple_rw: FAIL >> GTester: last random seed: R02S46a30a1ccd33dc104919118330810a85 >> (pid=13062) >> /x86_64/ide/bmdma/short_prdt: FAIL >> GTester: last random seed: R02S19fdcc95895b870371ed5ddcc8b77eda >> (pid=13063) >> >> [...] >> >> >> On 06/04/2013 10:13 PM, Paolo Bonzini wrote: >>> Add ref/unref calls at the following places: >>> >>> - places where memory regions are stashed by a listener and >>> used outside the BQL (including in Xen or KVM). >>> >>> - memory_region_find callsites >>> >>> - creation of aliases and containers (only the aliased/contained >>> region gets a reference to avoid loops) >>> >>> - around calls to del_subregion/add_subregion, where the region >>> could disappear after the first call >>> >>> Signed-off-by: Paolo Bonzini >>> --- >>> exec.c | 6 +++++- >>> hw/core/loader.c | 1 + >>> hw/display/exynos4210_fimd.c | 6 ++++++ >>> hw/display/framebuffer.c | 12 +++++++----- >>> hw/i386/kvmvapic.c | 1 + >>> hw/misc/vfio.c | 2 ++ >>> hw/virtio/dataplane/hostmem.c | 7 +++++++ >>> hw/virtio/vhost.c | 2 ++ >>> hw/virtio/virtio-balloon.c | 1 + >>> hw/xen/xen_pt.c | 4 ++++ >>> include/hw/virtio/dataplane/hostmem.h | 1 + >>> kvm-all.c | 2 ++ >>> memory.c | 20 ++++++++++++++++++++ >>> target-arm/kvm.c | 2 ++ >>> target-sparc/mmu_helper.c | 1 + >>> xen-all.c | 2 ++ >>> 16 files changed, 64 insertions(+), 6 deletions(-) > > -- Alexey