* [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
[not found] ` <20151113190014.GB18986@redhat.com>
@ 2015-11-16 8:11 ` Grundmann, Christian
2015-11-17 9:59 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 16+ messages in thread
From: Grundmann, Christian @ 2015-11-16 8:11 UTC (permalink / raw)
To: 'qemu-devel@nongnu.org'
Hi,
Dan sent me over to you,
please let me know if i can provide additional informations
Softwareversions:
ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso
qemu-img-ev-2.3.0-29.1.el7.x86_64
qemu-kvm-ev-2.3.0-29.1.el7.x86_64
qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
kernel-3.10.0-229.14.1.el7.x86_64
libvirt-daemon-driver-nwfilter-1.2.8-16.el7_1.4.x86_64
libvirt-lock-sanlock-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-kvm-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-config-nwfilter-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-secret-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-nodedev-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64
libvirt-python-1.2.8-7.el7_1.1.x86_64
libvirt-daemon-driver-interface-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-network-1.2.8-16.el7_1.4.x86_64
libvirt-daemon-driver-storage-1.2.8-16.el7_1.4.x86_64
libvirt-client-1.2.8-16.el7_1.4.x86_64
gperftools-libs-2.4-7.el7.x86_64
Commandline:
/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-5A43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2015-11-15T20:04:35,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84-8746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffaecf/d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,format=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Stack Trace:
gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt"
[New LWP 14750]
[New LWP 51911]
[New LWP 14758]
[New LWP 14759]
[New LWP 14754]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,usb=o'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4
Thread 5 (Thread 0x7fa8a1ce0700 (LWP 14754)):
#0 0x00007fa8a9eb9949 in syscall () from /lib64/libc.so.6
#1 0x00007fa8b19ed272 in qemu_event_wait ()
#2 0x00007fa8b19fb526 in call_rcu_thread ()
#3 0x00007fa8b02b1df5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fa8a9ebf1ad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fa8a04dd700 (LWP 14759)):
#0 0x00007fa8a9eb6257 in ioctl () from /lib64/libc.so.6
#1 0x00007fa8b17b8025 in kvm_vcpu_ioctl ()
#2 0x00007fa8b17b80de in kvm_cpu_exec ()
#3 0x00007fa8b17a5d7a in qemu_kvm_cpu_thread_fn ()
#4 0x00007fa8b02b1df5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fa8a9ebf1ad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fa8a0cde700 (LWP 14758)):
#0 0x00007fa8a9eb6257 in ioctl () from /lib64/libc.so.6
#1 0x00007fa8b17b8025 in kvm_vcpu_ioctl ()
#2 0x00007fa8b17b80de in kvm_cpu_exec ()
#3 0x00007fa8b17a5d7a in qemu_kvm_cpu_thread_fn ()
#4 0x00007fa8b02b1df5 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fa8a9ebf1ad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fa8a14df700 (LWP 51911)):
#0 0x00007fa8b02b78a0 in sem_timedwait () from /lib64/libpthread.so.0
#1 0x00007fa8b19ed0c7 in qemu_sem_timedwait ()
#2 0x00007fa8b19708ac in worker_thread ()
#3 0x00007fa8b02b1df5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fa8a9ebf1ad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)):
#0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4
#1 0x00007fa8b186b489 in malloc_and_trace ()
#2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0
#3 0x00007fa8afbd666e in g_slice_alloc () from /lib64/libglib-2.0.so.0
#4 0x00007fa8b17cbffd in virtio_blk_handle_output ()
#5 0x00007fa8b197e6b6 in qemu_iohandler_poll ()
#6 0x00007fa8b197e296 in main_loop_wait ()
#7 0x00007fa8b177da4e in main ()
Thx Christian
-----Ursprüngliche Nachricht-----
Von: Dan Kenigsberg [mailto:danken@redhat.com]
Gesendet: Freitag, 13. November 2015 20:00
An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
Cc: 'users@ovirt.org' <users@ovirt.org>
Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc
On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote:
> Hi,
> i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso" (is
> there something better to use?) fort he nodes, and have random crashes
> of VMs The dumps are always the Same
>
> gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump [Thread
> debugging using libthread_db enabled] Using host libthread_db library
> "/lib64/libthread_db.so.1".
> Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'.
> Program terminated with signal 11, Segmentation fault.
> #0 0x00007f0c559c4353 in
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::Fr
> eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
>
>
> Didn't have the Problem with 3.5 el6 nodes, so don't no if ist centos7
> or 3.6
Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt.
Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-11-16 8:11 ` [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc Grundmann, Christian
@ 2015-11-17 9:59 ` Dr. David Alan Gilbert
2015-11-17 10:36 ` Grundmann, Christian
0 siblings, 1 reply; 16+ messages in thread
From: Dr. David Alan Gilbert @ 2015-11-17 9:59 UTC (permalink / raw)
To: Grundmann, Christian; +Cc: 'qemu-devel@nongnu.org', stefanha
* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> Hi,
> Dan sent me over to you,
> please let me know if i can provide additional informations
Hi Christian,
Thanks for reporting this,
> Softwareversions:
> ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso
>
> qemu-img-ev-2.3.0-29.1.el7.x86_64
> qemu-kvm-ev-2.3.0-29.1.el7.x86_64
> qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
> qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
> ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
> kernel-3.10.0-229.14.1.el7.x86_64
> gperftools-libs-2.4-7.el7.x86_64
>
> Commandline:
> /usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-5A43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2015-11-15T20:04:35,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84-8746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffaecf/d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,format=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
>
> Stack Trace:
>
> gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt"
Can you please use a 'thread apply all bt full' the full gives a little more info.
Also, if you've not already got it installed can you please install the debuginfo package for
qemu, it gives a lot more information in backtraces.
> Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)):
> #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4
> #1 0x00007fa8b186b489 in malloc_and_trace ()
> #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0
> #3 0x00007fa8afbd666e in g_slice_alloc () from /lib64/libglib-2.0.so.0
> #4 0x00007fa8b17cbffd in virtio_blk_handle_output ()
> #5 0x00007fa8b197e6b6 in qemu_iohandler_poll ()
> #6 0x00007fa8b197e296 in main_loop_wait ()
> #7 0x00007fa8b177da4e in main ()
Does this part always look the same in your backtraces?
The segfault in tc_malloc is probably due to a heap corruption, or double free or similar -
although it can be a bit tricky to find out what did it, since the corruption might
have happened a bit before the place it crashed.
Some other ideas:
1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
2) Did you hit any IO errors and need to tell the VM to continue after a problem?
3) If this is pretty repeatable, then it would be interesting to try changing to a different
disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
Dave
>
>
> Thx Christian
>
> -----Ursprüngliche Nachricht-----
> Von: Dan Kenigsberg [mailto:danken@redhat.com]
> Gesendet: Freitag, 13. November 2015 20:00
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> Cc: 'users@ovirt.org' <users@ovirt.org>
> Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc
>
> On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote:
> > Hi,
> > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso" (is
> > there something better to use?) fort he nodes, and have random crashes
> > of VMs The dumps are always the Same
> >
> > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump [Thread
> > debugging using libthread_db enabled] Using host libthread_db library
> > "/lib64/libthread_db.so.1".
> > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'.
> > Program terminated with signal 11, Segmentation fault.
> > #0 0x00007f0c559c4353 in
> > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::Fr
> > eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> >
> >
> > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist centos7
> > or 3.6
>
> Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt.
>
> Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible.
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-11-17 9:59 ` Dr. David Alan Gilbert
@ 2015-11-17 10:36 ` Grundmann, Christian
2015-11-17 11:36 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 16+ messages in thread
From: Grundmann, Christian @ 2015-11-17 10:36 UTC (permalink / raw)
To: 'Dr. David Alan Gilbert'
Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com
Hi,
@ Can you please use a 'thread apply all bt full' the full gives a little more info.
gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
[New LWP 52281]
[New LWP 52288]
[New LWP 52286]
[New LWP 52291]
[New LWP 52292]
[New LWP 52287]
[New LWP 52293]
[New LWP 52290]
[New LWP 56455]
[New LWP 52289]
[New LWP 52282]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,us'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4
Thread 11 (Thread 0x7f6d47719700 (LWP 52282)):
#0 0x00007f6d4f8f2949 in syscall () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d57426272 in qemu_event_wait ()
No symbol table info available.
#2 0x00007f6d57434526 in call_rcu_thread ()
No symbol table info available.
#3 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)):
#0 0x00007f6d55cf08a0 in sem_timedwait () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007f6d574260c7 in qemu_sem_timedwait ()
No symbol table info available.
#2 0x00007f6d573a98ac in worker_thread ()
No symbol table info available.
#3 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 8 (Thread 0x7f6d44713700 (LWP 52290)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 5 (Thread 0x7f6d43711700 (LWP 52292)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 3 (Thread 0x7f6d46717700 (LWP 52286)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 2 (Thread 0x7f6d45715700 (LWP 52288)):
#0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl ()
No symbol table info available.
#2 0x00007f6d571f10de in kvm_cpu_exec ()
No symbol table info available.
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn ()
No symbol table info available.
#4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)):
#0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4
No symbol table info available.
#1 0x00007f6d572a4489 in malloc_and_trace ()
No symbol table info available.
#2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0
No symbol table info available.
#3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0
No symbol table info available.
#4 0x00007f6d57204ffd in virtio_blk_handle_output ()
No symbol table info available.
#5 0x00007f6d573b76b6 in qemu_iohandler_poll ()
No symbol table info available.
#6 0x00007f6d573b7296 in main_loop_wait ()
No symbol table info available.
#7 0x00007f6d571b6a4e in main ()
No symbol table info available.
@ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
Sorry it's a ovirt-node System where I can't you yum
@ Does this part always look the same in your backtraces?
The most are the same, found one a little bit different :
Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)):
#0 0x00007f3785d18353 in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
No symbol table info available.
#1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from /lib64/libtcmalloc.so.4
No symbol table info available.
#2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4
No symbol table info available.
#3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0
No symbol table info available.
#4 0x00007f37885fec89 in g_slice_free1 () from /lib64/libglib-2.0.so.0
No symbol table info available.
#5 0x00007f378a1f232e in virtio_blk_rw_complete ()
No symbol table info available.
#6 0x00007f378a39f1ae in bdrv_co_em_bh ()
No symbol table info available.
#7 0x00007f378a398394 in aio_bh_poll ()
No symbol table info available.
#8 0x00007f378a3a7409 in aio_dispatch_clients ()
No symbol table info available.
#9 0x00007f378a39820e in aio_ctx_dispatch ()
No symbol table info available.
#10 0x00007f37885e299a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
No symbol table info available.
#11 0x00007f378a3a6288 in main_loop_wait ()
No symbol table info available.
#12 0x00007f378a1a5a4e in main ()
No symbol table info available.
@ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
No nothing abnormal
@ 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot.
Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline.
@ 3) If this is pretty repeatable, then it would be interesting to try changing to a different
disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
Ok will try that and report
Thx Christian
-----Ursprüngliche Nachricht-----
Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
Gesendet: Dienstag, 17. November 2015 10:59
An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> Hi,
> Dan sent me over to you,
> please let me know if i can provide additional informations
Hi Christian,
Thanks for reporting this,
> Softwareversions:
> ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso
>
> qemu-img-ev-2.3.0-29.1.el7.x86_64
> qemu-kvm-ev-2.3.0-29.1.el7.x86_64
> qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
> qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
> ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
> kernel-3.10.0-229.14.1.el7.x86_64
> gperftools-libs-2.4-7.el7.x86_64
>
> Commandline:
> /usr/libexec/qemu-kvm -name myvmname -S -machine
> rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime mlock=off
> -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios
> type=1,manufacturer=oVirt,product=oVirt
> Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-5A
> 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319
> -nographic -no-user-config -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,serv
> er,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc
> base=2015-11-15T20:04:35,driftfix=slew -global
> kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on
> -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
> virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
> virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
> -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
> -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> -drive
> file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84-8
> 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffaecf
> /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,fo
> rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,werr
> or=stop,rerror=stop,aio=native -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id
> =virtio-disk0,bootindex=1 -netdev
> tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci.0
> ,addr=0x3 -chardev
> socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a
> 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=chann
> el0,name=com.redhat.rhevm.vdsm -chardev
> socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a
> 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait -device
> virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=chann
> el1,name=org.qemu.guest_agent.0 -device
> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
>
> Stack Trace:
>
> gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt"
Can you please use a 'thread apply all bt full' the full gives a little more info.
Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
> Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)):
> #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4
> #1 0x00007fa8b186b489 in malloc_and_trace ()
> #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0
> #3 0x00007fa8afbd666e in g_slice_alloc () from
> /lib64/libglib-2.0.so.0
> #4 0x00007fa8b17cbffd in virtio_blk_handle_output ()
> #5 0x00007fa8b197e6b6 in qemu_iohandler_poll ()
> #6 0x00007fa8b197e296 in main_loop_wait ()
> #7 0x00007fa8b177da4e in main ()
Does this part always look the same in your backtraces?
The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed.
Some other ideas:
1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
2) Did you hit any IO errors and need to tell the VM to continue after a problem?
3) If this is pretty repeatable, then it would be interesting to try changing to a different
disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
Dave
>
>
> Thx Christian
>
> -----Ursprüngliche Nachricht-----
> Von: Dan Kenigsberg [mailto:danken@redhat.com]
> Gesendet: Freitag, 13. November 2015 20:00
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> Cc: 'users@ovirt.org' <users@ovirt.org>
> Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc
>
> On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote:
> > Hi,
> > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso"
> > (is there something better to use?) fort he nodes, and have random
> > crashes of VMs The dumps are always the Same
> >
> > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump [Thread
> > debugging using libthread_db enabled] Using host libthread_db
> > library "/lib64/libthread_db.so.1".
> > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'.
> > Program terminated with signal 11, Segmentation fault.
> > #0 0x00007f0c559c4353 in
> > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::
> > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> >
> >
> > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist
> > centos7 or 3.6
>
> Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt.
>
> Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible.
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-11-17 10:36 ` Grundmann, Christian
@ 2015-11-17 11:36 ` Dr. David Alan Gilbert
2015-11-17 14:11 ` Grundmann, Christian
0 siblings, 1 reply; 16+ messages in thread
From: Dr. David Alan Gilbert @ 2015-11-17 11:36 UTC (permalink / raw)
To: Grundmann, Christian; +Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com
* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> Hi,
>
> @ Can you please use a 'thread apply all bt full' the full gives a little more info.
>
> gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
OK, it doesn't relaly give any more without the debuginfo package mentioned below.
<snip>
> @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
> Sorry it's a ovirt-node System where I can't you yum
Ah, although perhaps if you took the core dump, onto another machine with matching qemu and debuginfo you should
be able to get more detail.
> @ Does this part always look the same in your backtraces?
> The most are the same, found one a little bit different :
> Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)):
> #0 0x00007f3785d18353 in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0
> No symbol table info available.
> #4 0x00007f37885fec89 in g_slice_free1 () from /lib64/libglib-2.0.so.0
> No symbol table info available.
> #5 0x00007f378a1f232e in virtio_blk_rw_complete ()
> No symbol table info available.
> #6 0x00007f378a39f1ae in bdrv_co_em_bh ()
> No symbol table info available.
> #7 0x00007f378a398394 in aio_bh_poll ()
> No symbol table info available.
> #8 0x00007f378a3a7409 in aio_dispatch_clients ()
> No symbol table info available.
> #9 0x00007f378a39820e in aio_ctx_dispatch ()
> No symbol table info available.
> #10 0x00007f37885e299a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
> No symbol table info available.
> #11 0x00007f378a3a6288 in main_loop_wait ()
> No symbol table info available.
> #12 0x00007f378a1a5a4e in main ()
> No symbol table info available.
>
OK, that's a bit different but interesting....
> @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> No nothing abnormal
>
> @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot.
> Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline.
OK, that's interesting, because you may be hitting the following bug;
http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html
whose fix coincidentally just got accepted today; it's related to error cases with error=stop which
you are using.
Do you think you're only hitting these crashes on VMs that have been paused because of these space errors?
> disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
>
> Ok will try that and report
Thanks,
Dave
>
> Thx Christian
>
>
> -----Ursprüngliche Nachricht-----
> Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> Gesendet: Dienstag, 17. November 2015 10:59
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
> Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
>
> * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > Hi,
> > Dan sent me over to you,
> > please let me know if i can provide additional informations
>
> Hi Christian,
> Thanks for reporting this,
>
> > Softwareversions:
> > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso
> >
> > qemu-img-ev-2.3.0-29.1.el7.x86_64
> > qemu-kvm-ev-2.3.0-29.1.el7.x86_64
> > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
> > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
> > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
> > kernel-3.10.0-229.14.1.el7.x86_64
> > gperftools-libs-2.4-7.el7.x86_64
> >
> > Commandline:
> > /usr/libexec/qemu-kvm -name myvmname -S -machine
> > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime mlock=off
> > -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios
> > type=1,manufacturer=oVirt,product=oVirt
> > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-5A
> > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319
> > -nographic -no-user-config -nodefaults -chardev
> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,serv
> > er,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc
> > base=2015-11-15T20:04:35,driftfix=slew -global
> > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on
> > -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
> > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
> > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
> > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
> > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> > -drive
> > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84-8
> > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffaecf
> > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,fo
> > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,werr
> > or=stop,rerror=stop,aio=native -device
> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id
> > =virtio-disk0,bootindex=1 -netdev
> > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device
> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci.0
> > ,addr=0x3 -chardev
> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a
> > 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait -device
> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=chann
> > el0,name=com.redhat.rhevm.vdsm -chardev
> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a
> > 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait -device
> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=chann
> > el1,name=org.qemu.guest_agent.0 -device
> > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
> >
> > Stack Trace:
> >
> > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt"
>
> Can you please use a 'thread apply all bt full' the full gives a little more info.
> Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
>
> > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)):
> > #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4
> > #1 0x00007fa8b186b489 in malloc_and_trace ()
> > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0
> > #3 0x00007fa8afbd666e in g_slice_alloc () from
> > /lib64/libglib-2.0.so.0
> > #4 0x00007fa8b17cbffd in virtio_blk_handle_output ()
> > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll ()
> > #6 0x00007fa8b197e296 in main_loop_wait ()
> > #7 0x00007fa8b177da4e in main ()
>
> Does this part always look the same in your backtraces?
> The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed.
>
> Some other ideas:
> 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> 3) If this is pretty repeatable, then it would be interesting to try changing to a different
> disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
>
> Dave
> >
> >
> > Thx Christian
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Dan Kenigsberg [mailto:danken@redhat.com]
> > Gesendet: Freitag, 13. November 2015 20:00
> > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> > Cc: 'users@ovirt.org' <users@ovirt.org>
> > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc
> >
> > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote:
> > > Hi,
> > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso"
> > > (is there something better to use?) fort he nodes, and have random
> > > crashes of VMs The dumps are always the Same
> > >
> > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump [Thread
> > > debugging using libthread_db enabled] Using host libthread_db
> > > library "/lib64/libthread_db.so.1".
> > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'.
> > > Program terminated with signal 11, Segmentation fault.
> > > #0 0x00007f0c559c4353 in
> > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::
> > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> > >
> > >
> > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist
> > > centos7 or 3.6
> >
> > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt.
> >
> > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible.
> >
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-11-17 11:36 ` Dr. David Alan Gilbert
@ 2015-11-17 14:11 ` Grundmann, Christian
2015-11-17 14:20 ` Grundmann, Christian
0 siblings, 1 reply; 16+ messages in thread
From: Grundmann, Christian @ 2015-11-17 14:11 UTC (permalink / raw)
To: 'Dr. David Alan Gilbert'
Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com
Here you go
gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
[New LWP 52281]
[New LWP 52288]
[New LWP 52286]
[New LWP 52291]
[New LWP 52292]
[New LWP 52287]
[New LWP 52293]
[New LWP 52290]
[New LWP 56455]
[New LWP 52289]
[New LWP 52282]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,us'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4
Thread 11 (Thread 0x7f6d47719700 (LWP 52282)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
No locals.
#1 0x00007f6d57426272 in futex_wait (val=4294967295, ev=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301
No locals.
#2 qemu_event_wait (ev=ev@entry=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399
value = <optimized out>
#3 0x00007f6d57434526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233
tries = 0
n = <optimized out>
node = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d47719700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d47719700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107326789376, 5884348200482620104, 0, 140107326790080, 140107326789376, 140107592666688, -5804038895876586808, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8a4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d44f139e0, reg_save_area = 0x7f6d44f139a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570cf000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8a4000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44f14700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d44f14700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107284825856, 5884348200482620104, 0, 140107284826560, 140107284825856, 140726431086992, -5804033392412867896, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)):
#0 sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
No locals.
#1 0x00007f6d574260c7 in qemu_sem_timedwait (sem=sem@entry=0x7f6d5a1b9248, ms=ms@entry=10000) at util/qemu-thread-posix.c:254
rc = <optimized out>
ts = {tv_sec = 1447709021, tv_nsec = 21985000}
__func__ = "qemu_sem_timedwait"
#2 0x00007f6d573a98ac in worker_thread (opaque=0x7f6d5a1b91e0) at thread-pool.c:92
req = <optimized out>
ret = <optimized out>
pool = 0x7f6d5a1b91e0
#3 0x00007f6d55ceadf5 in start_thread (arg=0x7f6ab1dff700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6ab1dff700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140096227505920, 5884348200482620104, 0, 140096227506624, 140096227505920, 26, -5801757560646548792, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#4 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 8 (Thread 0x7f6d44713700 (LWP 52290)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8b8000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d447129e0, reg_save_area = 0x7f6d447129a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570cc000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8b8000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44713700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d44713700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107276433152, 5884348200482620104, 0, 140107276433856, 140107276433152, 140726431086992, -5804032293438111032, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8f4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d42f0f9e0, reg_save_area = 0x7f6d42f0f9a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570c3000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8f4000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d42f10700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d42f10700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107251255040, 5884348200482620104, 0, 140107251255744, 140107251255040, 140726431086992, -5804046580109950264, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c878000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d45f159e0, reg_save_area = 0x7f6d45f159a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570d5000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c878000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45f16700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d45f16700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107301611264, 5884348200482620104, 0, 140107301611968, 140107301611264, 140726431086992, -5804035590362381624, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 5 (Thread 0x7f6d43711700 (LWP 52292)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8e0000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d437109e0, reg_save_area = 0x7f6d437109a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570c6000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8e0000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43711700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d43711700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107259647744, 5884348200482620104, 0, 140107259648448, 140107259647744, 140726431086992, -5804047687674641720, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8cc000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d43f119e0, reg_save_area = 0x7f6d43f119a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570c9000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8cc000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43f12700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d43f12700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107268040448, 5884348200482620104, 0, 140107268041152, 140107268040448, 140726431086992, -5804048786649398584, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 3 (Thread 0x7f6d46717700 (LWP 52286)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c810000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d467169e0, reg_save_area = 0x7f6d467169a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570d8000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c810000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d46717700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d46717700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107310003968, 5884348200482620104, 0, 140107310004672, 140107310003968, 140726431086992, -5804036689337138488, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 2 (Thread 0x7f6d45715700 (LWP 52288)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c890000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d457149e0, reg_save_area = 0x7f6d457149a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570d2000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c890000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45715700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d45715700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107293218560, 5884348200482620104, 0, 140107293219264, 140107293218560, 140726431086992, -5804034491387624760, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)):
#0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4
No symbol table info available.
#1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=49280) at vl.c:2575
ptr = 0x7f6d59a346a0
#2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0
No symbol table info available.
#3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0
No symbol table info available.
#4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33
req = <optimized out>
#5 virtio_blk_get_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:192
req = <optimized out>
#6 virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604
s = 0x7f6d5de1ff40
__func__ = "virtio_blk_handle_output"
mrb = {reqs = {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7c000, 0x0 <repeats 29 times>}, num_reqs = 3, is_write = true}
#7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=0x7f6d5a1aea00, ret=62, ret@entry=1) at iohandler.c:143
revents = 1
#8 0x00007f6d573b7296 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:504
ret = 1
timeout = 4294967295
timeout_ns = <optimized out>
#9 0x00007f6d571b6a4e in main_loop () at vl.c:1818
nonblocking = <optimized out>
last_io = 0
#10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
i = <optimized out>
snapshot = <optimized out>
linux_boot = <optimized out>
initrd_filename = <optimized out>
kernel_filename = <optimized out>
kernel_cmdline = <optimized out>
boot_order = 0x7f6d57449a67 "cad"
boot_once = 0x0
cyls = <optimized out>
heads = <optimized out>
secs = <optimized out>
translation = <optimized out>
hda_opts = <optimized out>
opts = <optimized out>
machine_opts = <optimized out>
icount_opts = <optimized out>
olist = <optimized out>
optind = 69
optarg = 0x7f6d5a14b3a0 "rhel6.5.0"
loadvm = <optimized out>
machine_class = <optimized out>
cpu_model = <optimized out>
vga_model = 0x0
qtest_chrdev = <optimized out>
qtest_log = <optimized out>
pid_file = <optimized out>
incoming = <optimized out>
show_vnc_port = <optimized out>
defconfig = <optimized out>
userconfig = 179
log_mask = <optimized out>
log_file = <optimized out>
mem_trace = {malloc = 0x7f6d572a4480 <malloc_and_trace>, realloc = 0x7f6d572a4460 <realloc_and_trace>, free = 0x7f6d572a4450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
trace_events = <optimized out>
trace_file = <optimized out>
maxram_size = <optimized out>
ram_slots = <optimized out>
vmstate_dump_file = <optimized out>
main_loop_err = 0x0
__func__ = "main"
@ Do you think you're only hitting these crashes on VMs that have been paused because of these space errors?
Will have a look on that
Thx Christian
-----Ursprüngliche Nachricht-----
Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
Gesendet: Dienstag, 17. November 2015 12:36
An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> Hi,
>
> @ Can you please use a 'thread apply all bt full' the full gives a little more info.
>
> gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
OK, it doesn't relaly give any more without the debuginfo package mentioned below.
<snip>
> @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
> Sorry it's a ovirt-node System where I can't you yum
Ah, although perhaps if you took the core dump, onto another machine with matching qemu and debuginfo you should be able to get more detail.
> @ Does this part always look the same in your backtraces?
> The most are the same, found one a little bit different :
> Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)):
> #0 0x00007f3785d18353 in
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from
> /lib64/libtcmalloc.so.4 No symbol table info available.
> #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No
> symbol table info available.
> #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No
> symbol table info available.
> #4 0x00007f37885fec89 in g_slice_free1 () from
> /lib64/libglib-2.0.so.0 No symbol table info available.
> #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table
> info available.
> #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info
> available.
> #7 0x00007f378a398394 in aio_bh_poll () No symbol table info
> available.
> #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table info
> available.
> #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info
> available.
> #10 0x00007f37885e299a in g_main_context_dispatch () from
> /lib64/libglib-2.0.so.0 No symbol table info available.
> #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info
> available.
> #12 0x00007f378a1a5a4e in main ()
> No symbol table info available.
>
OK, that's a bit different but interesting....
> @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> No nothing abnormal
>
> @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot.
> Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline.
OK, that's interesting, because you may be hitting the following bug; http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html
whose fix coincidentally just got accepted today; it's related to error cases with error=stop which you are using.
Do you think you're only hitting these crashes on VMs that have been paused because of these space errors?
> disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
>
> Ok will try that and report
Thanks,
Dave
>
> Thx Christian
>
>
> -----Ursprüngliche Nachricht-----
> Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> Gesendet: Dienstag, 17. November 2015 10:59
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>;
> stefanha@redhat.com
> Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in
> libtcmalloc
>
> * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > Hi,
> > Dan sent me over to you,
> > please let me know if i can provide additional informations
>
> Hi Christian,
> Thanks for reporting this,
>
> > Softwareversions:
> > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso
> >
> > qemu-img-ev-2.3.0-29.1.el7.x86_64
> > qemu-kvm-ev-2.3.0-29.1.el7.x86_64
> > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
> > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
> > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
> > kernel-3.10.0-229.14.1.el7.x86_64
> > gperftools-libs-2.4-7.el7.x86_64
> >
> > Commandline:
> > /usr/libexec/qemu-kvm -name myvmname -S -machine
> > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime
> > mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios
> > type=1,manufacturer=oVirt,product=oVirt
> > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-
> > 5A
> > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319
> > -nographic -no-user-config -nodefaults -chardev
> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,se
> > rv er,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc
> > base=2015-11-15T20:04:35,driftfix=slew -global
> > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
> > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> > -device
> > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
> > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
> > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
> > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> > -drive
> > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84
> > -8
> > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffae
> > cf
> > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,
> > fo
> > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,we
> > rr or=stop,rerror=stop,aio=native -device
> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,
> > id
> > =virtio-disk0,bootindex=1 -netdev
> > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device
> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci
> > .0
> > ,addr=0x3 -chardev
> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-
> > 5a 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait
> > -device
> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=cha
> > nn el0,name=com.redhat.rhevm.vdsm -chardev
> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-
> > 5a 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait
> > -device
> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=cha
> > nn
> > el1,name=org.qemu.guest_agent.0 -device
> > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
> >
> > Stack Trace:
> >
> > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt"
>
> Can you please use a 'thread apply all bt full' the full gives a little more info.
> Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
>
> > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)):
> > #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4
> > #1 0x00007fa8b186b489 in malloc_and_trace ()
> > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0
> > #3 0x00007fa8afbd666e in g_slice_alloc () from
> > /lib64/libglib-2.0.so.0
> > #4 0x00007fa8b17cbffd in virtio_blk_handle_output ()
> > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll ()
> > #6 0x00007fa8b197e296 in main_loop_wait ()
> > #7 0x00007fa8b177da4e in main ()
>
> Does this part always look the same in your backtraces?
> The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed.
>
> Some other ideas:
> 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> 3) If this is pretty repeatable, then it would be interesting to try changing to a different
> disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
>
> Dave
> >
> >
> > Thx Christian
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Dan Kenigsberg [mailto:danken@redhat.com]
> > Gesendet: Freitag, 13. November 2015 20:00
> > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> > Cc: 'users@ovirt.org' <users@ovirt.org>
> > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc
> >
> > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote:
> > > Hi,
> > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso"
> > > (is there something better to use?) fort he nodes, and have random
> > > crashes of VMs The dumps are always the Same
> > >
> > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump
> > > [Thread debugging using libthread_db enabled] Using host
> > > libthread_db library "/lib64/libthread_db.so.1".
> > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'.
> > > Program terminated with signal 11, Segmentation fault.
> > > #0 0x00007f0c559c4353 in
> > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::
> > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> > >
> > >
> > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist
> > > centos7 or 3.6
> >
> > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt.
> >
> > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible.
> >
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-11-17 14:11 ` Grundmann, Christian
@ 2015-11-17 14:20 ` Grundmann, Christian
2015-11-17 14:42 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 16+ messages in thread
From: Grundmann, Christian @ 2015-11-17 14:20 UTC (permalink / raw)
To: 'Dr. David Alan Gilbert'
Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com
And here another one
[New LWP 1507]
[New LWP 1508]
[New LWP 1514]
[New LWP 1513]
[New LWP 2417]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -name myvmname2 -S -machine rhel6.5.0,accel=kvm,usb=o'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f5cff7e2e7d in tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from /lib64/libtcmalloc.so.4
Thread 5 (Thread 0x7f5cf39d5700 (LWP 2417)):
#0 0x00007f5d027aeac3 in pread64 () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f5d03ea52a3 in pread (__offset=<optimized out>, __nbytes=<optimized out>, __buf=0x7f5d061f8000, __fd=<optimized out>) at /usr/include/bits/unistd.h:99
No locals.
#2 handle_aiocb_rw_linear (aiocb=aiocb@entry=0x7f5d061e7340, buf=buf@entry=0x7f5d061f8000 "QF", <incomplete sequence \373>) at block/raw-posix.c:909
offset = 0
len = <optimized out>
#3 0x00007f5d03ea53d1 in handle_aiocb_rw (aiocb=0x7f5d061e7340) at block/raw-posix.c:992
nbytes = <optimized out>
buf = 0x7f5d061f8000 "QF", <incomplete sequence \373>
__PRETTY_FUNCTION__ = "handle_aiocb_rw"
#4 0x00007f5d03ea6945 in aio_worker (arg=0x7f5d061e7340) at block/raw-posix.c:1204
aiocb = 0x7f5d061e7340
ret = 0
#5 0x00007f5d03e6691b in worker_thread (opaque=0x7f5d062011e0) at thread-pool.c:105
req = 0x7f5d061e6a60
ret = <optimized out>
pool = 0x7f5d062011e0
#6 0x00007f5d027a7df5 in start_thread (arg=0x7f5cf39d5700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f5cf39d5700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140037200893696, 4764812136514238282, 0, 140037200894400, 140037200893696, 26, -4856599629847414966, -4856560931613919414}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#7 0x00007f5cfc3b51ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 4 (Thread 0x7f5cf31d4700 (LWP 1513)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1 0x00007f5d03ee2f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7f5d0438fd40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132
err = <optimized out>
__func__ = "qemu_cond_wait"
#2 0x00007f5d03c9bd3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912
No locals.
#3 qemu_kvm_cpu_thread_fn (arg=0x7f5d0757a000) at /usr/src/debug/qemu-2.3.0/cpus.c:949
cpu = 0x7f5d0757a000
r = <optimized out>
#4 0x00007f5d027a7df5 in start_thread (arg=0x7f5cf31d4700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f5cf31d4700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140037192500992, 4764812136514238282, 0, 140037192501696, 140037192500992, 140728281959696, -4856598527651432630, -4856560931613919414}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f5cfc3b51ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 3 (Thread 0x7f5cf29d3700 (LWP 1514)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1 0x00007f5d03ee2f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7f5d0438fd40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132
err = <optimized out>
__func__ = "qemu_cond_wait"
#2 0x00007f5d03c9bd3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912
No locals.
#3 qemu_kvm_cpu_thread_fn (arg=0x7f5d075e2000) at /usr/src/debug/qemu-2.3.0/cpus.c:949
cpu = 0x7f5d075e2000
r = <optimized out>
#4 0x00007f5d027a7df5 in start_thread (arg=0x7f5cf29d3700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f5cf29d3700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140037184108288, 4764812136514238282, 0, 140037184108992, 140037184108288, 140728281959696, -4856597427602933942, -4856560931613919414}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f5cfc3b51ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 2 (Thread 0x7f5cf41d6700 (LWP 1508)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
No locals.
#1 0x00007f5d03ee3272 in futex_wait (val=4294967295, ev=0x7f5d047adf44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301
No locals.
#2 qemu_event_wait (ev=ev@entry=0x7f5d047adf44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399
value = <optimized out>
#3 0x00007f5d03ef1526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233
tries = 0
n = <optimized out>
node = <optimized out>
#4 0x00007f5d027a7df5 in start_thread (arg=0x7f5cf41d6700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f5cf41d6700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140037209286400, 4764812136514238282, 0, 140037209287104, 140037209286400, 140037475163712, -4856591933802891446, -4856560931613919414}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f5cfc3b51ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 1 (Thread 0x7f5d03ba5c00 (LWP 1507)):
#0 0x00007f5cff7e2e7d in tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from /lib64/libtcmalloc.so.4
No symbol table info available.
#1 0x00007f5cff7e312a in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) () from /lib64/libtcmalloc.so.4
No symbol table info available.
#2 0x00007f5cff7e31dd in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /lib64/libtcmalloc.so.4
No symbol table info available.
#3 0x00007f5cff7e6235 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /lib64/libtcmalloc.so.4
No symbol table info available.
#4 0x00007f5cff7f4d5b in tc_malloc () from /lib64/libtcmalloc.so.4
No symbol table info available.
#5 0x00007f5d03d61489 in malloc_and_trace (n_bytes=18) at vl.c:2575
ptr = 0x1
#6 0x00007f5d020b647f in g_malloc () from /lib64/libglib-2.0.so.0
No symbol table info available.
#7 0x00007f5d020cdf7f in g_strdup () from /lib64/libglib-2.0.so.0
No symbol table info available.
#8 0x00007f5d03eddab5 in alloc_entry (value=0x7f5d088de6c0, key=0x7f5d03f5debb "wr_highest_offset") at qobject/qdict.c:79
entry = 0x7f5d088df480
#9 qdict_put_obj (qdict=0x7f5d06e10400, key=0x7f5d03f5debb "wr_highest_offset", value=0x7f5d088de6c0) at qobject/qdict.c:145
bucket = 81
entry = <optimized out>
#10 0x00007f5d03ebf34a in visit_type_BlockDeviceStats_fields (errp=0x7ffddb417ca0, obj=0x7f5d07f905a0, m=0x7f5d061fdea0) at qapi-visit.c:1542
err = 0x0
#11 visit_type_BlockDeviceStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f905a0, name=name@entry=0x7f5d03f032ec "stats", errp=errp@entry=0x7ffddb417ca0) at qapi-visit.c:1566
err = 0x0
#12 0x00007f5d03ebf5b1 in visit_type_BlockStats_fields (errp=0x7ffddb417cf0, obj=0x7f5d07f90650, m=0x7f5d061fdea0) at qapi-visit.c:1614
err = 0x0
#13 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f90650, name=name@entry=0x7f5d03f480f4 "parent", errp=errp@entry=0x7ffddb417cf0) at qapi-visit.c:1644
err = 0x0
#14 0x00007f5d03ebf6bd in visit_type_BlockStats_fields (errp=0x7ffddb417d38, obj=0x7f5d07c67a50, m=0x7f5d061fdea0) at qapi-visit.c:1620
err = 0x0
#15 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07c67a50, name=name@entry=0x0, errp=errp@entry=0x7ffddb417d38) at qapi-visit.c:1644
err = 0x0
#16 0x00007f5d03ebf760 in visit_type_BlockStatsList (m=0x7f5d061fdea0, obj=obj@entry=0x7ffddb417d98, name=name@entry=0x7f5d03f00e6e "unused", errp=errp@entry=0x7ffddb417da0) at qapi-visit.c:1665
native_i = <optimized out>
err = 0x0
i = 0x7f5d07c67a50
prev = 0x7ffddb417d40
#17 0x00007f5d03d674dd in qmp_marshal_output_query_blockstats (errp=0x7ffddb417d90, ret_out=0x7ffddb417e10, ret_in=0x7f5d07c67120) at qmp-marshal.c:182
local_err = 0x0
mo = 0x7f5d061fdea0
md = <optimized out>
v = <optimized out>
#18 qmp_marshal_input_query_blockstats (mon=<optimized out>, qdict=<optimized out>, ret=0x7ffddb417e10) at qmp-marshal.c:225
local_err = 0x0
args = <optimized out>
retval = <optimized out>
mi = 0x7f5d064e2000
md = <optimized out>
v = <optimized out>
has_query_nodes = false
query_nodes = false
#19 0x00007f5d03ca0531 in qmp_call_cmd (cmd=<optimized out>, params=0x7f5d075dd600, mon=0x7f5d06208320) at /usr/src/debug/qemu-2.3.0/monitor.c:5051
ret = <optimized out>
data = 0x0
#20 handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5113
err = <optimized out>
obj = <optimized out>
input = <optimized out>
args = 0x7f5d075dd600
cmd_name = <optimized out>
mon = 0x7f5d06208320
#21 0x00007f5d03edf4f2 in json_message_process_token (lexer=0x7f5d061f5d70, token=0x7f5d061991e0, type=JSON_OPERATOR, x=48, y=15) at qobject/json-streamer.c:87
parser = 0x7f5d061f5d68
dict = 0x7f5d088ea800
#22 0x00007f5d03ef191f in json_lexer_feed_char (lexer=lexer@entry=0x7f5d061f5d70, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303
new_state = 100
#23 0x00007f5d03ef19ee in json_lexer_feed (lexer=0x7f5d061f5d70, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356
err = <optimized out>
i = <optimized out>
#24 0x00007f5d03edf689 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110
No locals.
#25 0x00007f5d03c9e8cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134
old_mon = 0x0
#26 0x00007f5d03d5b1b0 in qemu_chr_be_write (len=<optimized out>, buf=0x7ffddb417f40 "}\177A\333\375\177", s=0x7f5d0625a2e0) at qemu-char.c:305
No locals.
#27 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f5d0625a2e0) at qemu-char.c:2870
chr = 0x7f5d0625a2e0
s = 0x7f5d061aa3f0
buf = "}\177A\333\375\177\000\000\360\360\355\003]\177\000\000\030\003\000\000\000\000\000\000\205\336\355\003]\177\000\000\000A\036\b]\177\000\000\263\325\355\003]\177\000\000\000A\036\b]\177", '\000' <repeats 18 times>, "`2\036\b]\177\000\000ЀA\333\375\177\000\000\000\000\000\000\000\000\000\000 \232\261\a]\177\000\000\000(\341\006]\177\000\000P\217A\333\375\177\000\000H\242<\374\\\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\240\200A\333\375\177\000\000\340\177A\333\375\177\000\000\r\000\000\000\000\000\000\000 Xz\b]\177\000\000\000@&\006]\177\000\000\340\365\215\b]\177\000\000Q\000\000\000\000\000\000\000\232"...
len = <optimized out>
size = <optimized out>
#28 0x00007f5d020b099a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
No symbol table info available.
#29 0x00007f5d03e74288 in glib_pollfds_poll () at main-loop.c:209
context = 0x7f5d06205140
pfds = <optimized out>
#30 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254
ret = 2
spin_counter = 0
#31 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503
ret = 2
timeout = 4294967295
timeout_ns = <optimized out>
#32 0x00007f5d03c73a4e in main_loop () at vl.c:1818
nonblocking = <optimized out>
last_io = 2
#33 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
i = <optimized out>
snapshot = <optimized out>
linux_boot = <optimized out>
initrd_filename = <optimized out>
kernel_filename = <optimized out>
kernel_cmdline = <optimized out>
boot_order = 0x7f5d03f06a67 "cad"
boot_once = 0x0
cyls = <optimized out>
heads = <optimized out>
secs = <optimized out>
translation = <optimized out>
hda_opts = <optimized out>
opts = <optimized out>
machine_opts = <optimized out>
icount_opts = <optimized out>
olist = <optimized out>
optind = 67
optarg = 0x7f5d06193570 "rhel6.5.0"
loadvm = <optimized out>
machine_class = <optimized out>
cpu_model = <optimized out>
vga_model = 0x0
qtest_chrdev = <optimized out>
qtest_log = <optimized out>
pid_file = <optimized out>
incoming = <optimized out>
show_vnc_port = <optimized out>
defconfig = <optimized out>
userconfig = 56
log_mask = <optimized out>
log_file = <optimized out>
mem_trace = {malloc = 0x7f5d03d61480 <malloc_and_trace>, realloc = 0x7f5d03d61460 <realloc_and_trace>, free = 0x7f5d03d61450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
trace_events = <optimized out>
trace_file = <optimized out>
maxram_size = <optimized out>
ram_slots = <optimized out>
vmstate_dump_file = <optimized out>
main_loop_err = 0x0
__func__ = "main"
-----Ursprüngliche Nachricht-----
Von: qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org [mailto:qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org] Im Auftrag von Grundmann, Christian
Gesendet: Dienstag, 17. November 2015 15:12
An: 'Dr. David Alan Gilbert' <dgilbert@redhat.com>
Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
Here you go
gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
[New LWP 52281]
[New LWP 52288]
[New LWP 52286]
[New LWP 52291]
[New LWP 52292]
[New LWP 52287]
[New LWP 52293]
[New LWP 52290]
[New LWP 56455]
[New LWP 52289]
[New LWP 52282]
[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,us'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4
Thread 11 (Thread 0x7f6d47719700 (LWP 52282)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
No locals.
#1 0x00007f6d57426272 in futex_wait (val=4294967295, ev=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals.
#2 qemu_event_wait (ev=ev@entry=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399
value = <optimized out>
#3 0x00007f6d57434526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233
tries = 0
n = <optimized out>
node = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d47719700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d47719700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107326789376, 5884348200482620104, 0, 140107326790080, 140107326789376, 140107592666688, -5804038895876586808, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8a4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d44f139e0, reg_save_area = 0x7f6d44f139a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570cf000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8a4000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44f14700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d44f14700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107284825856, 5884348200482620104, 0, 140107284826560, 140107284825856, 140726431086992, -5804033392412867896, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)):
#0 sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
No locals.
#1 0x00007f6d574260c7 in qemu_sem_timedwait (sem=sem@entry=0x7f6d5a1b9248, ms=ms@entry=10000) at util/qemu-thread-posix.c:254
rc = <optimized out>
ts = {tv_sec = 1447709021, tv_nsec = 21985000}
__func__ = "qemu_sem_timedwait"
#2 0x00007f6d573a98ac in worker_thread (opaque=0x7f6d5a1b91e0) at thread-pool.c:92
req = <optimized out>
ret = <optimized out>
pool = 0x7f6d5a1b91e0
#3 0x00007f6d55ceadf5 in start_thread (arg=0x7f6ab1dff700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6ab1dff700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140096227505920, 5884348200482620104, 0, 140096227506624, 140096227505920, 26, -5801757560646548792, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#4 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 8 (Thread 0x7f6d44713700 (LWP 52290)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8b8000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d447129e0, reg_save_area = 0x7f6d447129a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570cc000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8b8000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44713700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d44713700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107276433152, 5884348200482620104, 0, 140107276433856, 140107276433152, 140726431086992, -5804032293438111032, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8f4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d42f0f9e0, reg_save_area = 0x7f6d42f0f9a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570c3000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8f4000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d42f10700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d42f10700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107251255040, 5884348200482620104, 0, 140107251255744, 140107251255040, 140726431086992, -5804046580109950264, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c878000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d45f159e0, reg_save_area = 0x7f6d45f159a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570d5000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c878000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45f16700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d45f16700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107301611264, 5884348200482620104, 0, 140107301611968, 140107301611264, 140726431086992, -5804035590362381624, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 5 (Thread 0x7f6d43711700 (LWP 52292)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8e0000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d437109e0, reg_save_area = 0x7f6d437109a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570c6000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8e0000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43711700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d43711700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107259647744, 5884348200482620104, 0, 140107259648448, 140107259647744, 140726431086992, -5804047687674641720, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8cc000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d43f119e0, reg_save_area = 0x7f6d43f119a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570c9000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c8cc000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43f12700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d43f12700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107268040448, 5884348200482620104, 0, 140107268041152, 140107268040448, 140726431086992, -5804048786649398584, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 3 (Thread 0x7f6d46717700 (LWP 52286)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c810000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d467169e0, reg_save_area = 0x7f6d467169a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570d8000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c810000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d46717700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d46717700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107310003968, 5884348200482620104, 0, 140107310004672, 140107310003968, 140726431086992, -5804036689337138488, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 2 (Thread 0x7f6d45715700 (LWP 52288)):
#0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c890000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
ret = <optimized out>
arg = <optimized out>
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d457149e0, reg_save_area = 0x7f6d457149a0}}
#2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
run = 0x7f6d570d2000
ret = <optimized out>
run_ret = <optimized out>
#3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
cpu = 0x7f6d5c890000
r = <optimized out>
#4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45715700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7f6d45715700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107293218560, 5884348200482620104, 0, 140107293219264, 140107293218560, 140726431086992, -5804034491387624760, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)):
#0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 No symbol table info available.
#1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=49280) at vl.c:2575
ptr = 0x7f6d59a346a0
#2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0 No symbol table info available.
#3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0 No symbol table info available.
#4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33
req = <optimized out>
#5 virtio_blk_get_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:192
req = <optimized out>
#6 virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604
s = 0x7f6d5de1ff40
__func__ = "virtio_blk_handle_output"
mrb = {reqs = {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7c000, 0x0 <repeats 29 times>}, num_reqs = 3, is_write = true}
#7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=0x7f6d5a1aea00, ret=62, ret@entry=1) at iohandler.c:143
revents = 1
#8 0x00007f6d573b7296 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:504
ret = 1
timeout = 4294967295
timeout_ns = <optimized out>
#9 0x00007f6d571b6a4e in main_loop () at vl.c:1818
nonblocking = <optimized out>
last_io = 0
#10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
i = <optimized out>
snapshot = <optimized out>
linux_boot = <optimized out>
initrd_filename = <optimized out>
kernel_filename = <optimized out>
kernel_cmdline = <optimized out>
boot_order = 0x7f6d57449a67 "cad"
boot_once = 0x0
cyls = <optimized out>
heads = <optimized out>
secs = <optimized out>
translation = <optimized out>
hda_opts = <optimized out>
opts = <optimized out>
machine_opts = <optimized out>
icount_opts = <optimized out>
olist = <optimized out>
optind = 69
optarg = 0x7f6d5a14b3a0 "rhel6.5.0"
loadvm = <optimized out>
machine_class = <optimized out>
cpu_model = <optimized out>
vga_model = 0x0
qtest_chrdev = <optimized out>
qtest_log = <optimized out>
pid_file = <optimized out>
incoming = <optimized out>
show_vnc_port = <optimized out>
defconfig = <optimized out>
userconfig = 179
log_mask = <optimized out>
log_file = <optimized out>
mem_trace = {malloc = 0x7f6d572a4480 <malloc_and_trace>, realloc = 0x7f6d572a4460 <realloc_and_trace>, free = 0x7f6d572a4450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
trace_events = <optimized out>
trace_file = <optimized out>
maxram_size = <optimized out>
ram_slots = <optimized out>
vmstate_dump_file = <optimized out>
main_loop_err = 0x0
__func__ = "main"
@ Do you think you're only hitting these crashes on VMs that have been paused because of these space errors?
Will have a look on that
Thx Christian
-----Ursprüngliche Nachricht-----
Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
Gesendet: Dienstag, 17. November 2015 12:36
An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> Hi,
>
> @ Can you please use a 'thread apply all bt full' the full gives a little more info.
>
> gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
OK, it doesn't relaly give any more without the debuginfo package mentioned below.
<snip>
> @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
> Sorry it's a ovirt-node System where I can't you yum
Ah, although perhaps if you took the core dump, onto another machine with matching qemu and debuginfo you should be able to get more detail.
> @ Does this part always look the same in your backtraces?
> The most are the same, found one a little bit different :
> Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)):
> #0 0x00007f3785d18353 in
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from
> /lib64/libtcmalloc.so.4 No symbol table info available.
> #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No
> symbol table info available.
> #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No
> symbol table info available.
> #4 0x00007f37885fec89 in g_slice_free1 () from
> /lib64/libglib-2.0.so.0 No symbol table info available.
> #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table
> info available.
> #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info
> available.
> #7 0x00007f378a398394 in aio_bh_poll () No symbol table info
> available.
> #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table info
> available.
> #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info
> available.
> #10 0x00007f37885e299a in g_main_context_dispatch () from
> /lib64/libglib-2.0.so.0 No symbol table info available.
> #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info
> available.
> #12 0x00007f378a1a5a4e in main ()
> No symbol table info available.
>
OK, that's a bit different but interesting....
> @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> No nothing abnormal
>
> @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot.
> Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline.
OK, that's interesting, because you may be hitting the following bug; http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html
whose fix coincidentally just got accepted today; it's related to error cases with error=stop which you are using.
Do you think you're only hitting these crashes on VMs that have been paused because of these space errors?
> disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
>
> Ok will try that and report
Thanks,
Dave
>
> Thx Christian
>
>
> -----Ursprüngliche Nachricht-----
> Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> Gesendet: Dienstag, 17. November 2015 10:59
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>;
> stefanha@redhat.com
> Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in
> libtcmalloc
>
> * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > Hi,
> > Dan sent me over to you,
> > please let me know if i can provide additional informations
>
> Hi Christian,
> Thanks for reporting this,
>
> > Softwareversions:
> > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso
> >
> > qemu-img-ev-2.3.0-29.1.el7.x86_64
> > qemu-kvm-ev-2.3.0-29.1.el7.x86_64
> > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
> > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
> > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
> > kernel-3.10.0-229.14.1.el7.x86_64
> > gperftools-libs-2.4-7.el7.x86_64
> >
> > Commandline:
> > /usr/libexec/qemu-kvm -name myvmname -S -machine
> > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime
> > mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios
> > type=1,manufacturer=oVirt,product=oVirt
> > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-
> > 5A
> > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319
> > -nographic -no-user-config -nodefaults -chardev
> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,se
> > rv er,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc
> > base=2015-11-15T20:04:35,driftfix=slew -global
> > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
> > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> > -device
> > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
> > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
> > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
> > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> > -drive
> > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84
> > -8
> > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffae
> > cf
> > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,
> > fo
> > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,we
> > rr or=stop,rerror=stop,aio=native -device
> > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,
> > id
> > =virtio-disk0,bootindex=1 -netdev
> > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device
> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci
> > .0
> > ,addr=0x3 -chardev
> > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-
> > 5a 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait
> > -device
> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=cha
> > nn el0,name=com.redhat.rhevm.vdsm -chardev
> > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-
> > 5a 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait
> > -device
> > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=cha
> > nn
> > el1,name=org.qemu.guest_agent.0 -device
> > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
> >
> > Stack Trace:
> >
> > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt"
>
> Can you please use a 'thread apply all bt full' the full gives a little more info.
> Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
>
> > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)):
> > #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4
> > #1 0x00007fa8b186b489 in malloc_and_trace ()
> > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0
> > #3 0x00007fa8afbd666e in g_slice_alloc () from
> > /lib64/libglib-2.0.so.0
> > #4 0x00007fa8b17cbffd in virtio_blk_handle_output ()
> > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll ()
> > #6 0x00007fa8b197e296 in main_loop_wait ()
> > #7 0x00007fa8b177da4e in main ()
>
> Does this part always look the same in your backtraces?
> The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed.
>
> Some other ideas:
> 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> 3) If this is pretty repeatable, then it would be interesting to try changing to a different
> disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
>
> Dave
> >
> >
> > Thx Christian
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Dan Kenigsberg [mailto:danken@redhat.com]
> > Gesendet: Freitag, 13. November 2015 20:00
> > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> > Cc: 'users@ovirt.org' <users@ovirt.org>
> > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc
> >
> > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote:
> > > Hi,
> > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso"
> > > (is there something better to use?) fort he nodes, and have random
> > > crashes of VMs The dumps are always the Same
> > >
> > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump
> > > [Thread debugging using libthread_db enabled] Using host
> > > libthread_db library "/lib64/libthread_db.so.1".
> > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'.
> > > Program terminated with signal 11, Segmentation fault.
> > > #0 0x00007f0c559c4353 in
> > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::
> > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> > >
> > >
> > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist
> > > centos7 or 3.6
> >
> > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt.
> >
> > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible.
> >
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-11-17 14:20 ` Grundmann, Christian
@ 2015-11-17 14:42 ` Dr. David Alan Gilbert
2015-11-19 16:00 ` Grundmann, Christian
0 siblings, 1 reply; 16+ messages in thread
From: Dr. David Alan Gilbert @ 2015-11-17 14:42 UTC (permalink / raw)
To: Grundmann, Christian; +Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com
* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> And here another one
Oh this is a bit of a different one, from query-blockstats, although again if
the heap's corrupted it might have just been the first guy to trip over the
corrupt part afterwards.
Dave
> Thread 1 (Thread 0x7f5d03ba5c00 (LWP 1507)):
> #0 0x00007f5cff7e2e7d in tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #1 0x00007f5cff7e312a in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #2 0x00007f5cff7e31dd in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #3 0x00007f5cff7e6235 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #4 0x00007f5cff7f4d5b in tc_malloc () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #5 0x00007f5d03d61489 in malloc_and_trace (n_bytes=18) at vl.c:2575
> ptr = 0x1
> #6 0x00007f5d020b647f in g_malloc () from /lib64/libglib-2.0.so.0
> No symbol table info available.
> #7 0x00007f5d020cdf7f in g_strdup () from /lib64/libglib-2.0.so.0
> No symbol table info available.
> #8 0x00007f5d03eddab5 in alloc_entry (value=0x7f5d088de6c0, key=0x7f5d03f5debb "wr_highest_offset") at qobject/qdict.c:79
> entry = 0x7f5d088df480
> #9 qdict_put_obj (qdict=0x7f5d06e10400, key=0x7f5d03f5debb "wr_highest_offset", value=0x7f5d088de6c0) at qobject/qdict.c:145
> bucket = 81
> entry = <optimized out>
> #10 0x00007f5d03ebf34a in visit_type_BlockDeviceStats_fields (errp=0x7ffddb417ca0, obj=0x7f5d07f905a0, m=0x7f5d061fdea0) at qapi-visit.c:1542
> err = 0x0
> #11 visit_type_BlockDeviceStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f905a0, name=name@entry=0x7f5d03f032ec "stats", errp=errp@entry=0x7ffddb417ca0) at qapi-visit.c:1566
> err = 0x0
> #12 0x00007f5d03ebf5b1 in visit_type_BlockStats_fields (errp=0x7ffddb417cf0, obj=0x7f5d07f90650, m=0x7f5d061fdea0) at qapi-visit.c:1614
> err = 0x0
> #13 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f90650, name=name@entry=0x7f5d03f480f4 "parent", errp=errp@entry=0x7ffddb417cf0) at qapi-visit.c:1644
> err = 0x0
> #14 0x00007f5d03ebf6bd in visit_type_BlockStats_fields (errp=0x7ffddb417d38, obj=0x7f5d07c67a50, m=0x7f5d061fdea0) at qapi-visit.c:1620
> err = 0x0
> #15 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07c67a50, name=name@entry=0x0, errp=errp@entry=0x7ffddb417d38) at qapi-visit.c:1644
> err = 0x0
> #16 0x00007f5d03ebf760 in visit_type_BlockStatsList (m=0x7f5d061fdea0, obj=obj@entry=0x7ffddb417d98, name=name@entry=0x7f5d03f00e6e "unused", errp=errp@entry=0x7ffddb417da0) at qapi-visit.c:1665
> native_i = <optimized out>
> err = 0x0
> i = 0x7f5d07c67a50
> prev = 0x7ffddb417d40
> #17 0x00007f5d03d674dd in qmp_marshal_output_query_blockstats (errp=0x7ffddb417d90, ret_out=0x7ffddb417e10, ret_in=0x7f5d07c67120) at qmp-marshal.c:182
> local_err = 0x0
> mo = 0x7f5d061fdea0
> md = <optimized out>
> v = <optimized out>
> #18 qmp_marshal_input_query_blockstats (mon=<optimized out>, qdict=<optimized out>, ret=0x7ffddb417e10) at qmp-marshal.c:225
> local_err = 0x0
> args = <optimized out>
> retval = <optimized out>
> mi = 0x7f5d064e2000
> md = <optimized out>
> v = <optimized out>
> has_query_nodes = false
> query_nodes = false
> #19 0x00007f5d03ca0531 in qmp_call_cmd (cmd=<optimized out>, params=0x7f5d075dd600, mon=0x7f5d06208320) at /usr/src/debug/qemu-2.3.0/monitor.c:5051
> ret = <optimized out>
> data = 0x0
> #20 handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5113
> err = <optimized out>
> obj = <optimized out>
> input = <optimized out>
> args = 0x7f5d075dd600
> cmd_name = <optimized out>
> mon = 0x7f5d06208320
> #21 0x00007f5d03edf4f2 in json_message_process_token (lexer=0x7f5d061f5d70, token=0x7f5d061991e0, type=JSON_OPERATOR, x=48, y=15) at qobject/json-streamer.c:87
> parser = 0x7f5d061f5d68
> dict = 0x7f5d088ea800
> #22 0x00007f5d03ef191f in json_lexer_feed_char (lexer=lexer@entry=0x7f5d061f5d70, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303
> new_state = 100
> #23 0x00007f5d03ef19ee in json_lexer_feed (lexer=0x7f5d061f5d70, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356
> err = <optimized out>
> i = <optimized out>
> #24 0x00007f5d03edf689 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110
> No locals.
> #25 0x00007f5d03c9e8cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134
> old_mon = 0x0
> #26 0x00007f5d03d5b1b0 in qemu_chr_be_write (len=<optimized out>, buf=0x7ffddb417f40 "}\177A\333\375\177", s=0x7f5d0625a2e0) at qemu-char.c:305
> No locals.
> #27 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f5d0625a2e0) at qemu-char.c:2870
> chr = 0x7f5d0625a2e0
> s = 0x7f5d061aa3f0
> buf = "}\177A\333\375\177\000\000\360\360\355\003]\177\000\000\030\003\000\000\000\000\000\000\205\336\355\003]\177\000\000\000A\036\b]\177\000\000\263\325\355\003]\177\000\000\000A\036\b]\177", '\000' <repeats 18 times>, "`2\036\b]\177\000\000ЀA\333\375\177\000\000\000\000\000\000\000\000\000\000 \232\261\a]\177\000\000\000(\341\006]\177\000\000P\217A\333\375\177\000\000H\242<\374\\\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\240\200A\333\375\177\000\000\340\177A\333\375\177\000\000\r\000\000\000\000\000\000\000 Xz\b]\177\000\000\000@&\006]\177\000\000\340\365\215\b]\177\000\000Q\000\000\000\000\000\000\000\232"...
> len = <optimized out>
> size = <optimized out>
> #28 0x00007f5d020b099a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
> No symbol table info available.
> #29 0x00007f5d03e74288 in glib_pollfds_poll () at main-loop.c:209
> context = 0x7f5d06205140
> pfds = <optimized out>
> #30 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254
> ret = 2
> spin_counter = 0
> #31 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503
> ret = 2
> timeout = 4294967295
> timeout_ns = <optimized out>
> #32 0x00007f5d03c73a4e in main_loop () at vl.c:1818
> nonblocking = <optimized out>
> last_io = 2
> #33 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
> i = <optimized out>
> snapshot = <optimized out>
> linux_boot = <optimized out>
> initrd_filename = <optimized out>
> kernel_filename = <optimized out>
> kernel_cmdline = <optimized out>
> boot_order = 0x7f5d03f06a67 "cad"
> boot_once = 0x0
> cyls = <optimized out>
> heads = <optimized out>
> secs = <optimized out>
> translation = <optimized out>
> hda_opts = <optimized out>
> opts = <optimized out>
> machine_opts = <optimized out>
> icount_opts = <optimized out>
> olist = <optimized out>
> optind = 67
> optarg = 0x7f5d06193570 "rhel6.5.0"
> loadvm = <optimized out>
> machine_class = <optimized out>
> cpu_model = <optimized out>
> vga_model = 0x0
> qtest_chrdev = <optimized out>
> qtest_log = <optimized out>
> pid_file = <optimized out>
> incoming = <optimized out>
> show_vnc_port = <optimized out>
> defconfig = <optimized out>
> userconfig = 56
> log_mask = <optimized out>
> log_file = <optimized out>
> mem_trace = {malloc = 0x7f5d03d61480 <malloc_and_trace>, realloc = 0x7f5d03d61460 <realloc_and_trace>, free = 0x7f5d03d61450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
> trace_events = <optimized out>
> trace_file = <optimized out>
> maxram_size = <optimized out>
> ram_slots = <optimized out>
> vmstate_dump_file = <optimized out>
> main_loop_err = 0x0
> __func__ = "main"
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org [mailto:qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org] Im Auftrag von Grundmann, Christian
> Gesendet: Dienstag, 17. November 2015 15:12
> An: 'Dr. David Alan Gilbert' <dgilbert@redhat.com>
> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
> Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
>
> Here you go
>
>
> gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
> [New LWP 52281]
> [New LWP 52288]
> [New LWP 52286]
> [New LWP 52291]
> [New LWP 52292]
> [New LWP 52287]
> [New LWP 52293]
> [New LWP 52290]
> [New LWP 56455]
> [New LWP 52289]
> [New LWP 52282]
> [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,us'.
> Program terminated with signal 11, Segmentation fault.
> #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4
>
> Thread 11 (Thread 0x7f6d47719700 (LWP 52282)):
> #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
> No locals.
> #1 0x00007f6d57426272 in futex_wait (val=4294967295, ev=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals.
> #2 qemu_event_wait (ev=ev@entry=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399
> value = <optimized out>
> #3 0x00007f6d57434526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233
> tries = 0
> n = <optimized out>
> node = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d47719700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d47719700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107326789376, 5884348200482620104, 0, 140107326790080, 140107326789376, 140107592666688, -5804038895876586808, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8a4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d44f139e0, reg_save_area = 0x7f6d44f139a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570cf000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8a4000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44f14700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d44f14700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107284825856, 5884348200482620104, 0, 140107284826560, 140107284825856, 140726431086992, -5804033392412867896, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)):
> #0 sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
> No locals.
> #1 0x00007f6d574260c7 in qemu_sem_timedwait (sem=sem@entry=0x7f6d5a1b9248, ms=ms@entry=10000) at util/qemu-thread-posix.c:254
> rc = <optimized out>
> ts = {tv_sec = 1447709021, tv_nsec = 21985000}
> __func__ = "qemu_sem_timedwait"
> #2 0x00007f6d573a98ac in worker_thread (opaque=0x7f6d5a1b91e0) at thread-pool.c:92
> req = <optimized out>
> ret = <optimized out>
> pool = 0x7f6d5a1b91e0
> #3 0x00007f6d55ceadf5 in start_thread (arg=0x7f6ab1dff700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6ab1dff700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140096227505920, 5884348200482620104, 0, 140096227506624, 140096227505920, 26, -5801757560646548792, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #4 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 8 (Thread 0x7f6d44713700 (LWP 52290)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8b8000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d447129e0, reg_save_area = 0x7f6d447129a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570cc000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8b8000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44713700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d44713700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107276433152, 5884348200482620104, 0, 140107276433856, 140107276433152, 140726431086992, -5804032293438111032, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8f4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d42f0f9e0, reg_save_area = 0x7f6d42f0f9a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570c3000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8f4000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d42f10700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d42f10700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107251255040, 5884348200482620104, 0, 140107251255744, 140107251255040, 140726431086992, -5804046580109950264, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c878000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d45f159e0, reg_save_area = 0x7f6d45f159a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570d5000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c878000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45f16700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d45f16700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107301611264, 5884348200482620104, 0, 140107301611968, 140107301611264, 140726431086992, -5804035590362381624, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 5 (Thread 0x7f6d43711700 (LWP 52292)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8e0000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d437109e0, reg_save_area = 0x7f6d437109a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570c6000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8e0000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43711700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d43711700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107259647744, 5884348200482620104, 0, 140107259648448, 140107259647744, 140726431086992, -5804047687674641720, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8cc000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d43f119e0, reg_save_area = 0x7f6d43f119a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570c9000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8cc000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43f12700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d43f12700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107268040448, 5884348200482620104, 0, 140107268041152, 140107268040448, 140726431086992, -5804048786649398584, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 3 (Thread 0x7f6d46717700 (LWP 52286)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c810000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d467169e0, reg_save_area = 0x7f6d467169a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570d8000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c810000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d46717700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d46717700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107310003968, 5884348200482620104, 0, 140107310004672, 140107310003968, 140726431086992, -5804036689337138488, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 2 (Thread 0x7f6d45715700 (LWP 52288)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c890000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d457149e0, reg_save_area = 0x7f6d457149a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570d2000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c890000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45715700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d45715700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107293218560, 5884348200482620104, 0, 140107293219264, 140107293218560, 140726431086992, -5804034491387624760, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)):
> #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 No symbol table info available.
> #1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=49280) at vl.c:2575
> ptr = 0x7f6d59a346a0
> #2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0 No symbol table info available.
> #3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0 No symbol table info available.
> #4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33
> req = <optimized out>
> #5 virtio_blk_get_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:192
> req = <optimized out>
> #6 virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604
> s = 0x7f6d5de1ff40
> __func__ = "virtio_blk_handle_output"
> mrb = {reqs = {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7c000, 0x0 <repeats 29 times>}, num_reqs = 3, is_write = true}
> #7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=0x7f6d5a1aea00, ret=62, ret@entry=1) at iohandler.c:143
> revents = 1
> #8 0x00007f6d573b7296 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:504
> ret = 1
> timeout = 4294967295
> timeout_ns = <optimized out>
> #9 0x00007f6d571b6a4e in main_loop () at vl.c:1818
> nonblocking = <optimized out>
> last_io = 0
> #10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
> i = <optimized out>
> snapshot = <optimized out>
> linux_boot = <optimized out>
> initrd_filename = <optimized out>
> kernel_filename = <optimized out>
> kernel_cmdline = <optimized out>
> boot_order = 0x7f6d57449a67 "cad"
> boot_once = 0x0
> cyls = <optimized out>
> heads = <optimized out>
> secs = <optimized out>
> translation = <optimized out>
> hda_opts = <optimized out>
> opts = <optimized out>
> machine_opts = <optimized out>
> icount_opts = <optimized out>
> olist = <optimized out>
> optind = 69
> optarg = 0x7f6d5a14b3a0 "rhel6.5.0"
> loadvm = <optimized out>
> machine_class = <optimized out>
> cpu_model = <optimized out>
> vga_model = 0x0
> qtest_chrdev = <optimized out>
> qtest_log = <optimized out>
> pid_file = <optimized out>
> incoming = <optimized out>
> show_vnc_port = <optimized out>
> defconfig = <optimized out>
> userconfig = 179
> log_mask = <optimized out>
> log_file = <optimized out>
> mem_trace = {malloc = 0x7f6d572a4480 <malloc_and_trace>, realloc = 0x7f6d572a4460 <realloc_and_trace>, free = 0x7f6d572a4450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
> trace_events = <optimized out>
> trace_file = <optimized out>
> maxram_size = <optimized out>
> ram_slots = <optimized out>
> vmstate_dump_file = <optimized out>
> main_loop_err = 0x0
> __func__ = "main"
>
>
>
>
>
> @ Do you think you're only hitting these crashes on VMs that have been paused because of these space errors?
> Will have a look on that
>
> Thx Christian
>
> -----Ursprüngliche Nachricht-----
> Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> Gesendet: Dienstag, 17. November 2015 12:36
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
> Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
>
> * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > Hi,
> >
> > @ Can you please use a 'thread apply all bt full' the full gives a little more info.
> >
> > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
>
> OK, it doesn't relaly give any more without the debuginfo package mentioned below.
>
> <snip>
>
> > @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
> > Sorry it's a ovirt-node System where I can't you yum
>
> Ah, although perhaps if you took the core dump, onto another machine with matching qemu and debuginfo you should be able to get more detail.
>
> > @ Does this part always look the same in your backtraces?
> > The most are the same, found one a little bit different :
> > Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)):
> > #0 0x00007f3785d18353 in
> > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> > #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from
> > /lib64/libtcmalloc.so.4 No symbol table info available.
> > #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No
> > symbol table info available.
> > #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No
> > symbol table info available.
> > #4 0x00007f37885fec89 in g_slice_free1 () from
> > /lib64/libglib-2.0.so.0 No symbol table info available.
> > #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table
> > info available.
> > #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info
> > available.
> > #7 0x00007f378a398394 in aio_bh_poll () No symbol table info
> > available.
> > #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table info
> > available.
> > #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info
> > available.
> > #10 0x00007f37885e299a in g_main_context_dispatch () from
> > /lib64/libglib-2.0.so.0 No symbol table info available.
> > #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info
> > available.
> > #12 0x00007f378a1a5a4e in main ()
> > No symbol table info available.
> >
>
> OK, that's a bit different but interesting....
>
> > @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> > No nothing abnormal
> >
> > @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> > Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot.
> > Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline.
>
> OK, that's interesting, because you may be hitting the following bug; http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html
>
> whose fix coincidentally just got accepted today; it's related to error cases with error=stop which you are using.
>
> Do you think you're only hitting these crashes on VMs that have been paused because of these space errors?
>
> > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
> >
> > Ok will try that and report
>
> Thanks,
>
> Dave
>
> >
> > Thx Christian
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> > Gesendet: Dienstag, 17. November 2015 10:59
> > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>;
> > stefanha@redhat.com
> > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in
> > libtcmalloc
> >
> > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > > Hi,
> > > Dan sent me over to you,
> > > please let me know if i can provide additional informations
> >
> > Hi Christian,
> > Thanks for reporting this,
> >
> > > Softwareversions:
> > > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso
> > >
> > > qemu-img-ev-2.3.0-29.1.el7.x86_64
> > > qemu-kvm-ev-2.3.0-29.1.el7.x86_64
> > > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
> > > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
> > > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
> > > kernel-3.10.0-229.14.1.el7.x86_64
> > > gperftools-libs-2.4-7.el7.x86_64
> > >
> > > Commandline:
> > > /usr/libexec/qemu-kvm -name myvmname -S -machine
> > > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime
> > > mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> > > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios
> > > type=1,manufacturer=oVirt,product=oVirt
> > > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-
> > > 5A
> > > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319
> > > -nographic -no-user-config -nodefaults -chardev
> > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,se
> > > rv er,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc
> > > base=2015-11-15T20:04:35,driftfix=slew -global
> > > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
> > > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> > > -device
> > > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
> > > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
> > > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
> > > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> > > -drive
> > > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84
> > > -8
> > > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffae
> > > cf
> > > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,
> > > fo
> > > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,we
> > > rr or=stop,rerror=stop,aio=native -device
> > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,
> > > id
> > > =virtio-disk0,bootindex=1 -netdev
> > > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device
> > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci
> > > .0
> > > ,addr=0x3 -chardev
> > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-
> > > 5a 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait
> > > -device
> > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=cha
> > > nn el0,name=com.redhat.rhevm.vdsm -chardev
> > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-
> > > 5a 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait
> > > -device
> > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=cha
> > > nn
> > > el1,name=org.qemu.guest_agent.0 -device
> > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
> > >
> > > Stack Trace:
> > >
> > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt"
> >
> > Can you please use a 'thread apply all bt full' the full gives a little more info.
> > Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
> >
> > > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)):
> > > #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4
> > > #1 0x00007fa8b186b489 in malloc_and_trace ()
> > > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0
> > > #3 0x00007fa8afbd666e in g_slice_alloc () from
> > > /lib64/libglib-2.0.so.0
> > > #4 0x00007fa8b17cbffd in virtio_blk_handle_output ()
> > > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll ()
> > > #6 0x00007fa8b197e296 in main_loop_wait ()
> > > #7 0x00007fa8b177da4e in main ()
> >
> > Does this part always look the same in your backtraces?
> > The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed.
> >
> > Some other ideas:
> > 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> > 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> > 3) If this is pretty repeatable, then it would be interesting to try changing to a different
> > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
> >
> > Dave
> > >
> > >
> > > Thx Christian
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Dan Kenigsberg [mailto:danken@redhat.com]
> > > Gesendet: Freitag, 13. November 2015 20:00
> > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> > > Cc: 'users@ovirt.org' <users@ovirt.org>
> > > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc
> > >
> > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote:
> > > > Hi,
> > > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso"
> > > > (is there something better to use?) fort he nodes, and have random
> > > > crashes of VMs The dumps are always the Same
> > > >
> > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump
> > > > [Thread debugging using libthread_db enabled] Using host
> > > > libthread_db library "/lib64/libthread_db.so.1".
> > > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'.
> > > > Program terminated with signal 11, Segmentation fault.
> > > > #0 0x00007f0c559c4353 in
> > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::
> > > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> > > >
> > > >
> > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist
> > > > centos7 or 3.6
> > >
> > > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt.
> > >
> > > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible.
> > >
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-11-17 14:42 ` Dr. David Alan Gilbert
@ 2015-11-19 16:00 ` Grundmann, Christian
2015-11-19 17:02 ` Paolo Bonzini
2015-11-20 19:06 ` Dr. David Alan Gilbert
0 siblings, 2 replies; 16+ messages in thread
From: Grundmann, Christian @ 2015-11-19 16:00 UTC (permalink / raw)
To: 'Dr. David Alan Gilbert'
Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com
Hi,
it seems that using virtio-scsi did the trick,
But now the VMs are pausing without an coredump, so the underlying Problem (no storage Error) is not fixed,
As I am using Snapshots (and so the disks have to grow very fast) I try if tuning "volume_utilization_percent" and "volume_utilization_chunk_mb" will help (https://access.redhat.com/solutions/130843)
Thx Christian
-----Ursprüngliche Nachricht-----
Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
Gesendet: Dienstag, 17. November 2015 15:42
An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> And here another one
Oh this is a bit of a different one, from query-blockstats, although again if the heap's corrupted it might have just been the first guy to trip over the corrupt part afterwards.
Dave
> Thread 1 (Thread 0x7f5d03ba5c00 (LWP 1507)):
> #0 0x00007f5cff7e2e7d in
> tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> #1 0x00007f5cff7e312a in
> tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> #2 0x00007f5cff7e31dd in
> tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> #3 0x00007f5cff7e6235 in
> tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> #4 0x00007f5cff7f4d5b in tc_malloc () from /lib64/libtcmalloc.so.4 No
> symbol table info available.
> #5 0x00007f5d03d61489 in malloc_and_trace (n_bytes=18) at vl.c:2575
> ptr = 0x1
> #6 0x00007f5d020b647f in g_malloc () from /lib64/libglib-2.0.so.0 No
> symbol table info available.
> #7 0x00007f5d020cdf7f in g_strdup () from /lib64/libglib-2.0.so.0 No
> symbol table info available.
> #8 0x00007f5d03eddab5 in alloc_entry (value=0x7f5d088de6c0, key=0x7f5d03f5debb "wr_highest_offset") at qobject/qdict.c:79
> entry = 0x7f5d088df480
> #9 qdict_put_obj (qdict=0x7f5d06e10400, key=0x7f5d03f5debb "wr_highest_offset", value=0x7f5d088de6c0) at qobject/qdict.c:145
> bucket = 81
> entry = <optimized out>
> #10 0x00007f5d03ebf34a in visit_type_BlockDeviceStats_fields (errp=0x7ffddb417ca0, obj=0x7f5d07f905a0, m=0x7f5d061fdea0) at qapi-visit.c:1542
> err = 0x0
> #11 visit_type_BlockDeviceStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f905a0, name=name@entry=0x7f5d03f032ec "stats", errp=errp@entry=0x7ffddb417ca0) at qapi-visit.c:1566
> err = 0x0
> #12 0x00007f5d03ebf5b1 in visit_type_BlockStats_fields (errp=0x7ffddb417cf0, obj=0x7f5d07f90650, m=0x7f5d061fdea0) at qapi-visit.c:1614
> err = 0x0
> #13 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f90650, name=name@entry=0x7f5d03f480f4 "parent", errp=errp@entry=0x7ffddb417cf0) at qapi-visit.c:1644
> err = 0x0
> #14 0x00007f5d03ebf6bd in visit_type_BlockStats_fields (errp=0x7ffddb417d38, obj=0x7f5d07c67a50, m=0x7f5d061fdea0) at qapi-visit.c:1620
> err = 0x0
> #15 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07c67a50, name=name@entry=0x0, errp=errp@entry=0x7ffddb417d38) at qapi-visit.c:1644
> err = 0x0
> #16 0x00007f5d03ebf760 in visit_type_BlockStatsList (m=0x7f5d061fdea0, obj=obj@entry=0x7ffddb417d98, name=name@entry=0x7f5d03f00e6e "unused", errp=errp@entry=0x7ffddb417da0) at qapi-visit.c:1665
> native_i = <optimized out>
> err = 0x0
> i = 0x7f5d07c67a50
> prev = 0x7ffddb417d40
> #17 0x00007f5d03d674dd in qmp_marshal_output_query_blockstats (errp=0x7ffddb417d90, ret_out=0x7ffddb417e10, ret_in=0x7f5d07c67120) at qmp-marshal.c:182
> local_err = 0x0
> mo = 0x7f5d061fdea0
> md = <optimized out>
> v = <optimized out>
> #18 qmp_marshal_input_query_blockstats (mon=<optimized out>, qdict=<optimized out>, ret=0x7ffddb417e10) at qmp-marshal.c:225
> local_err = 0x0
> args = <optimized out>
> retval = <optimized out>
> mi = 0x7f5d064e2000
> md = <optimized out>
> v = <optimized out>
> has_query_nodes = false
> query_nodes = false
> #19 0x00007f5d03ca0531 in qmp_call_cmd (cmd=<optimized out>, params=0x7f5d075dd600, mon=0x7f5d06208320) at /usr/src/debug/qemu-2.3.0/monitor.c:5051
> ret = <optimized out>
> data = 0x0
> #20 handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5113
> err = <optimized out>
> obj = <optimized out>
> input = <optimized out>
> args = 0x7f5d075dd600
> cmd_name = <optimized out>
> mon = 0x7f5d06208320
> #21 0x00007f5d03edf4f2 in json_message_process_token (lexer=0x7f5d061f5d70, token=0x7f5d061991e0, type=JSON_OPERATOR, x=48, y=15) at qobject/json-streamer.c:87
> parser = 0x7f5d061f5d68
> dict = 0x7f5d088ea800
> #22 0x00007f5d03ef191f in json_lexer_feed_char (lexer=lexer@entry=0x7f5d061f5d70, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303
> new_state = 100
> #23 0x00007f5d03ef19ee in json_lexer_feed (lexer=0x7f5d061f5d70, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356
> err = <optimized out>
> i = <optimized out>
> #24 0x00007f5d03edf689 in json_message_parser_feed (parser=<optimized
> out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 No locals.
> #25 0x00007f5d03c9e8cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134
> old_mon = 0x0
> #26 0x00007f5d03d5b1b0 in qemu_chr_be_write (len=<optimized out>,
> buf=0x7ffddb417f40 "}\177A\333\375\177", s=0x7f5d0625a2e0) at qemu-char.c:305 No locals.
> #27 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f5d0625a2e0) at qemu-char.c:2870
> chr = 0x7f5d0625a2e0
> s = 0x7f5d061aa3f0
> buf = "}\177A\333\375\177\000\000\360\360\355\003]\177\000\000\030\003\000\000\000\000\000\000\205\336\355\003]\177\000\000\000A\036\b]\177\000\000\263\325\355\003]\177\000\000\000A\036\b]\177", '\000' <repeats 18 times>, "`2\036\b]\177\000\000ЀA\333\375\177\000\000\000\000\000\000\000\000\000\000 \232\261\a]\177\000\000\000(\341\006]\177\000\000P\217A\333\375\177\000\000H\242<\374\\\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\240\200A\333\375\177\000\000\340\177A\333\375\177\000\000\r\000\000\000\000\000\000\000 Xz\b]\177\000\000\000@&\006]\177\000\000\340\365\215\b]\177\000\000Q\000\000\000\000\000\000\000\232"...
> len = <optimized out>
> size = <optimized out>
> #28 0x00007f5d020b099a in g_main_context_dispatch () from
> /lib64/libglib-2.0.so.0 No symbol table info available.
> #29 0x00007f5d03e74288 in glib_pollfds_poll () at main-loop.c:209
> context = 0x7f5d06205140
> pfds = <optimized out>
> #30 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254
> ret = 2
> spin_counter = 0
> #31 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503
> ret = 2
> timeout = 4294967295
> timeout_ns = <optimized out>
> #32 0x00007f5d03c73a4e in main_loop () at vl.c:1818
> nonblocking = <optimized out>
> last_io = 2
> #33 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
> i = <optimized out>
> snapshot = <optimized out>
> linux_boot = <optimized out>
> initrd_filename = <optimized out>
> kernel_filename = <optimized out>
> kernel_cmdline = <optimized out>
> boot_order = 0x7f5d03f06a67 "cad"
> boot_once = 0x0
> cyls = <optimized out>
> heads = <optimized out>
> secs = <optimized out>
> translation = <optimized out>
> hda_opts = <optimized out>
> opts = <optimized out>
> machine_opts = <optimized out>
> icount_opts = <optimized out>
> olist = <optimized out>
> optind = 67
> optarg = 0x7f5d06193570 "rhel6.5.0"
> loadvm = <optimized out>
> machine_class = <optimized out>
> cpu_model = <optimized out>
> vga_model = 0x0
> qtest_chrdev = <optimized out>
> qtest_log = <optimized out>
> pid_file = <optimized out>
> incoming = <optimized out>
> show_vnc_port = <optimized out>
> defconfig = <optimized out>
> userconfig = 56
> log_mask = <optimized out>
> log_file = <optimized out>
> mem_trace = {malloc = 0x7f5d03d61480 <malloc_and_trace>, realloc = 0x7f5d03d61460 <realloc_and_trace>, free = 0x7f5d03d61450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
> trace_events = <optimized out>
> trace_file = <optimized out>
> maxram_size = <optimized out>
> ram_slots = <optimized out>
> vmstate_dump_file = <optimized out>
> main_loop_err = 0x0
> __func__ = "main"
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org
> [mailto:qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org
> ] Im Auftrag von Grundmann, Christian
> Gesendet: Dienstag, 17. November 2015 15:12
> An: 'Dr. David Alan Gilbert' <dgilbert@redhat.com>
> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>;
> stefanha@redhat.com
> Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in
> libtcmalloc
>
> Here you go
>
>
> gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
> [New LWP 52281]
> [New LWP 52288]
> [New LWP 52286]
> [New LWP 52291]
> [New LWP 52292]
> [New LWP 52287]
> [New LWP 52293]
> [New LWP 52290]
> [New LWP 56455]
> [New LWP 52289]
> [New LWP 52282]
> [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,us'.
> Program terminated with signal 11, Segmentation fault.
> #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4
>
> Thread 11 (Thread 0x7f6d47719700 (LWP 52282)):
> #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
> No locals.
> #1 0x00007f6d57426272 in futex_wait (val=4294967295, ev=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals.
> #2 qemu_event_wait (ev=ev@entry=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399
> value = <optimized out>
> #3 0x00007f6d57434526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233
> tries = 0
> n = <optimized out>
> node = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d47719700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d47719700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107326789376, 5884348200482620104, 0, 140107326790080, 140107326789376, 140107592666688, -5804038895876586808, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)):
> #0 0x00007f6d4f8ef257 in ioctl () at
> ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8a4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> 0x7f6d44f139e0, reg_save_area = 0x7f6d44f139a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570cf000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8a4000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44f14700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d44f14700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107284825856, 5884348200482620104, 0, 140107284826560, 140107284825856, 140726431086992, -5804033392412867896, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)):
> #0 sem_timedwait () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
> No locals.
> #1 0x00007f6d574260c7 in qemu_sem_timedwait (sem=sem@entry=0x7f6d5a1b9248, ms=ms@entry=10000) at util/qemu-thread-posix.c:254
> rc = <optimized out>
> ts = {tv_sec = 1447709021, tv_nsec = 21985000}
> __func__ = "qemu_sem_timedwait"
> #2 0x00007f6d573a98ac in worker_thread (opaque=0x7f6d5a1b91e0) at thread-pool.c:92
> req = <optimized out>
> ret = <optimized out>
> pool = 0x7f6d5a1b91e0
> #3 0x00007f6d55ceadf5 in start_thread (arg=0x7f6ab1dff700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6ab1dff700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140096227505920, 5884348200482620104, 0, 140096227506624, 140096227505920, 26, -5801757560646548792, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #4 0x00007f6d4f8f81ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 8 (Thread 0x7f6d44713700 (LWP 52290)):
> #0 0x00007f6d4f8ef257 in ioctl () at
> ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8b8000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> 0x7f6d447129e0, reg_save_area = 0x7f6d447129a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570cc000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8b8000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44713700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d44713700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107276433152, 5884348200482620104, 0, 140107276433856, 140107276433152, 140726431086992, -5804032293438111032, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)):
> #0 0x00007f6d4f8ef257 in ioctl () at
> ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8f4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> 0x7f6d42f0f9e0, reg_save_area = 0x7f6d42f0f9a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570c3000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8f4000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d42f10700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d42f10700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107251255040, 5884348200482620104, 0, 140107251255744, 140107251255040, 140726431086992, -5804046580109950264, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)):
> #0 0x00007f6d4f8ef257 in ioctl () at
> ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c878000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> 0x7f6d45f159e0, reg_save_area = 0x7f6d45f159a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570d5000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c878000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45f16700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d45f16700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107301611264, 5884348200482620104, 0, 140107301611968, 140107301611264, 140726431086992, -5804035590362381624, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 5 (Thread 0x7f6d43711700 (LWP 52292)):
> #0 0x00007f6d4f8ef257 in ioctl () at
> ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8e0000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> 0x7f6d437109e0, reg_save_area = 0x7f6d437109a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570c6000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8e0000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43711700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d43711700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107259647744, 5884348200482620104, 0, 140107259648448, 140107259647744, 140726431086992, -5804047687674641720, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)):
> #0 0x00007f6d4f8ef257 in ioctl () at
> ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8cc000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> 0x7f6d43f119e0, reg_save_area = 0x7f6d43f119a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570c9000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8cc000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43f12700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d43f12700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107268040448, 5884348200482620104, 0, 140107268041152, 140107268040448, 140726431086992, -5804048786649398584, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 3 (Thread 0x7f6d46717700 (LWP 52286)):
> #0 0x00007f6d4f8ef257 in ioctl () at
> ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c810000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> 0x7f6d467169e0, reg_save_area = 0x7f6d467169a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570d8000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c810000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d46717700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d46717700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107310003968, 5884348200482620104, 0, 140107310004672, 140107310003968, 140726431086992, -5804036689337138488, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 2 (Thread 0x7f6d45715700 (LWP 52288)):
> #0 0x00007f6d4f8ef257 in ioctl () at
> ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c890000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> 0x7f6d457149e0, reg_save_area = 0x7f6d457149a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570d2000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c890000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45715700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d45715700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107293218560, 5884348200482620104, 0, 140107293219264, 140107293218560, 140726431086992, -5804034491387624760, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)):
> #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 No symbol table info available.
> #1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=49280) at vl.c:2575
> ptr = 0x7f6d59a346a0
> #2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0 No symbol table info available.
> #3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0 No symbol table info available.
> #4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33
> req = <optimized out>
> #5 virtio_blk_get_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:192
> req = <optimized out>
> #6 virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604
> s = 0x7f6d5de1ff40
> __func__ = "virtio_blk_handle_output"
> mrb = {reqs = {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7c000,
> 0x0 <repeats 29 times>}, num_reqs = 3, is_write = true}
> #7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=0x7f6d5a1aea00, ret=62, ret@entry=1) at iohandler.c:143
> revents = 1
> #8 0x00007f6d573b7296 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:504
> ret = 1
> timeout = 4294967295
> timeout_ns = <optimized out>
> #9 0x00007f6d571b6a4e in main_loop () at vl.c:1818
> nonblocking = <optimized out>
> last_io = 0
> #10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
> i = <optimized out>
> snapshot = <optimized out>
> linux_boot = <optimized out>
> initrd_filename = <optimized out>
> kernel_filename = <optimized out>
> kernel_cmdline = <optimized out>
> boot_order = 0x7f6d57449a67 "cad"
> boot_once = 0x0
> cyls = <optimized out>
> heads = <optimized out>
> secs = <optimized out>
> translation = <optimized out>
> hda_opts = <optimized out>
> opts = <optimized out>
> machine_opts = <optimized out>
> icount_opts = <optimized out>
> olist = <optimized out>
> optind = 69
> optarg = 0x7f6d5a14b3a0 "rhel6.5.0"
> loadvm = <optimized out>
> machine_class = <optimized out>
> cpu_model = <optimized out>
> vga_model = 0x0
> qtest_chrdev = <optimized out>
> qtest_log = <optimized out>
> pid_file = <optimized out>
> incoming = <optimized out>
> show_vnc_port = <optimized out>
> defconfig = <optimized out>
> userconfig = 179
> log_mask = <optimized out>
> log_file = <optimized out>
> mem_trace = {malloc = 0x7f6d572a4480 <malloc_and_trace>, realloc = 0x7f6d572a4460 <realloc_and_trace>, free = 0x7f6d572a4450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
> trace_events = <optimized out>
> trace_file = <optimized out>
> maxram_size = <optimized out>
> ram_slots = <optimized out>
> vmstate_dump_file = <optimized out>
> main_loop_err = 0x0
> __func__ = "main"
>
>
>
>
>
> @ Do you think you're only hitting these crashes on VMs that have been paused because of these space errors?
> Will have a look on that
>
> Thx Christian
>
> -----Ursprüngliche Nachricht-----
> Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> Gesendet: Dienstag, 17. November 2015 12:36
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>;
> stefanha@redhat.com
> Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in
> libtcmalloc
>
> * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > Hi,
> >
> > @ Can you please use a 'thread apply all bt full' the full gives a little more info.
> >
> > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
>
> OK, it doesn't relaly give any more without the debuginfo package mentioned below.
>
> <snip>
>
> > @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
> > Sorry it's a ovirt-node System where I can't you yum
>
> Ah, although perhaps if you took the core dump, onto another machine with matching qemu and debuginfo you should be able to get more detail.
>
> > @ Does this part always look the same in your backtraces?
> > The most are the same, found one a little bit different :
> > Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)):
> > #0 0x00007f3785d18353 in
> > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> > #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from
> > /lib64/libtcmalloc.so.4 No symbol table info available.
> > #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No
> > symbol table info available.
> > #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No
> > symbol table info available.
> > #4 0x00007f37885fec89 in g_slice_free1 () from
> > /lib64/libglib-2.0.so.0 No symbol table info available.
> > #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table
> > info available.
> > #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info
> > available.
> > #7 0x00007f378a398394 in aio_bh_poll () No symbol table info
> > available.
> > #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table
> > info available.
> > #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info
> > available.
> > #10 0x00007f37885e299a in g_main_context_dispatch () from
> > /lib64/libglib-2.0.so.0 No symbol table info available.
> > #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info
> > available.
> > #12 0x00007f378a1a5a4e in main ()
> > No symbol table info available.
> >
>
> OK, that's a bit different but interesting....
>
> > @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> > No nothing abnormal
> >
> > @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> > Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot.
> > Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline.
>
> OK, that's interesting, because you may be hitting the following bug;
> http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html
>
> whose fix coincidentally just got accepted today; it's related to error cases with error=stop which you are using.
>
> Do you think you're only hitting these crashes on VMs that have been paused because of these space errors?
>
> > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
> >
> > Ok will try that and report
>
> Thanks,
>
> Dave
>
> >
> > Thx Christian
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> > Gesendet: Dienstag, 17. November 2015 10:59
> > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>;
> > stefanha@redhat.com
> > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in
> > libtcmalloc
> >
> > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > > Hi,
> > > Dan sent me over to you,
> > > please let me know if i can provide additional informations
> >
> > Hi Christian,
> > Thanks for reporting this,
> >
> > > Softwareversions:
> > > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso
> > >
> > > qemu-img-ev-2.3.0-29.1.el7.x86_64
> > > qemu-kvm-ev-2.3.0-29.1.el7.x86_64
> > > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
> > > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
> > > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
> > > kernel-3.10.0-229.14.1.el7.x86_64
> > > gperftools-libs-2.4-7.el7.x86_64
> > >
> > > Commandline:
> > > /usr/libexec/qemu-kvm -name myvmname -S -machine
> > > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime
> > > mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> > > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios
> > > type=1,manufacturer=oVirt,product=oVirt
> > > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-313
> > > 8-
> > > 5A
> > > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319
> > > -nographic -no-user-config -nodefaults -chardev
> > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,
> > > se rv er,nowait -mon chardev=charmonitor,id=monitor,mode=control
> > > -rtc base=2015-11-15T20:04:35,driftfix=slew -global
> > > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
> > > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> > > -device
> > > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
> > > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x
> > > 5 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
> > > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> > > -drive
> > > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b
> > > 84
> > > -8
> > > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baff
> > > ae
> > > cf
> > > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk
> > > 0,
> > > fo
> > > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,
> > > we rr or=stop,rerror=stop,aio=native -device
> > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk
> > > 0,
> > > id
> > > =virtio-disk0,bootindex=1 -netdev
> > > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device
> > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=p
> > > ci
> > > .0
> > > ,addr=0x3 -chardev
> > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b889
> > > 9- 5a
> > > 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait
> > > -device
> > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=c
> > > ha nn el0,name=com.redhat.rhevm.vdsm -chardev
> > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b889
> > > 9- 5a
> > > 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait
> > > -device
> > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=c
> > > ha
> > > nn
> > > el1,name=org.qemu.guest_agent.0 -device
> > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg
> > > timestamp=on
> > >
> > > Stack Trace:
> > >
> > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt"
> >
> > Can you please use a 'thread apply all bt full' the full gives a little more info.
> > Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
> >
> > > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)):
> > > #0 0x00007fa8ad2febe1 in tc_malloc () from
> > > /lib64/libtcmalloc.so.4
> > > #1 0x00007fa8b186b489 in malloc_and_trace ()
> > > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0
> > > #3 0x00007fa8afbd666e in g_slice_alloc () from
> > > /lib64/libglib-2.0.so.0
> > > #4 0x00007fa8b17cbffd in virtio_blk_handle_output ()
> > > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll ()
> > > #6 0x00007fa8b197e296 in main_loop_wait ()
> > > #7 0x00007fa8b177da4e in main ()
> >
> > Does this part always look the same in your backtraces?
> > The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed.
> >
> > Some other ideas:
> > 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> > 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> > 3) If this is pretty repeatable, then it would be interesting to try changing to a different
> > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
> >
> > Dave
> > >
> > >
> > > Thx Christian
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Dan Kenigsberg [mailto:danken@redhat.com]
> > > Gesendet: Freitag, 13. November 2015 20:00
> > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> > > Cc: 'users@ovirt.org' <users@ovirt.org>
> > > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc
> > >
> > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote:
> > > > Hi,
> > > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso"
> > > > (is there something better to use?) fort he nodes, and have
> > > > random crashes of VMs The dumps are always the Same
> > > >
> > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump
> > > > [Thread debugging using libthread_db enabled] Using host
> > > > libthread_db library "/lib64/libthread_db.so.1".
> > > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'.
> > > > Program terminated with signal 11, Segmentation fault.
> > > > #0 0x00007f0c559c4353 in
> > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::
> > > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> > > >
> > > >
> > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist
> > > > centos7 or 3.6
> > >
> > > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt.
> > >
> > > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible.
> > >
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-11-19 16:00 ` Grundmann, Christian
@ 2015-11-19 17:02 ` Paolo Bonzini
2015-12-03 8:18 ` Grundmann, Christian
2015-11-20 19:06 ` Dr. David Alan Gilbert
1 sibling, 1 reply; 16+ messages in thread
From: Paolo Bonzini @ 2015-11-19 17:02 UTC (permalink / raw)
To: Grundmann, Christian, 'Dr. David Alan Gilbert'
Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com
On 19/11/2015 17:00, Grundmann, Christian wrote:
> Hi, it seems that using virtio-scsi did the trick, But now the VMs
> are pausing without an coredump, so the underlying Problem (no
> storage Error) is not fixed, As I am using Snapshots (and so the
> disks have to grow very fast) I try if tuning
> "volume_utilization_percent" and "volume_utilization_chunk_mb" will
> help (https://access.redhat.com/solutions/130843)
The fix for virtio-blk is probably this patch:
http://article.gmane.org/gmane.comp.emulators.qemu.block/6380/raw
Paolo
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-11-19 16:00 ` Grundmann, Christian
2015-11-19 17:02 ` Paolo Bonzini
@ 2015-11-20 19:06 ` Dr. David Alan Gilbert
1 sibling, 0 replies; 16+ messages in thread
From: Dr. David Alan Gilbert @ 2015-11-20 19:06 UTC (permalink / raw)
To: Grundmann, Christian; +Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com
* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> Hi,
> it seems that using virtio-scsi did the trick,
> But now the VMs are pausing without an coredump, so the underlying Problem (no storage Error) is not fixed,
> As I am using Snapshots (and so the disks have to grow very fast) I try if tuning "volume_utilization_percent" and "volume_utilization_chunk_mb" will help (https://access.redhat.com/solutions/130843)
I don't know the oVirt stuff of what's supposed to happen with the auto extension
stuff at that level. I suggest you ask again on the oVirt side, but if they
say QEMU isn't providing the right info/state to them please come right back.
Dave
>
> Thx Christian
>
>
> -----Ursprüngliche Nachricht-----
> Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> Gesendet: Dienstag, 17. November 2015 15:42
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
> Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
>
> * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > And here another one
>
> Oh this is a bit of a different one, from query-blockstats, although again if the heap's corrupted it might have just been the first guy to trip over the corrupt part afterwards.
>
> Dave
>
> > Thread 1 (Thread 0x7f5d03ba5c00 (LWP 1507)):
> > #0 0x00007f5cff7e2e7d in
> > tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> > #1 0x00007f5cff7e312a in
> > tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> > #2 0x00007f5cff7e31dd in
> > tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> > #3 0x00007f5cff7e6235 in
> > tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> > #4 0x00007f5cff7f4d5b in tc_malloc () from /lib64/libtcmalloc.so.4 No
> > symbol table info available.
> > #5 0x00007f5d03d61489 in malloc_and_trace (n_bytes=18) at vl.c:2575
> > ptr = 0x1
> > #6 0x00007f5d020b647f in g_malloc () from /lib64/libglib-2.0.so.0 No
> > symbol table info available.
> > #7 0x00007f5d020cdf7f in g_strdup () from /lib64/libglib-2.0.so.0 No
> > symbol table info available.
> > #8 0x00007f5d03eddab5 in alloc_entry (value=0x7f5d088de6c0, key=0x7f5d03f5debb "wr_highest_offset") at qobject/qdict.c:79
> > entry = 0x7f5d088df480
> > #9 qdict_put_obj (qdict=0x7f5d06e10400, key=0x7f5d03f5debb "wr_highest_offset", value=0x7f5d088de6c0) at qobject/qdict.c:145
> > bucket = 81
> > entry = <optimized out>
> > #10 0x00007f5d03ebf34a in visit_type_BlockDeviceStats_fields (errp=0x7ffddb417ca0, obj=0x7f5d07f905a0, m=0x7f5d061fdea0) at qapi-visit.c:1542
> > err = 0x0
> > #11 visit_type_BlockDeviceStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f905a0, name=name@entry=0x7f5d03f032ec "stats", errp=errp@entry=0x7ffddb417ca0) at qapi-visit.c:1566
> > err = 0x0
> > #12 0x00007f5d03ebf5b1 in visit_type_BlockStats_fields (errp=0x7ffddb417cf0, obj=0x7f5d07f90650, m=0x7f5d061fdea0) at qapi-visit.c:1614
> > err = 0x0
> > #13 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f90650, name=name@entry=0x7f5d03f480f4 "parent", errp=errp@entry=0x7ffddb417cf0) at qapi-visit.c:1644
> > err = 0x0
> > #14 0x00007f5d03ebf6bd in visit_type_BlockStats_fields (errp=0x7ffddb417d38, obj=0x7f5d07c67a50, m=0x7f5d061fdea0) at qapi-visit.c:1620
> > err = 0x0
> > #15 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07c67a50, name=name@entry=0x0, errp=errp@entry=0x7ffddb417d38) at qapi-visit.c:1644
> > err = 0x0
> > #16 0x00007f5d03ebf760 in visit_type_BlockStatsList (m=0x7f5d061fdea0, obj=obj@entry=0x7ffddb417d98, name=name@entry=0x7f5d03f00e6e "unused", errp=errp@entry=0x7ffddb417da0) at qapi-visit.c:1665
> > native_i = <optimized out>
> > err = 0x0
> > i = 0x7f5d07c67a50
> > prev = 0x7ffddb417d40
> > #17 0x00007f5d03d674dd in qmp_marshal_output_query_blockstats (errp=0x7ffddb417d90, ret_out=0x7ffddb417e10, ret_in=0x7f5d07c67120) at qmp-marshal.c:182
> > local_err = 0x0
> > mo = 0x7f5d061fdea0
> > md = <optimized out>
> > v = <optimized out>
> > #18 qmp_marshal_input_query_blockstats (mon=<optimized out>, qdict=<optimized out>, ret=0x7ffddb417e10) at qmp-marshal.c:225
> > local_err = 0x0
> > args = <optimized out>
> > retval = <optimized out>
> > mi = 0x7f5d064e2000
> > md = <optimized out>
> > v = <optimized out>
> > has_query_nodes = false
> > query_nodes = false
> > #19 0x00007f5d03ca0531 in qmp_call_cmd (cmd=<optimized out>, params=0x7f5d075dd600, mon=0x7f5d06208320) at /usr/src/debug/qemu-2.3.0/monitor.c:5051
> > ret = <optimized out>
> > data = 0x0
> > #20 handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5113
> > err = <optimized out>
> > obj = <optimized out>
> > input = <optimized out>
> > args = 0x7f5d075dd600
> > cmd_name = <optimized out>
> > mon = 0x7f5d06208320
> > #21 0x00007f5d03edf4f2 in json_message_process_token (lexer=0x7f5d061f5d70, token=0x7f5d061991e0, type=JSON_OPERATOR, x=48, y=15) at qobject/json-streamer.c:87
> > parser = 0x7f5d061f5d68
> > dict = 0x7f5d088ea800
> > #22 0x00007f5d03ef191f in json_lexer_feed_char (lexer=lexer@entry=0x7f5d061f5d70, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303
> > new_state = 100
> > #23 0x00007f5d03ef19ee in json_lexer_feed (lexer=0x7f5d061f5d70, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356
> > err = <optimized out>
> > i = <optimized out>
> > #24 0x00007f5d03edf689 in json_message_parser_feed (parser=<optimized
> > out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 No locals.
> > #25 0x00007f5d03c9e8cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134
> > old_mon = 0x0
> > #26 0x00007f5d03d5b1b0 in qemu_chr_be_write (len=<optimized out>,
> > buf=0x7ffddb417f40 "}\177A\333\375\177", s=0x7f5d0625a2e0) at qemu-char.c:305 No locals.
> > #27 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f5d0625a2e0) at qemu-char.c:2870
> > chr = 0x7f5d0625a2e0
> > s = 0x7f5d061aa3f0
> > buf = "}\177A\333\375\177\000\000\360\360\355\003]\177\000\000\030\003\000\000\000\000\000\000\205\336\355\003]\177\000\000\000A\036\b]\177\000\000\263\325\355\003]\177\000\000\000A\036\b]\177", '\000' <repeats 18 times>, "`2\036\b]\177\000\000ЀA\333\375\177\000\000\000\000\000\000\000\000\000\000 \232\261\a]\177\000\000\000(\341\006]\177\000\000P\217A\333\375\177\000\000H\242<\374\\\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\240\200A\333\375\177\000\000\340\177A\333\375\177\000\000\r\000\000\000\000\000\000\000 Xz\b]\177\000\000\000@&\006]\177\000\000\340\365\215\b]\177\000\000Q\000\000\000\000\000\000\000\232"...
> > len = <optimized out>
> > size = <optimized out>
> > #28 0x00007f5d020b099a in g_main_context_dispatch () from
> > /lib64/libglib-2.0.so.0 No symbol table info available.
> > #29 0x00007f5d03e74288 in glib_pollfds_poll () at main-loop.c:209
> > context = 0x7f5d06205140
> > pfds = <optimized out>
> > #30 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254
> > ret = 2
> > spin_counter = 0
> > #31 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503
> > ret = 2
> > timeout = 4294967295
> > timeout_ns = <optimized out>
> > #32 0x00007f5d03c73a4e in main_loop () at vl.c:1818
> > nonblocking = <optimized out>
> > last_io = 2
> > #33 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
> > i = <optimized out>
> > snapshot = <optimized out>
> > linux_boot = <optimized out>
> > initrd_filename = <optimized out>
> > kernel_filename = <optimized out>
> > kernel_cmdline = <optimized out>
> > boot_order = 0x7f5d03f06a67 "cad"
> > boot_once = 0x0
> > cyls = <optimized out>
> > heads = <optimized out>
> > secs = <optimized out>
> > translation = <optimized out>
> > hda_opts = <optimized out>
> > opts = <optimized out>
> > machine_opts = <optimized out>
> > icount_opts = <optimized out>
> > olist = <optimized out>
> > optind = 67
> > optarg = 0x7f5d06193570 "rhel6.5.0"
> > loadvm = <optimized out>
> > machine_class = <optimized out>
> > cpu_model = <optimized out>
> > vga_model = 0x0
> > qtest_chrdev = <optimized out>
> > qtest_log = <optimized out>
> > pid_file = <optimized out>
> > incoming = <optimized out>
> > show_vnc_port = <optimized out>
> > defconfig = <optimized out>
> > userconfig = 56
> > log_mask = <optimized out>
> > log_file = <optimized out>
> > mem_trace = {malloc = 0x7f5d03d61480 <malloc_and_trace>, realloc = 0x7f5d03d61460 <realloc_and_trace>, free = 0x7f5d03d61450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
> > trace_events = <optimized out>
> > trace_file = <optimized out>
> > maxram_size = <optimized out>
> > ram_slots = <optimized out>
> > vmstate_dump_file = <optimized out>
> > main_loop_err = 0x0
> > __func__ = "main"
> >
> >
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org
> > [mailto:qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org
> > ] Im Auftrag von Grundmann, Christian
> > Gesendet: Dienstag, 17. November 2015 15:12
> > An: 'Dr. David Alan Gilbert' <dgilbert@redhat.com>
> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>;
> > stefanha@redhat.com
> > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in
> > libtcmalloc
> >
> > Here you go
> >
> >
> > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
> > [New LWP 52281]
> > [New LWP 52288]
> > [New LWP 52286]
> > [New LWP 52291]
> > [New LWP 52292]
> > [New LWP 52287]
> > [New LWP 52293]
> > [New LWP 52290]
> > [New LWP 56455]
> > [New LWP 52289]
> > [New LWP 52282]
> > [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1".
> > Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,us'.
> > Program terminated with signal 11, Segmentation fault.
> > #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4
> >
> > Thread 11 (Thread 0x7f6d47719700 (LWP 52282)):
> > #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
> > No locals.
> > #1 0x00007f6d57426272 in futex_wait (val=4294967295, ev=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals.
> > #2 qemu_event_wait (ev=ev@entry=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399
> > value = <optimized out>
> > #3 0x00007f6d57434526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233
> > tries = 0
> > n = <optimized out>
> > node = <optimized out>
> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d47719700) at pthread_create.c:308
> > __res = <optimized out>
> > pd = 0x7f6d47719700
> > now = <optimized out>
> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107326789376, 5884348200482620104, 0, 140107326790080, 140107326789376, 140107592666688, -5804038895876586808, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> > not_first_call = <optimized out>
> > pagesize_m1 = <optimized out>
> > sp = <optimized out>
> > freesize = <optimized out>
> > #5 0x00007f6d4f8f81ad in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> > No locals.
> >
> > Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)):
> > #0 0x00007f6d4f8ef257 in ioctl () at
> > ../sysdeps/unix/syscall-template.S:81
> > No locals.
> > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8a4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> > ret = <optimized out>
> > arg = <optimized out>
> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> > 0x7f6d44f139e0, reg_save_area = 0x7f6d44f139a0}}
> > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> > run = 0x7f6d570cf000
> > ret = <optimized out>
> > run_ret = <optimized out>
> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> > cpu = 0x7f6d5c8a4000
> > r = <optimized out>
> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44f14700) at pthread_create.c:308
> > __res = <optimized out>
> > pd = 0x7f6d44f14700
> > now = <optimized out>
> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107284825856, 5884348200482620104, 0, 140107284826560, 140107284825856, 140726431086992, -5804033392412867896, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> > not_first_call = <optimized out>
> > pagesize_m1 = <optimized out>
> > sp = <optimized out>
> > freesize = <optimized out>
> > #5 0x00007f6d4f8f81ad in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> > No locals.
> >
> > Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)):
> > #0 sem_timedwait () at
> > ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
> > No locals.
> > #1 0x00007f6d574260c7 in qemu_sem_timedwait (sem=sem@entry=0x7f6d5a1b9248, ms=ms@entry=10000) at util/qemu-thread-posix.c:254
> > rc = <optimized out>
> > ts = {tv_sec = 1447709021, tv_nsec = 21985000}
> > __func__ = "qemu_sem_timedwait"
> > #2 0x00007f6d573a98ac in worker_thread (opaque=0x7f6d5a1b91e0) at thread-pool.c:92
> > req = <optimized out>
> > ret = <optimized out>
> > pool = 0x7f6d5a1b91e0
> > #3 0x00007f6d55ceadf5 in start_thread (arg=0x7f6ab1dff700) at pthread_create.c:308
> > __res = <optimized out>
> > pd = 0x7f6ab1dff700
> > now = <optimized out>
> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140096227505920, 5884348200482620104, 0, 140096227506624, 140096227505920, 26, -5801757560646548792, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> > not_first_call = <optimized out>
> > pagesize_m1 = <optimized out>
> > sp = <optimized out>
> > freesize = <optimized out>
> > #4 0x00007f6d4f8f81ad in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> > No locals.
> >
> > Thread 8 (Thread 0x7f6d44713700 (LWP 52290)):
> > #0 0x00007f6d4f8ef257 in ioctl () at
> > ../sysdeps/unix/syscall-template.S:81
> > No locals.
> > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8b8000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> > ret = <optimized out>
> > arg = <optimized out>
> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> > 0x7f6d447129e0, reg_save_area = 0x7f6d447129a0}}
> > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> > run = 0x7f6d570cc000
> > ret = <optimized out>
> > run_ret = <optimized out>
> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> > cpu = 0x7f6d5c8b8000
> > r = <optimized out>
> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44713700) at pthread_create.c:308
> > __res = <optimized out>
> > pd = 0x7f6d44713700
> > now = <optimized out>
> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107276433152, 5884348200482620104, 0, 140107276433856, 140107276433152, 140726431086992, -5804032293438111032, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> > not_first_call = <optimized out>
> > pagesize_m1 = <optimized out>
> > sp = <optimized out>
> > freesize = <optimized out>
> > #5 0x00007f6d4f8f81ad in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> > No locals.
> >
> > Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)):
> > #0 0x00007f6d4f8ef257 in ioctl () at
> > ../sysdeps/unix/syscall-template.S:81
> > No locals.
> > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8f4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> > ret = <optimized out>
> > arg = <optimized out>
> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> > 0x7f6d42f0f9e0, reg_save_area = 0x7f6d42f0f9a0}}
> > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> > run = 0x7f6d570c3000
> > ret = <optimized out>
> > run_ret = <optimized out>
> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> > cpu = 0x7f6d5c8f4000
> > r = <optimized out>
> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d42f10700) at pthread_create.c:308
> > __res = <optimized out>
> > pd = 0x7f6d42f10700
> > now = <optimized out>
> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107251255040, 5884348200482620104, 0, 140107251255744, 140107251255040, 140726431086992, -5804046580109950264, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> > not_first_call = <optimized out>
> > pagesize_m1 = <optimized out>
> > sp = <optimized out>
> > freesize = <optimized out>
> > #5 0x00007f6d4f8f81ad in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> > No locals.
> >
> > Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)):
> > #0 0x00007f6d4f8ef257 in ioctl () at
> > ../sysdeps/unix/syscall-template.S:81
> > No locals.
> > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c878000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> > ret = <optimized out>
> > arg = <optimized out>
> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> > 0x7f6d45f159e0, reg_save_area = 0x7f6d45f159a0}}
> > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> > run = 0x7f6d570d5000
> > ret = <optimized out>
> > run_ret = <optimized out>
> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> > cpu = 0x7f6d5c878000
> > r = <optimized out>
> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45f16700) at pthread_create.c:308
> > __res = <optimized out>
> > pd = 0x7f6d45f16700
> > now = <optimized out>
> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107301611264, 5884348200482620104, 0, 140107301611968, 140107301611264, 140726431086992, -5804035590362381624, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> > not_first_call = <optimized out>
> > pagesize_m1 = <optimized out>
> > sp = <optimized out>
> > freesize = <optimized out>
> > #5 0x00007f6d4f8f81ad in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> > No locals.
> >
> > Thread 5 (Thread 0x7f6d43711700 (LWP 52292)):
> > #0 0x00007f6d4f8ef257 in ioctl () at
> > ../sysdeps/unix/syscall-template.S:81
> > No locals.
> > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8e0000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> > ret = <optimized out>
> > arg = <optimized out>
> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> > 0x7f6d437109e0, reg_save_area = 0x7f6d437109a0}}
> > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> > run = 0x7f6d570c6000
> > ret = <optimized out>
> > run_ret = <optimized out>
> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> > cpu = 0x7f6d5c8e0000
> > r = <optimized out>
> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43711700) at pthread_create.c:308
> > __res = <optimized out>
> > pd = 0x7f6d43711700
> > now = <optimized out>
> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107259647744, 5884348200482620104, 0, 140107259648448, 140107259647744, 140726431086992, -5804047687674641720, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> > not_first_call = <optimized out>
> > pagesize_m1 = <optimized out>
> > sp = <optimized out>
> > freesize = <optimized out>
> > #5 0x00007f6d4f8f81ad in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> > No locals.
> >
> > Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)):
> > #0 0x00007f6d4f8ef257 in ioctl () at
> > ../sysdeps/unix/syscall-template.S:81
> > No locals.
> > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8cc000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> > ret = <optimized out>
> > arg = <optimized out>
> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> > 0x7f6d43f119e0, reg_save_area = 0x7f6d43f119a0}}
> > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> > run = 0x7f6d570c9000
> > ret = <optimized out>
> > run_ret = <optimized out>
> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> > cpu = 0x7f6d5c8cc000
> > r = <optimized out>
> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43f12700) at pthread_create.c:308
> > __res = <optimized out>
> > pd = 0x7f6d43f12700
> > now = <optimized out>
> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107268040448, 5884348200482620104, 0, 140107268041152, 140107268040448, 140726431086992, -5804048786649398584, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> > not_first_call = <optimized out>
> > pagesize_m1 = <optimized out>
> > sp = <optimized out>
> > freesize = <optimized out>
> > #5 0x00007f6d4f8f81ad in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> > No locals.
> >
> > Thread 3 (Thread 0x7f6d46717700 (LWP 52286)):
> > #0 0x00007f6d4f8ef257 in ioctl () at
> > ../sysdeps/unix/syscall-template.S:81
> > No locals.
> > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c810000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> > ret = <optimized out>
> > arg = <optimized out>
> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> > 0x7f6d467169e0, reg_save_area = 0x7f6d467169a0}}
> > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> > run = 0x7f6d570d8000
> > ret = <optimized out>
> > run_ret = <optimized out>
> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> > cpu = 0x7f6d5c810000
> > r = <optimized out>
> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d46717700) at pthread_create.c:308
> > __res = <optimized out>
> > pd = 0x7f6d46717700
> > now = <optimized out>
> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107310003968, 5884348200482620104, 0, 140107310004672, 140107310003968, 140726431086992, -5804036689337138488, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> > not_first_call = <optimized out>
> > pagesize_m1 = <optimized out>
> > sp = <optimized out>
> > freesize = <optimized out>
> > #5 0x00007f6d4f8f81ad in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> > No locals.
> >
> > Thread 2 (Thread 0x7f6d45715700 (LWP 52288)):
> > #0 0x00007f6d4f8ef257 in ioctl () at
> > ../sysdeps/unix/syscall-template.S:81
> > No locals.
> > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c890000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> > ret = <optimized out>
> > arg = <optimized out>
> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area =
> > 0x7f6d457149e0, reg_save_area = 0x7f6d457149a0}}
> > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> > run = 0x7f6d570d2000
> > ret = <optimized out>
> > run_ret = <optimized out>
> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> > cpu = 0x7f6d5c890000
> > r = <optimized out>
> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45715700) at pthread_create.c:308
> > __res = <optimized out>
> > pd = 0x7f6d45715700
> > now = <optimized out>
> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107293218560, 5884348200482620104, 0, 140107293219264, 140107293218560, 140726431086992, -5804034491387624760, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> > not_first_call = <optimized out>
> > pagesize_m1 = <optimized out>
> > sp = <optimized out>
> > freesize = <optimized out>
> > #5 0x00007f6d4f8f81ad in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> > No locals.
> >
> > Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)):
> > #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 No symbol table info available.
> > #1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=49280) at vl.c:2575
> > ptr = 0x7f6d59a346a0
> > #2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0 No symbol table info available.
> > #3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0 No symbol table info available.
> > #4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33
> > req = <optimized out>
> > #5 virtio_blk_get_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:192
> > req = <optimized out>
> > #6 virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604
> > s = 0x7f6d5de1ff40
> > __func__ = "virtio_blk_handle_output"
> > mrb = {reqs = {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7c000,
> > 0x0 <repeats 29 times>}, num_reqs = 3, is_write = true}
> > #7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=0x7f6d5a1aea00, ret=62, ret@entry=1) at iohandler.c:143
> > revents = 1
> > #8 0x00007f6d573b7296 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:504
> > ret = 1
> > timeout = 4294967295
> > timeout_ns = <optimized out>
> > #9 0x00007f6d571b6a4e in main_loop () at vl.c:1818
> > nonblocking = <optimized out>
> > last_io = 0
> > #10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
> > i = <optimized out>
> > snapshot = <optimized out>
> > linux_boot = <optimized out>
> > initrd_filename = <optimized out>
> > kernel_filename = <optimized out>
> > kernel_cmdline = <optimized out>
> > boot_order = 0x7f6d57449a67 "cad"
> > boot_once = 0x0
> > cyls = <optimized out>
> > heads = <optimized out>
> > secs = <optimized out>
> > translation = <optimized out>
> > hda_opts = <optimized out>
> > opts = <optimized out>
> > machine_opts = <optimized out>
> > icount_opts = <optimized out>
> > olist = <optimized out>
> > optind = 69
> > optarg = 0x7f6d5a14b3a0 "rhel6.5.0"
> > loadvm = <optimized out>
> > machine_class = <optimized out>
> > cpu_model = <optimized out>
> > vga_model = 0x0
> > qtest_chrdev = <optimized out>
> > qtest_log = <optimized out>
> > pid_file = <optimized out>
> > incoming = <optimized out>
> > show_vnc_port = <optimized out>
> > defconfig = <optimized out>
> > userconfig = 179
> > log_mask = <optimized out>
> > log_file = <optimized out>
> > mem_trace = {malloc = 0x7f6d572a4480 <malloc_and_trace>, realloc = 0x7f6d572a4460 <realloc_and_trace>, free = 0x7f6d572a4450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
> > trace_events = <optimized out>
> > trace_file = <optimized out>
> > maxram_size = <optimized out>
> > ram_slots = <optimized out>
> > vmstate_dump_file = <optimized out>
> > main_loop_err = 0x0
> > __func__ = "main"
> >
> >
> >
> >
> >
> > @ Do you think you're only hitting these crashes on VMs that have been paused because of these space errors?
> > Will have a look on that
> >
> > Thx Christian
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> > Gesendet: Dienstag, 17. November 2015 12:36
> > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>;
> > stefanha@redhat.com
> > Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in
> > libtcmalloc
> >
> > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > > Hi,
> > >
> > > @ Can you please use a 'thread apply all bt full' the full gives a little more info.
> > >
> > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
> >
> > OK, it doesn't relaly give any more without the debuginfo package mentioned below.
> >
> > <snip>
> >
> > > @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
> > > Sorry it's a ovirt-node System where I can't you yum
> >
> > Ah, although perhaps if you took the core dump, onto another machine with matching qemu and debuginfo you should be able to get more detail.
> >
> > > @ Does this part always look the same in your backtraces?
> > > The most are the same, found one a little bit different :
> > > Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)):
> > > #0 0x00007f3785d18353 in
> > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> > > #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from
> > > /lib64/libtcmalloc.so.4 No symbol table info available.
> > > #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No
> > > symbol table info available.
> > > #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No
> > > symbol table info available.
> > > #4 0x00007f37885fec89 in g_slice_free1 () from
> > > /lib64/libglib-2.0.so.0 No symbol table info available.
> > > #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table
> > > info available.
> > > #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info
> > > available.
> > > #7 0x00007f378a398394 in aio_bh_poll () No symbol table info
> > > available.
> > > #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table
> > > info available.
> > > #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info
> > > available.
> > > #10 0x00007f37885e299a in g_main_context_dispatch () from
> > > /lib64/libglib-2.0.so.0 No symbol table info available.
> > > #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info
> > > available.
> > > #12 0x00007f378a1a5a4e in main ()
> > > No symbol table info available.
> > >
> >
> > OK, that's a bit different but interesting....
> >
> > > @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> > > No nothing abnormal
> > >
> > > @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> > > Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot.
> > > Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline.
> >
> > OK, that's interesting, because you may be hitting the following bug;
> > http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html
> >
> > whose fix coincidentally just got accepted today; it's related to error cases with error=stop which you are using.
> >
> > Do you think you're only hitting these crashes on VMs that have been paused because of these space errors?
> >
> > > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
> > >
> > > Ok will try that and report
> >
> > Thanks,
> >
> > Dave
> >
> > >
> > > Thx Christian
> > >
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> > > Gesendet: Dienstag, 17. November 2015 10:59
> > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> > > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>;
> > > stefanha@redhat.com
> > > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in
> > > libtcmalloc
> > >
> > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > > > Hi,
> > > > Dan sent me over to you,
> > > > please let me know if i can provide additional informations
> > >
> > > Hi Christian,
> > > Thanks for reporting this,
> > >
> > > > Softwareversions:
> > > > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso
> > > >
> > > > qemu-img-ev-2.3.0-29.1.el7.x86_64
> > > > qemu-kvm-ev-2.3.0-29.1.el7.x86_64
> > > > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
> > > > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
> > > > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
> > > > kernel-3.10.0-229.14.1.el7.x86_64
> > > > gperftools-libs-2.4-7.el7.x86_64
> > > >
> > > > Commandline:
> > > > /usr/libexec/qemu-kvm -name myvmname -S -machine
> > > > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime
> > > > mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> > > > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios
> > > > type=1,manufacturer=oVirt,product=oVirt
> > > > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-313
> > > > 8-
> > > > 5A
> > > > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319
> > > > -nographic -no-user-config -nodefaults -chardev
> > > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,
> > > > se rv er,nowait -mon chardev=charmonitor,id=monitor,mode=control
> > > > -rtc base=2015-11-15T20:04:35,driftfix=slew -global
> > > > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
> > > > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> > > > -device
> > > > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
> > > > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x
> > > > 5 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
> > > > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> > > > -drive
> > > > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b
> > > > 84
> > > > -8
> > > > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baff
> > > > ae
> > > > cf
> > > > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk
> > > > 0,
> > > > fo
> > > > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,
> > > > we rr or=stop,rerror=stop,aio=native -device
> > > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk
> > > > 0,
> > > > id
> > > > =virtio-disk0,bootindex=1 -netdev
> > > > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device
> > > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=p
> > > > ci
> > > > .0
> > > > ,addr=0x3 -chardev
> > > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b889
> > > > 9- 5a
> > > > 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait
> > > > -device
> > > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=c
> > > > ha nn el0,name=com.redhat.rhevm.vdsm -chardev
> > > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b889
> > > > 9- 5a
> > > > 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait
> > > > -device
> > > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=c
> > > > ha
> > > > nn
> > > > el1,name=org.qemu.guest_agent.0 -device
> > > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> > > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg
> > > > timestamp=on
> > > >
> > > > Stack Trace:
> > > >
> > > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt"
> > >
> > > Can you please use a 'thread apply all bt full' the full gives a little more info.
> > > Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
> > >
> > > > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)):
> > > > #0 0x00007fa8ad2febe1 in tc_malloc () from
> > > > /lib64/libtcmalloc.so.4
> > > > #1 0x00007fa8b186b489 in malloc_and_trace ()
> > > > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0
> > > > #3 0x00007fa8afbd666e in g_slice_alloc () from
> > > > /lib64/libglib-2.0.so.0
> > > > #4 0x00007fa8b17cbffd in virtio_blk_handle_output ()
> > > > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll ()
> > > > #6 0x00007fa8b197e296 in main_loop_wait ()
> > > > #7 0x00007fa8b177da4e in main ()
> > >
> > > Does this part always look the same in your backtraces?
> > > The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed.
> > >
> > > Some other ideas:
> > > 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> > > 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> > > 3) If this is pretty repeatable, then it would be interesting to try changing to a different
> > > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
> > >
> > > Dave
> > > >
> > > >
> > > > Thx Christian
> > > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: Dan Kenigsberg [mailto:danken@redhat.com]
> > > > Gesendet: Freitag, 13. November 2015 20:00
> > > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> > > > Cc: 'users@ovirt.org' <users@ovirt.org>
> > > > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc
> > > >
> > > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote:
> > > > > Hi,
> > > > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso"
> > > > > (is there something better to use?) fort he nodes, and have
> > > > > random crashes of VMs The dumps are always the Same
> > > > >
> > > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump
> > > > > [Thread debugging using libthread_db enabled] Using host
> > > > > libthread_db library "/lib64/libthread_db.so.1".
> > > > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'.
> > > > > Program terminated with signal 11, Segmentation fault.
> > > > > #0 0x00007f0c559c4353 in
> > > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::
> > > > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> > > > >
> > > > >
> > > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist
> > > > > centos7 or 3.6
> > > >
> > > > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt.
> > > >
> > > > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible.
> > > >
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-11-19 17:02 ` Paolo Bonzini
@ 2015-12-03 8:18 ` Grundmann, Christian
2015-12-03 9:04 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 16+ messages in thread
From: Grundmann, Christian @ 2015-12-03 8:18 UTC (permalink / raw)
To: 'Paolo Bonzini', 'Dr. David Alan Gilbert'
Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com
Hi again,
got a Segfault today without virtio :-( (one IDE Disk and one virtio-scsi)
Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine pc-i440fx-rhel7.2.0,accel='.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fb299cbd3ab in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
Thread 6 (Thread 0x7fb28d0c5700 (LWP 29423)):
#0 0x00007fb29cc85ac3 in pread64 () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007fb29e37c2a3 in pread (__offset=<optimized out>, __nbytes=<optimized out>, __buf=0x7fb2a3e21a00, __fd=<optimized out>) at /usr/include/bits/unistd.h:99
No locals.
#2 handle_aiocb_rw_linear (aiocb=aiocb@entry=0x7fb2a1474340, buf=buf@entry=0x7fb2a3e21a00 "QF", <incomplete sequence \373>) at block/raw-posix.c:909
offset = 0
len = <optimized out>
#3 0x00007fb29e37c3d1 in handle_aiocb_rw (aiocb=0x7fb2a1474340) at block/raw-posix.c:992
nbytes = <optimized out>
buf = 0x7fb2a3e21a00 "QF", <incomplete sequence \373>
__PRETTY_FUNCTION__ = "handle_aiocb_rw"
#4 0x00007fb29e37d945 in aio_worker (arg=0x7fb2a1474340) at block/raw-posix.c:1204
aiocb = 0x7fb2a1474340
ret = 0
#5 0x00007fb29e33d91b in worker_thread (opaque=0x7fb2a148d450) at thread-pool.c:105
req = 0x7fb2a1474b30
ret = <optimized out>
pool = 0x7fb2a148d450
#6 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28d0c5700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7fb28d0c5700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404847302400, -4522449750849005939, 0, 140404847303104, 140404847302400, 26, 4492373549408278157, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#7 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 5 (Thread 0x7fb104fff700 (LWP 29084)):
#0 0x00007fb296881b7d in poll () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x00007fb2977d6fe7 in red_worker_main () from /lib64/libspice-server.so.1
No symbol table info available.
#2 0x00007fb29cc7edf5 in start_thread (arg=0x7fb104fff700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7fb104fff700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140398269822720, -4522449750849005939, 0, 140398269823424, 140398269822720, 140405245697216, 4494326442046740109, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#3 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 4 (Thread 0x7fb28c8c4700 (LWP 29081)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1 0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132
err = <optimized out>
__func__ = "qemu_cond_wait"
#2 0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912
No locals.
#3 qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d2e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949
cpu = 0x7fb2a3d2e000
r = <optimized out>
#4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c8c4700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7fb28c8c4700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404838909696, -4522449750849005939, 0, 140404838910400, 140404838909696, 140735272359936, 4492374652678002317, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 3 (Thread 0x7fb28c0c3700 (LWP 29082)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1 0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132
err = <optimized out>
__func__ = "qemu_cond_wait"
#2 0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912
No locals.
#3 qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d7e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949
cpu = 0x7fb2a3d7e000
r = <optimized out>
#4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c0c3700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7fb28c0c3700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404830516992, -4522449750849005939, 0, 140404830517696, 140404830516992, 140735272359936, 4492375751652759181, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 2 (Thread 0x7fb28e6ad700 (LWP 29077)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
No locals.
#1 0x00007fb29e3ba272 in futex_wait (val=4294967295, ev=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301
No locals.
#2 qemu_event_wait (ev=ev@entry=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399
value = <optimized out>
#3 0x00007fb29e3c8526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233
tries = 0
n = <optimized out>
node = <optimized out>
#4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28e6ad700) at pthread_create.c:308
__res = <optimized out>
pd = 0x7fb28e6ad700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404870272768, -4522449750849005939, 0, 140404870273472, 140404870272768, 140405136150080, 4492370572995942029, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
No locals.
Thread 1 (Thread 0x7fb29e07cc00 (LWP 29076)):
#0 0x00007fb299cbd3ab in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
No symbol table info available.
#1 0x00007fb299cbd47b in tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) () from /lib64/libtcmalloc.so.4
No symbol table info available.
#2 0x00007fb299ccc070 in tc_free () from /lib64/libtcmalloc.so.4
No symbol table info available.
#3 0x00007fb29c58d58f in g_free () from /lib64/libglib-2.0.so.0
No symbol table info available.
#4 0x00007fb29e3b7721 in parser_context_free (ctxt=0x7fb2a531e0c0) at qobject/json-parser.c:358
i = <optimized out>
#5 json_parser_parse_err (tokens=<optimized out>, ap=ap@entry=0x0, errp=errp@entry=0x0) at qobject/json-parser.c:710
result = 0x7fb2a4bdf600
#6 0x00007fb29e3b7767 in json_parser_parse (tokens=<optimized out>, ap=ap@entry=0x0) at qobject/json-parser.c:694
No locals.
#7 0x00007fb29e176e04 in handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5068
err = <optimized out>
obj = <optimized out>
input = 0x0
args = 0x0
cmd_name = <optimized out>
mon = 0x7fb2a153e140
#8 0x00007fb29e3b64f2 in json_message_process_token (lexer=0x7fb2a1460040, token=0x7fb2a1424880, type=JSON_OPERATOR, x=49, y=104) at qobject/json-streamer.c:87
parser = 0x7fb2a1460038
dict = 0x7fb2a3e27200
#9 0x00007fb29e3c891f in json_lexer_feed_char (lexer=lexer@entry=0x7fb2a1460040, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303
new_state = 100
#10 0x00007fb29e3c89ee in json_lexer_feed (lexer=0x7fb2a1460040, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356
err = <optimized out>
i = <optimized out>
#11 0x00007fb29e3b6689 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110
No locals.
#12 0x00007fb29e1758cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134
old_mon = 0x0
#13 0x00007fb29e2321b0 in qemu_chr_be_write (len=<optimized out>, buf=0x7fff7bea8a30 "}\212\352{\377\177", s=0x7fb2a14442e0) at qemu-char.c:305
No locals.
#14 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7fb2a14442e0) at qemu-char.c:2870
chr = 0x7fb2a14442e0
s = 0x7fb2a14363f0
buf = "}\212\352{\377\177\000\000\360`;\236\262\177\000\000\030\003\000\000\000\000\000\000\205N;\236\262\177\000\000\240LB\241\262\177\000\000\263E;\236\262\177\000\000\240LB\241\262\177", '\000' <repeats 18 times>, "\360\017c\244\262\177\000\000\300\213\352{\377\177\000\000\000\000\000\000\000\000\000\000\060\356t\245\262\177\000\000\000$ᤲ\177\000\000@\232\352{\377\177\000\000H\022\212\226\262\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\220\213\352{\377\177\000\000Њ\352{\377\177\000\000\r\000\000\000\000\000\000\000\340\234\177\000\000\000d\023\245\262\177\000\000`\376\061\245\262\177\000\000Q\000\000\000\000\000\000\000\325b\004\000\000\000\000\000"...
len = <optimized out>
size = <optimized out>
#15 0x00007fb29c58799a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
No symbol table info available.
#16 0x00007fb29e34b288 in glib_pollfds_poll () at main-loop.c:209
context = 0x7fb2a1491140
pfds = <optimized out>
#17 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254
ret = 2
spin_counter = 0
#18 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503
ret = 2
timeout = 4294967295
timeout_ns = <optimized out>
#19 0x00007fb29e14aa4e in main_loop () at vl.c:1818
nonblocking = <optimized out>
last_io = 2
#20 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
i = <optimized out>
snapshot = <optimized out>
linux_boot = <optimized out>
initrd_filename = <optimized out>
kernel_filename = <optimized out>
kernel_cmdline = <optimized out>
boot_order = 0x7fb29e3dda67 "cad"
boot_once = 0x0
cyls = <optimized out>
heads = <optimized out>
secs = <optimized out>
translation = <optimized out>
hda_opts = <optimized out>
opts = <optimized out>
machine_opts = <optimized out>
icount_opts = <optimized out>
olist = <optimized out>
optind = 78
optarg = 0x7fb2a14ef8c0 "pc-i440fx-rhel7.2.0"
loadvm = <optimized out>
machine_class = <optimized out>
cpu_model = <optimized out>
vga_model = 0x0
qtest_chrdev = <optimized out>
qtest_log = <optimized out>
pid_file = <optimized out>
incoming = <optimized out>
show_vnc_port = <optimized out>
defconfig = <optimized out>
userconfig = 111
log_mask = <optimized out>
log_file = <optimized out>
mem_trace = {malloc = 0x7fb29e238480 <malloc_and_trace>, realloc = 0x7fb29e238460 <realloc_and_trace>, free = 0x7fb29e238450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
trace_events = <optimized out>
trace_file = <optimized out>
maxram_size = <optimized out>
ram_slots = <optimized out>
vmstate_dump_file = <optimized out>
main_loop_err = 0x0
__func__ = "main"
-----Ursprüngliche Nachricht-----
Von: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] Im Auftrag von Paolo Bonzini
Gesendet: Donnerstag, 19. November 2015 18:02
An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>; 'Dr. David Alan Gilbert' <dgilbert@redhat.com>
Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
Betreff: Re: WG: [ovirt-users] Segmentation fault in libtcmalloc
On 19/11/2015 17:00, Grundmann, Christian wrote:
> Hi, it seems that using virtio-scsi did the trick, But now the VMs are
> pausing without an coredump, so the underlying Problem (no storage
> Error) is not fixed, As I am using Snapshots (and so the disks have to
> grow very fast) I try if tuning "volume_utilization_percent" and
> "volume_utilization_chunk_mb" will help
> (https://access.redhat.com/solutions/130843)
The fix for virtio-blk is probably this patch:
http://article.gmane.org/gmane.comp.emulators.qemu.block/6380/raw
Paolo
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-12-03 8:18 ` Grundmann, Christian
@ 2015-12-03 9:04 ` Dr. David Alan Gilbert
2015-12-03 9:07 ` Grundmann, Christian
0 siblings, 1 reply; 16+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-03 9:04 UTC (permalink / raw)
To: Grundmann, Christian
Cc: 'Paolo Bonzini', 'qemu-devel@nongnu.org',
stefanha@redhat.com
* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> Hi again,
> got a Segfault today without virtio :-( (one IDE Disk and one virtio-scsi)
>
> Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine pc-i440fx-rhel7.2.0,accel='.
Can you confirm the package version you were using; if you're running the pc-i440fx-rhel7.2.0 machine
type it must be pretty new.
Dave
> Program terminated with signal 11, Segmentation fault.
> #0 0x00007fb299cbd3ab in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
>
> Thread 6 (Thread 0x7fb28d0c5700 (LWP 29423)):
> #0 0x00007fb29cc85ac3 in pread64 () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007fb29e37c2a3 in pread (__offset=<optimized out>, __nbytes=<optimized out>, __buf=0x7fb2a3e21a00, __fd=<optimized out>) at /usr/include/bits/unistd.h:99
> No locals.
> #2 handle_aiocb_rw_linear (aiocb=aiocb@entry=0x7fb2a1474340, buf=buf@entry=0x7fb2a3e21a00 "QF", <incomplete sequence \373>) at block/raw-posix.c:909
> offset = 0
> len = <optimized out>
> #3 0x00007fb29e37c3d1 in handle_aiocb_rw (aiocb=0x7fb2a1474340) at block/raw-posix.c:992
> nbytes = <optimized out>
> buf = 0x7fb2a3e21a00 "QF", <incomplete sequence \373>
> __PRETTY_FUNCTION__ = "handle_aiocb_rw"
> #4 0x00007fb29e37d945 in aio_worker (arg=0x7fb2a1474340) at block/raw-posix.c:1204
> aiocb = 0x7fb2a1474340
> ret = 0
> #5 0x00007fb29e33d91b in worker_thread (opaque=0x7fb2a148d450) at thread-pool.c:105
> req = 0x7fb2a1474b30
> ret = <optimized out>
> pool = 0x7fb2a148d450
> #6 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28d0c5700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7fb28d0c5700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404847302400, -4522449750849005939, 0, 140404847303104, 140404847302400, 26, 4492373549408278157, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #7 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 5 (Thread 0x7fb104fff700 (LWP 29084)):
> #0 0x00007fb296881b7d in poll () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007fb2977d6fe7 in red_worker_main () from /lib64/libspice-server.so.1
> No symbol table info available.
> #2 0x00007fb29cc7edf5 in start_thread (arg=0x7fb104fff700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7fb104fff700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140398269822720, -4522449750849005939, 0, 140398269823424, 140398269822720, 140405245697216, 4494326442046740109, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #3 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 4 (Thread 0x7fb28c8c4700 (LWP 29081)):
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> No locals.
> #1 0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132
> err = <optimized out>
> __func__ = "qemu_cond_wait"
> #2 0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912
> No locals.
> #3 qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d2e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949
> cpu = 0x7fb2a3d2e000
> r = <optimized out>
> #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c8c4700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7fb28c8c4700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404838909696, -4522449750849005939, 0, 140404838910400, 140404838909696, 140735272359936, 4492374652678002317, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 3 (Thread 0x7fb28c0c3700 (LWP 29082)):
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> No locals.
> #1 0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132
> err = <optimized out>
> __func__ = "qemu_cond_wait"
> #2 0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912
> No locals.
> #3 qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d7e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949
> cpu = 0x7fb2a3d7e000
> r = <optimized out>
> #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c0c3700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7fb28c0c3700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404830516992, -4522449750849005939, 0, 140404830517696, 140404830516992, 140735272359936, 4492375751652759181, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 2 (Thread 0x7fb28e6ad700 (LWP 29077)):
> #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
> No locals.
> #1 0x00007fb29e3ba272 in futex_wait (val=4294967295, ev=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301
> No locals.
> #2 qemu_event_wait (ev=ev@entry=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399
> value = <optimized out>
> #3 0x00007fb29e3c8526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233
> tries = 0
> n = <optimized out>
> node = <optimized out>
> #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28e6ad700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7fb28e6ad700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404870272768, -4522449750849005939, 0, 140404870273472, 140404870272768, 140405136150080, 4492370572995942029, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 1 (Thread 0x7fb29e07cc00 (LWP 29076)):
> #0 0x00007fb299cbd3ab in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #1 0x00007fb299cbd47b in tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #2 0x00007fb299ccc070 in tc_free () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #3 0x00007fb29c58d58f in g_free () from /lib64/libglib-2.0.so.0
> No symbol table info available.
> #4 0x00007fb29e3b7721 in parser_context_free (ctxt=0x7fb2a531e0c0) at qobject/json-parser.c:358
> i = <optimized out>
> #5 json_parser_parse_err (tokens=<optimized out>, ap=ap@entry=0x0, errp=errp@entry=0x0) at qobject/json-parser.c:710
> result = 0x7fb2a4bdf600
> #6 0x00007fb29e3b7767 in json_parser_parse (tokens=<optimized out>, ap=ap@entry=0x0) at qobject/json-parser.c:694
> No locals.
> #7 0x00007fb29e176e04 in handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5068
> err = <optimized out>
> obj = <optimized out>
> input = 0x0
> args = 0x0
> cmd_name = <optimized out>
> mon = 0x7fb2a153e140
> #8 0x00007fb29e3b64f2 in json_message_process_token (lexer=0x7fb2a1460040, token=0x7fb2a1424880, type=JSON_OPERATOR, x=49, y=104) at qobject/json-streamer.c:87
> parser = 0x7fb2a1460038
> dict = 0x7fb2a3e27200
> #9 0x00007fb29e3c891f in json_lexer_feed_char (lexer=lexer@entry=0x7fb2a1460040, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303
> new_state = 100
> #10 0x00007fb29e3c89ee in json_lexer_feed (lexer=0x7fb2a1460040, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356
> err = <optimized out>
> i = <optimized out>
> #11 0x00007fb29e3b6689 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110
> No locals.
> #12 0x00007fb29e1758cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134
> old_mon = 0x0
> #13 0x00007fb29e2321b0 in qemu_chr_be_write (len=<optimized out>, buf=0x7fff7bea8a30 "}\212\352{\377\177", s=0x7fb2a14442e0) at qemu-char.c:305
> No locals.
> #14 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7fb2a14442e0) at qemu-char.c:2870
> chr = 0x7fb2a14442e0
> s = 0x7fb2a14363f0
> buf = "}\212\352{\377\177\000\000\360`;\236\262\177\000\000\030\003\000\000\000\000\000\000\205N;\236\262\177\000\000\240LB\241\262\177\000\000\263E;\236\262\177\000\000\240LB\241\262\177", '\000' <repeats 18 times>, "\360\017c\244\262\177\000\000\300\213\352{\377\177\000\000\000\000\000\000\000\000\000\000\060\356t\245\262\177\000\000\000$ᤲ\177\000\000@\232\352{\377\177\000\000H\022\212\226\262\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\220\213\352{\377\177\000\000Њ\352{\377\177\000\000\r\000\000\000\000\000\000\000\340\234\177\000\000\000d\023\245\262\177\000\000`\376\061\245\262\177\000\000Q\000\000\000\000\000\000\000\325b\004\000\000\000\000\000"...
> len = <optimized out>
> size = <optimized out>
> #15 0x00007fb29c58799a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
> No symbol table info available.
> #16 0x00007fb29e34b288 in glib_pollfds_poll () at main-loop.c:209
> context = 0x7fb2a1491140
> pfds = <optimized out>
> #17 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254
> ret = 2
> spin_counter = 0
> #18 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503
> ret = 2
> timeout = 4294967295
> timeout_ns = <optimized out>
> #19 0x00007fb29e14aa4e in main_loop () at vl.c:1818
> nonblocking = <optimized out>
> last_io = 2
> #20 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
> i = <optimized out>
> snapshot = <optimized out>
> linux_boot = <optimized out>
> initrd_filename = <optimized out>
> kernel_filename = <optimized out>
> kernel_cmdline = <optimized out>
> boot_order = 0x7fb29e3dda67 "cad"
> boot_once = 0x0
> cyls = <optimized out>
> heads = <optimized out>
> secs = <optimized out>
> translation = <optimized out>
> hda_opts = <optimized out>
> opts = <optimized out>
> machine_opts = <optimized out>
> icount_opts = <optimized out>
> olist = <optimized out>
> optind = 78
> optarg = 0x7fb2a14ef8c0 "pc-i440fx-rhel7.2.0"
> loadvm = <optimized out>
> machine_class = <optimized out>
> cpu_model = <optimized out>
> vga_model = 0x0
> qtest_chrdev = <optimized out>
> qtest_log = <optimized out>
> pid_file = <optimized out>
> incoming = <optimized out>
> show_vnc_port = <optimized out>
> defconfig = <optimized out>
> userconfig = 111
> log_mask = <optimized out>
> log_file = <optimized out>
> mem_trace = {malloc = 0x7fb29e238480 <malloc_and_trace>, realloc = 0x7fb29e238460 <realloc_and_trace>, free = 0x7fb29e238450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
> trace_events = <optimized out>
> trace_file = <optimized out>
> maxram_size = <optimized out>
> ram_slots = <optimized out>
> vmstate_dump_file = <optimized out>
> main_loop_err = 0x0
> __func__ = "main"
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] Im Auftrag von Paolo Bonzini
> Gesendet: Donnerstag, 19. November 2015 18:02
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>; 'Dr. David Alan Gilbert' <dgilbert@redhat.com>
> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
> Betreff: Re: WG: [ovirt-users] Segmentation fault in libtcmalloc
>
>
>
> On 19/11/2015 17:00, Grundmann, Christian wrote:
> > Hi, it seems that using virtio-scsi did the trick, But now the VMs are
> > pausing without an coredump, so the underlying Problem (no storage
> > Error) is not fixed, As I am using Snapshots (and so the disks have to
> > grow very fast) I try if tuning "volume_utilization_percent" and
> > "volume_utilization_chunk_mb" will help
> > (https://access.redhat.com/solutions/130843)
>
> The fix for virtio-blk is probably this patch:
> http://article.gmane.org/gmane.comp.emulators.qemu.block/6380/raw
>
> Paolo
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-12-03 9:04 ` Dr. David Alan Gilbert
@ 2015-12-03 9:07 ` Grundmann, Christian
2015-12-10 12:38 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 16+ messages in thread
From: Grundmann, Christian @ 2015-12-03 9:07 UTC (permalink / raw)
To: 'Dr. David Alan Gilbert'
Cc: 'Paolo Bonzini', 'qemu-devel@nongnu.org',
stefanha@redhat.com
Hi,
qemu-img-ev-2.3.0-29.1.el7.x86_64
libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64
qemu-kvm-ev-2.3.0-29.1.el7.x86_64
qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
it seems pc-i440fx-rhel7.2.0 is the default for ovirt 3.6
I tried using only virtio-scsi disk but the VM wont boot (not bootable device) so i used IDE for the boot disk.
Thx Christian
-----Ursprüngliche Nachricht-----
Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
Gesendet: Donnerstag, 03. Dezember 2015 10:04
An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
Cc: 'Paolo Bonzini' <pbonzini@redhat.com>; 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
Betreff: Re: AW: WG: [ovirt-users] Segmentation fault in libtcmalloc
* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> Hi again,
> got a Segfault today without virtio :-( (one IDE Disk and one
> virtio-scsi)
>
> Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine pc-i440fx-rhel7.2.0,accel='.
Can you confirm the package version you were using; if you're running the pc-i440fx-rhel7.2.0 machine type it must be pretty new.
Dave
> Program terminated with signal 11, Segmentation fault.
> #0 0x00007fb299cbd3ab in
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::Fr
> eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
>
> Thread 6 (Thread 0x7fb28d0c5700 (LWP 29423)):
> #0 0x00007fb29cc85ac3 in pread64 () at
> ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007fb29e37c2a3 in pread (__offset=<optimized out>,
> __nbytes=<optimized out>, __buf=0x7fb2a3e21a00, __fd=<optimized out>) at /usr/include/bits/unistd.h:99 No locals.
> #2 handle_aiocb_rw_linear (aiocb=aiocb@entry=0x7fb2a1474340, buf=buf@entry=0x7fb2a3e21a00 "QF", <incomplete sequence \373>) at block/raw-posix.c:909
> offset = 0
> len = <optimized out>
> #3 0x00007fb29e37c3d1 in handle_aiocb_rw (aiocb=0x7fb2a1474340) at block/raw-posix.c:992
> nbytes = <optimized out>
> buf = 0x7fb2a3e21a00 "QF", <incomplete sequence \373>
> __PRETTY_FUNCTION__ = "handle_aiocb_rw"
> #4 0x00007fb29e37d945 in aio_worker (arg=0x7fb2a1474340) at block/raw-posix.c:1204
> aiocb = 0x7fb2a1474340
> ret = 0
> #5 0x00007fb29e33d91b in worker_thread (opaque=0x7fb2a148d450) at thread-pool.c:105
> req = 0x7fb2a1474b30
> ret = <optimized out>
> pool = 0x7fb2a148d450
> #6 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28d0c5700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7fb28d0c5700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404847302400, -4522449750849005939, 0, 140404847303104, 140404847302400, 26, 4492373549408278157, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #7 0x00007fb29688c1ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 5 (Thread 0x7fb104fff700 (LWP 29084)):
> #0 0x00007fb296881b7d in poll () at
> ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007fb2977d6fe7 in red_worker_main () from
> /lib64/libspice-server.so.1 No symbol table info available.
> #2 0x00007fb29cc7edf5 in start_thread (arg=0x7fb104fff700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7fb104fff700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140398269822720, -4522449750849005939, 0, 140398269823424, 140398269822720, 140405245697216, 4494326442046740109, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #3 0x00007fb29688c1ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 4 (Thread 0x7fb28c8c4700 (LWP 29081)):
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> No locals.
> #1 0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132
> err = <optimized out>
> __func__ = "qemu_cond_wait"
> #2 0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>)
> at /usr/src/debug/qemu-2.3.0/cpus.c:912
> No locals.
> #3 qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d2e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949
> cpu = 0x7fb2a3d2e000
> r = <optimized out>
> #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c8c4700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7fb28c8c4700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404838909696, -4522449750849005939, 0, 140404838910400, 140404838909696, 140735272359936, 4492374652678002317, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007fb29688c1ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 3 (Thread 0x7fb28c0c3700 (LWP 29082)):
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> No locals.
> #1 0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132
> err = <optimized out>
> __func__ = "qemu_cond_wait"
> #2 0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>)
> at /usr/src/debug/qemu-2.3.0/cpus.c:912
> No locals.
> #3 qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d7e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949
> cpu = 0x7fb2a3d7e000
> r = <optimized out>
> #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c0c3700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7fb28c0c3700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404830516992, -4522449750849005939, 0, 140404830517696, 140404830516992, 140735272359936, 4492375751652759181, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007fb29688c1ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 2 (Thread 0x7fb28e6ad700 (LWP 29077)):
> #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
> No locals.
> #1 0x00007fb29e3ba272 in futex_wait (val=4294967295,
> ev=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals.
> #2 qemu_event_wait (ev=ev@entry=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399
> value = <optimized out>
> #3 0x00007fb29e3c8526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233
> tries = 0
> n = <optimized out>
> node = <optimized out>
> #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28e6ad700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7fb28e6ad700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404870272768, -4522449750849005939, 0, 140404870273472, 140404870272768, 140405136150080, 4492370572995942029, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007fb29688c1ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 1 (Thread 0x7fb29e07cc00 (LWP 29076)):
> #0 0x00007fb299cbd3ab in
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> #1 0x00007fb299cbd47b in
> tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> #2 0x00007fb299ccc070 in tc_free () from /lib64/libtcmalloc.so.4 No
> symbol table info available.
> #3 0x00007fb29c58d58f in g_free () from /lib64/libglib-2.0.so.0 No
> symbol table info available.
> #4 0x00007fb29e3b7721 in parser_context_free (ctxt=0x7fb2a531e0c0) at qobject/json-parser.c:358
> i = <optimized out>
> #5 json_parser_parse_err (tokens=<optimized out>, ap=ap@entry=0x0, errp=errp@entry=0x0) at qobject/json-parser.c:710
> result = 0x7fb2a4bdf600
> #6 0x00007fb29e3b7767 in json_parser_parse (tokens=<optimized out>,
> ap=ap@entry=0x0) at qobject/json-parser.c:694 No locals.
> #7 0x00007fb29e176e04 in handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5068
> err = <optimized out>
> obj = <optimized out>
> input = 0x0
> args = 0x0
> cmd_name = <optimized out>
> mon = 0x7fb2a153e140
> #8 0x00007fb29e3b64f2 in json_message_process_token (lexer=0x7fb2a1460040, token=0x7fb2a1424880, type=JSON_OPERATOR, x=49, y=104) at qobject/json-streamer.c:87
> parser = 0x7fb2a1460038
> dict = 0x7fb2a3e27200
> #9 0x00007fb29e3c891f in json_lexer_feed_char (lexer=lexer@entry=0x7fb2a1460040, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303
> new_state = 100
> #10 0x00007fb29e3c89ee in json_lexer_feed (lexer=0x7fb2a1460040, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356
> err = <optimized out>
> i = <optimized out>
> #11 0x00007fb29e3b6689 in json_message_parser_feed (parser=<optimized
> out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 No locals.
> #12 0x00007fb29e1758cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134
> old_mon = 0x0
> #13 0x00007fb29e2321b0 in qemu_chr_be_write (len=<optimized out>,
> buf=0x7fff7bea8a30 "}\212\352{\377\177", s=0x7fb2a14442e0) at qemu-char.c:305 No locals.
> #14 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7fb2a14442e0) at qemu-char.c:2870
> chr = 0x7fb2a14442e0
> s = 0x7fb2a14363f0
> buf = "}\212\352{\377\177\000\000\360`;\236\262\177\000\000\030\003\000\000\000\000\000\000\205N;\236\262\177\000\000\240LB\241\262\177\000\000\263E;\236\262\177\000\000\240LB\241\262\177", '\000' <repeats 18 times>, "\360\017c\244\262\177\000\000\300\213\352{\377\177\000\000\000\000\000\000\000\000\000\000\060\356t\245\262\177\000\000\000$ᤲ\177\000\000@\232\352{\377\177\000\000H\022\212\226\262\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\220\213\352{\377\177\000\000Њ\352{\377\177\000\000\r\000\000\000\000\000\000\000\340\234\177\000\000\000d\023\245\262\177\000\000`\376\061\245\262\177\000\000Q\000\000\000\000\000\000\000\325b\004\000\000\000\000\000"...
> len = <optimized out>
> size = <optimized out>
> #15 0x00007fb29c58799a in g_main_context_dispatch () from
> /lib64/libglib-2.0.so.0 No symbol table info available.
> #16 0x00007fb29e34b288 in glib_pollfds_poll () at main-loop.c:209
> context = 0x7fb2a1491140
> pfds = <optimized out>
> #17 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254
> ret = 2
> spin_counter = 0
> #18 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503
> ret = 2
> timeout = 4294967295
> timeout_ns = <optimized out>
> #19 0x00007fb29e14aa4e in main_loop () at vl.c:1818
> nonblocking = <optimized out>
> last_io = 2
> #20 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
> i = <optimized out>
> snapshot = <optimized out>
> linux_boot = <optimized out>
> initrd_filename = <optimized out>
> kernel_filename = <optimized out>
> kernel_cmdline = <optimized out>
> boot_order = 0x7fb29e3dda67 "cad"
> boot_once = 0x0
> cyls = <optimized out>
> heads = <optimized out>
> secs = <optimized out>
> translation = <optimized out>
> hda_opts = <optimized out>
> opts = <optimized out>
> machine_opts = <optimized out>
> icount_opts = <optimized out>
> olist = <optimized out>
> optind = 78
> optarg = 0x7fb2a14ef8c0 "pc-i440fx-rhel7.2.0"
> loadvm = <optimized out>
> machine_class = <optimized out>
> cpu_model = <optimized out>
> vga_model = 0x0
> qtest_chrdev = <optimized out>
> qtest_log = <optimized out>
> pid_file = <optimized out>
> incoming = <optimized out>
> show_vnc_port = <optimized out>
> defconfig = <optimized out>
> userconfig = 111
> log_mask = <optimized out>
> log_file = <optimized out>
> mem_trace = {malloc = 0x7fb29e238480 <malloc_and_trace>, realloc = 0x7fb29e238460 <realloc_and_trace>, free = 0x7fb29e238450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
> trace_events = <optimized out>
> trace_file = <optimized out>
> maxram_size = <optimized out>
> ram_slots = <optimized out>
> vmstate_dump_file = <optimized out>
> main_loop_err = 0x0
> __func__ = "main"
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] Im Auftrag von
> Paolo Bonzini
> Gesendet: Donnerstag, 19. November 2015 18:02
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>; 'Dr.
> David Alan Gilbert' <dgilbert@redhat.com>
> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>;
> stefanha@redhat.com
> Betreff: Re: WG: [ovirt-users] Segmentation fault in libtcmalloc
>
>
>
> On 19/11/2015 17:00, Grundmann, Christian wrote:
> > Hi, it seems that using virtio-scsi did the trick, But now the VMs
> > are pausing without an coredump, so the underlying Problem (no
> > storage
> > Error) is not fixed, As I am using Snapshots (and so the disks have
> > to grow very fast) I try if tuning "volume_utilization_percent" and
> > "volume_utilization_chunk_mb" will help
> > (https://access.redhat.com/solutions/130843)
>
> The fix for virtio-blk is probably this patch:
> http://article.gmane.org/gmane.comp.emulators.qemu.block/6380/raw
>
> Paolo
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-12-03 9:07 ` Grundmann, Christian
@ 2015-12-10 12:38 ` Dr. David Alan Gilbert
2015-12-10 13:18 ` Markus Armbruster
0 siblings, 1 reply; 16+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-10 12:38 UTC (permalink / raw)
To: Grundmann, Christian
Cc: 'Paolo Bonzini', 'qemu-devel@nongnu.org',
stefanha@redhat.com
* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> Hi,
>
> qemu-img-ev-2.3.0-29.1.el7.x86_64
> libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64
> qemu-kvm-ev-2.3.0-29.1.el7.x86_64
> qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
> ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
> qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
>
>
> it seems pc-i440fx-rhel7.2.0 is the default for ovirt 3.6
>
> I tried using only virtio-scsi disk but the VM wont boot (not bootable device) so i used IDE for the boot disk.
I think this seg is actually quite different - although it depends where the actual corruption
happened - looking at the backtrace again the failing thread wasn't the io thread; it
failed in a call from the json parser in the main thread.
Dave
> a
> Thx Christian
>
> -----Ursprüngliche Nachricht-----
> Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> Gesendet: Donnerstag, 03. Dezember 2015 10:04
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> Cc: 'Paolo Bonzini' <pbonzini@redhat.com>; 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
> Betreff: Re: AW: WG: [ovirt-users] Segmentation fault in libtcmalloc
>
> * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > Hi again,
> > got a Segfault today without virtio :-( (one IDE Disk and one
> > virtio-scsi)
> >
> > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine pc-i440fx-rhel7.2.0,accel='.
>
> Can you confirm the package version you were using; if you're running the pc-i440fx-rhel7.2.0 machine type it must be pretty new.
>
> Dave
>
> > Program terminated with signal 11, Segmentation fault.
> > #0 0x00007fb299cbd3ab in
> > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::Fr
> > eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
<deleted>
It looks like it's the main thread in the json parser:
> > Thread 1 (Thread 0x7fb29e07cc00 (LWP 29076)):
> > #0 0x00007fb299cbd3ab in
> > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> > #1 0x00007fb299cbd47b in
> > tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> > #2 0x00007fb299ccc070 in tc_free () from /lib64/libtcmalloc.so.4 No
> > symbol table info available.
> > #3 0x00007fb29c58d58f in g_free () from /lib64/libglib-2.0.so.0 No
> > symbol table info available.
> > #4 0x00007fb29e3b7721 in parser_context_free (ctxt=0x7fb2a531e0c0) at qobject/json-parser.c:358
> > i = <optimized out>
> > #5 json_parser_parse_err (tokens=<optimized out>, ap=ap@entry=0x0, errp=errp@entry=0x0) at qobject/json-parser.c:710
> > result = 0x7fb2a4bdf600
> > #6 0x00007fb29e3b7767 in json_parser_parse (tokens=<optimized out>,
> > ap=ap@entry=0x0) at qobject/json-parser.c:694 No locals.
> > #7 0x00007fb29e176e04 in handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5068
> > err = <optimized out>
> > obj = <optimized out>
> > input = 0x0
> > args = 0x0
> > cmd_name = <optimized out>
> > mon = 0x7fb2a153e140
> > #8 0x00007fb29e3b64f2 in json_message_process_token (lexer=0x7fb2a1460040, token=0x7fb2a1424880, type=JSON_OPERATOR, x=49, y=104) at qobject/json-streamer.c:87
> > parser = 0x7fb2a1460038
> > dict = 0x7fb2a3e27200
> > #9 0x00007fb29e3c891f in json_lexer_feed_char (lexer=lexer@entry=0x7fb2a1460040, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303
> > new_state = 100
> > #10 0x00007fb29e3c89ee in json_lexer_feed (lexer=0x7fb2a1460040, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356
> > err = <optimized out>
> > i = <optimized out>
> > #11 0x00007fb29e3b6689 in json_message_parser_feed (parser=<optimized
> > out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 No locals.
> > #12 0x00007fb29e1758cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134
> > old_mon = 0x0
> > #13 0x00007fb29e2321b0 in qemu_chr_be_write (len=<optimized out>,
> > buf=0x7fff7bea8a30 "}\212\352{\377\177", s=0x7fb2a14442e0) at qemu-char.c:305 No locals.
> > #14 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7fb2a14442e0) at qemu-char.c:2870
> > chr = 0x7fb2a14442e0
> > s = 0x7fb2a14363f0
> > buf = "}\212\352{\377\177\000\000\360`;\236\262\177\000\000\030\003\000\000\000\000\000\000\205N;\236\262\177\000\000\240LB\241\262\177\000\000\263E;\236\262\177\000\000\240LB\241\262\177", '\000' <repeats 18 times>, "\360\017c\244\262\177\000\000\300\213\352{\377\177\000\000\000\000\000\000\000\000\000\000\060\356t\245\262\177\000\000\000$ᤲ\177\000\000@\232\352{\377\177\000\000H\022\212\226\262\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\220\213\352{\377\177\000\000Њ\352{\377\177\000\000\r\000\000\000\000\000\000\000\340\234\177\000\000\000d\023\245\262\177\000\000`\376\061\245\262\177\000\000Q\000\000\000\000\000\000\000\325b\004\000\000\000\000\000"...
> > len = <optimized out>
> > size = <optimized out>
> > #15 0x00007fb29c58799a in g_main_context_dispatch () from
> > /lib64/libglib-2.0.so.0 No symbol table info available.
> > #16 0x00007fb29e34b288 in glib_pollfds_poll () at main-loop.c:209
> > context = 0x7fb2a1491140
> > pfds = <optimized out>
> > #17 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254
> > ret = 2
> > spin_counter = 0
> > #18 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503
> > ret = 2
> > timeout = 4294967295
> > timeout_ns = <optimized out>
> > #19 0x00007fb29e14aa4e in main_loop () at vl.c:1818
> > nonblocking = <optimized out>
> > last_io = 2
> > #20 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
> > i = <optimized out>
> > snapshot = <optimized out>
> > linux_boot = <optimized out>
> > initrd_filename = <optimized out>
> > kernel_filename = <optimized out>
> > kernel_cmdline = <optimized out>
> > boot_order = 0x7fb29e3dda67 "cad"
> > boot_once = 0x0
> > cyls = <optimized out>
> > heads = <optimized out>
> > secs = <optimized out>
> > translation = <optimized out>
> > hda_opts = <optimized out>
> > opts = <optimized out>
> > machine_opts = <optimized out>
> > icount_opts = <optimized out>
> > olist = <optimized out>
> > optind = 78
> > optarg = 0x7fb2a14ef8c0 "pc-i440fx-rhel7.2.0"
> > loadvm = <optimized out>
> > machine_class = <optimized out>
> > cpu_model = <optimized out>
> > vga_model = 0x0
> > qtest_chrdev = <optimized out>
> > qtest_log = <optimized out>
> > pid_file = <optimized out>
> > incoming = <optimized out>
> > show_vnc_port = <optimized out>
> > defconfig = <optimized out>
> > userconfig = 111
> > log_mask = <optimized out>
> > log_file = <optimized out>
> > mem_trace = {malloc = 0x7fb29e238480 <malloc_and_trace>, realloc = 0x7fb29e238460 <realloc_and_trace>, free = 0x7fb29e238450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
> > trace_events = <optimized out>
> > trace_file = <optimized out>
> > maxram_size = <optimized out>
> > ram_slots = <optimized out>
> > vmstate_dump_file = <optimized out>
> > main_loop_err = 0x0
> > __func__ = "main"
> >
> >
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] Im Auftrag von
> > Paolo Bonzini
> > Gesendet: Donnerstag, 19. November 2015 18:02
> > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>; 'Dr.
> > David Alan Gilbert' <dgilbert@redhat.com>
> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>;
> > stefanha@redhat.com
> > Betreff: Re: WG: [ovirt-users] Segmentation fault in libtcmalloc
> >
> >
> >
> > On 19/11/2015 17:00, Grundmann, Christian wrote:
> > > Hi, it seems that using virtio-scsi did the trick, But now the VMs
> > > are pausing without an coredump, so the underlying Problem (no
> > > storage
> > > Error) is not fixed, As I am using Snapshots (and so the disks have
> > > to grow very fast) I try if tuning "volume_utilization_percent" and
> > > "volume_utilization_chunk_mb" will help
> > > (https://access.redhat.com/solutions/130843)
> >
> > The fix for virtio-blk is probably this patch:
> > http://article.gmane.org/gmane.comp.emulators.qemu.block/6380/raw
> >
> > Paolo
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-12-10 12:38 ` Dr. David Alan Gilbert
@ 2015-12-10 13:18 ` Markus Armbruster
2015-12-10 13:37 ` Grundmann, Christian
0 siblings, 1 reply; 16+ messages in thread
From: Markus Armbruster @ 2015-12-10 13:18 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: 'Paolo Bonzini', Grundmann, Christian,
'qemu-devel@nongnu.org', stefanha@redhat.com
"Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
> * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
>> Hi,
>>
>> qemu-img-ev-2.3.0-29.1.el7.x86_64
>> libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64
>> qemu-kvm-ev-2.3.0-29.1.el7.x86_64
>> qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
>> ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
>> qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
>>
>>
>> it seems pc-i440fx-rhel7.2.0 is the default for ovirt 3.6
>>
>> I tried using only virtio-scsi disk but the VM wont boot (not
>> bootable device) so i used IDE for the boot disk.
>
> I think this seg is actually quite different - although it depends
> where the actual corruption
> happened - looking at the backtrace again the failing thread wasn't
> the io thread; it
> failed in a call from the json parser in the main thread.
Yes, in a free on behalf of parser_context_free() on parser cleanup.
Smells like memory corruption. Habe you tried reproducing under
valgrind?
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
2015-12-10 13:18 ` Markus Armbruster
@ 2015-12-10 13:37 ` Grundmann, Christian
0 siblings, 0 replies; 16+ messages in thread
From: Grundmann, Christian @ 2015-12-10 13:37 UTC (permalink / raw)
To: Markus Armbruster
Cc: Paolo Bonzini, Dr. David Alan Gilbert, stefanha@redhat.com,
qemu-devel@nongnu.org
Sorry as this is my productionsystem i can't
> Am 10.12.2015 um 14:18 schrieb Markus Armbruster <armbru@redhat.com>:
>
> "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes:
>
>> * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
>>> Hi,
>>>
>>> qemu-img-ev-2.3.0-29.1.el7.x86_64
>>> libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64
>>> qemu-kvm-ev-2.3.0-29.1.el7.x86_64
>>> qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
>>> ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
>>> qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
>>>
>>>
>>> it seems pc-i440fx-rhel7.2.0 is the default for ovirt 3.6
>>>
>>> I tried using only virtio-scsi disk but the VM wont boot (not
>>> bootable device) so i used IDE for the boot disk.
>>
>> I think this seg is actually quite different - although it depends
>> where the actual corruption
>> happened - looking at the backtrace again the failing thread wasn't
>> the io thread; it
>> failed in a call from the json parser in the main thread.
>
> Yes, in a free on behalf of parser_context_free() on parser cleanup.
> Smells like memory corruption. Habe you tried reproducing under
> valgrind?
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2015-12-10 13:37 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <6A17C71B52524C408E7AAF69103E9E490F14400C@fabamailserver.fabagl.fabasoft.com>
[not found] ` <20151113190014.GB18986@redhat.com>
2015-11-16 8:11 ` [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc Grundmann, Christian
2015-11-17 9:59 ` Dr. David Alan Gilbert
2015-11-17 10:36 ` Grundmann, Christian
2015-11-17 11:36 ` Dr. David Alan Gilbert
2015-11-17 14:11 ` Grundmann, Christian
2015-11-17 14:20 ` Grundmann, Christian
2015-11-17 14:42 ` Dr. David Alan Gilbert
2015-11-19 16:00 ` Grundmann, Christian
2015-11-19 17:02 ` Paolo Bonzini
2015-12-03 8:18 ` Grundmann, Christian
2015-12-03 9:04 ` Dr. David Alan Gilbert
2015-12-03 9:07 ` Grundmann, Christian
2015-12-10 12:38 ` Dr. David Alan Gilbert
2015-12-10 13:18 ` Markus Armbruster
2015-12-10 13:37 ` Grundmann, Christian
2015-11-20 19:06 ` Dr. David Alan Gilbert
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).