* [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc [not found] ` <20151113190014.GB18986@redhat.com> @ 2015-11-16 8:11 ` Grundmann, Christian 2015-11-17 9:59 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 16+ messages in thread From: Grundmann, Christian @ 2015-11-16 8:11 UTC (permalink / raw) To: 'qemu-devel@nongnu.org' Hi, Dan sent me over to you, please let me know if i can provide additional informations Softwareversions: ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso qemu-img-ev-2.3.0-29.1.el7.x86_64 qemu-kvm-ev-2.3.0-29.1.el7.x86_64 qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch kernel-3.10.0-229.14.1.el7.x86_64 libvirt-daemon-driver-nwfilter-1.2.8-16.el7_1.4.x86_64 libvirt-lock-sanlock-1.2.8-16.el7_1.4.x86_64 libvirt-daemon-kvm-1.2.8-16.el7_1.4.x86_64 libvirt-daemon-1.2.8-16.el7_1.4.x86_64 libvirt-daemon-config-nwfilter-1.2.8-16.el7_1.4.x86_64 libvirt-daemon-driver-secret-1.2.8-16.el7_1.4.x86_64 libvirt-daemon-driver-nodedev-1.2.8-16.el7_1.4.x86_64 libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64 libvirt-python-1.2.8-7.el7_1.1.x86_64 libvirt-daemon-driver-interface-1.2.8-16.el7_1.4.x86_64 libvirt-daemon-driver-network-1.2.8-16.el7_1.4.x86_64 libvirt-daemon-driver-storage-1.2.8-16.el7_1.4.x86_64 libvirt-client-1.2.8-16.el7_1.4.x86_64 gperftools-libs-2.4-7.el7.x86_64 Commandline: /usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-5A43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2015-11-15T20:04:35,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84-8746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffaecf/d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,format=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on Stack Trace: gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt" [New LWP 14750] [New LWP 51911] [New LWP 14758] [New LWP 14759] [New LWP 14754] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,usb=o'. Program terminated with signal 11, Segmentation fault. #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4 Thread 5 (Thread 0x7fa8a1ce0700 (LWP 14754)): #0 0x00007fa8a9eb9949 in syscall () from /lib64/libc.so.6 #1 0x00007fa8b19ed272 in qemu_event_wait () #2 0x00007fa8b19fb526 in call_rcu_thread () #3 0x00007fa8b02b1df5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fa8a9ebf1ad in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fa8a04dd700 (LWP 14759)): #0 0x00007fa8a9eb6257 in ioctl () from /lib64/libc.so.6 #1 0x00007fa8b17b8025 in kvm_vcpu_ioctl () #2 0x00007fa8b17b80de in kvm_cpu_exec () #3 0x00007fa8b17a5d7a in qemu_kvm_cpu_thread_fn () #4 0x00007fa8b02b1df5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa8a9ebf1ad in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fa8a0cde700 (LWP 14758)): #0 0x00007fa8a9eb6257 in ioctl () from /lib64/libc.so.6 #1 0x00007fa8b17b8025 in kvm_vcpu_ioctl () #2 0x00007fa8b17b80de in kvm_cpu_exec () #3 0x00007fa8b17a5d7a in qemu_kvm_cpu_thread_fn () #4 0x00007fa8b02b1df5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa8a9ebf1ad in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fa8a14df700 (LWP 51911)): #0 0x00007fa8b02b78a0 in sem_timedwait () from /lib64/libpthread.so.0 #1 0x00007fa8b19ed0c7 in qemu_sem_timedwait () #2 0x00007fa8b19708ac in worker_thread () #3 0x00007fa8b02b1df5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fa8a9ebf1ad in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)): #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4 #1 0x00007fa8b186b489 in malloc_and_trace () #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0 #3 0x00007fa8afbd666e in g_slice_alloc () from /lib64/libglib-2.0.so.0 #4 0x00007fa8b17cbffd in virtio_blk_handle_output () #5 0x00007fa8b197e6b6 in qemu_iohandler_poll () #6 0x00007fa8b197e296 in main_loop_wait () #7 0x00007fa8b177da4e in main () Thx Christian -----Ursprüngliche Nachricht----- Von: Dan Kenigsberg [mailto:danken@redhat.com] Gesendet: Freitag, 13. November 2015 20:00 An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> Cc: 'users@ovirt.org' <users@ovirt.org> Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote: > Hi, > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso" (is > there something better to use?) fort he nodes, and have random crashes > of VMs The dumps are always the Same > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump [Thread > debugging using libthread_db enabled] Using host libthread_db library > "/lib64/libthread_db.so.1". > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007f0c559c4353 in > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::Fr > eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist centos7 > or 3.6 Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt. Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-11-16 8:11 ` [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc Grundmann, Christian @ 2015-11-17 9:59 ` Dr. David Alan Gilbert 2015-11-17 10:36 ` Grundmann, Christian 0 siblings, 1 reply; 16+ messages in thread From: Dr. David Alan Gilbert @ 2015-11-17 9:59 UTC (permalink / raw) To: Grundmann, Christian; +Cc: 'qemu-devel@nongnu.org', stefanha * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > Hi, > Dan sent me over to you, > please let me know if i can provide additional informations Hi Christian, Thanks for reporting this, > Softwareversions: > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso > > qemu-img-ev-2.3.0-29.1.el7.x86_64 > qemu-kvm-ev-2.3.0-29.1.el7.x86_64 > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch > kernel-3.10.0-229.14.1.el7.x86_64 > gperftools-libs-2.4-7.el7.x86_64 > > Commandline: > /usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-5A43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2015-11-15T20:04:35,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84-8746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffaecf/d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,format=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on > > Stack Trace: > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt" Can you please use a 'thread apply all bt full' the full gives a little more info. Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)): > #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4 > #1 0x00007fa8b186b489 in malloc_and_trace () > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0 > #3 0x00007fa8afbd666e in g_slice_alloc () from /lib64/libglib-2.0.so.0 > #4 0x00007fa8b17cbffd in virtio_blk_handle_output () > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll () > #6 0x00007fa8b197e296 in main_loop_wait () > #7 0x00007fa8b177da4e in main () Does this part always look the same in your backtraces? The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed. Some other ideas: 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? 2) Did you hit any IO errors and need to tell the VM to continue after a problem? 3) If this is pretty repeatable, then it would be interesting to try changing to a different disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. Dave > > > Thx Christian > > -----Ursprüngliche Nachricht----- > Von: Dan Kenigsberg [mailto:danken@redhat.com] > Gesendet: Freitag, 13. November 2015 20:00 > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > Cc: 'users@ovirt.org' <users@ovirt.org> > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote: > > Hi, > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso" (is > > there something better to use?) fort he nodes, and have random crashes > > of VMs The dumps are always the Same > > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump [Thread > > debugging using libthread_db enabled] Using host libthread_db library > > "/lib64/libthread_db.so.1". > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'. > > Program terminated with signal 11, Segmentation fault. > > #0 0x00007f0c559c4353 in > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::Fr > > eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > > > > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist centos7 > > or 3.6 > > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt. > > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible. > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-11-17 9:59 ` Dr. David Alan Gilbert @ 2015-11-17 10:36 ` Grundmann, Christian 2015-11-17 11:36 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 16+ messages in thread From: Grundmann, Christian @ 2015-11-17 10:36 UTC (permalink / raw) To: 'Dr. David Alan Gilbert' Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com Hi, @ Can you please use a 'thread apply all bt full' the full gives a little more info. gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full" [New LWP 52281] [New LWP 52288] [New LWP 52286] [New LWP 52291] [New LWP 52292] [New LWP 52287] [New LWP 52293] [New LWP 52290] [New LWP 56455] [New LWP 52289] [New LWP 52282] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,us'. Program terminated with signal 11, Segmentation fault. #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 Thread 11 (Thread 0x7f6d47719700 (LWP 52282)): #0 0x00007f6d4f8f2949 in syscall () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f6d57426272 in qemu_event_wait () No symbol table info available. #2 0x00007f6d57434526 in call_rcu_thread () No symbol table info available. #3 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6 No symbol table info available. Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)): #0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl () No symbol table info available. #2 0x00007f6d571f10de in kvm_cpu_exec () No symbol table info available. #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn () No symbol table info available. #4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6 No symbol table info available. Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)): #0 0x00007f6d55cf08a0 in sem_timedwait () from /lib64/libpthread.so.0 No symbol table info available. #1 0x00007f6d574260c7 in qemu_sem_timedwait () No symbol table info available. #2 0x00007f6d573a98ac in worker_thread () No symbol table info available. #3 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #4 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6 No symbol table info available. Thread 8 (Thread 0x7f6d44713700 (LWP 52290)): #0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl () No symbol table info available. #2 0x00007f6d571f10de in kvm_cpu_exec () No symbol table info available. #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn () No symbol table info available. #4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6 No symbol table info available. Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)): #0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl () No symbol table info available. #2 0x00007f6d571f10de in kvm_cpu_exec () No symbol table info available. #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn () No symbol table info available. #4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6 No symbol table info available. Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)): #0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl () No symbol table info available. #2 0x00007f6d571f10de in kvm_cpu_exec () No symbol table info available. #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn () No symbol table info available. #4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6 No symbol table info available. Thread 5 (Thread 0x7f6d43711700 (LWP 52292)): #0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl () No symbol table info available. #2 0x00007f6d571f10de in kvm_cpu_exec () No symbol table info available. #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn () No symbol table info available. #4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6 No symbol table info available. Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)): #0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl () No symbol table info available. #2 0x00007f6d571f10de in kvm_cpu_exec () No symbol table info available. #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn () No symbol table info available. #4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6 No symbol table info available. Thread 3 (Thread 0x7f6d46717700 (LWP 52286)): #0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl () No symbol table info available. #2 0x00007f6d571f10de in kvm_cpu_exec () No symbol table info available. #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn () No symbol table info available. #4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6 No symbol table info available. Thread 2 (Thread 0x7f6d45715700 (LWP 52288)): #0 0x00007f6d4f8ef257 in ioctl () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl () No symbol table info available. #2 0x00007f6d571f10de in kvm_cpu_exec () No symbol table info available. #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn () No symbol table info available. #4 0x00007f6d55ceadf5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f6d4f8f81ad in clone () from /lib64/libc.so.6 No symbol table info available. Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)): #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 No symbol table info available. #1 0x00007f6d572a4489 in malloc_and_trace () No symbol table info available. #2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0 No symbol table info available. #3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0 No symbol table info available. #4 0x00007f6d57204ffd in virtio_blk_handle_output () No symbol table info available. #5 0x00007f6d573b76b6 in qemu_iohandler_poll () No symbol table info available. #6 0x00007f6d573b7296 in main_loop_wait () No symbol table info available. #7 0x00007f6d571b6a4e in main () No symbol table info available. @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. Sorry it's a ovirt-node System where I can't you yum @ Does this part always look the same in your backtraces? The most are the same, found one a little bit different : Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)): #0 0x00007f3785d18353 in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available. #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from /lib64/libtcmalloc.so.4 No symbol table info available. #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No symbol table info available. #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No symbol table info available. #4 0x00007f37885fec89 in g_slice_free1 () from /lib64/libglib-2.0.so.0 No symbol table info available. #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table info available. #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info available. #7 0x00007f378a398394 in aio_bh_poll () No symbol table info available. #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table info available. #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info available. #10 0x00007f37885e299a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0 No symbol table info available. #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info available. #12 0x00007f378a1a5a4e in main () No symbol table info available. @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? No nothing abnormal @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem? Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot. Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline. @ 3) If this is pretty repeatable, then it would be interesting to try changing to a different disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. Ok will try that and report Thx Christian -----Ursprüngliche Nachricht----- Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] Gesendet: Dienstag, 17. November 2015 10:59 An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > Hi, > Dan sent me over to you, > please let me know if i can provide additional informations Hi Christian, Thanks for reporting this, > Softwareversions: > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso > > qemu-img-ev-2.3.0-29.1.el7.x86_64 > qemu-kvm-ev-2.3.0-29.1.el7.x86_64 > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch > kernel-3.10.0-229.14.1.el7.x86_64 > gperftools-libs-2.4-7.el7.x86_64 > > Commandline: > /usr/libexec/qemu-kvm -name myvmname -S -machine > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime mlock=off > -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios > type=1,manufacturer=oVirt,product=oVirt > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-5A > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319 > -nographic -no-user-config -nodefaults -chardev > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,serv > er,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc > base=2015-11-15T20:04:35,driftfix=slew -global > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on > -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > -drive > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84-8 > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffaecf > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,fo > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,werr > or=stop,rerror=stop,aio=native -device > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id > =virtio-disk0,bootindex=1 -netdev > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci.0 > ,addr=0x3 -chardev > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a > 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait -device > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=chann > el0,name=com.redhat.rhevm.vdsm -chardev > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a > 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait -device > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=chann > el1,name=org.qemu.guest_agent.0 -device > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on > > Stack Trace: > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt" Can you please use a 'thread apply all bt full' the full gives a little more info. Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)): > #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4 > #1 0x00007fa8b186b489 in malloc_and_trace () > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0 > #3 0x00007fa8afbd666e in g_slice_alloc () from > /lib64/libglib-2.0.so.0 > #4 0x00007fa8b17cbffd in virtio_blk_handle_output () > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll () > #6 0x00007fa8b197e296 in main_loop_wait () > #7 0x00007fa8b177da4e in main () Does this part always look the same in your backtraces? The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed. Some other ideas: 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? 2) Did you hit any IO errors and need to tell the VM to continue after a problem? 3) If this is pretty repeatable, then it would be interesting to try changing to a different disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. Dave > > > Thx Christian > > -----Ursprüngliche Nachricht----- > Von: Dan Kenigsberg [mailto:danken@redhat.com] > Gesendet: Freitag, 13. November 2015 20:00 > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > Cc: 'users@ovirt.org' <users@ovirt.org> > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote: > > Hi, > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso" > > (is there something better to use?) fort he nodes, and have random > > crashes of VMs The dumps are always the Same > > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump [Thread > > debugging using libthread_db enabled] Using host libthread_db > > library "/lib64/libthread_db.so.1". > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'. > > Program terminated with signal 11, Segmentation fault. > > #0 0x00007f0c559c4353 in > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache:: > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > > > > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist > > centos7 or 3.6 > > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt. > > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible. > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-11-17 10:36 ` Grundmann, Christian @ 2015-11-17 11:36 ` Dr. David Alan Gilbert 2015-11-17 14:11 ` Grundmann, Christian 0 siblings, 1 reply; 16+ messages in thread From: Dr. David Alan Gilbert @ 2015-11-17 11:36 UTC (permalink / raw) To: Grundmann, Christian; +Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > Hi, > > @ Can you please use a 'thread apply all bt full' the full gives a little more info. > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full" OK, it doesn't relaly give any more without the debuginfo package mentioned below. <snip> > @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > Sorry it's a ovirt-node System where I can't you yum Ah, although perhaps if you took the core dump, onto another machine with matching qemu and debuginfo you should be able to get more detail. > @ Does this part always look the same in your backtraces? > The most are the same, found one a little bit different : > Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)): > #0 0x00007f3785d18353 in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > No symbol table info available. > #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from /lib64/libtcmalloc.so.4 > No symbol table info available. > #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 > No symbol table info available. > #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 > No symbol table info available. > #4 0x00007f37885fec89 in g_slice_free1 () from /lib64/libglib-2.0.so.0 > No symbol table info available. > #5 0x00007f378a1f232e in virtio_blk_rw_complete () > No symbol table info available. > #6 0x00007f378a39f1ae in bdrv_co_em_bh () > No symbol table info available. > #7 0x00007f378a398394 in aio_bh_poll () > No symbol table info available. > #8 0x00007f378a3a7409 in aio_dispatch_clients () > No symbol table info available. > #9 0x00007f378a39820e in aio_ctx_dispatch () > No symbol table info available. > #10 0x00007f37885e299a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0 > No symbol table info available. > #11 0x00007f378a3a6288 in main_loop_wait () > No symbol table info available. > #12 0x00007f378a1a5a4e in main () > No symbol table info available. > OK, that's a bit different but interesting.... > @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > No nothing abnormal > > @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem? > Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot. > Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline. OK, that's interesting, because you may be hitting the following bug; http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html whose fix coincidentally just got accepted today; it's related to error cases with error=stop which you are using. Do you think you're only hitting these crashes on VMs that have been paused because of these space errors? > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. > > Ok will try that and report Thanks, Dave > > Thx Christian > > > -----Ursprüngliche Nachricht----- > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] > Gesendet: Dienstag, 17. November 2015 10:59 > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > Hi, > > Dan sent me over to you, > > please let me know if i can provide additional informations > > Hi Christian, > Thanks for reporting this, > > > Softwareversions: > > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso > > > > qemu-img-ev-2.3.0-29.1.el7.x86_64 > > qemu-kvm-ev-2.3.0-29.1.el7.x86_64 > > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 > > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 > > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch > > kernel-3.10.0-229.14.1.el7.x86_64 > > gperftools-libs-2.4-7.el7.x86_64 > > > > Commandline: > > /usr/libexec/qemu-kvm -name myvmname -S -machine > > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime mlock=off > > -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios > > type=1,manufacturer=oVirt,product=oVirt > > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-5A > > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319 > > -nographic -no-user-config -nodefaults -chardev > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,serv > > er,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc > > base=2015-11-15T20:04:35,driftfix=slew -global > > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on > > -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device > > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device > > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 > > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= > > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > > -drive > > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84-8 > > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffaecf > > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,fo > > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,werr > > or=stop,rerror=stop,aio=native -device > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id > > =virtio-disk0,bootindex=1 -netdev > > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci.0 > > ,addr=0x3 -chardev > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a > > 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait -device > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=chann > > el0,name=com.redhat.rhevm.vdsm -chardev > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-5a > > 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait -device > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=chann > > el1,name=org.qemu.guest_agent.0 -device > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on > > > > Stack Trace: > > > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt" > > Can you please use a 'thread apply all bt full' the full gives a little more info. > Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > > > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)): > > #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4 > > #1 0x00007fa8b186b489 in malloc_and_trace () > > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0 > > #3 0x00007fa8afbd666e in g_slice_alloc () from > > /lib64/libglib-2.0.so.0 > > #4 0x00007fa8b17cbffd in virtio_blk_handle_output () > > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll () > > #6 0x00007fa8b197e296 in main_loop_wait () > > #7 0x00007fa8b177da4e in main () > > Does this part always look the same in your backtraces? > The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed. > > Some other ideas: > 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > 2) Did you hit any IO errors and need to tell the VM to continue after a problem? > 3) If this is pretty repeatable, then it would be interesting to try changing to a different > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. > > Dave > > > > > > Thx Christian > > > > -----Ursprüngliche Nachricht----- > > Von: Dan Kenigsberg [mailto:danken@redhat.com] > > Gesendet: Freitag, 13. November 2015 20:00 > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > > Cc: 'users@ovirt.org' <users@ovirt.org> > > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc > > > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote: > > > Hi, > > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso" > > > (is there something better to use?) fort he nodes, and have random > > > crashes of VMs The dumps are always the Same > > > > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump [Thread > > > debugging using libthread_db enabled] Using host libthread_db > > > library "/lib64/libthread_db.so.1". > > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'. > > > Program terminated with signal 11, Segmentation fault. > > > #0 0x00007f0c559c4353 in > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache:: > > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > > > > > > > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist > > > centos7 or 3.6 > > > > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt. > > > > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible. > > > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-11-17 11:36 ` Dr. David Alan Gilbert @ 2015-11-17 14:11 ` Grundmann, Christian 2015-11-17 14:20 ` Grundmann, Christian 0 siblings, 1 reply; 16+ messages in thread From: Grundmann, Christian @ 2015-11-17 14:11 UTC (permalink / raw) To: 'Dr. David Alan Gilbert' Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com Here you go gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full" [New LWP 52281] [New LWP 52288] [New LWP 52286] [New LWP 52291] [New LWP 52292] [New LWP 52287] [New LWP 52293] [New LWP 52290] [New LWP 56455] [New LWP 52289] [New LWP 52282] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,us'. Program terminated with signal 11, Segmentation fault. #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 Thread 11 (Thread 0x7f6d47719700 (LWP 52282)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 No locals. #1 0x00007f6d57426272 in futex_wait (val=4294967295, ev=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals. #2 qemu_event_wait (ev=ev@entry=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399 value = <optimized out> #3 0x00007f6d57434526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233 tries = 0 n = <optimized out> node = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d47719700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d47719700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107326789376, 5884348200482620104, 0, 140107326790080, 140107326789376, 140107592666688, -5804038895876586808, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8a4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d44f139e0, reg_save_area = 0x7f6d44f139a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570cf000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c8a4000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44f14700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d44f14700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107284825856, 5884348200482620104, 0, 140107284826560, 140107284825856, 140726431086992, -5804033392412867896, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)): #0 sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101 No locals. #1 0x00007f6d574260c7 in qemu_sem_timedwait (sem=sem@entry=0x7f6d5a1b9248, ms=ms@entry=10000) at util/qemu-thread-posix.c:254 rc = <optimized out> ts = {tv_sec = 1447709021, tv_nsec = 21985000} __func__ = "qemu_sem_timedwait" #2 0x00007f6d573a98ac in worker_thread (opaque=0x7f6d5a1b91e0) at thread-pool.c:92 req = <optimized out> ret = <optimized out> pool = 0x7f6d5a1b91e0 #3 0x00007f6d55ceadf5 in start_thread (arg=0x7f6ab1dff700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6ab1dff700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140096227505920, 5884348200482620104, 0, 140096227506624, 140096227505920, 26, -5801757560646548792, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #4 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 8 (Thread 0x7f6d44713700 (LWP 52290)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8b8000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d447129e0, reg_save_area = 0x7f6d447129a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570cc000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c8b8000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44713700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d44713700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107276433152, 5884348200482620104, 0, 140107276433856, 140107276433152, 140726431086992, -5804032293438111032, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8f4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d42f0f9e0, reg_save_area = 0x7f6d42f0f9a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570c3000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c8f4000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d42f10700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d42f10700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107251255040, 5884348200482620104, 0, 140107251255744, 140107251255040, 140726431086992, -5804046580109950264, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c878000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d45f159e0, reg_save_area = 0x7f6d45f159a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570d5000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c878000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45f16700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d45f16700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107301611264, 5884348200482620104, 0, 140107301611968, 140107301611264, 140726431086992, -5804035590362381624, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 5 (Thread 0x7f6d43711700 (LWP 52292)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8e0000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d437109e0, reg_save_area = 0x7f6d437109a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570c6000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c8e0000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43711700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d43711700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107259647744, 5884348200482620104, 0, 140107259648448, 140107259647744, 140726431086992, -5804047687674641720, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8cc000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d43f119e0, reg_save_area = 0x7f6d43f119a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570c9000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c8cc000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43f12700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d43f12700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107268040448, 5884348200482620104, 0, 140107268041152, 140107268040448, 140726431086992, -5804048786649398584, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 3 (Thread 0x7f6d46717700 (LWP 52286)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c810000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d467169e0, reg_save_area = 0x7f6d467169a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570d8000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c810000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d46717700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d46717700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107310003968, 5884348200482620104, 0, 140107310004672, 140107310003968, 140726431086992, -5804036689337138488, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 2 (Thread 0x7f6d45715700 (LWP 52288)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c890000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d457149e0, reg_save_area = 0x7f6d457149a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570d2000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c890000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45715700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d45715700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107293218560, 5884348200482620104, 0, 140107293219264, 140107293218560, 140726431086992, -5804034491387624760, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)): #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 No symbol table info available. #1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=49280) at vl.c:2575 ptr = 0x7f6d59a346a0 #2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0 No symbol table info available. #3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0 No symbol table info available. #4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33 req = <optimized out> #5 virtio_blk_get_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:192 req = <optimized out> #6 virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604 s = 0x7f6d5de1ff40 __func__ = "virtio_blk_handle_output" mrb = {reqs = {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7c000, 0x0 <repeats 29 times>}, num_reqs = 3, is_write = true} #7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=0x7f6d5a1aea00, ret=62, ret@entry=1) at iohandler.c:143 revents = 1 #8 0x00007f6d573b7296 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:504 ret = 1 timeout = 4294967295 timeout_ns = <optimized out> #9 0x00007f6d571b6a4e in main_loop () at vl.c:1818 nonblocking = <optimized out> last_io = 0 #10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394 i = <optimized out> snapshot = <optimized out> linux_boot = <optimized out> initrd_filename = <optimized out> kernel_filename = <optimized out> kernel_cmdline = <optimized out> boot_order = 0x7f6d57449a67 "cad" boot_once = 0x0 cyls = <optimized out> heads = <optimized out> secs = <optimized out> translation = <optimized out> hda_opts = <optimized out> opts = <optimized out> machine_opts = <optimized out> icount_opts = <optimized out> olist = <optimized out> optind = 69 optarg = 0x7f6d5a14b3a0 "rhel6.5.0" loadvm = <optimized out> machine_class = <optimized out> cpu_model = <optimized out> vga_model = 0x0 qtest_chrdev = <optimized out> qtest_log = <optimized out> pid_file = <optimized out> incoming = <optimized out> show_vnc_port = <optimized out> defconfig = <optimized out> userconfig = 179 log_mask = <optimized out> log_file = <optimized out> mem_trace = {malloc = 0x7f6d572a4480 <malloc_and_trace>, realloc = 0x7f6d572a4460 <realloc_and_trace>, free = 0x7f6d572a4450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} trace_events = <optimized out> trace_file = <optimized out> maxram_size = <optimized out> ram_slots = <optimized out> vmstate_dump_file = <optimized out> main_loop_err = 0x0 __func__ = "main" @ Do you think you're only hitting these crashes on VMs that have been paused because of these space errors? Will have a look on that Thx Christian -----Ursprüngliche Nachricht----- Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] Gesendet: Dienstag, 17. November 2015 12:36 An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > Hi, > > @ Can you please use a 'thread apply all bt full' the full gives a little more info. > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full" OK, it doesn't relaly give any more without the debuginfo package mentioned below. <snip> > @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > Sorry it's a ovirt-node System where I can't you yum Ah, although perhaps if you took the core dump, onto another machine with matching qemu and debuginfo you should be able to get more detail. > @ Does this part always look the same in your backtraces? > The most are the same, found one a little bit different : > Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)): > #0 0x00007f3785d18353 in > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available. > #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from > /lib64/libtcmalloc.so.4 No symbol table info available. > #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No > symbol table info available. > #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No > symbol table info available. > #4 0x00007f37885fec89 in g_slice_free1 () from > /lib64/libglib-2.0.so.0 No symbol table info available. > #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table > info available. > #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info > available. > #7 0x00007f378a398394 in aio_bh_poll () No symbol table info > available. > #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table info > available. > #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info > available. > #10 0x00007f37885e299a in g_main_context_dispatch () from > /lib64/libglib-2.0.so.0 No symbol table info available. > #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info > available. > #12 0x00007f378a1a5a4e in main () > No symbol table info available. > OK, that's a bit different but interesting.... > @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > No nothing abnormal > > @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem? > Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot. > Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline. OK, that's interesting, because you may be hitting the following bug; http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html whose fix coincidentally just got accepted today; it's related to error cases with error=stop which you are using. Do you think you're only hitting these crashes on VMs that have been paused because of these space errors? > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. > > Ok will try that and report Thanks, Dave > > Thx Christian > > > -----Ursprüngliche Nachricht----- > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] > Gesendet: Dienstag, 17. November 2015 10:59 > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > stefanha@redhat.com > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in > libtcmalloc > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > Hi, > > Dan sent me over to you, > > please let me know if i can provide additional informations > > Hi Christian, > Thanks for reporting this, > > > Softwareversions: > > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso > > > > qemu-img-ev-2.3.0-29.1.el7.x86_64 > > qemu-kvm-ev-2.3.0-29.1.el7.x86_64 > > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 > > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 > > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch > > kernel-3.10.0-229.14.1.el7.x86_64 > > gperftools-libs-2.4-7.el7.x86_64 > > > > Commandline: > > /usr/libexec/qemu-kvm -name myvmname -S -machine > > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime > > mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios > > type=1,manufacturer=oVirt,product=oVirt > > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138- > > 5A > > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319 > > -nographic -no-user-config -nodefaults -chardev > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,se > > rv er,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc > > base=2015-11-15T20:04:35,driftfix=slew -global > > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot > > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 > > -device > > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device > > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 > > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= > > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > > -drive > > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84 > > -8 > > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffae > > cf > > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0, > > fo > > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,we > > rr or=stop,rerror=stop,aio=native -device > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0, > > id > > =virtio-disk0,bootindex=1 -netdev > > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci > > .0 > > ,addr=0x3 -chardev > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899- > > 5a 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait > > -device > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=cha > > nn el0,name=com.redhat.rhevm.vdsm -chardev > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899- > > 5a 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait > > -device > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=cha > > nn > > el1,name=org.qemu.guest_agent.0 -device > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on > > > > Stack Trace: > > > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt" > > Can you please use a 'thread apply all bt full' the full gives a little more info. > Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > > > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)): > > #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4 > > #1 0x00007fa8b186b489 in malloc_and_trace () > > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0 > > #3 0x00007fa8afbd666e in g_slice_alloc () from > > /lib64/libglib-2.0.so.0 > > #4 0x00007fa8b17cbffd in virtio_blk_handle_output () > > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll () > > #6 0x00007fa8b197e296 in main_loop_wait () > > #7 0x00007fa8b177da4e in main () > > Does this part always look the same in your backtraces? > The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed. > > Some other ideas: > 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > 2) Did you hit any IO errors and need to tell the VM to continue after a problem? > 3) If this is pretty repeatable, then it would be interesting to try changing to a different > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. > > Dave > > > > > > Thx Christian > > > > -----Ursprüngliche Nachricht----- > > Von: Dan Kenigsberg [mailto:danken@redhat.com] > > Gesendet: Freitag, 13. November 2015 20:00 > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > > Cc: 'users@ovirt.org' <users@ovirt.org> > > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc > > > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote: > > > Hi, > > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso" > > > (is there something better to use?) fort he nodes, and have random > > > crashes of VMs The dumps are always the Same > > > > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump > > > [Thread debugging using libthread_db enabled] Using host > > > libthread_db library "/lib64/libthread_db.so.1". > > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'. > > > Program terminated with signal 11, Segmentation fault. > > > #0 0x00007f0c559c4353 in > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache:: > > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > > > > > > > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist > > > centos7 or 3.6 > > > > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt. > > > > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible. > > > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-11-17 14:11 ` Grundmann, Christian @ 2015-11-17 14:20 ` Grundmann, Christian 2015-11-17 14:42 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 16+ messages in thread From: Grundmann, Christian @ 2015-11-17 14:20 UTC (permalink / raw) To: 'Dr. David Alan Gilbert' Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com And here another one [New LWP 1507] [New LWP 1508] [New LWP 1514] [New LWP 1513] [New LWP 2417] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/libexec/qemu-kvm -name myvmname2 -S -machine rhel6.5.0,accel=kvm,usb=o'. Program terminated with signal 11, Segmentation fault. #0 0x00007f5cff7e2e7d in tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from /lib64/libtcmalloc.so.4 Thread 5 (Thread 0x7f5cf39d5700 (LWP 2417)): #0 0x00007f5d027aeac3 in pread64 () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f5d03ea52a3 in pread (__offset=<optimized out>, __nbytes=<optimized out>, __buf=0x7f5d061f8000, __fd=<optimized out>) at /usr/include/bits/unistd.h:99 No locals. #2 handle_aiocb_rw_linear (aiocb=aiocb@entry=0x7f5d061e7340, buf=buf@entry=0x7f5d061f8000 "QF", <incomplete sequence \373>) at block/raw-posix.c:909 offset = 0 len = <optimized out> #3 0x00007f5d03ea53d1 in handle_aiocb_rw (aiocb=0x7f5d061e7340) at block/raw-posix.c:992 nbytes = <optimized out> buf = 0x7f5d061f8000 "QF", <incomplete sequence \373> __PRETTY_FUNCTION__ = "handle_aiocb_rw" #4 0x00007f5d03ea6945 in aio_worker (arg=0x7f5d061e7340) at block/raw-posix.c:1204 aiocb = 0x7f5d061e7340 ret = 0 #5 0x00007f5d03e6691b in worker_thread (opaque=0x7f5d062011e0) at thread-pool.c:105 req = 0x7f5d061e6a60 ret = <optimized out> pool = 0x7f5d062011e0 #6 0x00007f5d027a7df5 in start_thread (arg=0x7f5cf39d5700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f5cf39d5700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140037200893696, 4764812136514238282, 0, 140037200894400, 140037200893696, 26, -4856599629847414966, -4856560931613919414}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #7 0x00007f5cfc3b51ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 4 (Thread 0x7f5cf31d4700 (LWP 1513)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 No locals. #1 0x00007f5d03ee2f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7f5d0438fd40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132 err = <optimized out> __func__ = "qemu_cond_wait" #2 0x00007f5d03c9bd3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912 No locals. #3 qemu_kvm_cpu_thread_fn (arg=0x7f5d0757a000) at /usr/src/debug/qemu-2.3.0/cpus.c:949 cpu = 0x7f5d0757a000 r = <optimized out> #4 0x00007f5d027a7df5 in start_thread (arg=0x7f5cf31d4700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f5cf31d4700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140037192500992, 4764812136514238282, 0, 140037192501696, 140037192500992, 140728281959696, -4856598527651432630, -4856560931613919414}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f5cfc3b51ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 3 (Thread 0x7f5cf29d3700 (LWP 1514)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 No locals. #1 0x00007f5d03ee2f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7f5d0438fd40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132 err = <optimized out> __func__ = "qemu_cond_wait" #2 0x00007f5d03c9bd3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912 No locals. #3 qemu_kvm_cpu_thread_fn (arg=0x7f5d075e2000) at /usr/src/debug/qemu-2.3.0/cpus.c:949 cpu = 0x7f5d075e2000 r = <optimized out> #4 0x00007f5d027a7df5 in start_thread (arg=0x7f5cf29d3700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f5cf29d3700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140037184108288, 4764812136514238282, 0, 140037184108992, 140037184108288, 140728281959696, -4856597427602933942, -4856560931613919414}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f5cfc3b51ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 2 (Thread 0x7f5cf41d6700 (LWP 1508)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 No locals. #1 0x00007f5d03ee3272 in futex_wait (val=4294967295, ev=0x7f5d047adf44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals. #2 qemu_event_wait (ev=ev@entry=0x7f5d047adf44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399 value = <optimized out> #3 0x00007f5d03ef1526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233 tries = 0 n = <optimized out> node = <optimized out> #4 0x00007f5d027a7df5 in start_thread (arg=0x7f5cf41d6700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f5cf41d6700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140037209286400, 4764812136514238282, 0, 140037209287104, 140037209286400, 140037475163712, -4856591933802891446, -4856560931613919414}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f5cfc3b51ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 1 (Thread 0x7f5d03ba5c00 (LWP 1507)): #0 0x00007f5cff7e2e7d in tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from /lib64/libtcmalloc.so.4 No symbol table info available. #1 0x00007f5cff7e312a in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) () from /lib64/libtcmalloc.so.4 No symbol table info available. #2 0x00007f5cff7e31dd in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /lib64/libtcmalloc.so.4 No symbol table info available. #3 0x00007f5cff7e6235 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /lib64/libtcmalloc.so.4 No symbol table info available. #4 0x00007f5cff7f4d5b in tc_malloc () from /lib64/libtcmalloc.so.4 No symbol table info available. #5 0x00007f5d03d61489 in malloc_and_trace (n_bytes=18) at vl.c:2575 ptr = 0x1 #6 0x00007f5d020b647f in g_malloc () from /lib64/libglib-2.0.so.0 No symbol table info available. #7 0x00007f5d020cdf7f in g_strdup () from /lib64/libglib-2.0.so.0 No symbol table info available. #8 0x00007f5d03eddab5 in alloc_entry (value=0x7f5d088de6c0, key=0x7f5d03f5debb "wr_highest_offset") at qobject/qdict.c:79 entry = 0x7f5d088df480 #9 qdict_put_obj (qdict=0x7f5d06e10400, key=0x7f5d03f5debb "wr_highest_offset", value=0x7f5d088de6c0) at qobject/qdict.c:145 bucket = 81 entry = <optimized out> #10 0x00007f5d03ebf34a in visit_type_BlockDeviceStats_fields (errp=0x7ffddb417ca0, obj=0x7f5d07f905a0, m=0x7f5d061fdea0) at qapi-visit.c:1542 err = 0x0 #11 visit_type_BlockDeviceStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f905a0, name=name@entry=0x7f5d03f032ec "stats", errp=errp@entry=0x7ffddb417ca0) at qapi-visit.c:1566 err = 0x0 #12 0x00007f5d03ebf5b1 in visit_type_BlockStats_fields (errp=0x7ffddb417cf0, obj=0x7f5d07f90650, m=0x7f5d061fdea0) at qapi-visit.c:1614 err = 0x0 #13 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f90650, name=name@entry=0x7f5d03f480f4 "parent", errp=errp@entry=0x7ffddb417cf0) at qapi-visit.c:1644 err = 0x0 #14 0x00007f5d03ebf6bd in visit_type_BlockStats_fields (errp=0x7ffddb417d38, obj=0x7f5d07c67a50, m=0x7f5d061fdea0) at qapi-visit.c:1620 err = 0x0 #15 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07c67a50, name=name@entry=0x0, errp=errp@entry=0x7ffddb417d38) at qapi-visit.c:1644 err = 0x0 #16 0x00007f5d03ebf760 in visit_type_BlockStatsList (m=0x7f5d061fdea0, obj=obj@entry=0x7ffddb417d98, name=name@entry=0x7f5d03f00e6e "unused", errp=errp@entry=0x7ffddb417da0) at qapi-visit.c:1665 native_i = <optimized out> err = 0x0 i = 0x7f5d07c67a50 prev = 0x7ffddb417d40 #17 0x00007f5d03d674dd in qmp_marshal_output_query_blockstats (errp=0x7ffddb417d90, ret_out=0x7ffddb417e10, ret_in=0x7f5d07c67120) at qmp-marshal.c:182 local_err = 0x0 mo = 0x7f5d061fdea0 md = <optimized out> v = <optimized out> #18 qmp_marshal_input_query_blockstats (mon=<optimized out>, qdict=<optimized out>, ret=0x7ffddb417e10) at qmp-marshal.c:225 local_err = 0x0 args = <optimized out> retval = <optimized out> mi = 0x7f5d064e2000 md = <optimized out> v = <optimized out> has_query_nodes = false query_nodes = false #19 0x00007f5d03ca0531 in qmp_call_cmd (cmd=<optimized out>, params=0x7f5d075dd600, mon=0x7f5d06208320) at /usr/src/debug/qemu-2.3.0/monitor.c:5051 ret = <optimized out> data = 0x0 #20 handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5113 err = <optimized out> obj = <optimized out> input = <optimized out> args = 0x7f5d075dd600 cmd_name = <optimized out> mon = 0x7f5d06208320 #21 0x00007f5d03edf4f2 in json_message_process_token (lexer=0x7f5d061f5d70, token=0x7f5d061991e0, type=JSON_OPERATOR, x=48, y=15) at qobject/json-streamer.c:87 parser = 0x7f5d061f5d68 dict = 0x7f5d088ea800 #22 0x00007f5d03ef191f in json_lexer_feed_char (lexer=lexer@entry=0x7f5d061f5d70, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303 new_state = 100 #23 0x00007f5d03ef19ee in json_lexer_feed (lexer=0x7f5d061f5d70, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356 err = <optimized out> i = <optimized out> #24 0x00007f5d03edf689 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 No locals. #25 0x00007f5d03c9e8cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134 old_mon = 0x0 #26 0x00007f5d03d5b1b0 in qemu_chr_be_write (len=<optimized out>, buf=0x7ffddb417f40 "}\177A\333\375\177", s=0x7f5d0625a2e0) at qemu-char.c:305 No locals. #27 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f5d0625a2e0) at qemu-char.c:2870 chr = 0x7f5d0625a2e0 s = 0x7f5d061aa3f0 buf = "}\177A\333\375\177\000\000\360\360\355\003]\177\000\000\030\003\000\000\000\000\000\000\205\336\355\003]\177\000\000\000A\036\b]\177\000\000\263\325\355\003]\177\000\000\000A\036\b]\177", '\000' <repeats 18 times>, "`2\036\b]\177\000\000ЀA\333\375\177\000\000\000\000\000\000\000\000\000\000 \232\261\a]\177\000\000\000(\341\006]\177\000\000P\217A\333\375\177\000\000H\242<\374\\\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\240\200A\333\375\177\000\000\340\177A\333\375\177\000\000\r\000\000\000\000\000\000\000 Xz\b]\177\000\000\000@&\006]\177\000\000\340\365\215\b]\177\000\000Q\000\000\000\000\000\000\000\232"... len = <optimized out> size = <optimized out> #28 0x00007f5d020b099a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0 No symbol table info available. #29 0x00007f5d03e74288 in glib_pollfds_poll () at main-loop.c:209 context = 0x7f5d06205140 pfds = <optimized out> #30 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254 ret = 2 spin_counter = 0 #31 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503 ret = 2 timeout = 4294967295 timeout_ns = <optimized out> #32 0x00007f5d03c73a4e in main_loop () at vl.c:1818 nonblocking = <optimized out> last_io = 2 #33 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394 i = <optimized out> snapshot = <optimized out> linux_boot = <optimized out> initrd_filename = <optimized out> kernel_filename = <optimized out> kernel_cmdline = <optimized out> boot_order = 0x7f5d03f06a67 "cad" boot_once = 0x0 cyls = <optimized out> heads = <optimized out> secs = <optimized out> translation = <optimized out> hda_opts = <optimized out> opts = <optimized out> machine_opts = <optimized out> icount_opts = <optimized out> olist = <optimized out> optind = 67 optarg = 0x7f5d06193570 "rhel6.5.0" loadvm = <optimized out> machine_class = <optimized out> cpu_model = <optimized out> vga_model = 0x0 qtest_chrdev = <optimized out> qtest_log = <optimized out> pid_file = <optimized out> incoming = <optimized out> show_vnc_port = <optimized out> defconfig = <optimized out> userconfig = 56 log_mask = <optimized out> log_file = <optimized out> mem_trace = {malloc = 0x7f5d03d61480 <malloc_and_trace>, realloc = 0x7f5d03d61460 <realloc_and_trace>, free = 0x7f5d03d61450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} trace_events = <optimized out> trace_file = <optimized out> maxram_size = <optimized out> ram_slots = <optimized out> vmstate_dump_file = <optimized out> main_loop_err = 0x0 __func__ = "main" -----Ursprüngliche Nachricht----- Von: qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org [mailto:qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org] Im Auftrag von Grundmann, Christian Gesendet: Dienstag, 17. November 2015 15:12 An: 'Dr. David Alan Gilbert' <dgilbert@redhat.com> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc Here you go gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full" [New LWP 52281] [New LWP 52288] [New LWP 52286] [New LWP 52291] [New LWP 52292] [New LWP 52287] [New LWP 52293] [New LWP 52290] [New LWP 56455] [New LWP 52289] [New LWP 52282] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,us'. Program terminated with signal 11, Segmentation fault. #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 Thread 11 (Thread 0x7f6d47719700 (LWP 52282)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 No locals. #1 0x00007f6d57426272 in futex_wait (val=4294967295, ev=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals. #2 qemu_event_wait (ev=ev@entry=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399 value = <optimized out> #3 0x00007f6d57434526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233 tries = 0 n = <optimized out> node = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d47719700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d47719700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107326789376, 5884348200482620104, 0, 140107326790080, 140107326789376, 140107592666688, -5804038895876586808, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8a4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d44f139e0, reg_save_area = 0x7f6d44f139a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570cf000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c8a4000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44f14700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d44f14700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107284825856, 5884348200482620104, 0, 140107284826560, 140107284825856, 140726431086992, -5804033392412867896, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)): #0 sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101 No locals. #1 0x00007f6d574260c7 in qemu_sem_timedwait (sem=sem@entry=0x7f6d5a1b9248, ms=ms@entry=10000) at util/qemu-thread-posix.c:254 rc = <optimized out> ts = {tv_sec = 1447709021, tv_nsec = 21985000} __func__ = "qemu_sem_timedwait" #2 0x00007f6d573a98ac in worker_thread (opaque=0x7f6d5a1b91e0) at thread-pool.c:92 req = <optimized out> ret = <optimized out> pool = 0x7f6d5a1b91e0 #3 0x00007f6d55ceadf5 in start_thread (arg=0x7f6ab1dff700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6ab1dff700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140096227505920, 5884348200482620104, 0, 140096227506624, 140096227505920, 26, -5801757560646548792, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #4 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 8 (Thread 0x7f6d44713700 (LWP 52290)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8b8000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d447129e0, reg_save_area = 0x7f6d447129a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570cc000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c8b8000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44713700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d44713700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107276433152, 5884348200482620104, 0, 140107276433856, 140107276433152, 140726431086992, -5804032293438111032, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8f4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d42f0f9e0, reg_save_area = 0x7f6d42f0f9a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570c3000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c8f4000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d42f10700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d42f10700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107251255040, 5884348200482620104, 0, 140107251255744, 140107251255040, 140726431086992, -5804046580109950264, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c878000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d45f159e0, reg_save_area = 0x7f6d45f159a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570d5000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c878000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45f16700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d45f16700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107301611264, 5884348200482620104, 0, 140107301611968, 140107301611264, 140726431086992, -5804035590362381624, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 5 (Thread 0x7f6d43711700 (LWP 52292)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8e0000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d437109e0, reg_save_area = 0x7f6d437109a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570c6000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c8e0000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43711700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d43711700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107259647744, 5884348200482620104, 0, 140107259648448, 140107259647744, 140726431086992, -5804047687674641720, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8cc000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d43f119e0, reg_save_area = 0x7f6d43f119a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570c9000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c8cc000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43f12700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d43f12700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107268040448, 5884348200482620104, 0, 140107268041152, 140107268040448, 140726431086992, -5804048786649398584, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 3 (Thread 0x7f6d46717700 (LWP 52286)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c810000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d467169e0, reg_save_area = 0x7f6d467169a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570d8000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c810000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d46717700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d46717700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107310003968, 5884348200482620104, 0, 140107310004672, 140107310003968, 140726431086992, -5804036689337138488, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 2 (Thread 0x7f6d45715700 (LWP 52288)): #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c890000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 ret = <optimized out> arg = <optimized out> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d457149e0, reg_save_area = 0x7f6d457149a0}} #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 run = 0x7f6d570d2000 ret = <optimized out> run_ret = <optimized out> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 cpu = 0x7f6d5c890000 r = <optimized out> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45715700) at pthread_create.c:308 __res = <optimized out> pd = 0x7f6d45715700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107293218560, 5884348200482620104, 0, 140107293219264, 140107293218560, 140726431086992, -5804034491387624760, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)): #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 No symbol table info available. #1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=49280) at vl.c:2575 ptr = 0x7f6d59a346a0 #2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0 No symbol table info available. #3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0 No symbol table info available. #4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33 req = <optimized out> #5 virtio_blk_get_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:192 req = <optimized out> #6 virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604 s = 0x7f6d5de1ff40 __func__ = "virtio_blk_handle_output" mrb = {reqs = {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7c000, 0x0 <repeats 29 times>}, num_reqs = 3, is_write = true} #7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=0x7f6d5a1aea00, ret=62, ret@entry=1) at iohandler.c:143 revents = 1 #8 0x00007f6d573b7296 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:504 ret = 1 timeout = 4294967295 timeout_ns = <optimized out> #9 0x00007f6d571b6a4e in main_loop () at vl.c:1818 nonblocking = <optimized out> last_io = 0 #10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394 i = <optimized out> snapshot = <optimized out> linux_boot = <optimized out> initrd_filename = <optimized out> kernel_filename = <optimized out> kernel_cmdline = <optimized out> boot_order = 0x7f6d57449a67 "cad" boot_once = 0x0 cyls = <optimized out> heads = <optimized out> secs = <optimized out> translation = <optimized out> hda_opts = <optimized out> opts = <optimized out> machine_opts = <optimized out> icount_opts = <optimized out> olist = <optimized out> optind = 69 optarg = 0x7f6d5a14b3a0 "rhel6.5.0" loadvm = <optimized out> machine_class = <optimized out> cpu_model = <optimized out> vga_model = 0x0 qtest_chrdev = <optimized out> qtest_log = <optimized out> pid_file = <optimized out> incoming = <optimized out> show_vnc_port = <optimized out> defconfig = <optimized out> userconfig = 179 log_mask = <optimized out> log_file = <optimized out> mem_trace = {malloc = 0x7f6d572a4480 <malloc_and_trace>, realloc = 0x7f6d572a4460 <realloc_and_trace>, free = 0x7f6d572a4450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} trace_events = <optimized out> trace_file = <optimized out> maxram_size = <optimized out> ram_slots = <optimized out> vmstate_dump_file = <optimized out> main_loop_err = 0x0 __func__ = "main" @ Do you think you're only hitting these crashes on VMs that have been paused because of these space errors? Will have a look on that Thx Christian -----Ursprüngliche Nachricht----- Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] Gesendet: Dienstag, 17. November 2015 12:36 An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > Hi, > > @ Can you please use a 'thread apply all bt full' the full gives a little more info. > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full" OK, it doesn't relaly give any more without the debuginfo package mentioned below. <snip> > @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > Sorry it's a ovirt-node System where I can't you yum Ah, although perhaps if you took the core dump, onto another machine with matching qemu and debuginfo you should be able to get more detail. > @ Does this part always look the same in your backtraces? > The most are the same, found one a little bit different : > Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)): > #0 0x00007f3785d18353 in > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available. > #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from > /lib64/libtcmalloc.so.4 No symbol table info available. > #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No > symbol table info available. > #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No > symbol table info available. > #4 0x00007f37885fec89 in g_slice_free1 () from > /lib64/libglib-2.0.so.0 No symbol table info available. > #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table > info available. > #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info > available. > #7 0x00007f378a398394 in aio_bh_poll () No symbol table info > available. > #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table info > available. > #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info > available. > #10 0x00007f37885e299a in g_main_context_dispatch () from > /lib64/libglib-2.0.so.0 No symbol table info available. > #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info > available. > #12 0x00007f378a1a5a4e in main () > No symbol table info available. > OK, that's a bit different but interesting.... > @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > No nothing abnormal > > @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem? > Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot. > Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline. OK, that's interesting, because you may be hitting the following bug; http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html whose fix coincidentally just got accepted today; it's related to error cases with error=stop which you are using. Do you think you're only hitting these crashes on VMs that have been paused because of these space errors? > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. > > Ok will try that and report Thanks, Dave > > Thx Christian > > > -----Ursprüngliche Nachricht----- > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] > Gesendet: Dienstag, 17. November 2015 10:59 > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > stefanha@redhat.com > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in > libtcmalloc > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > Hi, > > Dan sent me over to you, > > please let me know if i can provide additional informations > > Hi Christian, > Thanks for reporting this, > > > Softwareversions: > > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso > > > > qemu-img-ev-2.3.0-29.1.el7.x86_64 > > qemu-kvm-ev-2.3.0-29.1.el7.x86_64 > > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 > > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 > > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch > > kernel-3.10.0-229.14.1.el7.x86_64 > > gperftools-libs-2.4-7.el7.x86_64 > > > > Commandline: > > /usr/libexec/qemu-kvm -name myvmname -S -machine > > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime > > mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios > > type=1,manufacturer=oVirt,product=oVirt > > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138- > > 5A > > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319 > > -nographic -no-user-config -nodefaults -chardev > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,se > > rv er,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc > > base=2015-11-15T20:04:35,driftfix=slew -global > > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot > > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 > > -device > > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device > > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 > > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= > > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > > -drive > > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84 > > -8 > > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffae > > cf > > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0, > > fo > > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,we > > rr or=stop,rerror=stop,aio=native -device > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0, > > id > > =virtio-disk0,bootindex=1 -netdev > > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci > > .0 > > ,addr=0x3 -chardev > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899- > > 5a 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait > > -device > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=cha > > nn el0,name=com.redhat.rhevm.vdsm -chardev > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899- > > 5a 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait > > -device > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=cha > > nn > > el1,name=org.qemu.guest_agent.0 -device > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on > > > > Stack Trace: > > > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt" > > Can you please use a 'thread apply all bt full' the full gives a little more info. > Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > > > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)): > > #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4 > > #1 0x00007fa8b186b489 in malloc_and_trace () > > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0 > > #3 0x00007fa8afbd666e in g_slice_alloc () from > > /lib64/libglib-2.0.so.0 > > #4 0x00007fa8b17cbffd in virtio_blk_handle_output () > > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll () > > #6 0x00007fa8b197e296 in main_loop_wait () > > #7 0x00007fa8b177da4e in main () > > Does this part always look the same in your backtraces? > The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed. > > Some other ideas: > 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > 2) Did you hit any IO errors and need to tell the VM to continue after a problem? > 3) If this is pretty repeatable, then it would be interesting to try changing to a different > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. > > Dave > > > > > > Thx Christian > > > > -----Ursprüngliche Nachricht----- > > Von: Dan Kenigsberg [mailto:danken@redhat.com] > > Gesendet: Freitag, 13. November 2015 20:00 > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > > Cc: 'users@ovirt.org' <users@ovirt.org> > > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc > > > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote: > > > Hi, > > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso" > > > (is there something better to use?) fort he nodes, and have random > > > crashes of VMs The dumps are always the Same > > > > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump > > > [Thread debugging using libthread_db enabled] Using host > > > libthread_db library "/lib64/libthread_db.so.1". > > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'. > > > Program terminated with signal 11, Segmentation fault. > > > #0 0x00007f0c559c4353 in > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache:: > > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > > > > > > > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist > > > centos7 or 3.6 > > > > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt. > > > > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible. > > > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-11-17 14:20 ` Grundmann, Christian @ 2015-11-17 14:42 ` Dr. David Alan Gilbert 2015-11-19 16:00 ` Grundmann, Christian 0 siblings, 1 reply; 16+ messages in thread From: Dr. David Alan Gilbert @ 2015-11-17 14:42 UTC (permalink / raw) To: Grundmann, Christian; +Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > And here another one Oh this is a bit of a different one, from query-blockstats, although again if the heap's corrupted it might have just been the first guy to trip over the corrupt part afterwards. Dave > Thread 1 (Thread 0x7f5d03ba5c00 (LWP 1507)): > #0 0x00007f5cff7e2e7d in tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from /lib64/libtcmalloc.so.4 > No symbol table info available. > #1 0x00007f5cff7e312a in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) () from /lib64/libtcmalloc.so.4 > No symbol table info available. > #2 0x00007f5cff7e31dd in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /lib64/libtcmalloc.so.4 > No symbol table info available. > #3 0x00007f5cff7e6235 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /lib64/libtcmalloc.so.4 > No symbol table info available. > #4 0x00007f5cff7f4d5b in tc_malloc () from /lib64/libtcmalloc.so.4 > No symbol table info available. > #5 0x00007f5d03d61489 in malloc_and_trace (n_bytes=18) at vl.c:2575 > ptr = 0x1 > #6 0x00007f5d020b647f in g_malloc () from /lib64/libglib-2.0.so.0 > No symbol table info available. > #7 0x00007f5d020cdf7f in g_strdup () from /lib64/libglib-2.0.so.0 > No symbol table info available. > #8 0x00007f5d03eddab5 in alloc_entry (value=0x7f5d088de6c0, key=0x7f5d03f5debb "wr_highest_offset") at qobject/qdict.c:79 > entry = 0x7f5d088df480 > #9 qdict_put_obj (qdict=0x7f5d06e10400, key=0x7f5d03f5debb "wr_highest_offset", value=0x7f5d088de6c0) at qobject/qdict.c:145 > bucket = 81 > entry = <optimized out> > #10 0x00007f5d03ebf34a in visit_type_BlockDeviceStats_fields (errp=0x7ffddb417ca0, obj=0x7f5d07f905a0, m=0x7f5d061fdea0) at qapi-visit.c:1542 > err = 0x0 > #11 visit_type_BlockDeviceStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f905a0, name=name@entry=0x7f5d03f032ec "stats", errp=errp@entry=0x7ffddb417ca0) at qapi-visit.c:1566 > err = 0x0 > #12 0x00007f5d03ebf5b1 in visit_type_BlockStats_fields (errp=0x7ffddb417cf0, obj=0x7f5d07f90650, m=0x7f5d061fdea0) at qapi-visit.c:1614 > err = 0x0 > #13 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f90650, name=name@entry=0x7f5d03f480f4 "parent", errp=errp@entry=0x7ffddb417cf0) at qapi-visit.c:1644 > err = 0x0 > #14 0x00007f5d03ebf6bd in visit_type_BlockStats_fields (errp=0x7ffddb417d38, obj=0x7f5d07c67a50, m=0x7f5d061fdea0) at qapi-visit.c:1620 > err = 0x0 > #15 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07c67a50, name=name@entry=0x0, errp=errp@entry=0x7ffddb417d38) at qapi-visit.c:1644 > err = 0x0 > #16 0x00007f5d03ebf760 in visit_type_BlockStatsList (m=0x7f5d061fdea0, obj=obj@entry=0x7ffddb417d98, name=name@entry=0x7f5d03f00e6e "unused", errp=errp@entry=0x7ffddb417da0) at qapi-visit.c:1665 > native_i = <optimized out> > err = 0x0 > i = 0x7f5d07c67a50 > prev = 0x7ffddb417d40 > #17 0x00007f5d03d674dd in qmp_marshal_output_query_blockstats (errp=0x7ffddb417d90, ret_out=0x7ffddb417e10, ret_in=0x7f5d07c67120) at qmp-marshal.c:182 > local_err = 0x0 > mo = 0x7f5d061fdea0 > md = <optimized out> > v = <optimized out> > #18 qmp_marshal_input_query_blockstats (mon=<optimized out>, qdict=<optimized out>, ret=0x7ffddb417e10) at qmp-marshal.c:225 > local_err = 0x0 > args = <optimized out> > retval = <optimized out> > mi = 0x7f5d064e2000 > md = <optimized out> > v = <optimized out> > has_query_nodes = false > query_nodes = false > #19 0x00007f5d03ca0531 in qmp_call_cmd (cmd=<optimized out>, params=0x7f5d075dd600, mon=0x7f5d06208320) at /usr/src/debug/qemu-2.3.0/monitor.c:5051 > ret = <optimized out> > data = 0x0 > #20 handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5113 > err = <optimized out> > obj = <optimized out> > input = <optimized out> > args = 0x7f5d075dd600 > cmd_name = <optimized out> > mon = 0x7f5d06208320 > #21 0x00007f5d03edf4f2 in json_message_process_token (lexer=0x7f5d061f5d70, token=0x7f5d061991e0, type=JSON_OPERATOR, x=48, y=15) at qobject/json-streamer.c:87 > parser = 0x7f5d061f5d68 > dict = 0x7f5d088ea800 > #22 0x00007f5d03ef191f in json_lexer_feed_char (lexer=lexer@entry=0x7f5d061f5d70, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303 > new_state = 100 > #23 0x00007f5d03ef19ee in json_lexer_feed (lexer=0x7f5d061f5d70, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356 > err = <optimized out> > i = <optimized out> > #24 0x00007f5d03edf689 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 > No locals. > #25 0x00007f5d03c9e8cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134 > old_mon = 0x0 > #26 0x00007f5d03d5b1b0 in qemu_chr_be_write (len=<optimized out>, buf=0x7ffddb417f40 "}\177A\333\375\177", s=0x7f5d0625a2e0) at qemu-char.c:305 > No locals. > #27 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f5d0625a2e0) at qemu-char.c:2870 > chr = 0x7f5d0625a2e0 > s = 0x7f5d061aa3f0 > buf = "}\177A\333\375\177\000\000\360\360\355\003]\177\000\000\030\003\000\000\000\000\000\000\205\336\355\003]\177\000\000\000A\036\b]\177\000\000\263\325\355\003]\177\000\000\000A\036\b]\177", '\000' <repeats 18 times>, "`2\036\b]\177\000\000ЀA\333\375\177\000\000\000\000\000\000\000\000\000\000 \232\261\a]\177\000\000\000(\341\006]\177\000\000P\217A\333\375\177\000\000H\242<\374\\\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\240\200A\333\375\177\000\000\340\177A\333\375\177\000\000\r\000\000\000\000\000\000\000 Xz\b]\177\000\000\000@&\006]\177\000\000\340\365\215\b]\177\000\000Q\000\000\000\000\000\000\000\232"... > len = <optimized out> > size = <optimized out> > #28 0x00007f5d020b099a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0 > No symbol table info available. > #29 0x00007f5d03e74288 in glib_pollfds_poll () at main-loop.c:209 > context = 0x7f5d06205140 > pfds = <optimized out> > #30 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254 > ret = 2 > spin_counter = 0 > #31 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503 > ret = 2 > timeout = 4294967295 > timeout_ns = <optimized out> > #32 0x00007f5d03c73a4e in main_loop () at vl.c:1818 > nonblocking = <optimized out> > last_io = 2 > #33 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394 > i = <optimized out> > snapshot = <optimized out> > linux_boot = <optimized out> > initrd_filename = <optimized out> > kernel_filename = <optimized out> > kernel_cmdline = <optimized out> > boot_order = 0x7f5d03f06a67 "cad" > boot_once = 0x0 > cyls = <optimized out> > heads = <optimized out> > secs = <optimized out> > translation = <optimized out> > hda_opts = <optimized out> > opts = <optimized out> > machine_opts = <optimized out> > icount_opts = <optimized out> > olist = <optimized out> > optind = 67 > optarg = 0x7f5d06193570 "rhel6.5.0" > loadvm = <optimized out> > machine_class = <optimized out> > cpu_model = <optimized out> > vga_model = 0x0 > qtest_chrdev = <optimized out> > qtest_log = <optimized out> > pid_file = <optimized out> > incoming = <optimized out> > show_vnc_port = <optimized out> > defconfig = <optimized out> > userconfig = 56 > log_mask = <optimized out> > log_file = <optimized out> > mem_trace = {malloc = 0x7f5d03d61480 <malloc_and_trace>, realloc = 0x7f5d03d61460 <realloc_and_trace>, free = 0x7f5d03d61450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} > trace_events = <optimized out> > trace_file = <optimized out> > maxram_size = <optimized out> > ram_slots = <optimized out> > vmstate_dump_file = <optimized out> > main_loop_err = 0x0 > __func__ = "main" > > > > > -----Ursprüngliche Nachricht----- > Von: qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org [mailto:qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org] Im Auftrag von Grundmann, Christian > Gesendet: Dienstag, 17. November 2015 15:12 > An: 'Dr. David Alan Gilbert' <dgilbert@redhat.com> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc > > Here you go > > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full" > [New LWP 52281] > [New LWP 52288] > [New LWP 52286] > [New LWP 52291] > [New LWP 52292] > [New LWP 52287] > [New LWP 52293] > [New LWP 52290] > [New LWP 56455] > [New LWP 52289] > [New LWP 52282] > [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,us'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 > > Thread 11 (Thread 0x7f6d47719700 (LWP 52282)): > #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 > No locals. > #1 0x00007f6d57426272 in futex_wait (val=4294967295, ev=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals. > #2 qemu_event_wait (ev=ev@entry=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399 > value = <optimized out> > #3 0x00007f6d57434526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233 > tries = 0 > n = <optimized out> > node = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d47719700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d47719700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107326789376, 5884348200482620104, 0, 140107326790080, 140107326789376, 140107592666688, -5804038895876586808, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)): > #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8a4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d44f139e0, reg_save_area = 0x7f6d44f139a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570cf000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8a4000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44f14700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d44f14700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107284825856, 5884348200482620104, 0, 140107284826560, 140107284825856, 140726431086992, -5804033392412867896, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)): > #0 sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101 > No locals. > #1 0x00007f6d574260c7 in qemu_sem_timedwait (sem=sem@entry=0x7f6d5a1b9248, ms=ms@entry=10000) at util/qemu-thread-posix.c:254 > rc = <optimized out> > ts = {tv_sec = 1447709021, tv_nsec = 21985000} > __func__ = "qemu_sem_timedwait" > #2 0x00007f6d573a98ac in worker_thread (opaque=0x7f6d5a1b91e0) at thread-pool.c:92 > req = <optimized out> > ret = <optimized out> > pool = 0x7f6d5a1b91e0 > #3 0x00007f6d55ceadf5 in start_thread (arg=0x7f6ab1dff700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6ab1dff700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140096227505920, 5884348200482620104, 0, 140096227506624, 140096227505920, 26, -5801757560646548792, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #4 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 8 (Thread 0x7f6d44713700 (LWP 52290)): > #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8b8000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d447129e0, reg_save_area = 0x7f6d447129a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570cc000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8b8000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44713700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d44713700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107276433152, 5884348200482620104, 0, 140107276433856, 140107276433152, 140726431086992, -5804032293438111032, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)): > #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8f4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d42f0f9e0, reg_save_area = 0x7f6d42f0f9a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570c3000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8f4000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d42f10700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d42f10700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107251255040, 5884348200482620104, 0, 140107251255744, 140107251255040, 140726431086992, -5804046580109950264, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)): > #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c878000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d45f159e0, reg_save_area = 0x7f6d45f159a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570d5000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c878000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45f16700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d45f16700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107301611264, 5884348200482620104, 0, 140107301611968, 140107301611264, 140726431086992, -5804035590362381624, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 5 (Thread 0x7f6d43711700 (LWP 52292)): > #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8e0000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d437109e0, reg_save_area = 0x7f6d437109a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570c6000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8e0000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43711700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d43711700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107259647744, 5884348200482620104, 0, 140107259648448, 140107259647744, 140726431086992, -5804047687674641720, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)): > #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8cc000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d43f119e0, reg_save_area = 0x7f6d43f119a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570c9000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8cc000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43f12700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d43f12700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107268040448, 5884348200482620104, 0, 140107268041152, 140107268040448, 140726431086992, -5804048786649398584, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 3 (Thread 0x7f6d46717700 (LWP 52286)): > #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c810000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d467169e0, reg_save_area = 0x7f6d467169a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570d8000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c810000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d46717700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d46717700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107310003968, 5884348200482620104, 0, 140107310004672, 140107310003968, 140726431086992, -5804036689337138488, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 2 (Thread 0x7f6d45715700 (LWP 52288)): > #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c890000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d457149e0, reg_save_area = 0x7f6d457149a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570d2000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c890000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45715700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d45715700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107293218560, 5884348200482620104, 0, 140107293219264, 140107293218560, 140726431086992, -5804034491387624760, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)): > #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 No symbol table info available. > #1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=49280) at vl.c:2575 > ptr = 0x7f6d59a346a0 > #2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0 No symbol table info available. > #3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0 No symbol table info available. > #4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33 > req = <optimized out> > #5 virtio_blk_get_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:192 > req = <optimized out> > #6 virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604 > s = 0x7f6d5de1ff40 > __func__ = "virtio_blk_handle_output" > mrb = {reqs = {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7c000, 0x0 <repeats 29 times>}, num_reqs = 3, is_write = true} > #7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=0x7f6d5a1aea00, ret=62, ret@entry=1) at iohandler.c:143 > revents = 1 > #8 0x00007f6d573b7296 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:504 > ret = 1 > timeout = 4294967295 > timeout_ns = <optimized out> > #9 0x00007f6d571b6a4e in main_loop () at vl.c:1818 > nonblocking = <optimized out> > last_io = 0 > #10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394 > i = <optimized out> > snapshot = <optimized out> > linux_boot = <optimized out> > initrd_filename = <optimized out> > kernel_filename = <optimized out> > kernel_cmdline = <optimized out> > boot_order = 0x7f6d57449a67 "cad" > boot_once = 0x0 > cyls = <optimized out> > heads = <optimized out> > secs = <optimized out> > translation = <optimized out> > hda_opts = <optimized out> > opts = <optimized out> > machine_opts = <optimized out> > icount_opts = <optimized out> > olist = <optimized out> > optind = 69 > optarg = 0x7f6d5a14b3a0 "rhel6.5.0" > loadvm = <optimized out> > machine_class = <optimized out> > cpu_model = <optimized out> > vga_model = 0x0 > qtest_chrdev = <optimized out> > qtest_log = <optimized out> > pid_file = <optimized out> > incoming = <optimized out> > show_vnc_port = <optimized out> > defconfig = <optimized out> > userconfig = 179 > log_mask = <optimized out> > log_file = <optimized out> > mem_trace = {malloc = 0x7f6d572a4480 <malloc_and_trace>, realloc = 0x7f6d572a4460 <realloc_and_trace>, free = 0x7f6d572a4450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} > trace_events = <optimized out> > trace_file = <optimized out> > maxram_size = <optimized out> > ram_slots = <optimized out> > vmstate_dump_file = <optimized out> > main_loop_err = 0x0 > __func__ = "main" > > > > > > @ Do you think you're only hitting these crashes on VMs that have been paused because of these space errors? > Will have a look on that > > Thx Christian > > -----Ursprüngliche Nachricht----- > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] > Gesendet: Dienstag, 17. November 2015 12:36 > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com > Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > Hi, > > > > @ Can you please use a 'thread apply all bt full' the full gives a little more info. > > > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full" > > OK, it doesn't relaly give any more without the debuginfo package mentioned below. > > <snip> > > > @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > > Sorry it's a ovirt-node System where I can't you yum > > Ah, although perhaps if you took the core dump, onto another machine with matching qemu and debuginfo you should be able to get more detail. > > > @ Does this part always look the same in your backtraces? > > The most are the same, found one a little bit different : > > Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)): > > #0 0x00007f3785d18353 in > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available. > > #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from > > /lib64/libtcmalloc.so.4 No symbol table info available. > > #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No > > symbol table info available. > > #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No > > symbol table info available. > > #4 0x00007f37885fec89 in g_slice_free1 () from > > /lib64/libglib-2.0.so.0 No symbol table info available. > > #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table > > info available. > > #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info > > available. > > #7 0x00007f378a398394 in aio_bh_poll () No symbol table info > > available. > > #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table info > > available. > > #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info > > available. > > #10 0x00007f37885e299a in g_main_context_dispatch () from > > /lib64/libglib-2.0.so.0 No symbol table info available. > > #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info > > available. > > #12 0x00007f378a1a5a4e in main () > > No symbol table info available. > > > > OK, that's a bit different but interesting.... > > > @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > > No nothing abnormal > > > > @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem? > > Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot. > > Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline. > > OK, that's interesting, because you may be hitting the following bug; http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html > > whose fix coincidentally just got accepted today; it's related to error cases with error=stop which you are using. > > Do you think you're only hitting these crashes on VMs that have been paused because of these space errors? > > > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. > > > > Ok will try that and report > > Thanks, > > Dave > > > > > Thx Christian > > > > > > -----Ursprüngliche Nachricht----- > > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] > > Gesendet: Dienstag, 17. November 2015 10:59 > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > > stefanha@redhat.com > > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in > > libtcmalloc > > > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > > Hi, > > > Dan sent me over to you, > > > please let me know if i can provide additional informations > > > > Hi Christian, > > Thanks for reporting this, > > > > > Softwareversions: > > > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso > > > > > > qemu-img-ev-2.3.0-29.1.el7.x86_64 > > > qemu-kvm-ev-2.3.0-29.1.el7.x86_64 > > > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 > > > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 > > > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch > > > kernel-3.10.0-229.14.1.el7.x86_64 > > > gperftools-libs-2.4-7.el7.x86_64 > > > > > > Commandline: > > > /usr/libexec/qemu-kvm -name myvmname -S -machine > > > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime > > > mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > > > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios > > > type=1,manufacturer=oVirt,product=oVirt > > > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138- > > > 5A > > > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319 > > > -nographic -no-user-config -nodefaults -chardev > > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,se > > > rv er,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc > > > base=2015-11-15T20:04:35,driftfix=slew -global > > > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot > > > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 > > > -device > > > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device > > > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 > > > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= > > > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > > > -drive > > > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84 > > > -8 > > > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffae > > > cf > > > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0, > > > fo > > > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,we > > > rr or=stop,rerror=stop,aio=native -device > > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0, > > > id > > > =virtio-disk0,bootindex=1 -netdev > > > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device > > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci > > > .0 > > > ,addr=0x3 -chardev > > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899- > > > 5a 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait > > > -device > > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=cha > > > nn el0,name=com.redhat.rhevm.vdsm -chardev > > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899- > > > 5a 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait > > > -device > > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=cha > > > nn > > > el1,name=org.qemu.guest_agent.0 -device > > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device > > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on > > > > > > Stack Trace: > > > > > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt" > > > > Can you please use a 'thread apply all bt full' the full gives a little more info. > > Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > > > > > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)): > > > #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4 > > > #1 0x00007fa8b186b489 in malloc_and_trace () > > > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0 > > > #3 0x00007fa8afbd666e in g_slice_alloc () from > > > /lib64/libglib-2.0.so.0 > > > #4 0x00007fa8b17cbffd in virtio_blk_handle_output () > > > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll () > > > #6 0x00007fa8b197e296 in main_loop_wait () > > > #7 0x00007fa8b177da4e in main () > > > > Does this part always look the same in your backtraces? > > The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed. > > > > Some other ideas: > > 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > > 2) Did you hit any IO errors and need to tell the VM to continue after a problem? > > 3) If this is pretty repeatable, then it would be interesting to try changing to a different > > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. > > > > Dave > > > > > > > > > Thx Christian > > > > > > -----Ursprüngliche Nachricht----- > > > Von: Dan Kenigsberg [mailto:danken@redhat.com] > > > Gesendet: Freitag, 13. November 2015 20:00 > > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > > > Cc: 'users@ovirt.org' <users@ovirt.org> > > > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc > > > > > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote: > > > > Hi, > > > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso" > > > > (is there something better to use?) fort he nodes, and have random > > > > crashes of VMs The dumps are always the Same > > > > > > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump > > > > [Thread debugging using libthread_db enabled] Using host > > > > libthread_db library "/lib64/libthread_db.so.1". > > > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'. > > > > Program terminated with signal 11, Segmentation fault. > > > > #0 0x00007f0c559c4353 in > > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache:: > > > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > > > > > > > > > > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist > > > > centos7 or 3.6 > > > > > > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt. > > > > > > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible. > > > > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-11-17 14:42 ` Dr. David Alan Gilbert @ 2015-11-19 16:00 ` Grundmann, Christian 2015-11-19 17:02 ` Paolo Bonzini 2015-11-20 19:06 ` Dr. David Alan Gilbert 0 siblings, 2 replies; 16+ messages in thread From: Grundmann, Christian @ 2015-11-19 16:00 UTC (permalink / raw) To: 'Dr. David Alan Gilbert' Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com Hi, it seems that using virtio-scsi did the trick, But now the VMs are pausing without an coredump, so the underlying Problem (no storage Error) is not fixed, As I am using Snapshots (and so the disks have to grow very fast) I try if tuning "volume_utilization_percent" and "volume_utilization_chunk_mb" will help (https://access.redhat.com/solutions/130843) Thx Christian -----Ursprüngliche Nachricht----- Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] Gesendet: Dienstag, 17. November 2015 15:42 An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > And here another one Oh this is a bit of a different one, from query-blockstats, although again if the heap's corrupted it might have just been the first guy to trip over the corrupt part afterwards. Dave > Thread 1 (Thread 0x7f5d03ba5c00 (LWP 1507)): > #0 0x00007f5cff7e2e7d in > tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from /lib64/libtcmalloc.so.4 No symbol table info available. > #1 0x00007f5cff7e312a in > tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) () from /lib64/libtcmalloc.so.4 No symbol table info available. > #2 0x00007f5cff7e31dd in > tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /lib64/libtcmalloc.so.4 No symbol table info available. > #3 0x00007f5cff7e6235 in > tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /lib64/libtcmalloc.so.4 No symbol table info available. > #4 0x00007f5cff7f4d5b in tc_malloc () from /lib64/libtcmalloc.so.4 No > symbol table info available. > #5 0x00007f5d03d61489 in malloc_and_trace (n_bytes=18) at vl.c:2575 > ptr = 0x1 > #6 0x00007f5d020b647f in g_malloc () from /lib64/libglib-2.0.so.0 No > symbol table info available. > #7 0x00007f5d020cdf7f in g_strdup () from /lib64/libglib-2.0.so.0 No > symbol table info available. > #8 0x00007f5d03eddab5 in alloc_entry (value=0x7f5d088de6c0, key=0x7f5d03f5debb "wr_highest_offset") at qobject/qdict.c:79 > entry = 0x7f5d088df480 > #9 qdict_put_obj (qdict=0x7f5d06e10400, key=0x7f5d03f5debb "wr_highest_offset", value=0x7f5d088de6c0) at qobject/qdict.c:145 > bucket = 81 > entry = <optimized out> > #10 0x00007f5d03ebf34a in visit_type_BlockDeviceStats_fields (errp=0x7ffddb417ca0, obj=0x7f5d07f905a0, m=0x7f5d061fdea0) at qapi-visit.c:1542 > err = 0x0 > #11 visit_type_BlockDeviceStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f905a0, name=name@entry=0x7f5d03f032ec "stats", errp=errp@entry=0x7ffddb417ca0) at qapi-visit.c:1566 > err = 0x0 > #12 0x00007f5d03ebf5b1 in visit_type_BlockStats_fields (errp=0x7ffddb417cf0, obj=0x7f5d07f90650, m=0x7f5d061fdea0) at qapi-visit.c:1614 > err = 0x0 > #13 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f90650, name=name@entry=0x7f5d03f480f4 "parent", errp=errp@entry=0x7ffddb417cf0) at qapi-visit.c:1644 > err = 0x0 > #14 0x00007f5d03ebf6bd in visit_type_BlockStats_fields (errp=0x7ffddb417d38, obj=0x7f5d07c67a50, m=0x7f5d061fdea0) at qapi-visit.c:1620 > err = 0x0 > #15 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07c67a50, name=name@entry=0x0, errp=errp@entry=0x7ffddb417d38) at qapi-visit.c:1644 > err = 0x0 > #16 0x00007f5d03ebf760 in visit_type_BlockStatsList (m=0x7f5d061fdea0, obj=obj@entry=0x7ffddb417d98, name=name@entry=0x7f5d03f00e6e "unused", errp=errp@entry=0x7ffddb417da0) at qapi-visit.c:1665 > native_i = <optimized out> > err = 0x0 > i = 0x7f5d07c67a50 > prev = 0x7ffddb417d40 > #17 0x00007f5d03d674dd in qmp_marshal_output_query_blockstats (errp=0x7ffddb417d90, ret_out=0x7ffddb417e10, ret_in=0x7f5d07c67120) at qmp-marshal.c:182 > local_err = 0x0 > mo = 0x7f5d061fdea0 > md = <optimized out> > v = <optimized out> > #18 qmp_marshal_input_query_blockstats (mon=<optimized out>, qdict=<optimized out>, ret=0x7ffddb417e10) at qmp-marshal.c:225 > local_err = 0x0 > args = <optimized out> > retval = <optimized out> > mi = 0x7f5d064e2000 > md = <optimized out> > v = <optimized out> > has_query_nodes = false > query_nodes = false > #19 0x00007f5d03ca0531 in qmp_call_cmd (cmd=<optimized out>, params=0x7f5d075dd600, mon=0x7f5d06208320) at /usr/src/debug/qemu-2.3.0/monitor.c:5051 > ret = <optimized out> > data = 0x0 > #20 handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5113 > err = <optimized out> > obj = <optimized out> > input = <optimized out> > args = 0x7f5d075dd600 > cmd_name = <optimized out> > mon = 0x7f5d06208320 > #21 0x00007f5d03edf4f2 in json_message_process_token (lexer=0x7f5d061f5d70, token=0x7f5d061991e0, type=JSON_OPERATOR, x=48, y=15) at qobject/json-streamer.c:87 > parser = 0x7f5d061f5d68 > dict = 0x7f5d088ea800 > #22 0x00007f5d03ef191f in json_lexer_feed_char (lexer=lexer@entry=0x7f5d061f5d70, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303 > new_state = 100 > #23 0x00007f5d03ef19ee in json_lexer_feed (lexer=0x7f5d061f5d70, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356 > err = <optimized out> > i = <optimized out> > #24 0x00007f5d03edf689 in json_message_parser_feed (parser=<optimized > out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 No locals. > #25 0x00007f5d03c9e8cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134 > old_mon = 0x0 > #26 0x00007f5d03d5b1b0 in qemu_chr_be_write (len=<optimized out>, > buf=0x7ffddb417f40 "}\177A\333\375\177", s=0x7f5d0625a2e0) at qemu-char.c:305 No locals. > #27 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f5d0625a2e0) at qemu-char.c:2870 > chr = 0x7f5d0625a2e0 > s = 0x7f5d061aa3f0 > buf = "}\177A\333\375\177\000\000\360\360\355\003]\177\000\000\030\003\000\000\000\000\000\000\205\336\355\003]\177\000\000\000A\036\b]\177\000\000\263\325\355\003]\177\000\000\000A\036\b]\177", '\000' <repeats 18 times>, "`2\036\b]\177\000\000ЀA\333\375\177\000\000\000\000\000\000\000\000\000\000 \232\261\a]\177\000\000\000(\341\006]\177\000\000P\217A\333\375\177\000\000H\242<\374\\\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\240\200A\333\375\177\000\000\340\177A\333\375\177\000\000\r\000\000\000\000\000\000\000 Xz\b]\177\000\000\000@&\006]\177\000\000\340\365\215\b]\177\000\000Q\000\000\000\000\000\000\000\232"... > len = <optimized out> > size = <optimized out> > #28 0x00007f5d020b099a in g_main_context_dispatch () from > /lib64/libglib-2.0.so.0 No symbol table info available. > #29 0x00007f5d03e74288 in glib_pollfds_poll () at main-loop.c:209 > context = 0x7f5d06205140 > pfds = <optimized out> > #30 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254 > ret = 2 > spin_counter = 0 > #31 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503 > ret = 2 > timeout = 4294967295 > timeout_ns = <optimized out> > #32 0x00007f5d03c73a4e in main_loop () at vl.c:1818 > nonblocking = <optimized out> > last_io = 2 > #33 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394 > i = <optimized out> > snapshot = <optimized out> > linux_boot = <optimized out> > initrd_filename = <optimized out> > kernel_filename = <optimized out> > kernel_cmdline = <optimized out> > boot_order = 0x7f5d03f06a67 "cad" > boot_once = 0x0 > cyls = <optimized out> > heads = <optimized out> > secs = <optimized out> > translation = <optimized out> > hda_opts = <optimized out> > opts = <optimized out> > machine_opts = <optimized out> > icount_opts = <optimized out> > olist = <optimized out> > optind = 67 > optarg = 0x7f5d06193570 "rhel6.5.0" > loadvm = <optimized out> > machine_class = <optimized out> > cpu_model = <optimized out> > vga_model = 0x0 > qtest_chrdev = <optimized out> > qtest_log = <optimized out> > pid_file = <optimized out> > incoming = <optimized out> > show_vnc_port = <optimized out> > defconfig = <optimized out> > userconfig = 56 > log_mask = <optimized out> > log_file = <optimized out> > mem_trace = {malloc = 0x7f5d03d61480 <malloc_and_trace>, realloc = 0x7f5d03d61460 <realloc_and_trace>, free = 0x7f5d03d61450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} > trace_events = <optimized out> > trace_file = <optimized out> > maxram_size = <optimized out> > ram_slots = <optimized out> > vmstate_dump_file = <optimized out> > main_loop_err = 0x0 > __func__ = "main" > > > > > -----Ursprüngliche Nachricht----- > Von: qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org > [mailto:qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org > ] Im Auftrag von Grundmann, Christian > Gesendet: Dienstag, 17. November 2015 15:12 > An: 'Dr. David Alan Gilbert' <dgilbert@redhat.com> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > stefanha@redhat.com > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in > libtcmalloc > > Here you go > > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full" > [New LWP 52281] > [New LWP 52288] > [New LWP 52286] > [New LWP 52291] > [New LWP 52292] > [New LWP 52287] > [New LWP 52293] > [New LWP 52290] > [New LWP 56455] > [New LWP 52289] > [New LWP 52282] > [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,us'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 > > Thread 11 (Thread 0x7f6d47719700 (LWP 52282)): > #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 > No locals. > #1 0x00007f6d57426272 in futex_wait (val=4294967295, ev=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals. > #2 qemu_event_wait (ev=ev@entry=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399 > value = <optimized out> > #3 0x00007f6d57434526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233 > tries = 0 > n = <optimized out> > node = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d47719700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d47719700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107326789376, 5884348200482620104, 0, 140107326790080, 140107326789376, 140107592666688, -5804038895876586808, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8a4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d44f139e0, reg_save_area = 0x7f6d44f139a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570cf000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8a4000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44f14700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d44f14700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107284825856, 5884348200482620104, 0, 140107284826560, 140107284825856, 140726431086992, -5804033392412867896, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)): > #0 sem_timedwait () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101 > No locals. > #1 0x00007f6d574260c7 in qemu_sem_timedwait (sem=sem@entry=0x7f6d5a1b9248, ms=ms@entry=10000) at util/qemu-thread-posix.c:254 > rc = <optimized out> > ts = {tv_sec = 1447709021, tv_nsec = 21985000} > __func__ = "qemu_sem_timedwait" > #2 0x00007f6d573a98ac in worker_thread (opaque=0x7f6d5a1b91e0) at thread-pool.c:92 > req = <optimized out> > ret = <optimized out> > pool = 0x7f6d5a1b91e0 > #3 0x00007f6d55ceadf5 in start_thread (arg=0x7f6ab1dff700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6ab1dff700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140096227505920, 5884348200482620104, 0, 140096227506624, 140096227505920, 26, -5801757560646548792, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #4 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 8 (Thread 0x7f6d44713700 (LWP 52290)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8b8000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d447129e0, reg_save_area = 0x7f6d447129a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570cc000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8b8000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44713700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d44713700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107276433152, 5884348200482620104, 0, 140107276433856, 140107276433152, 140726431086992, -5804032293438111032, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8f4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d42f0f9e0, reg_save_area = 0x7f6d42f0f9a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570c3000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8f4000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d42f10700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d42f10700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107251255040, 5884348200482620104, 0, 140107251255744, 140107251255040, 140726431086992, -5804046580109950264, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c878000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d45f159e0, reg_save_area = 0x7f6d45f159a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570d5000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c878000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45f16700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d45f16700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107301611264, 5884348200482620104, 0, 140107301611968, 140107301611264, 140726431086992, -5804035590362381624, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 5 (Thread 0x7f6d43711700 (LWP 52292)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8e0000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d437109e0, reg_save_area = 0x7f6d437109a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570c6000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8e0000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43711700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d43711700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107259647744, 5884348200482620104, 0, 140107259648448, 140107259647744, 140726431086992, -5804047687674641720, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8cc000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d43f119e0, reg_save_area = 0x7f6d43f119a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570c9000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c8cc000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43f12700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d43f12700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107268040448, 5884348200482620104, 0, 140107268041152, 140107268040448, 140726431086992, -5804048786649398584, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 3 (Thread 0x7f6d46717700 (LWP 52286)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c810000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d467169e0, reg_save_area = 0x7f6d467169a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570d8000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c810000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d46717700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d46717700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107310003968, 5884348200482620104, 0, 140107310004672, 140107310003968, 140726431086992, -5804036689337138488, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 2 (Thread 0x7f6d45715700 (LWP 52288)): > #0 0x00007f6d4f8ef257 in ioctl () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c890000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > ret = <optimized out> > arg = <optimized out> > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > 0x7f6d457149e0, reg_save_area = 0x7f6d457149a0}} > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > run = 0x7f6d570d2000 > ret = <optimized out> > run_ret = <optimized out> > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > cpu = 0x7f6d5c890000 > r = <optimized out> > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45715700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7f6d45715700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107293218560, 5884348200482620104, 0, 140107293219264, 140107293218560, 140726431086992, -5804034491387624760, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007f6d4f8f81ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)): > #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 No symbol table info available. > #1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=49280) at vl.c:2575 > ptr = 0x7f6d59a346a0 > #2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0 No symbol table info available. > #3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0 No symbol table info available. > #4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33 > req = <optimized out> > #5 virtio_blk_get_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:192 > req = <optimized out> > #6 virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604 > s = 0x7f6d5de1ff40 > __func__ = "virtio_blk_handle_output" > mrb = {reqs = {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7c000, > 0x0 <repeats 29 times>}, num_reqs = 3, is_write = true} > #7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=0x7f6d5a1aea00, ret=62, ret@entry=1) at iohandler.c:143 > revents = 1 > #8 0x00007f6d573b7296 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:504 > ret = 1 > timeout = 4294967295 > timeout_ns = <optimized out> > #9 0x00007f6d571b6a4e in main_loop () at vl.c:1818 > nonblocking = <optimized out> > last_io = 0 > #10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394 > i = <optimized out> > snapshot = <optimized out> > linux_boot = <optimized out> > initrd_filename = <optimized out> > kernel_filename = <optimized out> > kernel_cmdline = <optimized out> > boot_order = 0x7f6d57449a67 "cad" > boot_once = 0x0 > cyls = <optimized out> > heads = <optimized out> > secs = <optimized out> > translation = <optimized out> > hda_opts = <optimized out> > opts = <optimized out> > machine_opts = <optimized out> > icount_opts = <optimized out> > olist = <optimized out> > optind = 69 > optarg = 0x7f6d5a14b3a0 "rhel6.5.0" > loadvm = <optimized out> > machine_class = <optimized out> > cpu_model = <optimized out> > vga_model = 0x0 > qtest_chrdev = <optimized out> > qtest_log = <optimized out> > pid_file = <optimized out> > incoming = <optimized out> > show_vnc_port = <optimized out> > defconfig = <optimized out> > userconfig = 179 > log_mask = <optimized out> > log_file = <optimized out> > mem_trace = {malloc = 0x7f6d572a4480 <malloc_and_trace>, realloc = 0x7f6d572a4460 <realloc_and_trace>, free = 0x7f6d572a4450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} > trace_events = <optimized out> > trace_file = <optimized out> > maxram_size = <optimized out> > ram_slots = <optimized out> > vmstate_dump_file = <optimized out> > main_loop_err = 0x0 > __func__ = "main" > > > > > > @ Do you think you're only hitting these crashes on VMs that have been paused because of these space errors? > Will have a look on that > > Thx Christian > > -----Ursprüngliche Nachricht----- > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] > Gesendet: Dienstag, 17. November 2015 12:36 > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > stefanha@redhat.com > Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in > libtcmalloc > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > Hi, > > > > @ Can you please use a 'thread apply all bt full' the full gives a little more info. > > > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full" > > OK, it doesn't relaly give any more without the debuginfo package mentioned below. > > <snip> > > > @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > > Sorry it's a ovirt-node System where I can't you yum > > Ah, although perhaps if you took the core dump, onto another machine with matching qemu and debuginfo you should be able to get more detail. > > > @ Does this part always look the same in your backtraces? > > The most are the same, found one a little bit different : > > Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)): > > #0 0x00007f3785d18353 in > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available. > > #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from > > /lib64/libtcmalloc.so.4 No symbol table info available. > > #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No > > symbol table info available. > > #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No > > symbol table info available. > > #4 0x00007f37885fec89 in g_slice_free1 () from > > /lib64/libglib-2.0.so.0 No symbol table info available. > > #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table > > info available. > > #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info > > available. > > #7 0x00007f378a398394 in aio_bh_poll () No symbol table info > > available. > > #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table > > info available. > > #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info > > available. > > #10 0x00007f37885e299a in g_main_context_dispatch () from > > /lib64/libglib-2.0.so.0 No symbol table info available. > > #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info > > available. > > #12 0x00007f378a1a5a4e in main () > > No symbol table info available. > > > > OK, that's a bit different but interesting.... > > > @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > > No nothing abnormal > > > > @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem? > > Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot. > > Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline. > > OK, that's interesting, because you may be hitting the following bug; > http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html > > whose fix coincidentally just got accepted today; it's related to error cases with error=stop which you are using. > > Do you think you're only hitting these crashes on VMs that have been paused because of these space errors? > > > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. > > > > Ok will try that and report > > Thanks, > > Dave > > > > > Thx Christian > > > > > > -----Ursprüngliche Nachricht----- > > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] > > Gesendet: Dienstag, 17. November 2015 10:59 > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > > stefanha@redhat.com > > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in > > libtcmalloc > > > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > > Hi, > > > Dan sent me over to you, > > > please let me know if i can provide additional informations > > > > Hi Christian, > > Thanks for reporting this, > > > > > Softwareversions: > > > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso > > > > > > qemu-img-ev-2.3.0-29.1.el7.x86_64 > > > qemu-kvm-ev-2.3.0-29.1.el7.x86_64 > > > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 > > > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 > > > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch > > > kernel-3.10.0-229.14.1.el7.x86_64 > > > gperftools-libs-2.4-7.el7.x86_64 > > > > > > Commandline: > > > /usr/libexec/qemu-kvm -name myvmname -S -machine > > > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime > > > mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > > > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios > > > type=1,manufacturer=oVirt,product=oVirt > > > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-313 > > > 8- > > > 5A > > > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319 > > > -nographic -no-user-config -nodefaults -chardev > > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor, > > > se rv er,nowait -mon chardev=charmonitor,id=monitor,mode=control > > > -rtc base=2015-11-15T20:04:35,driftfix=slew -global > > > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot > > > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 > > > -device > > > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device > > > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x > > > 5 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= > > > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > > > -drive > > > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b > > > 84 > > > -8 > > > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baff > > > ae > > > cf > > > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk > > > 0, > > > fo > > > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none, > > > we rr or=stop,rerror=stop,aio=native -device > > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk > > > 0, > > > id > > > =virtio-disk0,bootindex=1 -netdev > > > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device > > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=p > > > ci > > > .0 > > > ,addr=0x3 -chardev > > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b889 > > > 9- 5a > > > 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait > > > -device > > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=c > > > ha nn el0,name=com.redhat.rhevm.vdsm -chardev > > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b889 > > > 9- 5a > > > 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait > > > -device > > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=c > > > ha > > > nn > > > el1,name=org.qemu.guest_agent.0 -device > > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device > > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg > > > timestamp=on > > > > > > Stack Trace: > > > > > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt" > > > > Can you please use a 'thread apply all bt full' the full gives a little more info. > > Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > > > > > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)): > > > #0 0x00007fa8ad2febe1 in tc_malloc () from > > > /lib64/libtcmalloc.so.4 > > > #1 0x00007fa8b186b489 in malloc_and_trace () > > > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0 > > > #3 0x00007fa8afbd666e in g_slice_alloc () from > > > /lib64/libglib-2.0.so.0 > > > #4 0x00007fa8b17cbffd in virtio_blk_handle_output () > > > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll () > > > #6 0x00007fa8b197e296 in main_loop_wait () > > > #7 0x00007fa8b177da4e in main () > > > > Does this part always look the same in your backtraces? > > The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed. > > > > Some other ideas: > > 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > > 2) Did you hit any IO errors and need to tell the VM to continue after a problem? > > 3) If this is pretty repeatable, then it would be interesting to try changing to a different > > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. > > > > Dave > > > > > > > > > Thx Christian > > > > > > -----Ursprüngliche Nachricht----- > > > Von: Dan Kenigsberg [mailto:danken@redhat.com] > > > Gesendet: Freitag, 13. November 2015 20:00 > > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > > > Cc: 'users@ovirt.org' <users@ovirt.org> > > > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc > > > > > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote: > > > > Hi, > > > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso" > > > > (is there something better to use?) fort he nodes, and have > > > > random crashes of VMs The dumps are always the Same > > > > > > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump > > > > [Thread debugging using libthread_db enabled] Using host > > > > libthread_db library "/lib64/libthread_db.so.1". > > > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'. > > > > Program terminated with signal 11, Segmentation fault. > > > > #0 0x00007f0c559c4353 in > > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache:: > > > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > > > > > > > > > > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist > > > > centos7 or 3.6 > > > > > > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt. > > > > > > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible. > > > > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-11-19 16:00 ` Grundmann, Christian @ 2015-11-19 17:02 ` Paolo Bonzini 2015-12-03 8:18 ` Grundmann, Christian 2015-11-20 19:06 ` Dr. David Alan Gilbert 1 sibling, 1 reply; 16+ messages in thread From: Paolo Bonzini @ 2015-11-19 17:02 UTC (permalink / raw) To: Grundmann, Christian, 'Dr. David Alan Gilbert' Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com On 19/11/2015 17:00, Grundmann, Christian wrote: > Hi, it seems that using virtio-scsi did the trick, But now the VMs > are pausing without an coredump, so the underlying Problem (no > storage Error) is not fixed, As I am using Snapshots (and so the > disks have to grow very fast) I try if tuning > "volume_utilization_percent" and "volume_utilization_chunk_mb" will > help (https://access.redhat.com/solutions/130843) The fix for virtio-blk is probably this patch: http://article.gmane.org/gmane.comp.emulators.qemu.block/6380/raw Paolo ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-11-19 17:02 ` Paolo Bonzini @ 2015-12-03 8:18 ` Grundmann, Christian 2015-12-03 9:04 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 16+ messages in thread From: Grundmann, Christian @ 2015-12-03 8:18 UTC (permalink / raw) To: 'Paolo Bonzini', 'Dr. David Alan Gilbert' Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com Hi again, got a Segfault today without virtio :-( (one IDE Disk and one virtio-scsi) Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine pc-i440fx-rhel7.2.0,accel='. Program terminated with signal 11, Segmentation fault. #0 0x00007fb299cbd3ab in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 Thread 6 (Thread 0x7fb28d0c5700 (LWP 29423)): #0 0x00007fb29cc85ac3 in pread64 () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007fb29e37c2a3 in pread (__offset=<optimized out>, __nbytes=<optimized out>, __buf=0x7fb2a3e21a00, __fd=<optimized out>) at /usr/include/bits/unistd.h:99 No locals. #2 handle_aiocb_rw_linear (aiocb=aiocb@entry=0x7fb2a1474340, buf=buf@entry=0x7fb2a3e21a00 "QF", <incomplete sequence \373>) at block/raw-posix.c:909 offset = 0 len = <optimized out> #3 0x00007fb29e37c3d1 in handle_aiocb_rw (aiocb=0x7fb2a1474340) at block/raw-posix.c:992 nbytes = <optimized out> buf = 0x7fb2a3e21a00 "QF", <incomplete sequence \373> __PRETTY_FUNCTION__ = "handle_aiocb_rw" #4 0x00007fb29e37d945 in aio_worker (arg=0x7fb2a1474340) at block/raw-posix.c:1204 aiocb = 0x7fb2a1474340 ret = 0 #5 0x00007fb29e33d91b in worker_thread (opaque=0x7fb2a148d450) at thread-pool.c:105 req = 0x7fb2a1474b30 ret = <optimized out> pool = 0x7fb2a148d450 #6 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28d0c5700) at pthread_create.c:308 __res = <optimized out> pd = 0x7fb28d0c5700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404847302400, -4522449750849005939, 0, 140404847303104, 140404847302400, 26, 4492373549408278157, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #7 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 5 (Thread 0x7fb104fff700 (LWP 29084)): #0 0x00007fb296881b7d in poll () at ../sysdeps/unix/syscall-template.S:81 No locals. #1 0x00007fb2977d6fe7 in red_worker_main () from /lib64/libspice-server.so.1 No symbol table info available. #2 0x00007fb29cc7edf5 in start_thread (arg=0x7fb104fff700) at pthread_create.c:308 __res = <optimized out> pd = 0x7fb104fff700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140398269822720, -4522449750849005939, 0, 140398269823424, 140398269822720, 140405245697216, 4494326442046740109, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #3 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 4 (Thread 0x7fb28c8c4700 (LWP 29081)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 No locals. #1 0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132 err = <optimized out> __func__ = "qemu_cond_wait" #2 0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912 No locals. #3 qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d2e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949 cpu = 0x7fb2a3d2e000 r = <optimized out> #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c8c4700) at pthread_create.c:308 __res = <optimized out> pd = 0x7fb28c8c4700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404838909696, -4522449750849005939, 0, 140404838910400, 140404838909696, 140735272359936, 4492374652678002317, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 3 (Thread 0x7fb28c0c3700 (LWP 29082)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 No locals. #1 0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132 err = <optimized out> __func__ = "qemu_cond_wait" #2 0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912 No locals. #3 qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d7e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949 cpu = 0x7fb2a3d7e000 r = <optimized out> #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c0c3700) at pthread_create.c:308 __res = <optimized out> pd = 0x7fb28c0c3700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404830516992, -4522449750849005939, 0, 140404830517696, 140404830516992, 140735272359936, 4492375751652759181, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 2 (Thread 0x7fb28e6ad700 (LWP 29077)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 No locals. #1 0x00007fb29e3ba272 in futex_wait (val=4294967295, ev=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals. #2 qemu_event_wait (ev=ev@entry=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399 value = <optimized out> #3 0x00007fb29e3c8526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233 tries = 0 n = <optimized out> node = <optimized out> #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28e6ad700) at pthread_create.c:308 __res = <optimized out> pd = 0x7fb28e6ad700 now = <optimized out> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404870272768, -4522449750849005939, 0, 140404870273472, 140404870272768, 140405136150080, 4492370572995942029, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = <optimized out> pagesize_m1 = <optimized out> sp = <optimized out> freesize = <optimized out> #5 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 No locals. Thread 1 (Thread 0x7fb29e07cc00 (LWP 29076)): #0 0x00007fb299cbd3ab in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available. #1 0x00007fb299cbd47b in tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) () from /lib64/libtcmalloc.so.4 No symbol table info available. #2 0x00007fb299ccc070 in tc_free () from /lib64/libtcmalloc.so.4 No symbol table info available. #3 0x00007fb29c58d58f in g_free () from /lib64/libglib-2.0.so.0 No symbol table info available. #4 0x00007fb29e3b7721 in parser_context_free (ctxt=0x7fb2a531e0c0) at qobject/json-parser.c:358 i = <optimized out> #5 json_parser_parse_err (tokens=<optimized out>, ap=ap@entry=0x0, errp=errp@entry=0x0) at qobject/json-parser.c:710 result = 0x7fb2a4bdf600 #6 0x00007fb29e3b7767 in json_parser_parse (tokens=<optimized out>, ap=ap@entry=0x0) at qobject/json-parser.c:694 No locals. #7 0x00007fb29e176e04 in handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5068 err = <optimized out> obj = <optimized out> input = 0x0 args = 0x0 cmd_name = <optimized out> mon = 0x7fb2a153e140 #8 0x00007fb29e3b64f2 in json_message_process_token (lexer=0x7fb2a1460040, token=0x7fb2a1424880, type=JSON_OPERATOR, x=49, y=104) at qobject/json-streamer.c:87 parser = 0x7fb2a1460038 dict = 0x7fb2a3e27200 #9 0x00007fb29e3c891f in json_lexer_feed_char (lexer=lexer@entry=0x7fb2a1460040, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303 new_state = 100 #10 0x00007fb29e3c89ee in json_lexer_feed (lexer=0x7fb2a1460040, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356 err = <optimized out> i = <optimized out> #11 0x00007fb29e3b6689 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 No locals. #12 0x00007fb29e1758cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134 old_mon = 0x0 #13 0x00007fb29e2321b0 in qemu_chr_be_write (len=<optimized out>, buf=0x7fff7bea8a30 "}\212\352{\377\177", s=0x7fb2a14442e0) at qemu-char.c:305 No locals. #14 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7fb2a14442e0) at qemu-char.c:2870 chr = 0x7fb2a14442e0 s = 0x7fb2a14363f0 buf = "}\212\352{\377\177\000\000\360`;\236\262\177\000\000\030\003\000\000\000\000\000\000\205N;\236\262\177\000\000\240LB\241\262\177\000\000\263E;\236\262\177\000\000\240LB\241\262\177", '\000' <repeats 18 times>, "\360\017c\244\262\177\000\000\300\213\352{\377\177\000\000\000\000\000\000\000\000\000\000\060\356t\245\262\177\000\000\000$ᤲ\177\000\000@\232\352{\377\177\000\000H\022\212\226\262\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\220\213\352{\377\177\000\000Њ\352{\377\177\000\000\r\000\000\000\000\000\000\000\340\234\177\000\000\000d\023\245\262\177\000\000`\376\061\245\262\177\000\000Q\000\000\000\000\000\000\000\325b\004\000\000\000\000\000"... len = <optimized out> size = <optimized out> #15 0x00007fb29c58799a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0 No symbol table info available. #16 0x00007fb29e34b288 in glib_pollfds_poll () at main-loop.c:209 context = 0x7fb2a1491140 pfds = <optimized out> #17 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254 ret = 2 spin_counter = 0 #18 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503 ret = 2 timeout = 4294967295 timeout_ns = <optimized out> #19 0x00007fb29e14aa4e in main_loop () at vl.c:1818 nonblocking = <optimized out> last_io = 2 #20 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394 i = <optimized out> snapshot = <optimized out> linux_boot = <optimized out> initrd_filename = <optimized out> kernel_filename = <optimized out> kernel_cmdline = <optimized out> boot_order = 0x7fb29e3dda67 "cad" boot_once = 0x0 cyls = <optimized out> heads = <optimized out> secs = <optimized out> translation = <optimized out> hda_opts = <optimized out> opts = <optimized out> machine_opts = <optimized out> icount_opts = <optimized out> olist = <optimized out> optind = 78 optarg = 0x7fb2a14ef8c0 "pc-i440fx-rhel7.2.0" loadvm = <optimized out> machine_class = <optimized out> cpu_model = <optimized out> vga_model = 0x0 qtest_chrdev = <optimized out> qtest_log = <optimized out> pid_file = <optimized out> incoming = <optimized out> show_vnc_port = <optimized out> defconfig = <optimized out> userconfig = 111 log_mask = <optimized out> log_file = <optimized out> mem_trace = {malloc = 0x7fb29e238480 <malloc_and_trace>, realloc = 0x7fb29e238460 <realloc_and_trace>, free = 0x7fb29e238450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} trace_events = <optimized out> trace_file = <optimized out> maxram_size = <optimized out> ram_slots = <optimized out> vmstate_dump_file = <optimized out> main_loop_err = 0x0 __func__ = "main" -----Ursprüngliche Nachricht----- Von: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] Im Auftrag von Paolo Bonzini Gesendet: Donnerstag, 19. November 2015 18:02 An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>; 'Dr. David Alan Gilbert' <dgilbert@redhat.com> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com Betreff: Re: WG: [ovirt-users] Segmentation fault in libtcmalloc On 19/11/2015 17:00, Grundmann, Christian wrote: > Hi, it seems that using virtio-scsi did the trick, But now the VMs are > pausing without an coredump, so the underlying Problem (no storage > Error) is not fixed, As I am using Snapshots (and so the disks have to > grow very fast) I try if tuning "volume_utilization_percent" and > "volume_utilization_chunk_mb" will help > (https://access.redhat.com/solutions/130843) The fix for virtio-blk is probably this patch: http://article.gmane.org/gmane.comp.emulators.qemu.block/6380/raw Paolo ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-12-03 8:18 ` Grundmann, Christian @ 2015-12-03 9:04 ` Dr. David Alan Gilbert 2015-12-03 9:07 ` Grundmann, Christian 0 siblings, 1 reply; 16+ messages in thread From: Dr. David Alan Gilbert @ 2015-12-03 9:04 UTC (permalink / raw) To: Grundmann, Christian Cc: 'Paolo Bonzini', 'qemu-devel@nongnu.org', stefanha@redhat.com * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > Hi again, > got a Segfault today without virtio :-( (one IDE Disk and one virtio-scsi) > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine pc-i440fx-rhel7.2.0,accel='. Can you confirm the package version you were using; if you're running the pc-i440fx-rhel7.2.0 machine type it must be pretty new. Dave > Program terminated with signal 11, Segmentation fault. > #0 0x00007fb299cbd3ab in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > > Thread 6 (Thread 0x7fb28d0c5700 (LWP 29423)): > #0 0x00007fb29cc85ac3 in pread64 () at ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007fb29e37c2a3 in pread (__offset=<optimized out>, __nbytes=<optimized out>, __buf=0x7fb2a3e21a00, __fd=<optimized out>) at /usr/include/bits/unistd.h:99 > No locals. > #2 handle_aiocb_rw_linear (aiocb=aiocb@entry=0x7fb2a1474340, buf=buf@entry=0x7fb2a3e21a00 "QF", <incomplete sequence \373>) at block/raw-posix.c:909 > offset = 0 > len = <optimized out> > #3 0x00007fb29e37c3d1 in handle_aiocb_rw (aiocb=0x7fb2a1474340) at block/raw-posix.c:992 > nbytes = <optimized out> > buf = 0x7fb2a3e21a00 "QF", <incomplete sequence \373> > __PRETTY_FUNCTION__ = "handle_aiocb_rw" > #4 0x00007fb29e37d945 in aio_worker (arg=0x7fb2a1474340) at block/raw-posix.c:1204 > aiocb = 0x7fb2a1474340 > ret = 0 > #5 0x00007fb29e33d91b in worker_thread (opaque=0x7fb2a148d450) at thread-pool.c:105 > req = 0x7fb2a1474b30 > ret = <optimized out> > pool = 0x7fb2a148d450 > #6 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28d0c5700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7fb28d0c5700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404847302400, -4522449750849005939, 0, 140404847303104, 140404847302400, 26, 4492373549408278157, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #7 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 5 (Thread 0x7fb104fff700 (LWP 29084)): > #0 0x00007fb296881b7d in poll () at ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007fb2977d6fe7 in red_worker_main () from /lib64/libspice-server.so.1 > No symbol table info available. > #2 0x00007fb29cc7edf5 in start_thread (arg=0x7fb104fff700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7fb104fff700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140398269822720, -4522449750849005939, 0, 140398269823424, 140398269822720, 140405245697216, 4494326442046740109, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #3 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 4 (Thread 0x7fb28c8c4700 (LWP 29081)): > #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > No locals. > #1 0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132 > err = <optimized out> > __func__ = "qemu_cond_wait" > #2 0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912 > No locals. > #3 qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d2e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949 > cpu = 0x7fb2a3d2e000 > r = <optimized out> > #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c8c4700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7fb28c8c4700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404838909696, -4522449750849005939, 0, 140404838910400, 140404838909696, 140735272359936, 4492374652678002317, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 3 (Thread 0x7fb28c0c3700 (LWP 29082)): > #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > No locals. > #1 0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132 > err = <optimized out> > __func__ = "qemu_cond_wait" > #2 0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912 > No locals. > #3 qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d7e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949 > cpu = 0x7fb2a3d7e000 > r = <optimized out> > #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c0c3700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7fb28c0c3700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404830516992, -4522449750849005939, 0, 140404830517696, 140404830516992, 140735272359936, 4492375751652759181, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 2 (Thread 0x7fb28e6ad700 (LWP 29077)): > #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 > No locals. > #1 0x00007fb29e3ba272 in futex_wait (val=4294967295, ev=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 > No locals. > #2 qemu_event_wait (ev=ev@entry=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399 > value = <optimized out> > #3 0x00007fb29e3c8526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233 > tries = 0 > n = <optimized out> > node = <optimized out> > #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28e6ad700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7fb28e6ad700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404870272768, -4522449750849005939, 0, 140404870273472, 140404870272768, 140405136150080, 4492370572995942029, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 1 (Thread 0x7fb29e07cc00 (LWP 29076)): > #0 0x00007fb299cbd3ab in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > No symbol table info available. > #1 0x00007fb299cbd47b in tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) () from /lib64/libtcmalloc.so.4 > No symbol table info available. > #2 0x00007fb299ccc070 in tc_free () from /lib64/libtcmalloc.so.4 > No symbol table info available. > #3 0x00007fb29c58d58f in g_free () from /lib64/libglib-2.0.so.0 > No symbol table info available. > #4 0x00007fb29e3b7721 in parser_context_free (ctxt=0x7fb2a531e0c0) at qobject/json-parser.c:358 > i = <optimized out> > #5 json_parser_parse_err (tokens=<optimized out>, ap=ap@entry=0x0, errp=errp@entry=0x0) at qobject/json-parser.c:710 > result = 0x7fb2a4bdf600 > #6 0x00007fb29e3b7767 in json_parser_parse (tokens=<optimized out>, ap=ap@entry=0x0) at qobject/json-parser.c:694 > No locals. > #7 0x00007fb29e176e04 in handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5068 > err = <optimized out> > obj = <optimized out> > input = 0x0 > args = 0x0 > cmd_name = <optimized out> > mon = 0x7fb2a153e140 > #8 0x00007fb29e3b64f2 in json_message_process_token (lexer=0x7fb2a1460040, token=0x7fb2a1424880, type=JSON_OPERATOR, x=49, y=104) at qobject/json-streamer.c:87 > parser = 0x7fb2a1460038 > dict = 0x7fb2a3e27200 > #9 0x00007fb29e3c891f in json_lexer_feed_char (lexer=lexer@entry=0x7fb2a1460040, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303 > new_state = 100 > #10 0x00007fb29e3c89ee in json_lexer_feed (lexer=0x7fb2a1460040, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356 > err = <optimized out> > i = <optimized out> > #11 0x00007fb29e3b6689 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 > No locals. > #12 0x00007fb29e1758cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134 > old_mon = 0x0 > #13 0x00007fb29e2321b0 in qemu_chr_be_write (len=<optimized out>, buf=0x7fff7bea8a30 "}\212\352{\377\177", s=0x7fb2a14442e0) at qemu-char.c:305 > No locals. > #14 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7fb2a14442e0) at qemu-char.c:2870 > chr = 0x7fb2a14442e0 > s = 0x7fb2a14363f0 > buf = "}\212\352{\377\177\000\000\360`;\236\262\177\000\000\030\003\000\000\000\000\000\000\205N;\236\262\177\000\000\240LB\241\262\177\000\000\263E;\236\262\177\000\000\240LB\241\262\177", '\000' <repeats 18 times>, "\360\017c\244\262\177\000\000\300\213\352{\377\177\000\000\000\000\000\000\000\000\000\000\060\356t\245\262\177\000\000\000$ᤲ\177\000\000@\232\352{\377\177\000\000H\022\212\226\262\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\220\213\352{\377\177\000\000Њ\352{\377\177\000\000\r\000\000\000\000\000\000\000\340\234\177\000\000\000d\023\245\262\177\000\000`\376\061\245\262\177\000\000Q\000\000\000\000\000\000\000\325b\004\000\000\000\000\000"... > len = <optimized out> > size = <optimized out> > #15 0x00007fb29c58799a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0 > No symbol table info available. > #16 0x00007fb29e34b288 in glib_pollfds_poll () at main-loop.c:209 > context = 0x7fb2a1491140 > pfds = <optimized out> > #17 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254 > ret = 2 > spin_counter = 0 > #18 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503 > ret = 2 > timeout = 4294967295 > timeout_ns = <optimized out> > #19 0x00007fb29e14aa4e in main_loop () at vl.c:1818 > nonblocking = <optimized out> > last_io = 2 > #20 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394 > i = <optimized out> > snapshot = <optimized out> > linux_boot = <optimized out> > initrd_filename = <optimized out> > kernel_filename = <optimized out> > kernel_cmdline = <optimized out> > boot_order = 0x7fb29e3dda67 "cad" > boot_once = 0x0 > cyls = <optimized out> > heads = <optimized out> > secs = <optimized out> > translation = <optimized out> > hda_opts = <optimized out> > opts = <optimized out> > machine_opts = <optimized out> > icount_opts = <optimized out> > olist = <optimized out> > optind = 78 > optarg = 0x7fb2a14ef8c0 "pc-i440fx-rhel7.2.0" > loadvm = <optimized out> > machine_class = <optimized out> > cpu_model = <optimized out> > vga_model = 0x0 > qtest_chrdev = <optimized out> > qtest_log = <optimized out> > pid_file = <optimized out> > incoming = <optimized out> > show_vnc_port = <optimized out> > defconfig = <optimized out> > userconfig = 111 > log_mask = <optimized out> > log_file = <optimized out> > mem_trace = {malloc = 0x7fb29e238480 <malloc_and_trace>, realloc = 0x7fb29e238460 <realloc_and_trace>, free = 0x7fb29e238450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} > trace_events = <optimized out> > trace_file = <optimized out> > maxram_size = <optimized out> > ram_slots = <optimized out> > vmstate_dump_file = <optimized out> > main_loop_err = 0x0 > __func__ = "main" > > > > > -----Ursprüngliche Nachricht----- > Von: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] Im Auftrag von Paolo Bonzini > Gesendet: Donnerstag, 19. November 2015 18:02 > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>; 'Dr. David Alan Gilbert' <dgilbert@redhat.com> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com > Betreff: Re: WG: [ovirt-users] Segmentation fault in libtcmalloc > > > > On 19/11/2015 17:00, Grundmann, Christian wrote: > > Hi, it seems that using virtio-scsi did the trick, But now the VMs are > > pausing without an coredump, so the underlying Problem (no storage > > Error) is not fixed, As I am using Snapshots (and so the disks have to > > grow very fast) I try if tuning "volume_utilization_percent" and > > "volume_utilization_chunk_mb" will help > > (https://access.redhat.com/solutions/130843) > > The fix for virtio-blk is probably this patch: > http://article.gmane.org/gmane.comp.emulators.qemu.block/6380/raw > > Paolo -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-12-03 9:04 ` Dr. David Alan Gilbert @ 2015-12-03 9:07 ` Grundmann, Christian 2015-12-10 12:38 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 16+ messages in thread From: Grundmann, Christian @ 2015-12-03 9:07 UTC (permalink / raw) To: 'Dr. David Alan Gilbert' Cc: 'Paolo Bonzini', 'qemu-devel@nongnu.org', stefanha@redhat.com Hi, qemu-img-ev-2.3.0-29.1.el7.x86_64 libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64 qemu-kvm-ev-2.3.0-29.1.el7.x86_64 qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 it seems pc-i440fx-rhel7.2.0 is the default for ovirt 3.6 I tried using only virtio-scsi disk but the VM wont boot (not bootable device) so i used IDE for the boot disk. Thx Christian -----Ursprüngliche Nachricht----- Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] Gesendet: Donnerstag, 03. Dezember 2015 10:04 An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> Cc: 'Paolo Bonzini' <pbonzini@redhat.com>; 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com Betreff: Re: AW: WG: [ovirt-users] Segmentation fault in libtcmalloc * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > Hi again, > got a Segfault today without virtio :-( (one IDE Disk and one > virtio-scsi) > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine pc-i440fx-rhel7.2.0,accel='. Can you confirm the package version you were using; if you're running the pc-i440fx-rhel7.2.0 machine type it must be pretty new. Dave > Program terminated with signal 11, Segmentation fault. > #0 0x00007fb299cbd3ab in > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::Fr > eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > > Thread 6 (Thread 0x7fb28d0c5700 (LWP 29423)): > #0 0x00007fb29cc85ac3 in pread64 () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007fb29e37c2a3 in pread (__offset=<optimized out>, > __nbytes=<optimized out>, __buf=0x7fb2a3e21a00, __fd=<optimized out>) at /usr/include/bits/unistd.h:99 No locals. > #2 handle_aiocb_rw_linear (aiocb=aiocb@entry=0x7fb2a1474340, buf=buf@entry=0x7fb2a3e21a00 "QF", <incomplete sequence \373>) at block/raw-posix.c:909 > offset = 0 > len = <optimized out> > #3 0x00007fb29e37c3d1 in handle_aiocb_rw (aiocb=0x7fb2a1474340) at block/raw-posix.c:992 > nbytes = <optimized out> > buf = 0x7fb2a3e21a00 "QF", <incomplete sequence \373> > __PRETTY_FUNCTION__ = "handle_aiocb_rw" > #4 0x00007fb29e37d945 in aio_worker (arg=0x7fb2a1474340) at block/raw-posix.c:1204 > aiocb = 0x7fb2a1474340 > ret = 0 > #5 0x00007fb29e33d91b in worker_thread (opaque=0x7fb2a148d450) at thread-pool.c:105 > req = 0x7fb2a1474b30 > ret = <optimized out> > pool = 0x7fb2a148d450 > #6 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28d0c5700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7fb28d0c5700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404847302400, -4522449750849005939, 0, 140404847303104, 140404847302400, 26, 4492373549408278157, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #7 0x00007fb29688c1ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 5 (Thread 0x7fb104fff700 (LWP 29084)): > #0 0x00007fb296881b7d in poll () at > ../sysdeps/unix/syscall-template.S:81 > No locals. > #1 0x00007fb2977d6fe7 in red_worker_main () from > /lib64/libspice-server.so.1 No symbol table info available. > #2 0x00007fb29cc7edf5 in start_thread (arg=0x7fb104fff700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7fb104fff700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140398269822720, -4522449750849005939, 0, 140398269823424, 140398269822720, 140405245697216, 4494326442046740109, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #3 0x00007fb29688c1ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 4 (Thread 0x7fb28c8c4700 (LWP 29081)): > #0 pthread_cond_wait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > No locals. > #1 0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132 > err = <optimized out> > __func__ = "qemu_cond_wait" > #2 0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) > at /usr/src/debug/qemu-2.3.0/cpus.c:912 > No locals. > #3 qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d2e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949 > cpu = 0x7fb2a3d2e000 > r = <optimized out> > #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c8c4700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7fb28c8c4700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404838909696, -4522449750849005939, 0, 140404838910400, 140404838909696, 140735272359936, 4492374652678002317, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007fb29688c1ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 3 (Thread 0x7fb28c0c3700 (LWP 29082)): > #0 pthread_cond_wait@@GLIBC_2.3.2 () at > ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > No locals. > #1 0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132 > err = <optimized out> > __func__ = "qemu_cond_wait" > #2 0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) > at /usr/src/debug/qemu-2.3.0/cpus.c:912 > No locals. > #3 qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d7e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949 > cpu = 0x7fb2a3d7e000 > r = <optimized out> > #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c0c3700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7fb28c0c3700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404830516992, -4522449750849005939, 0, 140404830517696, 140404830516992, 140735272359936, 4492375751652759181, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007fb29688c1ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 2 (Thread 0x7fb28e6ad700 (LWP 29077)): > #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 > No locals. > #1 0x00007fb29e3ba272 in futex_wait (val=4294967295, > ev=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals. > #2 qemu_event_wait (ev=ev@entry=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399 > value = <optimized out> > #3 0x00007fb29e3c8526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233 > tries = 0 > n = <optimized out> > node = <optimized out> > #4 0x00007fb29cc7edf5 in start_thread (arg=0x7fb28e6ad700) at pthread_create.c:308 > __res = <optimized out> > pd = 0x7fb28e6ad700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404870272768, -4522449750849005939, 0, 140404870273472, 140404870272768, 140405136150080, 4492370572995942029, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > #5 0x00007fb29688c1ad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > No locals. > > Thread 1 (Thread 0x7fb29e07cc00 (LWP 29076)): > #0 0x00007fb299cbd3ab in > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available. > #1 0x00007fb299cbd47b in > tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) () from /lib64/libtcmalloc.so.4 No symbol table info available. > #2 0x00007fb299ccc070 in tc_free () from /lib64/libtcmalloc.so.4 No > symbol table info available. > #3 0x00007fb29c58d58f in g_free () from /lib64/libglib-2.0.so.0 No > symbol table info available. > #4 0x00007fb29e3b7721 in parser_context_free (ctxt=0x7fb2a531e0c0) at qobject/json-parser.c:358 > i = <optimized out> > #5 json_parser_parse_err (tokens=<optimized out>, ap=ap@entry=0x0, errp=errp@entry=0x0) at qobject/json-parser.c:710 > result = 0x7fb2a4bdf600 > #6 0x00007fb29e3b7767 in json_parser_parse (tokens=<optimized out>, > ap=ap@entry=0x0) at qobject/json-parser.c:694 No locals. > #7 0x00007fb29e176e04 in handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5068 > err = <optimized out> > obj = <optimized out> > input = 0x0 > args = 0x0 > cmd_name = <optimized out> > mon = 0x7fb2a153e140 > #8 0x00007fb29e3b64f2 in json_message_process_token (lexer=0x7fb2a1460040, token=0x7fb2a1424880, type=JSON_OPERATOR, x=49, y=104) at qobject/json-streamer.c:87 > parser = 0x7fb2a1460038 > dict = 0x7fb2a3e27200 > #9 0x00007fb29e3c891f in json_lexer_feed_char (lexer=lexer@entry=0x7fb2a1460040, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303 > new_state = 100 > #10 0x00007fb29e3c89ee in json_lexer_feed (lexer=0x7fb2a1460040, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356 > err = <optimized out> > i = <optimized out> > #11 0x00007fb29e3b6689 in json_message_parser_feed (parser=<optimized > out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 No locals. > #12 0x00007fb29e1758cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134 > old_mon = 0x0 > #13 0x00007fb29e2321b0 in qemu_chr_be_write (len=<optimized out>, > buf=0x7fff7bea8a30 "}\212\352{\377\177", s=0x7fb2a14442e0) at qemu-char.c:305 No locals. > #14 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7fb2a14442e0) at qemu-char.c:2870 > chr = 0x7fb2a14442e0 > s = 0x7fb2a14363f0 > buf = "}\212\352{\377\177\000\000\360`;\236\262\177\000\000\030\003\000\000\000\000\000\000\205N;\236\262\177\000\000\240LB\241\262\177\000\000\263E;\236\262\177\000\000\240LB\241\262\177", '\000' <repeats 18 times>, "\360\017c\244\262\177\000\000\300\213\352{\377\177\000\000\000\000\000\000\000\000\000\000\060\356t\245\262\177\000\000\000$ᤲ\177\000\000@\232\352{\377\177\000\000H\022\212\226\262\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\220\213\352{\377\177\000\000Њ\352{\377\177\000\000\r\000\000\000\000\000\000\000\340\234\177\000\000\000d\023\245\262\177\000\000`\376\061\245\262\177\000\000Q\000\000\000\000\000\000\000\325b\004\000\000\000\000\000"... > len = <optimized out> > size = <optimized out> > #15 0x00007fb29c58799a in g_main_context_dispatch () from > /lib64/libglib-2.0.so.0 No symbol table info available. > #16 0x00007fb29e34b288 in glib_pollfds_poll () at main-loop.c:209 > context = 0x7fb2a1491140 > pfds = <optimized out> > #17 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254 > ret = 2 > spin_counter = 0 > #18 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503 > ret = 2 > timeout = 4294967295 > timeout_ns = <optimized out> > #19 0x00007fb29e14aa4e in main_loop () at vl.c:1818 > nonblocking = <optimized out> > last_io = 2 > #20 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394 > i = <optimized out> > snapshot = <optimized out> > linux_boot = <optimized out> > initrd_filename = <optimized out> > kernel_filename = <optimized out> > kernel_cmdline = <optimized out> > boot_order = 0x7fb29e3dda67 "cad" > boot_once = 0x0 > cyls = <optimized out> > heads = <optimized out> > secs = <optimized out> > translation = <optimized out> > hda_opts = <optimized out> > opts = <optimized out> > machine_opts = <optimized out> > icount_opts = <optimized out> > olist = <optimized out> > optind = 78 > optarg = 0x7fb2a14ef8c0 "pc-i440fx-rhel7.2.0" > loadvm = <optimized out> > machine_class = <optimized out> > cpu_model = <optimized out> > vga_model = 0x0 > qtest_chrdev = <optimized out> > qtest_log = <optimized out> > pid_file = <optimized out> > incoming = <optimized out> > show_vnc_port = <optimized out> > defconfig = <optimized out> > userconfig = 111 > log_mask = <optimized out> > log_file = <optimized out> > mem_trace = {malloc = 0x7fb29e238480 <malloc_and_trace>, realloc = 0x7fb29e238460 <realloc_and_trace>, free = 0x7fb29e238450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} > trace_events = <optimized out> > trace_file = <optimized out> > maxram_size = <optimized out> > ram_slots = <optimized out> > vmstate_dump_file = <optimized out> > main_loop_err = 0x0 > __func__ = "main" > > > > > -----Ursprüngliche Nachricht----- > Von: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] Im Auftrag von > Paolo Bonzini > Gesendet: Donnerstag, 19. November 2015 18:02 > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>; 'Dr. > David Alan Gilbert' <dgilbert@redhat.com> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > stefanha@redhat.com > Betreff: Re: WG: [ovirt-users] Segmentation fault in libtcmalloc > > > > On 19/11/2015 17:00, Grundmann, Christian wrote: > > Hi, it seems that using virtio-scsi did the trick, But now the VMs > > are pausing without an coredump, so the underlying Problem (no > > storage > > Error) is not fixed, As I am using Snapshots (and so the disks have > > to grow very fast) I try if tuning "volume_utilization_percent" and > > "volume_utilization_chunk_mb" will help > > (https://access.redhat.com/solutions/130843) > > The fix for virtio-blk is probably this patch: > http://article.gmane.org/gmane.comp.emulators.qemu.block/6380/raw > > Paolo -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-12-03 9:07 ` Grundmann, Christian @ 2015-12-10 12:38 ` Dr. David Alan Gilbert 2015-12-10 13:18 ` Markus Armbruster 0 siblings, 1 reply; 16+ messages in thread From: Dr. David Alan Gilbert @ 2015-12-10 12:38 UTC (permalink / raw) To: Grundmann, Christian Cc: 'Paolo Bonzini', 'qemu-devel@nongnu.org', stefanha@redhat.com * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > Hi, > > qemu-img-ev-2.3.0-29.1.el7.x86_64 > libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64 > qemu-kvm-ev-2.3.0-29.1.el7.x86_64 > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 > > > it seems pc-i440fx-rhel7.2.0 is the default for ovirt 3.6 > > I tried using only virtio-scsi disk but the VM wont boot (not bootable device) so i used IDE for the boot disk. I think this seg is actually quite different - although it depends where the actual corruption happened - looking at the backtrace again the failing thread wasn't the io thread; it failed in a call from the json parser in the main thread. Dave > a > Thx Christian > > -----Ursprüngliche Nachricht----- > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] > Gesendet: Donnerstag, 03. Dezember 2015 10:04 > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > Cc: 'Paolo Bonzini' <pbonzini@redhat.com>; 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com > Betreff: Re: AW: WG: [ovirt-users] Segmentation fault in libtcmalloc > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > Hi again, > > got a Segfault today without virtio :-( (one IDE Disk and one > > virtio-scsi) > > > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine pc-i440fx-rhel7.2.0,accel='. > > Can you confirm the package version you were using; if you're running the pc-i440fx-rhel7.2.0 machine type it must be pretty new. > > Dave > > > Program terminated with signal 11, Segmentation fault. > > #0 0x00007fb299cbd3ab in > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::Fr > > eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 <deleted> It looks like it's the main thread in the json parser: > > Thread 1 (Thread 0x7fb29e07cc00 (LWP 29076)): > > #0 0x00007fb299cbd3ab in > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available. > > #1 0x00007fb299cbd47b in > > tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) () from /lib64/libtcmalloc.so.4 No symbol table info available. > > #2 0x00007fb299ccc070 in tc_free () from /lib64/libtcmalloc.so.4 No > > symbol table info available. > > #3 0x00007fb29c58d58f in g_free () from /lib64/libglib-2.0.so.0 No > > symbol table info available. > > #4 0x00007fb29e3b7721 in parser_context_free (ctxt=0x7fb2a531e0c0) at qobject/json-parser.c:358 > > i = <optimized out> > > #5 json_parser_parse_err (tokens=<optimized out>, ap=ap@entry=0x0, errp=errp@entry=0x0) at qobject/json-parser.c:710 > > result = 0x7fb2a4bdf600 > > #6 0x00007fb29e3b7767 in json_parser_parse (tokens=<optimized out>, > > ap=ap@entry=0x0) at qobject/json-parser.c:694 No locals. > > #7 0x00007fb29e176e04 in handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5068 > > err = <optimized out> > > obj = <optimized out> > > input = 0x0 > > args = 0x0 > > cmd_name = <optimized out> > > mon = 0x7fb2a153e140 > > #8 0x00007fb29e3b64f2 in json_message_process_token (lexer=0x7fb2a1460040, token=0x7fb2a1424880, type=JSON_OPERATOR, x=49, y=104) at qobject/json-streamer.c:87 > > parser = 0x7fb2a1460038 > > dict = 0x7fb2a3e27200 > > #9 0x00007fb29e3c891f in json_lexer_feed_char (lexer=lexer@entry=0x7fb2a1460040, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303 > > new_state = 100 > > #10 0x00007fb29e3c89ee in json_lexer_feed (lexer=0x7fb2a1460040, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356 > > err = <optimized out> > > i = <optimized out> > > #11 0x00007fb29e3b6689 in json_message_parser_feed (parser=<optimized > > out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 No locals. > > #12 0x00007fb29e1758cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134 > > old_mon = 0x0 > > #13 0x00007fb29e2321b0 in qemu_chr_be_write (len=<optimized out>, > > buf=0x7fff7bea8a30 "}\212\352{\377\177", s=0x7fb2a14442e0) at qemu-char.c:305 No locals. > > #14 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7fb2a14442e0) at qemu-char.c:2870 > > chr = 0x7fb2a14442e0 > > s = 0x7fb2a14363f0 > > buf = "}\212\352{\377\177\000\000\360`;\236\262\177\000\000\030\003\000\000\000\000\000\000\205N;\236\262\177\000\000\240LB\241\262\177\000\000\263E;\236\262\177\000\000\240LB\241\262\177", '\000' <repeats 18 times>, "\360\017c\244\262\177\000\000\300\213\352{\377\177\000\000\000\000\000\000\000\000\000\000\060\356t\245\262\177\000\000\000$ᤲ\177\000\000@\232\352{\377\177\000\000H\022\212\226\262\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\220\213\352{\377\177\000\000Њ\352{\377\177\000\000\r\000\000\000\000\000\000\000\340\234\177\000\000\000d\023\245\262\177\000\000`\376\061\245\262\177\000\000Q\000\000\000\000\000\000\000\325b\004\000\000\000\000\000"... > > len = <optimized out> > > size = <optimized out> > > #15 0x00007fb29c58799a in g_main_context_dispatch () from > > /lib64/libglib-2.0.so.0 No symbol table info available. > > #16 0x00007fb29e34b288 in glib_pollfds_poll () at main-loop.c:209 > > context = 0x7fb2a1491140 > > pfds = <optimized out> > > #17 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254 > > ret = 2 > > spin_counter = 0 > > #18 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503 > > ret = 2 > > timeout = 4294967295 > > timeout_ns = <optimized out> > > #19 0x00007fb29e14aa4e in main_loop () at vl.c:1818 > > nonblocking = <optimized out> > > last_io = 2 > > #20 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394 > > i = <optimized out> > > snapshot = <optimized out> > > linux_boot = <optimized out> > > initrd_filename = <optimized out> > > kernel_filename = <optimized out> > > kernel_cmdline = <optimized out> > > boot_order = 0x7fb29e3dda67 "cad" > > boot_once = 0x0 > > cyls = <optimized out> > > heads = <optimized out> > > secs = <optimized out> > > translation = <optimized out> > > hda_opts = <optimized out> > > opts = <optimized out> > > machine_opts = <optimized out> > > icount_opts = <optimized out> > > olist = <optimized out> > > optind = 78 > > optarg = 0x7fb2a14ef8c0 "pc-i440fx-rhel7.2.0" > > loadvm = <optimized out> > > machine_class = <optimized out> > > cpu_model = <optimized out> > > vga_model = 0x0 > > qtest_chrdev = <optimized out> > > qtest_log = <optimized out> > > pid_file = <optimized out> > > incoming = <optimized out> > > show_vnc_port = <optimized out> > > defconfig = <optimized out> > > userconfig = 111 > > log_mask = <optimized out> > > log_file = <optimized out> > > mem_trace = {malloc = 0x7fb29e238480 <malloc_and_trace>, realloc = 0x7fb29e238460 <realloc_and_trace>, free = 0x7fb29e238450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} > > trace_events = <optimized out> > > trace_file = <optimized out> > > maxram_size = <optimized out> > > ram_slots = <optimized out> > > vmstate_dump_file = <optimized out> > > main_loop_err = 0x0 > > __func__ = "main" > > > > > > > > > > -----Ursprüngliche Nachricht----- > > Von: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] Im Auftrag von > > Paolo Bonzini > > Gesendet: Donnerstag, 19. November 2015 18:02 > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>; 'Dr. > > David Alan Gilbert' <dgilbert@redhat.com> > > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > > stefanha@redhat.com > > Betreff: Re: WG: [ovirt-users] Segmentation fault in libtcmalloc > > > > > > > > On 19/11/2015 17:00, Grundmann, Christian wrote: > > > Hi, it seems that using virtio-scsi did the trick, But now the VMs > > > are pausing without an coredump, so the underlying Problem (no > > > storage > > > Error) is not fixed, As I am using Snapshots (and so the disks have > > > to grow very fast) I try if tuning "volume_utilization_percent" and > > > "volume_utilization_chunk_mb" will help > > > (https://access.redhat.com/solutions/130843) > > > > The fix for virtio-blk is probably this patch: > > http://article.gmane.org/gmane.comp.emulators.qemu.block/6380/raw > > > > Paolo > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-12-10 12:38 ` Dr. David Alan Gilbert @ 2015-12-10 13:18 ` Markus Armbruster 2015-12-10 13:37 ` Grundmann, Christian 0 siblings, 1 reply; 16+ messages in thread From: Markus Armbruster @ 2015-12-10 13:18 UTC (permalink / raw) To: Dr. David Alan Gilbert Cc: 'Paolo Bonzini', Grundmann, Christian, 'qemu-devel@nongnu.org', stefanha@redhat.com "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes: > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: >> Hi, >> >> qemu-img-ev-2.3.0-29.1.el7.x86_64 >> libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64 >> qemu-kvm-ev-2.3.0-29.1.el7.x86_64 >> qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 >> ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch >> qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 >> >> >> it seems pc-i440fx-rhel7.2.0 is the default for ovirt 3.6 >> >> I tried using only virtio-scsi disk but the VM wont boot (not >> bootable device) so i used IDE for the boot disk. > > I think this seg is actually quite different - although it depends > where the actual corruption > happened - looking at the backtrace again the failing thread wasn't > the io thread; it > failed in a call from the json parser in the main thread. Yes, in a free on behalf of parser_context_free() on parser cleanup. Smells like memory corruption. Habe you tried reproducing under valgrind? ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-12-10 13:18 ` Markus Armbruster @ 2015-12-10 13:37 ` Grundmann, Christian 0 siblings, 0 replies; 16+ messages in thread From: Grundmann, Christian @ 2015-12-10 13:37 UTC (permalink / raw) To: Markus Armbruster Cc: Paolo Bonzini, Dr. David Alan Gilbert, stefanha@redhat.com, qemu-devel@nongnu.org Sorry as this is my productionsystem i can't > Am 10.12.2015 um 14:18 schrieb Markus Armbruster <armbru@redhat.com>: > > "Dr. David Alan Gilbert" <dgilbert@redhat.com> writes: > >> * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: >>> Hi, >>> >>> qemu-img-ev-2.3.0-29.1.el7.x86_64 >>> libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64 >>> qemu-kvm-ev-2.3.0-29.1.el7.x86_64 >>> qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 >>> ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch >>> qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 >>> >>> >>> it seems pc-i440fx-rhel7.2.0 is the default for ovirt 3.6 >>> >>> I tried using only virtio-scsi disk but the VM wont boot (not >>> bootable device) so i used IDE for the boot disk. >> >> I think this seg is actually quite different - although it depends >> where the actual corruption >> happened - looking at the backtrace again the failing thread wasn't >> the io thread; it >> failed in a call from the json parser in the main thread. > > Yes, in a free on behalf of parser_context_free() on parser cleanup. > Smells like memory corruption. Habe you tried reproducing under > valgrind? ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc 2015-11-19 16:00 ` Grundmann, Christian 2015-11-19 17:02 ` Paolo Bonzini @ 2015-11-20 19:06 ` Dr. David Alan Gilbert 1 sibling, 0 replies; 16+ messages in thread From: Dr. David Alan Gilbert @ 2015-11-20 19:06 UTC (permalink / raw) To: Grundmann, Christian; +Cc: 'qemu-devel@nongnu.org', stefanha@redhat.com * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > Hi, > it seems that using virtio-scsi did the trick, > But now the VMs are pausing without an coredump, so the underlying Problem (no storage Error) is not fixed, > As I am using Snapshots (and so the disks have to grow very fast) I try if tuning "volume_utilization_percent" and "volume_utilization_chunk_mb" will help (https://access.redhat.com/solutions/130843) I don't know the oVirt stuff of what's supposed to happen with the auto extension stuff at that level. I suggest you ask again on the oVirt side, but if they say QEMU isn't providing the right info/state to them please come right back. Dave > > Thx Christian > > > -----Ursprüngliche Nachricht----- > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] > Gesendet: Dienstag, 17. November 2015 15:42 > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com > Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > And here another one > > Oh this is a bit of a different one, from query-blockstats, although again if the heap's corrupted it might have just been the first guy to trip over the corrupt part afterwards. > > Dave > > > Thread 1 (Thread 0x7f5d03ba5c00 (LWP 1507)): > > #0 0x00007f5cff7e2e7d in > > tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from /lib64/libtcmalloc.so.4 No symbol table info available. > > #1 0x00007f5cff7e312a in > > tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) () from /lib64/libtcmalloc.so.4 No symbol table info available. > > #2 0x00007f5cff7e31dd in > > tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /lib64/libtcmalloc.so.4 No symbol table info available. > > #3 0x00007f5cff7e6235 in > > tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /lib64/libtcmalloc.so.4 No symbol table info available. > > #4 0x00007f5cff7f4d5b in tc_malloc () from /lib64/libtcmalloc.so.4 No > > symbol table info available. > > #5 0x00007f5d03d61489 in malloc_and_trace (n_bytes=18) at vl.c:2575 > > ptr = 0x1 > > #6 0x00007f5d020b647f in g_malloc () from /lib64/libglib-2.0.so.0 No > > symbol table info available. > > #7 0x00007f5d020cdf7f in g_strdup () from /lib64/libglib-2.0.so.0 No > > symbol table info available. > > #8 0x00007f5d03eddab5 in alloc_entry (value=0x7f5d088de6c0, key=0x7f5d03f5debb "wr_highest_offset") at qobject/qdict.c:79 > > entry = 0x7f5d088df480 > > #9 qdict_put_obj (qdict=0x7f5d06e10400, key=0x7f5d03f5debb "wr_highest_offset", value=0x7f5d088de6c0) at qobject/qdict.c:145 > > bucket = 81 > > entry = <optimized out> > > #10 0x00007f5d03ebf34a in visit_type_BlockDeviceStats_fields (errp=0x7ffddb417ca0, obj=0x7f5d07f905a0, m=0x7f5d061fdea0) at qapi-visit.c:1542 > > err = 0x0 > > #11 visit_type_BlockDeviceStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f905a0, name=name@entry=0x7f5d03f032ec "stats", errp=errp@entry=0x7ffddb417ca0) at qapi-visit.c:1566 > > err = 0x0 > > #12 0x00007f5d03ebf5b1 in visit_type_BlockStats_fields (errp=0x7ffddb417cf0, obj=0x7f5d07f90650, m=0x7f5d061fdea0) at qapi-visit.c:1614 > > err = 0x0 > > #13 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f90650, name=name@entry=0x7f5d03f480f4 "parent", errp=errp@entry=0x7ffddb417cf0) at qapi-visit.c:1644 > > err = 0x0 > > #14 0x00007f5d03ebf6bd in visit_type_BlockStats_fields (errp=0x7ffddb417d38, obj=0x7f5d07c67a50, m=0x7f5d061fdea0) at qapi-visit.c:1620 > > err = 0x0 > > #15 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07c67a50, name=name@entry=0x0, errp=errp@entry=0x7ffddb417d38) at qapi-visit.c:1644 > > err = 0x0 > > #16 0x00007f5d03ebf760 in visit_type_BlockStatsList (m=0x7f5d061fdea0, obj=obj@entry=0x7ffddb417d98, name=name@entry=0x7f5d03f00e6e "unused", errp=errp@entry=0x7ffddb417da0) at qapi-visit.c:1665 > > native_i = <optimized out> > > err = 0x0 > > i = 0x7f5d07c67a50 > > prev = 0x7ffddb417d40 > > #17 0x00007f5d03d674dd in qmp_marshal_output_query_blockstats (errp=0x7ffddb417d90, ret_out=0x7ffddb417e10, ret_in=0x7f5d07c67120) at qmp-marshal.c:182 > > local_err = 0x0 > > mo = 0x7f5d061fdea0 > > md = <optimized out> > > v = <optimized out> > > #18 qmp_marshal_input_query_blockstats (mon=<optimized out>, qdict=<optimized out>, ret=0x7ffddb417e10) at qmp-marshal.c:225 > > local_err = 0x0 > > args = <optimized out> > > retval = <optimized out> > > mi = 0x7f5d064e2000 > > md = <optimized out> > > v = <optimized out> > > has_query_nodes = false > > query_nodes = false > > #19 0x00007f5d03ca0531 in qmp_call_cmd (cmd=<optimized out>, params=0x7f5d075dd600, mon=0x7f5d06208320) at /usr/src/debug/qemu-2.3.0/monitor.c:5051 > > ret = <optimized out> > > data = 0x0 > > #20 handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5113 > > err = <optimized out> > > obj = <optimized out> > > input = <optimized out> > > args = 0x7f5d075dd600 > > cmd_name = <optimized out> > > mon = 0x7f5d06208320 > > #21 0x00007f5d03edf4f2 in json_message_process_token (lexer=0x7f5d061f5d70, token=0x7f5d061991e0, type=JSON_OPERATOR, x=48, y=15) at qobject/json-streamer.c:87 > > parser = 0x7f5d061f5d68 > > dict = 0x7f5d088ea800 > > #22 0x00007f5d03ef191f in json_lexer_feed_char (lexer=lexer@entry=0x7f5d061f5d70, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303 > > new_state = 100 > > #23 0x00007f5d03ef19ee in json_lexer_feed (lexer=0x7f5d061f5d70, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356 > > err = <optimized out> > > i = <optimized out> > > #24 0x00007f5d03edf689 in json_message_parser_feed (parser=<optimized > > out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 No locals. > > #25 0x00007f5d03c9e8cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134 > > old_mon = 0x0 > > #26 0x00007f5d03d5b1b0 in qemu_chr_be_write (len=<optimized out>, > > buf=0x7ffddb417f40 "}\177A\333\375\177", s=0x7f5d0625a2e0) at qemu-char.c:305 No locals. > > #27 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f5d0625a2e0) at qemu-char.c:2870 > > chr = 0x7f5d0625a2e0 > > s = 0x7f5d061aa3f0 > > buf = "}\177A\333\375\177\000\000\360\360\355\003]\177\000\000\030\003\000\000\000\000\000\000\205\336\355\003]\177\000\000\000A\036\b]\177\000\000\263\325\355\003]\177\000\000\000A\036\b]\177", '\000' <repeats 18 times>, "`2\036\b]\177\000\000ЀA\333\375\177\000\000\000\000\000\000\000\000\000\000 \232\261\a]\177\000\000\000(\341\006]\177\000\000P\217A\333\375\177\000\000H\242<\374\\\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\240\200A\333\375\177\000\000\340\177A\333\375\177\000\000\r\000\000\000\000\000\000\000 Xz\b]\177\000\000\000@&\006]\177\000\000\340\365\215\b]\177\000\000Q\000\000\000\000\000\000\000\232"... > > len = <optimized out> > > size = <optimized out> > > #28 0x00007f5d020b099a in g_main_context_dispatch () from > > /lib64/libglib-2.0.so.0 No symbol table info available. > > #29 0x00007f5d03e74288 in glib_pollfds_poll () at main-loop.c:209 > > context = 0x7f5d06205140 > > pfds = <optimized out> > > #30 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254 > > ret = 2 > > spin_counter = 0 > > #31 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503 > > ret = 2 > > timeout = 4294967295 > > timeout_ns = <optimized out> > > #32 0x00007f5d03c73a4e in main_loop () at vl.c:1818 > > nonblocking = <optimized out> > > last_io = 2 > > #33 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394 > > i = <optimized out> > > snapshot = <optimized out> > > linux_boot = <optimized out> > > initrd_filename = <optimized out> > > kernel_filename = <optimized out> > > kernel_cmdline = <optimized out> > > boot_order = 0x7f5d03f06a67 "cad" > > boot_once = 0x0 > > cyls = <optimized out> > > heads = <optimized out> > > secs = <optimized out> > > translation = <optimized out> > > hda_opts = <optimized out> > > opts = <optimized out> > > machine_opts = <optimized out> > > icount_opts = <optimized out> > > olist = <optimized out> > > optind = 67 > > optarg = 0x7f5d06193570 "rhel6.5.0" > > loadvm = <optimized out> > > machine_class = <optimized out> > > cpu_model = <optimized out> > > vga_model = 0x0 > > qtest_chrdev = <optimized out> > > qtest_log = <optimized out> > > pid_file = <optimized out> > > incoming = <optimized out> > > show_vnc_port = <optimized out> > > defconfig = <optimized out> > > userconfig = 56 > > log_mask = <optimized out> > > log_file = <optimized out> > > mem_trace = {malloc = 0x7f5d03d61480 <malloc_and_trace>, realloc = 0x7f5d03d61460 <realloc_and_trace>, free = 0x7f5d03d61450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} > > trace_events = <optimized out> > > trace_file = <optimized out> > > maxram_size = <optimized out> > > ram_slots = <optimized out> > > vmstate_dump_file = <optimized out> > > main_loop_err = 0x0 > > __func__ = "main" > > > > > > > > > > -----Ursprüngliche Nachricht----- > > Von: qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org > > [mailto:qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org > > ] Im Auftrag von Grundmann, Christian > > Gesendet: Dienstag, 17. November 2015 15:12 > > An: 'Dr. David Alan Gilbert' <dgilbert@redhat.com> > > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > > stefanha@redhat.com > > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in > > libtcmalloc > > > > Here you go > > > > > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full" > > [New LWP 52281] > > [New LWP 52288] > > [New LWP 52286] > > [New LWP 52291] > > [New LWP 52292] > > [New LWP 52287] > > [New LWP 52293] > > [New LWP 52290] > > [New LWP 56455] > > [New LWP 52289] > > [New LWP 52282] > > [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". > > Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,us'. > > Program terminated with signal 11, Segmentation fault. > > #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 > > > > Thread 11 (Thread 0x7f6d47719700 (LWP 52282)): > > #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 > > No locals. > > #1 0x00007f6d57426272 in futex_wait (val=4294967295, ev=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals. > > #2 qemu_event_wait (ev=ev@entry=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399 > > value = <optimized out> > > #3 0x00007f6d57434526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233 > > tries = 0 > > n = <optimized out> > > node = <optimized out> > > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d47719700) at pthread_create.c:308 > > __res = <optimized out> > > pd = 0x7f6d47719700 > > now = <optimized out> > > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107326789376, 5884348200482620104, 0, 140107326790080, 140107326789376, 140107592666688, -5804038895876586808, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > > not_first_call = <optimized out> > > pagesize_m1 = <optimized out> > > sp = <optimized out> > > freesize = <optimized out> > > #5 0x00007f6d4f8f81ad in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > > > > Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)): > > #0 0x00007f6d4f8ef257 in ioctl () at > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8a4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > > ret = <optimized out> > > arg = <optimized out> > > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > > 0x7f6d44f139e0, reg_save_area = 0x7f6d44f139a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run = 0x7f6d570cf000 > > ret = <optimized out> > > run_ret = <optimized out> > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu = 0x7f6d5c8a4000 > > r = <optimized out> > > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44f14700) at pthread_create.c:308 > > __res = <optimized out> > > pd = 0x7f6d44f14700 > > now = <optimized out> > > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107284825856, 5884348200482620104, 0, 140107284826560, 140107284825856, 140726431086992, -5804033392412867896, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > > not_first_call = <optimized out> > > pagesize_m1 = <optimized out> > > sp = <optimized out> > > freesize = <optimized out> > > #5 0x00007f6d4f8f81ad in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > > > > Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)): > > #0 sem_timedwait () at > > ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101 > > No locals. > > #1 0x00007f6d574260c7 in qemu_sem_timedwait (sem=sem@entry=0x7f6d5a1b9248, ms=ms@entry=10000) at util/qemu-thread-posix.c:254 > > rc = <optimized out> > > ts = {tv_sec = 1447709021, tv_nsec = 21985000} > > __func__ = "qemu_sem_timedwait" > > #2 0x00007f6d573a98ac in worker_thread (opaque=0x7f6d5a1b91e0) at thread-pool.c:92 > > req = <optimized out> > > ret = <optimized out> > > pool = 0x7f6d5a1b91e0 > > #3 0x00007f6d55ceadf5 in start_thread (arg=0x7f6ab1dff700) at pthread_create.c:308 > > __res = <optimized out> > > pd = 0x7f6ab1dff700 > > now = <optimized out> > > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140096227505920, 5884348200482620104, 0, 140096227506624, 140096227505920, 26, -5801757560646548792, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > > not_first_call = <optimized out> > > pagesize_m1 = <optimized out> > > sp = <optimized out> > > freesize = <optimized out> > > #4 0x00007f6d4f8f81ad in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > > > > Thread 8 (Thread 0x7f6d44713700 (LWP 52290)): > > #0 0x00007f6d4f8ef257 in ioctl () at > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8b8000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > > ret = <optimized out> > > arg = <optimized out> > > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > > 0x7f6d447129e0, reg_save_area = 0x7f6d447129a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run = 0x7f6d570cc000 > > ret = <optimized out> > > run_ret = <optimized out> > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu = 0x7f6d5c8b8000 > > r = <optimized out> > > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44713700) at pthread_create.c:308 > > __res = <optimized out> > > pd = 0x7f6d44713700 > > now = <optimized out> > > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107276433152, 5884348200482620104, 0, 140107276433856, 140107276433152, 140726431086992, -5804032293438111032, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > > not_first_call = <optimized out> > > pagesize_m1 = <optimized out> > > sp = <optimized out> > > freesize = <optimized out> > > #5 0x00007f6d4f8f81ad in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > > > > Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)): > > #0 0x00007f6d4f8ef257 in ioctl () at > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8f4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > > ret = <optimized out> > > arg = <optimized out> > > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > > 0x7f6d42f0f9e0, reg_save_area = 0x7f6d42f0f9a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run = 0x7f6d570c3000 > > ret = <optimized out> > > run_ret = <optimized out> > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu = 0x7f6d5c8f4000 > > r = <optimized out> > > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d42f10700) at pthread_create.c:308 > > __res = <optimized out> > > pd = 0x7f6d42f10700 > > now = <optimized out> > > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107251255040, 5884348200482620104, 0, 140107251255744, 140107251255040, 140726431086992, -5804046580109950264, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > > not_first_call = <optimized out> > > pagesize_m1 = <optimized out> > > sp = <optimized out> > > freesize = <optimized out> > > #5 0x00007f6d4f8f81ad in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > > > > Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)): > > #0 0x00007f6d4f8ef257 in ioctl () at > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c878000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > > ret = <optimized out> > > arg = <optimized out> > > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > > 0x7f6d45f159e0, reg_save_area = 0x7f6d45f159a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run = 0x7f6d570d5000 > > ret = <optimized out> > > run_ret = <optimized out> > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu = 0x7f6d5c878000 > > r = <optimized out> > > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45f16700) at pthread_create.c:308 > > __res = <optimized out> > > pd = 0x7f6d45f16700 > > now = <optimized out> > > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107301611264, 5884348200482620104, 0, 140107301611968, 140107301611264, 140726431086992, -5804035590362381624, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > > not_first_call = <optimized out> > > pagesize_m1 = <optimized out> > > sp = <optimized out> > > freesize = <optimized out> > > #5 0x00007f6d4f8f81ad in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > > > > Thread 5 (Thread 0x7f6d43711700 (LWP 52292)): > > #0 0x00007f6d4f8ef257 in ioctl () at > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8e0000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > > ret = <optimized out> > > arg = <optimized out> > > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > > 0x7f6d437109e0, reg_save_area = 0x7f6d437109a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run = 0x7f6d570c6000 > > ret = <optimized out> > > run_ret = <optimized out> > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu = 0x7f6d5c8e0000 > > r = <optimized out> > > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43711700) at pthread_create.c:308 > > __res = <optimized out> > > pd = 0x7f6d43711700 > > now = <optimized out> > > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107259647744, 5884348200482620104, 0, 140107259648448, 140107259647744, 140726431086992, -5804047687674641720, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > > not_first_call = <optimized out> > > pagesize_m1 = <optimized out> > > sp = <optimized out> > > freesize = <optimized out> > > #5 0x00007f6d4f8f81ad in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > > > > Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)): > > #0 0x00007f6d4f8ef257 in ioctl () at > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8cc000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > > ret = <optimized out> > > arg = <optimized out> > > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > > 0x7f6d43f119e0, reg_save_area = 0x7f6d43f119a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run = 0x7f6d570c9000 > > ret = <optimized out> > > run_ret = <optimized out> > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu = 0x7f6d5c8cc000 > > r = <optimized out> > > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43f12700) at pthread_create.c:308 > > __res = <optimized out> > > pd = 0x7f6d43f12700 > > now = <optimized out> > > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107268040448, 5884348200482620104, 0, 140107268041152, 140107268040448, 140726431086992, -5804048786649398584, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > > not_first_call = <optimized out> > > pagesize_m1 = <optimized out> > > sp = <optimized out> > > freesize = <optimized out> > > #5 0x00007f6d4f8f81ad in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > > > > Thread 3 (Thread 0x7f6d46717700 (LWP 52286)): > > #0 0x00007f6d4f8ef257 in ioctl () at > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c810000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > > ret = <optimized out> > > arg = <optimized out> > > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > > 0x7f6d467169e0, reg_save_area = 0x7f6d467169a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run = 0x7f6d570d8000 > > ret = <optimized out> > > run_ret = <optimized out> > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu = 0x7f6d5c810000 > > r = <optimized out> > > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d46717700) at pthread_create.c:308 > > __res = <optimized out> > > pd = 0x7f6d46717700 > > now = <optimized out> > > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107310003968, 5884348200482620104, 0, 140107310004672, 140107310003968, 140726431086992, -5804036689337138488, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > > not_first_call = <optimized out> > > pagesize_m1 = <optimized out> > > sp = <optimized out> > > freesize = <optimized out> > > #5 0x00007f6d4f8f81ad in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > > > > Thread 2 (Thread 0x7f6d45715700 (LWP 52288)): > > #0 0x00007f6d4f8ef257 in ioctl () at > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c890000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969 > > ret = <optimized out> > > arg = <optimized out> > > ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = > > 0x7f6d457149e0, reg_save_area = 0x7f6d457149a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run = 0x7f6d570d2000 > > ret = <optimized out> > > run_ret = <optimized out> > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu = 0x7f6d5c890000 > > r = <optimized out> > > #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45715700) at pthread_create.c:308 > > __res = <optimized out> > > pd = 0x7f6d45715700 > > now = <optimized out> > > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107293218560, 5884348200482620104, 0, 140107293219264, 140107293218560, 140726431086992, -5804034491387624760, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} > > not_first_call = <optimized out> > > pagesize_m1 = <optimized out> > > sp = <optimized out> > > freesize = <optimized out> > > #5 0x00007f6d4f8f81ad in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > > > > Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)): > > #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 No symbol table info available. > > #1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=49280) at vl.c:2575 > > ptr = 0x7f6d59a346a0 > > #2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0 No symbol table info available. > > #3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0 No symbol table info available. > > #4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33 > > req = <optimized out> > > #5 virtio_blk_get_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:192 > > req = <optimized out> > > #6 virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604 > > s = 0x7f6d5de1ff40 > > __func__ = "virtio_blk_handle_output" > > mrb = {reqs = {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7c000, > > 0x0 <repeats 29 times>}, num_reqs = 3, is_write = true} > > #7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=0x7f6d5a1aea00, ret=62, ret@entry=1) at iohandler.c:143 > > revents = 1 > > #8 0x00007f6d573b7296 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:504 > > ret = 1 > > timeout = 4294967295 > > timeout_ns = <optimized out> > > #9 0x00007f6d571b6a4e in main_loop () at vl.c:1818 > > nonblocking = <optimized out> > > last_io = 0 > > #10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394 > > i = <optimized out> > > snapshot = <optimized out> > > linux_boot = <optimized out> > > initrd_filename = <optimized out> > > kernel_filename = <optimized out> > > kernel_cmdline = <optimized out> > > boot_order = 0x7f6d57449a67 "cad" > > boot_once = 0x0 > > cyls = <optimized out> > > heads = <optimized out> > > secs = <optimized out> > > translation = <optimized out> > > hda_opts = <optimized out> > > opts = <optimized out> > > machine_opts = <optimized out> > > icount_opts = <optimized out> > > olist = <optimized out> > > optind = 69 > > optarg = 0x7f6d5a14b3a0 "rhel6.5.0" > > loadvm = <optimized out> > > machine_class = <optimized out> > > cpu_model = <optimized out> > > vga_model = 0x0 > > qtest_chrdev = <optimized out> > > qtest_log = <optimized out> > > pid_file = <optimized out> > > incoming = <optimized out> > > show_vnc_port = <optimized out> > > defconfig = <optimized out> > > userconfig = 179 > > log_mask = <optimized out> > > log_file = <optimized out> > > mem_trace = {malloc = 0x7f6d572a4480 <malloc_and_trace>, realloc = 0x7f6d572a4460 <realloc_and_trace>, free = 0x7f6d572a4450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0} > > trace_events = <optimized out> > > trace_file = <optimized out> > > maxram_size = <optimized out> > > ram_slots = <optimized out> > > vmstate_dump_file = <optimized out> > > main_loop_err = 0x0 > > __func__ = "main" > > > > > > > > > > > > @ Do you think you're only hitting these crashes on VMs that have been paused because of these space errors? > > Will have a look on that > > > > Thx Christian > > > > -----Ursprüngliche Nachricht----- > > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] > > Gesendet: Dienstag, 17. November 2015 12:36 > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > > stefanha@redhat.com > > Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in > > libtcmalloc > > > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > > Hi, > > > > > > @ Can you please use a 'thread apply all bt full' the full gives a little more info. > > > > > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full" > > > > OK, it doesn't relaly give any more without the debuginfo package mentioned below. > > > > <snip> > > > > > @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > > > Sorry it's a ovirt-node System where I can't you yum > > > > Ah, although perhaps if you took the core dump, onto another machine with matching qemu and debuginfo you should be able to get more detail. > > > > > @ Does this part always look the same in your backtraces? > > > The most are the same, found one a little bit different : > > > Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)): > > > #0 0x00007f3785d18353 in > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available. > > > #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from > > > /lib64/libtcmalloc.so.4 No symbol table info available. > > > #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No > > > symbol table info available. > > > #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No > > > symbol table info available. > > > #4 0x00007f37885fec89 in g_slice_free1 () from > > > /lib64/libglib-2.0.so.0 No symbol table info available. > > > #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table > > > info available. > > > #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info > > > available. > > > #7 0x00007f378a398394 in aio_bh_poll () No symbol table info > > > available. > > > #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table > > > info available. > > > #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info > > > available. > > > #10 0x00007f37885e299a in g_main_context_dispatch () from > > > /lib64/libglib-2.0.so.0 No symbol table info available. > > > #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info > > > available. > > > #12 0x00007f378a1a5a4e in main () > > > No symbol table info available. > > > > > > > OK, that's a bit different but interesting.... > > > > > @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > > > No nothing abnormal > > > > > > @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem? > > > Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot. > > > Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline. > > > > OK, that's interesting, because you may be hitting the following bug; > > http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html > > > > whose fix coincidentally just got accepted today; it's related to error cases with error=stop which you are using. > > > > Do you think you're only hitting these crashes on VMs that have been paused because of these space errors? > > > > > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. > > > > > > Ok will try that and report > > > > Thanks, > > > > Dave > > > > > > > > Thx Christian > > > > > > > > > -----Ursprüngliche Nachricht----- > > > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] > > > Gesendet: Dienstag, 17. November 2015 10:59 > > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > > > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; > > > stefanha@redhat.com > > > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in > > > libtcmalloc > > > > > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > > > Hi, > > > > Dan sent me over to you, > > > > please let me know if i can provide additional informations > > > > > > Hi Christian, > > > Thanks for reporting this, > > > > > > > Softwareversions: > > > > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso > > > > > > > > qemu-img-ev-2.3.0-29.1.el7.x86_64 > > > > qemu-kvm-ev-2.3.0-29.1.el7.x86_64 > > > > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 > > > > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 > > > > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch > > > > kernel-3.10.0-229.14.1.el7.x86_64 > > > > gperftools-libs-2.4-7.el7.x86_64 > > > > > > > > Commandline: > > > > /usr/libexec/qemu-kvm -name myvmname -S -machine > > > > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime > > > > mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid > > > > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios > > > > type=1,manufacturer=oVirt,product=oVirt > > > > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-313 > > > > 8- > > > > 5A > > > > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319 > > > > -nographic -no-user-config -nodefaults -chardev > > > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor, > > > > se rv er,nowait -mon chardev=charmonitor,id=monitor,mode=control > > > > -rtc base=2015-11-15T20:04:35,driftfix=slew -global > > > > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot > > > > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 > > > > -device > > > > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device > > > > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x > > > > 5 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= > > > > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 > > > > -drive > > > > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b > > > > 84 > > > > -8 > > > > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baff > > > > ae > > > > cf > > > > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk > > > > 0, > > > > fo > > > > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none, > > > > we rr or=stop,rerror=stop,aio=native -device > > > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk > > > > 0, > > > > id > > > > =virtio-disk0,bootindex=1 -netdev > > > > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device > > > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=p > > > > ci > > > > .0 > > > > ,addr=0x3 -chardev > > > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b889 > > > > 9- 5a > > > > 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait > > > > -device > > > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=c > > > > ha nn el0,name=com.redhat.rhevm.vdsm -chardev > > > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b889 > > > > 9- 5a > > > > 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait > > > > -device > > > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=c > > > > ha > > > > nn > > > > el1,name=org.qemu.guest_agent.0 -device > > > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device > > > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg > > > > timestamp=on > > > > > > > > Stack Trace: > > > > > > > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt" > > > > > > Can you please use a 'thread apply all bt full' the full gives a little more info. > > > Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces. > > > > > > > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)): > > > > #0 0x00007fa8ad2febe1 in tc_malloc () from > > > > /lib64/libtcmalloc.so.4 > > > > #1 0x00007fa8b186b489 in malloc_and_trace () > > > > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0 > > > > #3 0x00007fa8afbd666e in g_slice_alloc () from > > > > /lib64/libglib-2.0.so.0 > > > > #4 0x00007fa8b17cbffd in virtio_blk_handle_output () > > > > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll () > > > > #6 0x00007fa8b197e296 in main_loop_wait () > > > > #7 0x00007fa8b177da4e in main () > > > > > > Does this part always look the same in your backtraces? > > > The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed. > > > > > > Some other ideas: > > > 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? > > > 2) Did you hit any IO errors and need to tell the VM to continue after a problem? > > > 3) If this is pretty repeatable, then it would be interesting to try changing to a different > > > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try. > > > > > > Dave > > > > > > > > > > > > Thx Christian > > > > > > > > -----Ursprüngliche Nachricht----- > > > > Von: Dan Kenigsberg [mailto:danken@redhat.com] > > > > Gesendet: Freitag, 13. November 2015 20:00 > > > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com> > > > > Cc: 'users@ovirt.org' <users@ovirt.org> > > > > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc > > > > > > > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote: > > > > > Hi, > > > > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso" > > > > > (is there something better to use?) fort he nodes, and have > > > > > random crashes of VMs The dumps are always the Same > > > > > > > > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump > > > > > [Thread debugging using libthread_db enabled] Using host > > > > > libthread_db library "/lib64/libthread_db.so.1". > > > > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'. > > > > > Program terminated with signal 11, Segmentation fault. > > > > > #0 0x00007f0c559c4353 in > > > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache:: > > > > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > > > > > > > > > > > > > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist > > > > > centos7 or 3.6 > > > > > > > > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt. > > > > > > > > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible. > > > > > > > -- > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2015-12-10 13:37 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <6A17C71B52524C408E7AAF69103E9E490F14400C@fabamailserver.fabagl.fabasoft.com>
[not found] ` <20151113190014.GB18986@redhat.com>
2015-11-16 8:11 ` [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc Grundmann, Christian
2015-11-17 9:59 ` Dr. David Alan Gilbert
2015-11-17 10:36 ` Grundmann, Christian
2015-11-17 11:36 ` Dr. David Alan Gilbert
2015-11-17 14:11 ` Grundmann, Christian
2015-11-17 14:20 ` Grundmann, Christian
2015-11-17 14:42 ` Dr. David Alan Gilbert
2015-11-19 16:00 ` Grundmann, Christian
2015-11-19 17:02 ` Paolo Bonzini
2015-12-03 8:18 ` Grundmann, Christian
2015-12-03 9:04 ` Dr. David Alan Gilbert
2015-12-03 9:07 ` Grundmann, Christian
2015-12-10 12:38 ` Dr. David Alan Gilbert
2015-12-10 13:18 ` Markus Armbruster
2015-12-10 13:37 ` Grundmann, Christian
2015-11-20 19:06 ` Dr. David Alan Gilbert
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).