From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46392) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zyd2u-00068d-TI for qemu-devel@nongnu.org; Tue, 17 Nov 2015 04:59:30 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Zyd2r-0000Uy-KQ for qemu-devel@nongnu.org; Tue, 17 Nov 2015 04:59:28 -0500 Received: from mx1.redhat.com ([209.132.183.28]:37351) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zyd2r-0000Up-D2 for qemu-devel@nongnu.org; Tue, 17 Nov 2015 04:59:25 -0500 Date: Tue, 17 Nov 2015 09:59:20 +0000 From: "Dr. David Alan Gilbert" Message-ID: <20151117095920.GB2498@work-vm> References: <6A17C71B52524C408E7AAF69103E9E490F14400C@fabamailserver.fabagl.fabasoft.com> <20151113190014.GB18986@redhat.com> <6A17C71B52524C408E7AAF69103E9E490F14E9F4@fabamailserver.fabagl.fabasoft.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <6A17C71B52524C408E7AAF69103E9E490F14E9F4@fabamailserver.fabagl.fabasoft.com> Subject: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Grundmann, Christian" Cc: "'qemu-devel@nongnu.org'" , stefanha@redhat.com * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > Hi, > Dan sent me over to you, > please let me know if i can provide additional informations Hi Christian, Thanks for reporting this, > Softwareversions: > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso >=20 > qemu-img-ev-2.3.0-29.1.el7.x86_64 > qemu-kvm-ev-2.3.0-29.1.el7.x86_64 > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch > kernel-3.10.0-229.14.1.el7.x86_64 > gperftools-libs-2.4-7.el7.x86_64 >=20 > Commandline: > /usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=3Dkvm,us= b=3Doff -cpu Westmere -m 7168 -realtime mlock=3Doff -smp 2,maxcpus=3D16,soc= kets=3D16,cores=3D1,threads=3D1 -uuid 5b6b8899-5a9d-4c07-a6aa-6171527ad319 = -smbios type=3D1,manufacturer=3DoVirt,product=3DoVirt Node,version=3D3.6-0.= 999.201510221942.el7.centos,serial=3D30343536-3138-5A43-4A34-323630303253,u= uid=3D5b6b8899-5a9d-4c07-a6aa-6171527ad319 -nographic -no-user-config -node= faults -chardev socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/myvmna= me.monitor,server,nowait -mon chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcon= trol -rtc base=3D2015-11-15T20:04:35,driftfix=3Dslew -global kvm-pit.lost_t= ick_policy=3Ddiscard -no-hpet -no-shutdown -boot strict=3Don -device piix3-= usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2 -device virtio-scsi-pci,id=3Ds= csi0,bus=3Dpci.0,addr=3D0x4 -device virtio-serial-pci,id=3Dvirtio-serial0,m= ax_ports=3D16,bus=3Dpci.0,addr=3D0x5 -drive if=3Dnone,id=3Ddrive-ide0-1-0,r= eadonly=3Don,format=3Draw,serial=3D -device ide-cd,bus=3Dide.1,unit=3D0,dri= ve=3Ddrive-ide0-1-0,id=3Dide0-1-0 -drive file=3D/rhev/data-center/00000002-= 0002-0002-0002-0000000000e2/5df61b84-8746-4460-b148-65cc0eb8d29c/images/820= 2b81d-6191-495f-8c9d-7d90baffaecf/d7665e07-1786-4051-aa26-0a3e1c9d2574,if= =3Dnone,id=3Ddrive-virtio-disk0,format=3Dqcow2,serial=3D8202b81d-6191-495f-= 8c9d-7d90baffaecf,cache=3Dnone,werror=3Dstop,rerror=3Dstop,aio=3Dnative -de= vice virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x6,drive=3Ddrive-virtio-= disk0,id=3Dvirtio-disk0,bootindex=3D1 -netdev tap,fd=3D39,id=3Dhostnet0,vho= st=3Don,vhostfd=3D65 -device virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac= =3D52:54:00:83:a2:0e,bus=3Dpci.0,addr=3D0x3 -chardev socket,id=3Dcharchanne= l0,path=3D/var/lib/libvirt/qemu/channels/5b6b8899-5a9d-4c07-a6aa-6171527ad3= 19.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=3Dvirtio-= serial0.0,nr=3D1,chardev=3Dcharchannel0,id=3Dchannel0,name=3Dcom.redhat.rhe= vm.vdsm -chardev socket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/chan= nels/5b6b8899-5a9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,now= ait -device virtserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dcharchan= nel1,id=3Dchannel1,name=3Dorg.qemu.guest_agent.0 -device cirrus-vga,id=3Dvi= deo0,bus=3Dpci.0,addr=3D0x2 -device virtio-balloon-pci,id=3Dballoon0,bus=3D= pci.0,addr=3D0x7 -msg timestamp=3Don >=20 > Stack Trace: >=20 > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pag= ination off" -ex "thread apply all bt" Can you please use a 'thread apply all bt full' the full gives a little m= ore info. Also, if you've not already got it installed can you please install the deb= uginfo package for qemu, it gives a lot more information in backtraces. > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)): > #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4 > #1 0x00007fa8b186b489 in malloc_and_trace () > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0 > #3 0x00007fa8afbd666e in g_slice_alloc () from /lib64/libglib-2.0.so.0 > #4 0x00007fa8b17cbffd in virtio_blk_handle_output () > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll () > #6 0x00007fa8b197e296 in main_loop_wait () > #7 0x00007fa8b177da4e in main () Does this part always look the same in your backtraces? The segfault in tc_malloc is probably due to a heap corruption, or double f= ree or similar - although it can be a bit tricky to find out what did it, since the corrupti= on might have happened a bit before the place it crashed. Some other ideas: 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ? 2) Did you hit any IO errors and need to tell the VM to continue after a = problem? 3) If this is pretty repeatable, then it would be interesting to try chan= ging to a different disk emulation and see if the problem goes away - e.g. virtio-scsi wou= ld be a good one to try. Dave >=20 >=20 > Thx Christian >=20 > -----Urspr=FCngliche Nachricht----- > Von: Dan Kenigsberg [mailto:danken@redhat.com]=20 > Gesendet: Freitag, 13. November 2015 20:00 > An: Grundmann, Christian > Cc: 'users@ovirt.org' > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc >=20 > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote: > > Hi, > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso" (is= =20 > > there something better to use?) fort he nodes, and have random crashes= =20 > > of VMs The dumps are always the Same > >=20 > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump [Thread=20 > > debugging using libthread_db enabled] Using host libthread_db library= =20 > > "/lib64/libthread_db.so.1". > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine r= hel6.5.0,accel=3Dkvm,usb=3Do'. > > Program terminated with signal 11, Segmentation fault. > > #0 0x00007f0c559c4353 in=20 > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::Fr > > eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > >=20 > >=20 > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist centos7= =20 > > or 3.6 >=20 > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/= libtcmalloc malloc bug, and not directly related to ovirt. >=20 > Please report the precise version of qemu,kernel,libvirt and gperftools-l= ibs to qemu-devel mailing list and the complete stack trace and qemu comman= d line, if possible. >=20 -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK