From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41453) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zzr0d-0003fZ-Lv for qemu-devel@nongnu.org; Fri, 20 Nov 2015 14:06:16 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Zzr0Z-0000se-61 for qemu-devel@nongnu.org; Fri, 20 Nov 2015 14:06:11 -0500 Received: from mx1.redhat.com ([209.132.183.28]:37423) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zzr0Y-0000sU-Pv for qemu-devel@nongnu.org; Fri, 20 Nov 2015 14:06:07 -0500 Date: Fri, 20 Nov 2015 19:06:00 +0000 From: "Dr. David Alan Gilbert" Message-ID: <20151120190559.GB2517@work-vm> References: <6A17C71B52524C408E7AAF69103E9E490F14400C@fabamailserver.fabagl.fabasoft.com> <20151113190014.GB18986@redhat.com> <6A17C71B52524C408E7AAF69103E9E490F14E9F4@fabamailserver.fabagl.fabasoft.com> <20151117095920.GB2498@work-vm> <6A17C71B52524C408E7AAF69103E9E490F153F45@fabamailserver.fabagl.fabasoft.com> <20151117113601.GD2498@work-vm> <6A17C71B52524C408E7AAF69103E9E490F15520E@fabamailserver.fabagl.fabasoft.com> <6A17C71B52524C408E7AAF69103E9E490F1552E7@fabamailserver.fabagl.fabasoft.com> <20151117144225.GH2498@work-vm> <6A17C71B52524C408E7AAF69103E9E490F15ECE9@fabamailserver.fabagl.fabasoft.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <6A17C71B52524C408E7AAF69103E9E490F15ECE9@fabamailserver.fabagl.fabasoft.com> Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Grundmann, Christian" Cc: "'qemu-devel@nongnu.org'" , "stefanha@redhat.com" * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > Hi, > it seems that using virtio-scsi did the trick, > But now the VMs are pausing without an coredump, so the underlying Prob= lem (no storage Error) is not fixed,=20 > As I am using Snapshots (and so the disks have to grow very fast) I try= if tuning "volume_utilization_percent" and "volume_utilization_chunk_mb"= will help (https://access.redhat.com/solutions/130843) I don't know the oVirt stuff of what's supposed to happen with the auto e= xtension stuff at that level. I suggest you ask again on the oVirt side, but if t= hey say QEMU isn't providing the right info/state to them please come right b= ack. Dave >=20 > Thx Christian >=20 >=20 > -----Urspr=C3=BCngliche Nachricht----- > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]=20 > Gesendet: Dienstag, 17. November 2015 15:42 > An: Grundmann, Christian > Cc: 'qemu-devel@nongnu.org' ; stefanha@redhat.co= m > Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in l= ibtcmalloc >=20 > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > And here another one >=20 > Oh this is a bit of a different one, from query-blockstats, although ag= ain if the heap's corrupted it might have just been the first guy to trip= over the corrupt part afterwards. >=20 > Dave >=20 > > Thread 1 (Thread 0x7f5d03ba5c00 (LWP 1507)): > > #0 0x00007f5cff7e2e7d in=20 > > tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () = from /lib64/libtcmalloc.so.4 No symbol table info available. > > #1 0x00007f5cff7e312a in=20 > > tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)= () from /lib64/libtcmalloc.so.4 No symbol table info available. > > #2 0x00007f5cff7e31dd in=20 > > tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /= lib64/libtcmalloc.so.4 No symbol table info available. > > #3 0x00007f5cff7e6235 in=20 > > tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned = long) () from /lib64/libtcmalloc.so.4 No symbol table info available. > > #4 0x00007f5cff7f4d5b in tc_malloc () from /lib64/libtcmalloc.so.4 N= o=20 > > symbol table info available. > > #5 0x00007f5d03d61489 in malloc_and_trace (n_bytes=3D18) at vl.c:257= 5 > > ptr =3D 0x1 > > #6 0x00007f5d020b647f in g_malloc () from /lib64/libglib-2.0.so.0 No= =20 > > symbol table info available. > > #7 0x00007f5d020cdf7f in g_strdup () from /lib64/libglib-2.0.so.0 No= =20 > > symbol table info available. > > #8 0x00007f5d03eddab5 in alloc_entry (value=3D0x7f5d088de6c0, key=3D= 0x7f5d03f5debb "wr_highest_offset") at qobject/qdict.c:79 > > entry =3D 0x7f5d088df480 > > #9 qdict_put_obj (qdict=3D0x7f5d06e10400, key=3D0x7f5d03f5debb "wr_h= ighest_offset", value=3D0x7f5d088de6c0) at qobject/qdict.c:145 > > bucket =3D 81 > > entry =3D > > #10 0x00007f5d03ebf34a in visit_type_BlockDeviceStats_fields (errp=3D= 0x7ffddb417ca0, obj=3D0x7f5d07f905a0, m=3D0x7f5d061fdea0) at qapi-visit.c= :1542 > > err =3D 0x0 > > #11 visit_type_BlockDeviceStats (m=3Dm@entry=3D0x7f5d061fdea0, obj=3D= 0x7f5d07f905a0, name=3Dname@entry=3D0x7f5d03f032ec "stats", errp=3Derrp@e= ntry=3D0x7ffddb417ca0) at qapi-visit.c:1566 > > err =3D 0x0 > > #12 0x00007f5d03ebf5b1 in visit_type_BlockStats_fields (errp=3D0x7ffd= db417cf0, obj=3D0x7f5d07f90650, m=3D0x7f5d061fdea0) at qapi-visit.c:1614 > > err =3D 0x0 > > #13 visit_type_BlockStats (m=3Dm@entry=3D0x7f5d061fdea0, obj=3D0x7f5d= 07f90650, name=3Dname@entry=3D0x7f5d03f480f4 "parent", errp=3Derrp@entry=3D= 0x7ffddb417cf0) at qapi-visit.c:1644 > > err =3D 0x0 > > #14 0x00007f5d03ebf6bd in visit_type_BlockStats_fields (errp=3D0x7ffd= db417d38, obj=3D0x7f5d07c67a50, m=3D0x7f5d061fdea0) at qapi-visit.c:1620 > > err =3D 0x0 > > #15 visit_type_BlockStats (m=3Dm@entry=3D0x7f5d061fdea0, obj=3D0x7f5d= 07c67a50, name=3Dname@entry=3D0x0, errp=3Derrp@entry=3D0x7ffddb417d38) at= qapi-visit.c:1644 > > err =3D 0x0 > > #16 0x00007f5d03ebf760 in visit_type_BlockStatsList (m=3D0x7f5d061fde= a0, obj=3Dobj@entry=3D0x7ffddb417d98, name=3Dname@entry=3D0x7f5d03f00e6e = "unused", errp=3Derrp@entry=3D0x7ffddb417da0) at qapi-visit.c:1665 > > native_i =3D > > err =3D 0x0 > > i =3D 0x7f5d07c67a50 > > prev =3D 0x7ffddb417d40 > > #17 0x00007f5d03d674dd in qmp_marshal_output_query_blockstats (errp=3D= 0x7ffddb417d90, ret_out=3D0x7ffddb417e10, ret_in=3D0x7f5d07c67120) at qmp= -marshal.c:182 > > local_err =3D 0x0 > > mo =3D 0x7f5d061fdea0 > > md =3D > > v =3D > > #18 qmp_marshal_input_query_blockstats (mon=3D, qdict=3D= , ret=3D0x7ffddb417e10) at qmp-marshal.c:225 > > local_err =3D 0x0 > > args =3D > > retval =3D > > mi =3D 0x7f5d064e2000 > > md =3D > > v =3D > > has_query_nodes =3D false > > query_nodes =3D false > > #19 0x00007f5d03ca0531 in qmp_call_cmd (cmd=3D, params= =3D0x7f5d075dd600, mon=3D0x7f5d06208320) at /usr/src/debug/qemu-2.3.0/mon= itor.c:5051 > > ret =3D > > data =3D 0x0 > > #20 handle_qmp_command (parser=3D, tokens=3D) at /usr/src/debug/qemu-2.3.0/monitor.c:5113 > > err =3D > > obj =3D > > input =3D > > args =3D 0x7f5d075dd600 > > cmd_name =3D > > mon =3D 0x7f5d06208320 > > #21 0x00007f5d03edf4f2 in json_message_process_token (lexer=3D0x7f5d0= 61f5d70, token=3D0x7f5d061991e0, type=3DJSON_OPERATOR, x=3D48, y=3D15) at= qobject/json-streamer.c:87 > > parser =3D 0x7f5d061f5d68 > > dict =3D 0x7f5d088ea800 > > #22 0x00007f5d03ef191f in json_lexer_feed_char (lexer=3Dlexer@entry=3D= 0x7f5d061f5d70, ch=3D, flush=3Dflush@entry=3Dfalse) at qob= ject/json-lexer.c:303 > > new_state =3D 100 > > #23 0x00007f5d03ef19ee in json_lexer_feed (lexer=3D0x7f5d061f5d70, bu= ffer=3D, size=3D) at qobject/json-lexer.c:3= 56 > > err =3D > > i =3D > > #24 0x00007f5d03edf689 in json_message_parser_feed (parser=3D > out>, buffer=3D, size=3D) at qobject/js= on-streamer.c:110 No locals. > > #25 0x00007f5d03c9e8cf in monitor_control_read (opaque=3D, buf=3D, size=3D) at /usr/src/debug/qem= u-2.3.0/monitor.c:5134 > > old_mon =3D 0x0 > > #26 0x00007f5d03d5b1b0 in qemu_chr_be_write (len=3D,=20 > > buf=3D0x7ffddb417f40 "}\177A\333\375\177", s=3D0x7f5d0625a2e0) at qem= u-char.c:305 No locals. > > #27 tcp_chr_read (chan=3D, cond=3D, opa= que=3D0x7f5d0625a2e0) at qemu-char.c:2870 > > chr =3D 0x7f5d0625a2e0 > > s =3D 0x7f5d061aa3f0 > > buf =3D "}\177A\333\375\177\000\000\360\360\355\003]\177\000\= 000\030\003\000\000\000\000\000\000\205\336\355\003]\177\000\000\000A\036= \b]\177\000\000\263\325\355\003]\177\000\000\000A\036\b]\177", '\000' , "`2\036\b]\177\000\000=D0=80A\333\375\177\000\000\000\00= 0\000\000\000\000\000\000 \232\261\a]\177\000\000\000(\341\006]\177\000\0= 00P\217A\333\375\177\000\000H\242<\374\\\177\000\000]\000\000\000\000\000= \000\000\060\000\000\000\060\000\000\000\240\200A\333\375\177\000\000\340= \177A\333\375\177\000\000\r\000\000\000\000\000\000\000 Xz\b]\177\000\000= \000@&\006]\177\000\000\340\365\215\b]\177\000\000Q\000\000\000\000\000\0= 00\000\232"... > > len =3D > > size =3D > > #28 0x00007f5d020b099a in g_main_context_dispatch () from=20 > > /lib64/libglib-2.0.so.0 No symbol table info available. > > #29 0x00007f5d03e74288 in glib_pollfds_poll () at main-loop.c:209 > > context =3D 0x7f5d06205140 > > pfds =3D > > #30 os_host_main_loop_wait (timeout=3D) at main-loop.c= :254 > > ret =3D 2 > > spin_counter =3D 0 > > #31 main_loop_wait (nonblocking=3D) at main-loop.c:503 > > ret =3D 2 > > timeout =3D 4294967295 > > timeout_ns =3D > > #32 0x00007f5d03c73a4e in main_loop () at vl.c:1818 > > nonblocking =3D > > last_io =3D 2 > > #33 main (argc=3D, argv=3D, envp=3D) at vl.c:4394 > > i =3D > > snapshot =3D > > linux_boot =3D > > initrd_filename =3D > > kernel_filename =3D > > kernel_cmdline =3D > > boot_order =3D 0x7f5d03f06a67 "cad" > > boot_once =3D 0x0 > > cyls =3D > > heads =3D > > secs =3D > > translation =3D > > hda_opts =3D > > opts =3D > > machine_opts =3D > > icount_opts =3D > > olist =3D > > optind =3D 67 > > optarg =3D 0x7f5d06193570 "rhel6.5.0" > > loadvm =3D > > machine_class =3D > > cpu_model =3D > > vga_model =3D 0x0 > > qtest_chrdev =3D > > qtest_log =3D > > pid_file =3D > > incoming =3D > > show_vnc_port =3D > > defconfig =3D > > userconfig =3D 56 > > log_mask =3D > > log_file =3D > > mem_trace =3D {malloc =3D 0x7f5d03d61480 , = realloc =3D 0x7f5d03d61460 , free =3D 0x7f5d03d61450 <= free_and_trace>, calloc =3D 0x0, try_malloc =3D 0x0, try_realloc =3D 0x0} > > trace_events =3D > > trace_file =3D > > maxram_size =3D > > ram_slots =3D > > vmstate_dump_file =3D > > main_loop_err =3D 0x0 > > __func__ =3D "main" > >=20 > >=20 > >=20 > >=20 > > -----Urspr=C3=BCngliche Nachricht----- > > Von: qemu-devel-bounces+christian.grundmann=3Dfabasoft.com@nongnu.org= =20 > > [mailto:qemu-devel-bounces+christian.grundmann=3Dfabasoft.com@nongnu.= org > > ] Im Auftrag von Grundmann, Christian > > Gesendet: Dienstag, 17. November 2015 15:12 > > An: 'Dr. David Alan Gilbert' > > Cc: 'qemu-devel@nongnu.org' ;=20 > > stefanha@redhat.com > > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in=20 > > libtcmalloc > >=20 > > Here you go > >=20 > >=20 > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set= pagination off" -ex "thread apply all bt full" > > [New LWP 52281] > > [New LWP 52288] > > [New LWP 52286] > > [New LWP 52291] > > [New LWP 52292] > > [New LWP 52287] > > [New LWP 52293] > > [New LWP 52290] > > [New LWP 56455] > > [New LWP 52289] > > [New LWP 52282] > > [Thread debugging using libthread_db enabled] Using host libthread_db= library "/lib64/libthread_db.so.1". > > Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machi= ne rhel6.5.0,accel=3Dkvm,us'. > > Program terminated with signal 11, Segmentation fault. > > #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 > >=20 > > Thread 11 (Thread 0x7f6d47719700 (LWP 52282)): > > #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 > > No locals. > > #1 0x00007f6d57426272 in futex_wait (val=3D4294967295, ev=3D0x7f6d57= cf0f44 ) at util/qemu-thread-posix.c:301 No locals. > > #2 qemu_event_wait (ev=3Dev@entry=3D0x7f6d57cf0f44 ) at util/qemu-thread-posix.c:399 > > value =3D > > #3 0x00007f6d57434526 in call_rcu_thread (opaque=3D) = at util/rcu.c:233 > > tries =3D 0 > > n =3D > > node =3D > > #4 0x00007f6d55ceadf5 in start_thread (arg=3D0x7f6d47719700) at pthr= ead_create.c:308 > > __res =3D > > pd =3D 0x7f6d47719700 > > now =3D > > unwind_buf =3D {cancel_jmp_buf =3D {{jmp_buf =3D {14010732678= 9376, 5884348200482620104, 0, 140107326790080, 140107326789376, 140107592= 666688, -5804038895876586808, -5804071064002379064}, mask_was_saved =3D 0= }}, priv =3D {pad =3D {0x0, 0x0, 0x0, 0x0}, data =3D {prev =3D 0x0, clean= up =3D 0x0, canceltype =3D 0}}} > > not_first_call =3D > > pagesize_m1 =3D > > sp =3D > > freesize =3D > > #5 0x00007f6d4f8f81ad in clone () at=20 > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > >=20 > > Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)): > > #0 0x00007f6d4f8ef257 in ioctl () at=20 > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=3Dcpu@entry=3D0x7f6d5c8= a4000, type=3Dtype@entry=3D44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:= 1969 > > ret =3D > > arg =3D > > ap =3D {{gp_offset =3D 16, fp_offset =3D 0, overflow_arg_area= =3D=20 > > 0x7f6d44f139e0, reg_save_area =3D 0x7f6d44f139a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=3Dcpu@entry=3D0x7f6d5c8a4= 000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run =3D 0x7f6d570cf000 > > ret =3D > > run_ret =3D > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=3D0x7f6d5c8a400= 0) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu =3D 0x7f6d5c8a4000 > > r =3D > > #4 0x00007f6d55ceadf5 in start_thread (arg=3D0x7f6d44f14700) at pthr= ead_create.c:308 > > __res =3D > > pd =3D 0x7f6d44f14700 > > now =3D > > unwind_buf =3D {cancel_jmp_buf =3D {{jmp_buf =3D {14010728482= 5856, 5884348200482620104, 0, 140107284826560, 140107284825856, 140726431= 086992, -5804033392412867896, -5804071064002379064}, mask_was_saved =3D 0= }}, priv =3D {pad =3D {0x0, 0x0, 0x0, 0x0}, data =3D {prev =3D 0x0, clean= up =3D 0x0, canceltype =3D 0}}} > > not_first_call =3D > > pagesize_m1 =3D > > sp =3D > > freesize =3D > > #5 0x00007f6d4f8f81ad in clone () at=20 > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > >=20 > > Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)): > > #0 sem_timedwait () at=20 > > ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101 > > No locals. > > #1 0x00007f6d574260c7 in qemu_sem_timedwait (sem=3Dsem@entry=3D0x7f6= d5a1b9248, ms=3Dms@entry=3D10000) at util/qemu-thread-posix.c:254 > > rc =3D > > ts =3D {tv_sec =3D 1447709021, tv_nsec =3D 21985000} > > __func__ =3D "qemu_sem_timedwait" > > #2 0x00007f6d573a98ac in worker_thread (opaque=3D0x7f6d5a1b91e0) at = thread-pool.c:92 > > req =3D > > ret =3D > > pool =3D 0x7f6d5a1b91e0 > > #3 0x00007f6d55ceadf5 in start_thread (arg=3D0x7f6ab1dff700) at pthr= ead_create.c:308 > > __res =3D > > pd =3D 0x7f6ab1dff700 > > now =3D > > unwind_buf =3D {cancel_jmp_buf =3D {{jmp_buf =3D {14009622750= 5920, 5884348200482620104, 0, 140096227506624, 140096227505920, 26, -5801= 757560646548792, -5804071064002379064}, mask_was_saved =3D 0}}, priv =3D = {pad =3D {0x0, 0x0, 0x0, 0x0}, data =3D {prev =3D 0x0, cleanup =3D 0x0, c= anceltype =3D 0}}} > > not_first_call =3D > > pagesize_m1 =3D > > sp =3D > > freesize =3D > > #4 0x00007f6d4f8f81ad in clone () at=20 > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > >=20 > > Thread 8 (Thread 0x7f6d44713700 (LWP 52290)): > > #0 0x00007f6d4f8ef257 in ioctl () at=20 > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=3Dcpu@entry=3D0x7f6d5c8= b8000, type=3Dtype@entry=3D44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:= 1969 > > ret =3D > > arg =3D > > ap =3D {{gp_offset =3D 16, fp_offset =3D 0, overflow_arg_area= =3D=20 > > 0x7f6d447129e0, reg_save_area =3D 0x7f6d447129a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=3Dcpu@entry=3D0x7f6d5c8b8= 000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run =3D 0x7f6d570cc000 > > ret =3D > > run_ret =3D > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=3D0x7f6d5c8b800= 0) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu =3D 0x7f6d5c8b8000 > > r =3D > > #4 0x00007f6d55ceadf5 in start_thread (arg=3D0x7f6d44713700) at pthr= ead_create.c:308 > > __res =3D > > pd =3D 0x7f6d44713700 > > now =3D > > unwind_buf =3D {cancel_jmp_buf =3D {{jmp_buf =3D {14010727643= 3152, 5884348200482620104, 0, 140107276433856, 140107276433152, 140726431= 086992, -5804032293438111032, -5804071064002379064}, mask_was_saved =3D 0= }}, priv =3D {pad =3D {0x0, 0x0, 0x0, 0x0}, data =3D {prev =3D 0x0, clean= up =3D 0x0, canceltype =3D 0}}} > > not_first_call =3D > > pagesize_m1 =3D > > sp =3D > > freesize =3D > > #5 0x00007f6d4f8f81ad in clone () at=20 > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > >=20 > > Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)): > > #0 0x00007f6d4f8ef257 in ioctl () at=20 > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=3Dcpu@entry=3D0x7f6d5c8= f4000, type=3Dtype@entry=3D44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:= 1969 > > ret =3D > > arg =3D > > ap =3D {{gp_offset =3D 16, fp_offset =3D 0, overflow_arg_area= =3D=20 > > 0x7f6d42f0f9e0, reg_save_area =3D 0x7f6d42f0f9a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=3Dcpu@entry=3D0x7f6d5c8f4= 000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run =3D 0x7f6d570c3000 > > ret =3D > > run_ret =3D > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=3D0x7f6d5c8f400= 0) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu =3D 0x7f6d5c8f4000 > > r =3D > > #4 0x00007f6d55ceadf5 in start_thread (arg=3D0x7f6d42f10700) at pthr= ead_create.c:308 > > __res =3D > > pd =3D 0x7f6d42f10700 > > now =3D > > unwind_buf =3D {cancel_jmp_buf =3D {{jmp_buf =3D {14010725125= 5040, 5884348200482620104, 0, 140107251255744, 140107251255040, 140726431= 086992, -5804046580109950264, -5804071064002379064}, mask_was_saved =3D 0= }}, priv =3D {pad =3D {0x0, 0x0, 0x0, 0x0}, data =3D {prev =3D 0x0, clean= up =3D 0x0, canceltype =3D 0}}} > > not_first_call =3D > > pagesize_m1 =3D > > sp =3D > > freesize =3D > > #5 0x00007f6d4f8f81ad in clone () at=20 > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > >=20 > > Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)): > > #0 0x00007f6d4f8ef257 in ioctl () at=20 > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=3Dcpu@entry=3D0x7f6d5c8= 78000, type=3Dtype@entry=3D44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:= 1969 > > ret =3D > > arg =3D > > ap =3D {{gp_offset =3D 16, fp_offset =3D 0, overflow_arg_area= =3D=20 > > 0x7f6d45f159e0, reg_save_area =3D 0x7f6d45f159a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=3Dcpu@entry=3D0x7f6d5c878= 000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run =3D 0x7f6d570d5000 > > ret =3D > > run_ret =3D > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=3D0x7f6d5c87800= 0) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu =3D 0x7f6d5c878000 > > r =3D > > #4 0x00007f6d55ceadf5 in start_thread (arg=3D0x7f6d45f16700) at pthr= ead_create.c:308 > > __res =3D > > pd =3D 0x7f6d45f16700 > > now =3D > > unwind_buf =3D {cancel_jmp_buf =3D {{jmp_buf =3D {14010730161= 1264, 5884348200482620104, 0, 140107301611968, 140107301611264, 140726431= 086992, -5804035590362381624, -5804071064002379064}, mask_was_saved =3D 0= }}, priv =3D {pad =3D {0x0, 0x0, 0x0, 0x0}, data =3D {prev =3D 0x0, clean= up =3D 0x0, canceltype =3D 0}}} > > not_first_call =3D > > pagesize_m1 =3D > > sp =3D > > freesize =3D > > #5 0x00007f6d4f8f81ad in clone () at=20 > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > >=20 > > Thread 5 (Thread 0x7f6d43711700 (LWP 52292)): > > #0 0x00007f6d4f8ef257 in ioctl () at=20 > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=3Dcpu@entry=3D0x7f6d5c8= e0000, type=3Dtype@entry=3D44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:= 1969 > > ret =3D > > arg =3D > > ap =3D {{gp_offset =3D 16, fp_offset =3D 0, overflow_arg_area= =3D=20 > > 0x7f6d437109e0, reg_save_area =3D 0x7f6d437109a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=3Dcpu@entry=3D0x7f6d5c8e0= 000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run =3D 0x7f6d570c6000 > > ret =3D > > run_ret =3D > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=3D0x7f6d5c8e000= 0) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu =3D 0x7f6d5c8e0000 > > r =3D > > #4 0x00007f6d55ceadf5 in start_thread (arg=3D0x7f6d43711700) at pthr= ead_create.c:308 > > __res =3D > > pd =3D 0x7f6d43711700 > > now =3D > > unwind_buf =3D {cancel_jmp_buf =3D {{jmp_buf =3D {14010725964= 7744, 5884348200482620104, 0, 140107259648448, 140107259647744, 140726431= 086992, -5804047687674641720, -5804071064002379064}, mask_was_saved =3D 0= }}, priv =3D {pad =3D {0x0, 0x0, 0x0, 0x0}, data =3D {prev =3D 0x0, clean= up =3D 0x0, canceltype =3D 0}}} > > not_first_call =3D > > pagesize_m1 =3D > > sp =3D > > freesize =3D > > #5 0x00007f6d4f8f81ad in clone () at=20 > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > >=20 > > Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)): > > #0 0x00007f6d4f8ef257 in ioctl () at=20 > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=3Dcpu@entry=3D0x7f6d5c8= cc000, type=3Dtype@entry=3D44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:= 1969 > > ret =3D > > arg =3D > > ap =3D {{gp_offset =3D 16, fp_offset =3D 0, overflow_arg_area= =3D=20 > > 0x7f6d43f119e0, reg_save_area =3D 0x7f6d43f119a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=3Dcpu@entry=3D0x7f6d5c8cc= 000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run =3D 0x7f6d570c9000 > > ret =3D > > run_ret =3D > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=3D0x7f6d5c8cc00= 0) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu =3D 0x7f6d5c8cc000 > > r =3D > > #4 0x00007f6d55ceadf5 in start_thread (arg=3D0x7f6d43f12700) at pthr= ead_create.c:308 > > __res =3D > > pd =3D 0x7f6d43f12700 > > now =3D > > unwind_buf =3D {cancel_jmp_buf =3D {{jmp_buf =3D {14010726804= 0448, 5884348200482620104, 0, 140107268041152, 140107268040448, 140726431= 086992, -5804048786649398584, -5804071064002379064}, mask_was_saved =3D 0= }}, priv =3D {pad =3D {0x0, 0x0, 0x0, 0x0}, data =3D {prev =3D 0x0, clean= up =3D 0x0, canceltype =3D 0}}} > > not_first_call =3D > > pagesize_m1 =3D > > sp =3D > > freesize =3D > > #5 0x00007f6d4f8f81ad in clone () at=20 > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > >=20 > > Thread 3 (Thread 0x7f6d46717700 (LWP 52286)): > > #0 0x00007f6d4f8ef257 in ioctl () at=20 > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=3Dcpu@entry=3D0x7f6d5c8= 10000, type=3Dtype@entry=3D44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:= 1969 > > ret =3D > > arg =3D > > ap =3D {{gp_offset =3D 16, fp_offset =3D 0, overflow_arg_area= =3D=20 > > 0x7f6d467169e0, reg_save_area =3D 0x7f6d467169a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=3Dcpu@entry=3D0x7f6d5c810= 000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run =3D 0x7f6d570d8000 > > ret =3D > > run_ret =3D > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=3D0x7f6d5c81000= 0) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu =3D 0x7f6d5c810000 > > r =3D > > #4 0x00007f6d55ceadf5 in start_thread (arg=3D0x7f6d46717700) at pthr= ead_create.c:308 > > __res =3D > > pd =3D 0x7f6d46717700 > > now =3D > > unwind_buf =3D {cancel_jmp_buf =3D {{jmp_buf =3D {14010731000= 3968, 5884348200482620104, 0, 140107310004672, 140107310003968, 140726431= 086992, -5804036689337138488, -5804071064002379064}, mask_was_saved =3D 0= }}, priv =3D {pad =3D {0x0, 0x0, 0x0, 0x0}, data =3D {prev =3D 0x0, clean= up =3D 0x0, canceltype =3D 0}}} > > not_first_call =3D > > pagesize_m1 =3D > > sp =3D > > freesize =3D > > #5 0x00007f6d4f8f81ad in clone () at=20 > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > >=20 > > Thread 2 (Thread 0x7f6d45715700 (LWP 52288)): > > #0 0x00007f6d4f8ef257 in ioctl () at=20 > > ../sysdeps/unix/syscall-template.S:81 > > No locals. > > #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=3Dcpu@entry=3D0x7f6d5c8= 90000, type=3Dtype@entry=3D44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:= 1969 > > ret =3D > > arg =3D > > ap =3D {{gp_offset =3D 16, fp_offset =3D 0, overflow_arg_area= =3D=20 > > 0x7f6d457149e0, reg_save_area =3D 0x7f6d457149a0}} > > #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=3Dcpu@entry=3D0x7f6d5c890= 000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829 > > run =3D 0x7f6d570d2000 > > ret =3D > > run_ret =3D > > #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=3D0x7f6d5c89000= 0) at /usr/src/debug/qemu-2.3.0/cpus.c:944 > > cpu =3D 0x7f6d5c890000 > > r =3D > > #4 0x00007f6d55ceadf5 in start_thread (arg=3D0x7f6d45715700) at pthr= ead_create.c:308 > > __res =3D > > pd =3D 0x7f6d45715700 > > now =3D > > unwind_buf =3D {cancel_jmp_buf =3D {{jmp_buf =3D {14010729321= 8560, 5884348200482620104, 0, 140107293219264, 140107293218560, 140726431= 086992, -5804034491387624760, -5804071064002379064}, mask_was_saved =3D 0= }}, priv =3D {pad =3D {0x0, 0x0, 0x0, 0x0}, data =3D {prev =3D 0x0, clean= up =3D 0x0, canceltype =3D 0}}} > > not_first_call =3D > > pagesize_m1 =3D > > sp =3D > > freesize =3D > > #5 0x00007f6d4f8f81ad in clone () at=20 > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 > > No locals. > >=20 > > Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)): > > #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 N= o symbol table info available. > > #1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=3D49280) at vl.c:= 2575 > > ptr =3D 0x7f6d59a346a0 > > #2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0 No= symbol table info available. > > #3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so= .0 No symbol table info available. > > #4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=3D0x7f6d5de1ff4= 0) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33 > > req =3D > > #5 virtio_blk_get_request (s=3D0x7f6d5de1ff40) at /usr/src/debug/qem= u-2.3.0/hw/block/virtio-blk.c:192 > > req =3D > > #6 virtio_blk_handle_output (vdev=3D, vq=3D) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604 > > s =3D 0x7f6d5de1ff40 > > __func__ =3D "virtio_blk_handle_output" > > mrb =3D {reqs =3D {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7= c000,=20 > > 0x0 }, num_reqs =3D 3, is_write =3D true} > > #7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=3D0x7f6d5a1aea= 00, ret=3D62, ret@entry=3D1) at iohandler.c:143 > > revents =3D 1 > > #8 0x00007f6d573b7296 in main_loop_wait (nonblocking=3D) at main-loop.c:504 > > ret =3D 1 > > timeout =3D 4294967295 > > timeout_ns =3D > > #9 0x00007f6d571b6a4e in main_loop () at vl.c:1818 > > nonblocking =3D > > last_io =3D 0 > > #10 main (argc=3D, argv=3D, envp=3D) at vl.c:4394 > > i =3D > > snapshot =3D > > linux_boot =3D > > initrd_filename =3D > > kernel_filename =3D > > kernel_cmdline =3D > > boot_order =3D 0x7f6d57449a67 "cad" > > boot_once =3D 0x0 > > cyls =3D > > heads =3D > > secs =3D > > translation =3D > > hda_opts =3D > > opts =3D > > machine_opts =3D > > icount_opts =3D > > olist =3D > > optind =3D 69 > > optarg =3D 0x7f6d5a14b3a0 "rhel6.5.0" > > loadvm =3D > > machine_class =3D > > cpu_model =3D > > vga_model =3D 0x0 > > qtest_chrdev =3D > > qtest_log =3D > > pid_file =3D > > incoming =3D > > show_vnc_port =3D > > defconfig =3D > > userconfig =3D 179 > > log_mask =3D > > log_file =3D > > mem_trace =3D {malloc =3D 0x7f6d572a4480 , = realloc =3D 0x7f6d572a4460 , free =3D 0x7f6d572a4450 <= free_and_trace>, calloc =3D 0x0, try_malloc =3D 0x0, try_realloc =3D 0x0} > > trace_events =3D > > trace_file =3D > > maxram_size =3D > > ram_slots =3D > > vmstate_dump_file =3D > > main_loop_err =3D 0x0 > > __func__ =3D "main" > >=20 > >=20 > >=20 > >=20 > >=20 > > @ Do you think you're only hitting these crashes on VMs that have bee= n paused because of these space errors? > > Will have a look on that > >=20 > > Thx Christian > >=20 > > -----Urspr=C3=BCngliche Nachricht----- > > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] > > Gesendet: Dienstag, 17. November 2015 12:36 > > An: Grundmann, Christian > > Cc: 'qemu-devel@nongnu.org' ;=20 > > stefanha@redhat.com > > Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in= =20 > > libtcmalloc > >=20 > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > > Hi, > > >=20 > > > @ Can you please use a 'thread apply all bt full' the full gives = a little more info. > > >=20 > > > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "s= et pagination off" -ex "thread apply all bt full" > >=20 > > OK, it doesn't relaly give any more without the debuginfo package men= tioned below. > >=20 > > > >=20 > > > @ Also, if you've not already got it installed can you please insta= ll the debuginfo package for qemu, it gives a lot more information in bac= ktraces. > > > Sorry it's a ovirt-node System where I can't you yum > >=20 > > Ah, although perhaps if you took the core dump, onto another machine = with matching qemu and debuginfo you should be able to get more detail. > >=20 > > > @ Does this part always look the same in your backtraces? > > > The most are the same, found one a little bit different : > > > Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)): > > > #0 0x00007f3785d18353 in > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache:= :FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol= table info available. > > > #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from > > > /lib64/libtcmalloc.so.4 No symbol table info available. > > > #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 N= o=20 > > > symbol table info available. > > > #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No= =20 > > > symbol table info available. > > > #4 0x00007f37885fec89 in g_slice_free1 () from > > > /lib64/libglib-2.0.so.0 No symbol table info available. > > > #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table= =20 > > > info available. > > > #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info=20 > > > available. > > > #7 0x00007f378a398394 in aio_bh_poll () No symbol table info=20 > > > available. > > > #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table=20 > > > info available. > > > #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info=20 > > > available. > > > #10 0x00007f37885e299a in g_main_context_dispatch () from > > > /lib64/libglib-2.0.so.0 No symbol table info available. > > > #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info=20 > > > available. > > > #12 0x00007f378a1a5a4e in main () > > > No symbol table info available. > > >=20 > >=20 > > OK, that's a bit different but interesting.... > >=20 > > > @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmn= ame.log ? > > > No nothing abnormal > > >=20 > > > @ 2) Did you hit any IO errors and need to tell the VM to continue= after a problem? > > > Ovirt tells me "no Storage space error". Which is something like th= e disk is growing to fast i think. I use Snapshots so on heavy write the = disk has to grow a lot. > > > Sometimes the VM is paused and resumed from ovirt. Sometimes the VM= stays offline. > >=20 > > OK, that's interesting, because you may be hitting the following bug;= =20 > > http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html > >=20 > > whose fix coincidentally just got accepted today; it's related to err= or cases with error=3Dstop which you are using. > >=20 > > Do you think you're only hitting these crashes on VMs that have been = paused because of these space errors? > >=20 > > > disk emulation and see if the problem goes away - e.g. virtio-= scsi would be a good one to try. > > >=20 > > > Ok will try that and report > >=20 > > Thanks, > >=20 > > Dave > >=20 > > >=20 > > > Thx Christian > > >=20 > > >=20 > > > -----Urspr=C3=BCngliche Nachricht----- > > > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] > > > Gesendet: Dienstag, 17. November 2015 10:59 > > > An: Grundmann, Christian > > > Cc: 'qemu-devel@nongnu.org' ;=20 > > > stefanha@redhat.com > > > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in=20 > > > libtcmalloc > > >=20 > > > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote: > > > > Hi, > > > > Dan sent me over to you, > > > > please let me know if i can provide additional informations > > >=20 > > > Hi Christian, > > > Thanks for reporting this, > > >=20 > > > > Softwareversions: > > > > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso > > > >=20 > > > > qemu-img-ev-2.3.0-29.1.el7.x86_64 > > > > qemu-kvm-ev-2.3.0-29.1.el7.x86_64 > > > > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64 > > > > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64 > > > > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch > > > > kernel-3.10.0-229.14.1.el7.x86_64 > > > > gperftools-libs-2.4-7.el7.x86_64 > > > >=20 > > > > Commandline: > > > > /usr/libexec/qemu-kvm -name myvmname -S -machine=20 > > > > rhel6.5.0,accel=3Dkvm,usb=3Doff -cpu Westmere -m 7168 -realtime=20 > > > > mlock=3Doff -smp 2,maxcpus=3D16,sockets=3D16,cores=3D1,threads=3D= 1 -uuid > > > > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios=20 > > > > type=3D1,manufacturer=3DoVirt,product=3DoVirt > > > > Node,version=3D3.6-0.999.201510221942.el7.centos,serial=3D3034353= 6-313 > > > > 8- > > > > 5A > > > > 43-4A34-323630303253,uuid=3D5b6b8899-5a9d-4c07-a6aa-6171527ad319 > > > > -nographic -no-user-config -nodefaults -chardev=20 > > > > socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/myvmname.mon= itor, > > > > se rv er,nowait -mon chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dco= ntrol=20 > > > > -rtc base=3D2015-11-15T20:04:35,driftfix=3Dslew -global=20 > > > > kvm-pit.lost_tick_policy=3Ddiscard -no-hpet -no-shutdown -boot=20 > > > > strict=3Don -device piix3-usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x= 1.0x2 > > > > -device > > > > virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0x4 -device > > > > virtio-serial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpci.0,= addr=3D0x > > > > 5 -drive if=3Dnone,id=3Ddrive-ide0-1-0,readonly=3Don,format=3Draw= ,serial=3D > > > > -device ide-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Di= de0-1-0 > > > > -drive > > > > file=3D/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df= 61b > > > > 84 > > > > -8 > > > > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baf= f > > > > ae > > > > cf > > > > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=3Dnone,id=3Ddrive-virtio= -disk > > > > 0, > > > > fo > > > > rmat=3Dqcow2,serial=3D8202b81d-6191-495f-8c9d-7d90baffaecf,cache=3D= none, > > > > we rr or=3Dstop,rerror=3Dstop,aio=3Dnative -device=20 > > > > virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x6,drive=3Ddrive-vi= rtio-disk > > > > 0, > > > > id > > > > =3Dvirtio-disk0,bootindex=3D1 -netdev > > > > tap,fd=3D39,id=3Dhostnet0,vhost=3Don,vhostfd=3D65 -device=20 > > > > virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D52:54:00:83:a2:0= e,bus=3Dp > > > > ci > > > > .0 > > > > ,addr=3D0x3 -chardev > > > > socket,id=3Dcharchannel0,path=3D/var/lib/libvirt/qemu/channels/5b= 6b889 > > > > 9- 5a=20 > > > > 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait > > > > -device > > > > virtserialport,bus=3Dvirtio-serial0.0,nr=3D1,chardev=3Dcharchanne= l0,id=3Dc > > > > ha nn el0,name=3Dcom.redhat.rhevm.vdsm -chardev > > > > socket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/channels/5b= 6b889 > > > > 9- 5a=20 > > > > 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait > > > > -device > > > > virtserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dcharchanne= l1,id=3Dc > > > > ha > > > > nn > > > > el1,name=3Dorg.qemu.guest_agent.0 -device > > > > cirrus-vga,id=3Dvideo0,bus=3Dpci.0,addr=3D0x2 -device > > > > virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x7 -msg=20 > > > > timestamp=3Don > > > >=20 > > > > Stack Trace: > > > >=20 > > > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex = "set pagination off" -ex "thread apply all bt" > > >=20 > > > Can you please use a 'thread apply all bt full' the full gives a = little more info. > > > Also, if you've not already got it installed can you please install= the debuginfo package for qemu, it gives a lot more information in backt= races. > > >=20 > > > > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)): > > > > #0 0x00007fa8ad2febe1 in tc_malloc () from=20 > > > > /lib64/libtcmalloc.so.4 > > > > #1 0x00007fa8b186b489 in malloc_and_trace () > > > > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.= 0 > > > > #3 0x00007fa8afbd666e in g_slice_alloc () from > > > > /lib64/libglib-2.0.so.0 > > > > #4 0x00007fa8b17cbffd in virtio_blk_handle_output () > > > > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll () > > > > #6 0x00007fa8b197e296 in main_loop_wait () > > > > #7 0x00007fa8b177da4e in main () > > >=20 > > > Does this part always look the same in your backtraces? > > > The segfault in tc_malloc is probably due to a heap corruption, or = double free or similar - although it can be a bit tricky to find out what= did it, since the corruption might have happened a bit before the place = it crashed. > > >=20 > > > Some other ideas: > > > 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmna= me.log ? > > > 2) Did you hit any IO errors and need to tell the VM to continue = after a problem? > > > 3) If this is pretty repeatable, then it would be interesting to = try changing to a different > > > disk emulation and see if the problem goes away - e.g. virtio-= scsi would be a good one to try. > > >=20 > > > Dave > > > >=20 > > > >=20 > > > > Thx Christian > > > >=20 > > > > -----Urspr=C3=BCngliche Nachricht----- > > > > Von: Dan Kenigsberg [mailto:danken@redhat.com] > > > > Gesendet: Freitag, 13. November 2015 20:00 > > > > An: Grundmann, Christian > > > > Cc: 'users@ovirt.org' > > > > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc > > > >=20 > > > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wr= ote: > > > > > Hi, > > > > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.is= o"=20 > > > > > (is there something better to use?) fort he nodes, and have=20 > > > > > random crashes of VMs The dumps are always the Same > > > > >=20 > > > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump=20 > > > > > [Thread debugging using libthread_db enabled] Using host=20 > > > > > libthread_db library "/lib64/libthread_db.so.1". > > > > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -m= achine rhel6.5.0,accel=3Dkvm,usb=3Do'. > > > > > Program terminated with signal 11, Segmentation fault. > > > > > #0 0x00007f0c559c4353 in > > > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCa= che:: > > > > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 > > > > >=20 > > > > >=20 > > > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist > > > > > centos7 or 3.6 > > > >=20 > > > > Due to the low-leveled-ness of the problem, I'd guess it's a qemu= //lib64/libtcmalloc malloc bug, and not directly related to ovirt. > > > >=20 > > > > Please report the precise version of qemu,kernel,libvirt and gper= ftools-libs to qemu-devel mailing list and the complete stack trace and q= emu command line, if possible. > > > >=20 > > > -- > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > >=20 > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK