From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
To: "Grundmann, Christian" <Christian.Grundmann@fabasoft.com>
Cc: "'qemu-devel@nongnu.org'" <qemu-devel@nongnu.org>,
"stefanha@redhat.com" <stefanha@redhat.com>
Subject: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
Date: Tue, 17 Nov 2015 14:42:26 +0000 [thread overview]
Message-ID: <20151117144225.GH2498@work-vm> (raw)
In-Reply-To: <6A17C71B52524C408E7AAF69103E9E490F1552E7@fabamailserver.fabagl.fabasoft.com>
* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> And here another one
Oh this is a bit of a different one, from query-blockstats, although again if
the heap's corrupted it might have just been the first guy to trip over the
corrupt part afterwards.
Dave
> Thread 1 (Thread 0x7f5d03ba5c00 (LWP 1507)):
> #0 0x00007f5cff7e2e7d in tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #1 0x00007f5cff7e312a in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #2 0x00007f5cff7e31dd in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #3 0x00007f5cff7e6235 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #4 0x00007f5cff7f4d5b in tc_malloc () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #5 0x00007f5d03d61489 in malloc_and_trace (n_bytes=18) at vl.c:2575
> ptr = 0x1
> #6 0x00007f5d020b647f in g_malloc () from /lib64/libglib-2.0.so.0
> No symbol table info available.
> #7 0x00007f5d020cdf7f in g_strdup () from /lib64/libglib-2.0.so.0
> No symbol table info available.
> #8 0x00007f5d03eddab5 in alloc_entry (value=0x7f5d088de6c0, key=0x7f5d03f5debb "wr_highest_offset") at qobject/qdict.c:79
> entry = 0x7f5d088df480
> #9 qdict_put_obj (qdict=0x7f5d06e10400, key=0x7f5d03f5debb "wr_highest_offset", value=0x7f5d088de6c0) at qobject/qdict.c:145
> bucket = 81
> entry = <optimized out>
> #10 0x00007f5d03ebf34a in visit_type_BlockDeviceStats_fields (errp=0x7ffddb417ca0, obj=0x7f5d07f905a0, m=0x7f5d061fdea0) at qapi-visit.c:1542
> err = 0x0
> #11 visit_type_BlockDeviceStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f905a0, name=name@entry=0x7f5d03f032ec "stats", errp=errp@entry=0x7ffddb417ca0) at qapi-visit.c:1566
> err = 0x0
> #12 0x00007f5d03ebf5b1 in visit_type_BlockStats_fields (errp=0x7ffddb417cf0, obj=0x7f5d07f90650, m=0x7f5d061fdea0) at qapi-visit.c:1614
> err = 0x0
> #13 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07f90650, name=name@entry=0x7f5d03f480f4 "parent", errp=errp@entry=0x7ffddb417cf0) at qapi-visit.c:1644
> err = 0x0
> #14 0x00007f5d03ebf6bd in visit_type_BlockStats_fields (errp=0x7ffddb417d38, obj=0x7f5d07c67a50, m=0x7f5d061fdea0) at qapi-visit.c:1620
> err = 0x0
> #15 visit_type_BlockStats (m=m@entry=0x7f5d061fdea0, obj=0x7f5d07c67a50, name=name@entry=0x0, errp=errp@entry=0x7ffddb417d38) at qapi-visit.c:1644
> err = 0x0
> #16 0x00007f5d03ebf760 in visit_type_BlockStatsList (m=0x7f5d061fdea0, obj=obj@entry=0x7ffddb417d98, name=name@entry=0x7f5d03f00e6e "unused", errp=errp@entry=0x7ffddb417da0) at qapi-visit.c:1665
> native_i = <optimized out>
> err = 0x0
> i = 0x7f5d07c67a50
> prev = 0x7ffddb417d40
> #17 0x00007f5d03d674dd in qmp_marshal_output_query_blockstats (errp=0x7ffddb417d90, ret_out=0x7ffddb417e10, ret_in=0x7f5d07c67120) at qmp-marshal.c:182
> local_err = 0x0
> mo = 0x7f5d061fdea0
> md = <optimized out>
> v = <optimized out>
> #18 qmp_marshal_input_query_blockstats (mon=<optimized out>, qdict=<optimized out>, ret=0x7ffddb417e10) at qmp-marshal.c:225
> local_err = 0x0
> args = <optimized out>
> retval = <optimized out>
> mi = 0x7f5d064e2000
> md = <optimized out>
> v = <optimized out>
> has_query_nodes = false
> query_nodes = false
> #19 0x00007f5d03ca0531 in qmp_call_cmd (cmd=<optimized out>, params=0x7f5d075dd600, mon=0x7f5d06208320) at /usr/src/debug/qemu-2.3.0/monitor.c:5051
> ret = <optimized out>
> data = 0x0
> #20 handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5113
> err = <optimized out>
> obj = <optimized out>
> input = <optimized out>
> args = 0x7f5d075dd600
> cmd_name = <optimized out>
> mon = 0x7f5d06208320
> #21 0x00007f5d03edf4f2 in json_message_process_token (lexer=0x7f5d061f5d70, token=0x7f5d061991e0, type=JSON_OPERATOR, x=48, y=15) at qobject/json-streamer.c:87
> parser = 0x7f5d061f5d68
> dict = 0x7f5d088ea800
> #22 0x00007f5d03ef191f in json_lexer_feed_char (lexer=lexer@entry=0x7f5d061f5d70, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303
> new_state = 100
> #23 0x00007f5d03ef19ee in json_lexer_feed (lexer=0x7f5d061f5d70, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356
> err = <optimized out>
> i = <optimized out>
> #24 0x00007f5d03edf689 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110
> No locals.
> #25 0x00007f5d03c9e8cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134
> old_mon = 0x0
> #26 0x00007f5d03d5b1b0 in qemu_chr_be_write (len=<optimized out>, buf=0x7ffddb417f40 "}\177A\333\375\177", s=0x7f5d0625a2e0) at qemu-char.c:305
> No locals.
> #27 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7f5d0625a2e0) at qemu-char.c:2870
> chr = 0x7f5d0625a2e0
> s = 0x7f5d061aa3f0
> buf = "}\177A\333\375\177\000\000\360\360\355\003]\177\000\000\030\003\000\000\000\000\000\000\205\336\355\003]\177\000\000\000A\036\b]\177\000\000\263\325\355\003]\177\000\000\000A\036\b]\177", '\000' <repeats 18 times>, "`2\036\b]\177\000\000ЀA\333\375\177\000\000\000\000\000\000\000\000\000\000 \232\261\a]\177\000\000\000(\341\006]\177\000\000P\217A\333\375\177\000\000H\242<\374\\\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\240\200A\333\375\177\000\000\340\177A\333\375\177\000\000\r\000\000\000\000\000\000\000 Xz\b]\177\000\000\000@&\006]\177\000\000\340\365\215\b]\177\000\000Q\000\000\000\000\000\000\000\232"...
> len = <optimized out>
> size = <optimized out>
> #28 0x00007f5d020b099a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
> No symbol table info available.
> #29 0x00007f5d03e74288 in glib_pollfds_poll () at main-loop.c:209
> context = 0x7f5d06205140
> pfds = <optimized out>
> #30 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254
> ret = 2
> spin_counter = 0
> #31 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503
> ret = 2
> timeout = 4294967295
> timeout_ns = <optimized out>
> #32 0x00007f5d03c73a4e in main_loop () at vl.c:1818
> nonblocking = <optimized out>
> last_io = 2
> #33 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
> i = <optimized out>
> snapshot = <optimized out>
> linux_boot = <optimized out>
> initrd_filename = <optimized out>
> kernel_filename = <optimized out>
> kernel_cmdline = <optimized out>
> boot_order = 0x7f5d03f06a67 "cad"
> boot_once = 0x0
> cyls = <optimized out>
> heads = <optimized out>
> secs = <optimized out>
> translation = <optimized out>
> hda_opts = <optimized out>
> opts = <optimized out>
> machine_opts = <optimized out>
> icount_opts = <optimized out>
> olist = <optimized out>
> optind = 67
> optarg = 0x7f5d06193570 "rhel6.5.0"
> loadvm = <optimized out>
> machine_class = <optimized out>
> cpu_model = <optimized out>
> vga_model = 0x0
> qtest_chrdev = <optimized out>
> qtest_log = <optimized out>
> pid_file = <optimized out>
> incoming = <optimized out>
> show_vnc_port = <optimized out>
> defconfig = <optimized out>
> userconfig = 56
> log_mask = <optimized out>
> log_file = <optimized out>
> mem_trace = {malloc = 0x7f5d03d61480 <malloc_and_trace>, realloc = 0x7f5d03d61460 <realloc_and_trace>, free = 0x7f5d03d61450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
> trace_events = <optimized out>
> trace_file = <optimized out>
> maxram_size = <optimized out>
> ram_slots = <optimized out>
> vmstate_dump_file = <optimized out>
> main_loop_err = 0x0
> __func__ = "main"
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org [mailto:qemu-devel-bounces+christian.grundmann=fabasoft.com@nongnu.org] Im Auftrag von Grundmann, Christian
> Gesendet: Dienstag, 17. November 2015 15:12
> An: 'Dr. David Alan Gilbert' <dgilbert@redhat.com>
> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
> Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
>
> Here you go
>
>
> gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
> [New LWP 52281]
> [New LWP 52288]
> [New LWP 52286]
> [New LWP 52291]
> [New LWP 52292]
> [New LWP 52287]
> [New LWP 52293]
> [New LWP 52290]
> [New LWP 56455]
> [New LWP 52289]
> [New LWP 52282]
> [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/usr/libexec/qemu-kvm -name myvmname -S -machine rhel6.5.0,accel=kvm,us'.
> Program terminated with signal 11, Segmentation fault.
> #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4
>
> Thread 11 (Thread 0x7f6d47719700 (LWP 52282)):
> #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
> No locals.
> #1 0x00007f6d57426272 in futex_wait (val=4294967295, ev=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301 No locals.
> #2 qemu_event_wait (ev=ev@entry=0x7f6d57cf0f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399
> value = <optimized out>
> #3 0x00007f6d57434526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233
> tries = 0
> n = <optimized out>
> node = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d47719700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d47719700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107326789376, 5884348200482620104, 0, 140107326790080, 140107326789376, 140107592666688, -5804038895876586808, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 10 (Thread 0x7f6d44f14700 (LWP 52289)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8a4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d44f139e0, reg_save_area = 0x7f6d44f139a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570cf000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8a4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8a4000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44f14700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d44f14700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107284825856, 5884348200482620104, 0, 140107284826560, 140107284825856, 140726431086992, -5804033392412867896, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 9 (Thread 0x7f6ab1dff700 (LWP 56455)):
> #0 sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
> No locals.
> #1 0x00007f6d574260c7 in qemu_sem_timedwait (sem=sem@entry=0x7f6d5a1b9248, ms=ms@entry=10000) at util/qemu-thread-posix.c:254
> rc = <optimized out>
> ts = {tv_sec = 1447709021, tv_nsec = 21985000}
> __func__ = "qemu_sem_timedwait"
> #2 0x00007f6d573a98ac in worker_thread (opaque=0x7f6d5a1b91e0) at thread-pool.c:92
> req = <optimized out>
> ret = <optimized out>
> pool = 0x7f6d5a1b91e0
> #3 0x00007f6d55ceadf5 in start_thread (arg=0x7f6ab1dff700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6ab1dff700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140096227505920, 5884348200482620104, 0, 140096227506624, 140096227505920, 26, -5801757560646548792, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #4 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 8 (Thread 0x7f6d44713700 (LWP 52290)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8b8000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d447129e0, reg_save_area = 0x7f6d447129a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570cc000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8b8000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8b8000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d44713700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d44713700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107276433152, 5884348200482620104, 0, 140107276433856, 140107276433152, 140726431086992, -5804032293438111032, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 7 (Thread 0x7f6d42f10700 (LWP 52293)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8f4000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d42f0f9e0, reg_save_area = 0x7f6d42f0f9a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570c3000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8f4000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8f4000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d42f10700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d42f10700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107251255040, 5884348200482620104, 0, 140107251255744, 140107251255040, 140726431086992, -5804046580109950264, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 6 (Thread 0x7f6d45f16700 (LWP 52287)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c878000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d45f159e0, reg_save_area = 0x7f6d45f159a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570d5000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c878000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c878000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45f16700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d45f16700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107301611264, 5884348200482620104, 0, 140107301611968, 140107301611264, 140726431086992, -5804035590362381624, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 5 (Thread 0x7f6d43711700 (LWP 52292)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8e0000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d437109e0, reg_save_area = 0x7f6d437109a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570c6000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8e0000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8e0000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43711700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d43711700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107259647744, 5884348200482620104, 0, 140107259648448, 140107259647744, 140726431086992, -5804047687674641720, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 4 (Thread 0x7f6d43f12700 (LWP 52291)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c8cc000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d43f119e0, reg_save_area = 0x7f6d43f119a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570c9000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c8cc000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c8cc000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d43f12700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d43f12700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107268040448, 5884348200482620104, 0, 140107268041152, 140107268040448, 140726431086992, -5804048786649398584, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 3 (Thread 0x7f6d46717700 (LWP 52286)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c810000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d467169e0, reg_save_area = 0x7f6d467169a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570d8000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c810000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c810000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d46717700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d46717700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107310003968, 5884348200482620104, 0, 140107310004672, 140107310003968, 140726431086992, -5804036689337138488, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 2 (Thread 0x7f6d45715700 (LWP 52288)):
> #0 0x00007f6d4f8ef257 in ioctl () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1 0x00007f6d571f1025 in kvm_vcpu_ioctl (cpu=cpu@entry=0x7f6d5c890000, type=type@entry=44672) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1969
> ret = <optimized out>
> arg = <optimized out>
> ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7f6d457149e0, reg_save_area = 0x7f6d457149a0}}
> #2 0x00007f6d571f10de in kvm_cpu_exec (cpu=cpu@entry=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/kvm-all.c:1829
> run = 0x7f6d570d2000
> ret = <optimized out>
> run_ret = <optimized out>
> #3 0x00007f6d571ded7a in qemu_kvm_cpu_thread_fn (arg=0x7f6d5c890000) at /usr/src/debug/qemu-2.3.0/cpus.c:944
> cpu = 0x7f6d5c890000
> r = <optimized out>
> #4 0x00007f6d55ceadf5 in start_thread (arg=0x7f6d45715700) at pthread_create.c:308
> __res = <optimized out>
> pd = 0x7f6d45715700
> now = <optimized out>
> unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140107293218560, 5884348200482620104, 0, 140107293219264, 140107293218560, 140726431086992, -5804034491387624760, -5804071064002379064}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
> not_first_call = <optimized out>
> pagesize_m1 = <optimized out>
> sp = <optimized out>
> freesize = <optimized out>
> #5 0x00007f6d4f8f81ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
>
> Thread 1 (Thread 0x7f6d570e8c00 (LWP 52281)):
> #0 0x00007f6d52d37be1 in tc_malloc () from /lib64/libtcmalloc.so.4 No symbol table info available.
> #1 0x00007f6d572a4489 in malloc_and_trace (n_bytes=49280) at vl.c:2575
> ptr = 0x7f6d59a346a0
> #2 0x00007f6d555f947f in g_malloc () from /lib64/libglib-2.0.so.0 No symbol table info available.
> #3 0x00007f6d5560f66e in g_slice_alloc () from /lib64/libglib-2.0.so.0 No symbol table info available.
> #4 0x00007f6d57204ffd in virtio_blk_alloc_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:33
> req = <optimized out>
> #5 virtio_blk_get_request (s=0x7f6d5de1ff40) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:192
> req = <optimized out>
> #6 virtio_blk_handle_output (vdev=<optimized out>, vq=<optimized out>) at /usr/src/debug/qemu-2.3.0/hw/block/virtio-blk.c:604
> s = 0x7f6d5de1ff40
> __func__ = "virtio_blk_handle_output"
> mrb = {reqs = {0x7f6d5ca40000, 0x7f6d5db7c000, 0x7f6d5db7c000, 0x0 <repeats 29 times>}, num_reqs = 3, is_write = true}
> #7 0x00007f6d573b76b6 in qemu_iohandler_poll (pollfds=0x7f6d5a1aea00, ret=62, ret@entry=1) at iohandler.c:143
> revents = 1
> #8 0x00007f6d573b7296 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:504
> ret = 1
> timeout = 4294967295
> timeout_ns = <optimized out>
> #9 0x00007f6d571b6a4e in main_loop () at vl.c:1818
> nonblocking = <optimized out>
> last_io = 0
> #10 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
> i = <optimized out>
> snapshot = <optimized out>
> linux_boot = <optimized out>
> initrd_filename = <optimized out>
> kernel_filename = <optimized out>
> kernel_cmdline = <optimized out>
> boot_order = 0x7f6d57449a67 "cad"
> boot_once = 0x0
> cyls = <optimized out>
> heads = <optimized out>
> secs = <optimized out>
> translation = <optimized out>
> hda_opts = <optimized out>
> opts = <optimized out>
> machine_opts = <optimized out>
> icount_opts = <optimized out>
> olist = <optimized out>
> optind = 69
> optarg = 0x7f6d5a14b3a0 "rhel6.5.0"
> loadvm = <optimized out>
> machine_class = <optimized out>
> cpu_model = <optimized out>
> vga_model = 0x0
> qtest_chrdev = <optimized out>
> qtest_log = <optimized out>
> pid_file = <optimized out>
> incoming = <optimized out>
> show_vnc_port = <optimized out>
> defconfig = <optimized out>
> userconfig = 179
> log_mask = <optimized out>
> log_file = <optimized out>
> mem_trace = {malloc = 0x7f6d572a4480 <malloc_and_trace>, realloc = 0x7f6d572a4460 <realloc_and_trace>, free = 0x7f6d572a4450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
> trace_events = <optimized out>
> trace_file = <optimized out>
> maxram_size = <optimized out>
> ram_slots = <optimized out>
> vmstate_dump_file = <optimized out>
> main_loop_err = 0x0
> __func__ = "main"
>
>
>
>
>
> @ Do you think you're only hitting these crashes on VMs that have been paused because of these space errors?
> Will have a look on that
>
> Thx Christian
>
> -----Ursprüngliche Nachricht-----
> Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> Gesendet: Dienstag, 17. November 2015 12:36
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
> Betreff: Re: AW: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc
>
> * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > Hi,
> >
> > @ Can you please use a 'thread apply all bt full' the full gives a little more info.
> >
> > gdb --batch /usr/libexec/qemu-kvm core.52281.1447709011.dump -ex "set pagination off" -ex "thread apply all bt full"
>
> OK, it doesn't relaly give any more without the debuginfo package mentioned below.
>
> <snip>
>
> > @ Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
> > Sorry it's a ovirt-node System where I can't you yum
>
> Ah, although perhaps if you took the core dump, onto another machine with matching qemu and debuginfo you should be able to get more detail.
>
> > @ Does this part always look the same in your backtraces?
> > The most are the same, found one a little bit different :
> > Thread 1 (Thread 0x7f378a0d7c00 (LWP 6658)):
> > #0 0x00007f3785d18353 in
> > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> > #1 0x00007f3785d186b0 in tcmalloc::ThreadCache::Scavenge() () from
> > /lib64/libtcmalloc.so.4 No symbol table info available.
> > #2 0x00007f3785d27057 in tc_free () from /lib64/libtcmalloc.so.4 No
> > symbol table info available.
> > #3 0x00007f37885e858f in g_free () from /lib64/libglib-2.0.so.0 No
> > symbol table info available.
> > #4 0x00007f37885fec89 in g_slice_free1 () from
> > /lib64/libglib-2.0.so.0 No symbol table info available.
> > #5 0x00007f378a1f232e in virtio_blk_rw_complete () No symbol table
> > info available.
> > #6 0x00007f378a39f1ae in bdrv_co_em_bh () No symbol table info
> > available.
> > #7 0x00007f378a398394 in aio_bh_poll () No symbol table info
> > available.
> > #8 0x00007f378a3a7409 in aio_dispatch_clients () No symbol table info
> > available.
> > #9 0x00007f378a39820e in aio_ctx_dispatch () No symbol table info
> > available.
> > #10 0x00007f37885e299a in g_main_context_dispatch () from
> > /lib64/libglib-2.0.so.0 No symbol table info available.
> > #11 0x00007f378a3a6288 in main_loop_wait () No symbol table info
> > available.
> > #12 0x00007f378a1a5a4e in main ()
> > No symbol table info available.
> >
>
> OK, that's a bit different but interesting....
>
> > @ 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> > No nothing abnormal
> >
> > @ 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> > Ovirt tells me "no Storage space error". Which is something like the disk is growing to fast i think. I use Snapshots so on heavy write the disk has to grow a lot.
> > Sometimes the VM is paused and resumed from ovirt. Sometimes the VM stays offline.
>
> OK, that's interesting, because you may be hitting the following bug; http://lists.nongnu.org/archive/html/qemu-block/2015-11/msg00585.html
>
> whose fix coincidentally just got accepted today; it's related to error cases with error=stop which you are using.
>
> Do you think you're only hitting these crashes on VMs that have been paused because of these space errors?
>
> > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
> >
> > Ok will try that and report
>
> Thanks,
>
> Dave
>
> >
> > Thx Christian
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> > Gesendet: Dienstag, 17. November 2015 10:59
> > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>;
> > stefanha@redhat.com
> > Betreff: Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in
> > libtcmalloc
> >
> > * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > > Hi,
> > > Dan sent me over to you,
> > > please let me know if i can provide additional informations
> >
> > Hi Christian,
> > Thanks for reporting this,
> >
> > > Softwareversions:
> > > ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso
> > >
> > > qemu-img-ev-2.3.0-29.1.el7.x86_64
> > > qemu-kvm-ev-2.3.0-29.1.el7.x86_64
> > > qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
> > > qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
> > > ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
> > > kernel-3.10.0-229.14.1.el7.x86_64
> > > gperftools-libs-2.4-7.el7.x86_64
> > >
> > > Commandline:
> > > /usr/libexec/qemu-kvm -name myvmname -S -machine
> > > rhel6.5.0,accel=kvm,usb=off -cpu Westmere -m 7168 -realtime
> > > mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
> > > 5b6b8899-5a9d-4c07-a6aa-6171527ad319 -smbios
> > > type=1,manufacturer=oVirt,product=oVirt
> > > Node,version=3.6-0.999.201510221942.el7.centos,serial=30343536-3138-
> > > 5A
> > > 43-4A34-323630303253,uuid=5b6b8899-5a9d-4c07-a6aa-6171527ad319
> > > -nographic -no-user-config -nodefaults -chardev
> > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/myvmname.monitor,se
> > > rv er,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc
> > > base=2015-11-15T20:04:35,driftfix=slew -global
> > > kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
> > > strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
> > > -device
> > > virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
> > > virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
> > > -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
> > > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> > > -drive
> > > file=/rhev/data-center/00000002-0002-0002-0002-0000000000e2/5df61b84
> > > -8
> > > 746-4460-b148-65cc0eb8d29c/images/8202b81d-6191-495f-8c9d-7d90baffae
> > > cf
> > > /d7665e07-1786-4051-aa26-0a3e1c9d2574,if=none,id=drive-virtio-disk0,
> > > fo
> > > rmat=qcow2,serial=8202b81d-6191-495f-8c9d-7d90baffaecf,cache=none,we
> > > rr or=stop,rerror=stop,aio=native -device
> > > virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,
> > > id
> > > =virtio-disk0,bootindex=1 -netdev
> > > tap,fd=39,id=hostnet0,vhost=on,vhostfd=65 -device
> > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:83:a2:0e,bus=pci
> > > .0
> > > ,addr=0x3 -chardev
> > > socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/5b6b8899-
> > > 5a 9d-4c07-a6aa-6171527ad319.com.redhat.rhevm.vdsm,server,nowait
> > > -device
> > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=cha
> > > nn el0,name=com.redhat.rhevm.vdsm -chardev
> > > socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/5b6b8899-
> > > 5a 9d-4c07-a6aa-6171527ad319.org.qemu.guest_agent.0,server,nowait
> > > -device
> > > virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=cha
> > > nn
> > > el1,name=org.qemu.guest_agent.0 -device
> > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
> > >
> > > Stack Trace:
> > >
> > > gdb --batch /usr/libexec/qemu-kvm core.14750.1447544080.dump -ex "set pagination off" -ex "thread apply all bt"
> >
> > Can you please use a 'thread apply all bt full' the full gives a little more info.
> > Also, if you've not already got it installed can you please install the debuginfo package for qemu, it gives a lot more information in backtraces.
> >
> > > Thread 1 (Thread 0x7fa8b16afc00 (LWP 14750)):
> > > #0 0x00007fa8ad2febe1 in tc_malloc () from /lib64/libtcmalloc.so.4
> > > #1 0x00007fa8b186b489 in malloc_and_trace ()
> > > #2 0x00007fa8afbc047f in g_malloc () from /lib64/libglib-2.0.so.0
> > > #3 0x00007fa8afbd666e in g_slice_alloc () from
> > > /lib64/libglib-2.0.so.0
> > > #4 0x00007fa8b17cbffd in virtio_blk_handle_output ()
> > > #5 0x00007fa8b197e6b6 in qemu_iohandler_poll ()
> > > #6 0x00007fa8b197e296 in main_loop_wait ()
> > > #7 0x00007fa8b177da4e in main ()
> >
> > Does this part always look the same in your backtraces?
> > The segfault in tc_malloc is probably due to a heap corruption, or double free or similar - although it can be a bit tricky to find out what did it, since the corruption might have happened a bit before the place it crashed.
> >
> > Some other ideas:
> > 1) Was there anything nasty in the /var/log/libvirt/qemu/yourvmname.log ?
> > 2) Did you hit any IO errors and need to tell the VM to continue after a problem?
> > 3) If this is pretty repeatable, then it would be interesting to try changing to a different
> > disk emulation and see if the problem goes away - e.g. virtio-scsi would be a good one to try.
> >
> > Dave
> > >
> > >
> > > Thx Christian
> > >
> > > -----Ursprüngliche Nachricht-----
> > > Von: Dan Kenigsberg [mailto:danken@redhat.com]
> > > Gesendet: Freitag, 13. November 2015 20:00
> > > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> > > Cc: 'users@ovirt.org' <users@ovirt.org>
> > > Betreff: Re: [ovirt-users] Segmentation fault in libtcmalloc
> > >
> > > On Fri, Nov 13, 2015 at 07:56:14AM +0000, Grundmann, Christian wrote:
> > > > Hi,
> > > > i am using "ovirt-node-iso-3.6-0.999.201510221942.el7.centos.iso"
> > > > (is there something better to use?) fort he nodes, and have random
> > > > crashes of VMs The dumps are always the Same
> > > >
> > > > gdb --batch /usr/libexec/qemu-kvm core.45902.1447199164.dump
> > > > [Thread debugging using libthread_db enabled] Using host
> > > > libthread_db library "/lib64/libthread_db.so.1".
> > > > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine rhel6.5.0,accel=kvm,usb=o'.
> > > > Program terminated with signal 11, Segmentation fault.
> > > > #0 0x00007f0c559c4353 in
> > > > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::
> > > > Fr eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> > > >
> > > >
> > > > Didn't have the Problem with 3.5 el6 nodes, so don't no if ist
> > > > centos7 or 3.6
> > >
> > > Due to the low-leveled-ness of the problem, I'd guess it's a qemu//lib64/libtcmalloc malloc bug, and not directly related to ovirt.
> > >
> > > Please report the precise version of qemu,kernel,libvirt and gperftools-libs to qemu-devel mailing list and the complete stack trace and qemu command line, if possible.
> > >
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
next prev parent reply other threads:[~2015-11-17 14:42 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <6A17C71B52524C408E7AAF69103E9E490F14400C@fabamailserver.fabagl.fabasoft.com>
[not found] ` <20151113190014.GB18986@redhat.com>
2015-11-16 8:11 ` [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc Grundmann, Christian
2015-11-17 9:59 ` Dr. David Alan Gilbert
2015-11-17 10:36 ` Grundmann, Christian
2015-11-17 11:36 ` Dr. David Alan Gilbert
2015-11-17 14:11 ` Grundmann, Christian
2015-11-17 14:20 ` Grundmann, Christian
2015-11-17 14:42 ` Dr. David Alan Gilbert [this message]
2015-11-19 16:00 ` Grundmann, Christian
2015-11-19 17:02 ` Paolo Bonzini
2015-12-03 8:18 ` Grundmann, Christian
2015-12-03 9:04 ` Dr. David Alan Gilbert
2015-12-03 9:07 ` Grundmann, Christian
2015-12-10 12:38 ` Dr. David Alan Gilbert
2015-12-10 13:18 ` Markus Armbruster
2015-12-10 13:37 ` Grundmann, Christian
2015-11-20 19:06 ` Dr. David Alan Gilbert
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20151117144225.GH2498@work-vm \
--to=dgilbert@redhat.com \
--cc=Christian.Grundmann@fabasoft.com \
--cc=qemu-devel@nongnu.org \
--cc=stefanha@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.