Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> Hi again,
> got a Segfault today without virtio :-( (one IDE Disk and one virtio-scsi)
> 
> Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine pc-i440fx-rhel7.2.0,accel='.

Can you confirm the package version you were using; if you're running the pc-i440fx-rhel7.2.0 machine
type it must be pretty new.

Dave

> Program terminated with signal 11, Segmentation fault.
> #0  0x00007fb299cbd3ab in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> 
> Thread 6 (Thread 0x7fb28d0c5700 (LWP 29423)):
> #0  0x00007fb29cc85ac3 in pread64 () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1  0x00007fb29e37c2a3 in pread (__offset=<optimized out>, __nbytes=<optimized out>, __buf=0x7fb2a3e21a00, __fd=<optimized out>) at /usr/include/bits/unistd.h:99
> No locals.
> #2  handle_aiocb_rw_linear (aiocb=aiocb@entry=0x7fb2a1474340, buf=buf@entry=0x7fb2a3e21a00 "QF", <incomplete sequence \373>) at block/raw-posix.c:909
>         offset = 0
>         len = <optimized out>
> #3  0x00007fb29e37c3d1 in handle_aiocb_rw (aiocb=0x7fb2a1474340) at block/raw-posix.c:992
>         nbytes = <optimized out>
>         buf = 0x7fb2a3e21a00 "QF", <incomplete sequence \373>
>         __PRETTY_FUNCTION__ = "handle_aiocb_rw"
> #4  0x00007fb29e37d945 in aio_worker (arg=0x7fb2a1474340) at block/raw-posix.c:1204
>         aiocb = 0x7fb2a1474340
>         ret = 0
> #5  0x00007fb29e33d91b in worker_thread (opaque=0x7fb2a148d450) at thread-pool.c:105
>         req = 0x7fb2a1474b30
>         ret = <optimized out>
>         pool = 0x7fb2a148d450
> #6  0x00007fb29cc7edf5 in start_thread (arg=0x7fb28d0c5700) at pthread_create.c:308
>         __res = <optimized out>
>         pd = 0x7fb28d0c5700
>         now = <optimized out>
>         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404847302400, -4522449750849005939, 0, 140404847303104, 140404847302400, 26, 4492373549408278157, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
>         not_first_call = <optimized out>
>         pagesize_m1 = <optimized out>
>         sp = <optimized out>
>         freesize = <optimized out>
> #7  0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
> 
> Thread 5 (Thread 0x7fb104fff700 (LWP 29084)):
> #0  0x00007fb296881b7d in poll () at ../sysdeps/unix/syscall-template.S:81
> No locals.
> #1  0x00007fb2977d6fe7 in red_worker_main () from /lib64/libspice-server.so.1
> No symbol table info available.
> #2  0x00007fb29cc7edf5 in start_thread (arg=0x7fb104fff700) at pthread_create.c:308
>         __res = <optimized out>
>         pd = 0x7fb104fff700
>         now = <optimized out>
>         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140398269822720, -4522449750849005939, 0, 140398269823424, 140398269822720, 140405245697216, 4494326442046740109, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
>         not_first_call = <optimized out>
>         pagesize_m1 = <optimized out>
>         sp = <optimized out>
>         freesize = <optimized out>
> #3  0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
> 
> Thread 4 (Thread 0x7fb28c8c4700 (LWP 29081)):
> #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> No locals.
> #1  0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132
>         err = <optimized out>
>         __func__ = "qemu_cond_wait"
> #2  0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912
> No locals.
> #3  qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d2e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949
>         cpu = 0x7fb2a3d2e000
>         r = <optimized out>
> #4  0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c8c4700) at pthread_create.c:308
>         __res = <optimized out>
>         pd = 0x7fb28c8c4700
>         now = <optimized out>
>         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404838909696, -4522449750849005939, 0, 140404838910400, 140404838909696, 140735272359936, 4492374652678002317, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
>         not_first_call = <optimized out>
>         pagesize_m1 = <optimized out>
>         sp = <optimized out>
>         freesize = <optimized out>
> #5  0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
> 
> Thread 3 (Thread 0x7fb28c0c3700 (LWP 29082)):
> #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> No locals.
> #1  0x00007fb29e3b9f79 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x7fb29e866d40 <qemu_global_mutex>) at util/qemu-thread-posix.c:132
>         err = <optimized out>
>         __func__ = "qemu_cond_wait"
> #2  0x00007fb29e172d3b in qemu_kvm_wait_io_event (cpu=<optimized out>) at /usr/src/debug/qemu-2.3.0/cpus.c:912
> No locals.
> #3  qemu_kvm_cpu_thread_fn (arg=0x7fb2a3d7e000) at /usr/src/debug/qemu-2.3.0/cpus.c:949
>         cpu = 0x7fb2a3d7e000
>         r = <optimized out>
> #4  0x00007fb29cc7edf5 in start_thread (arg=0x7fb28c0c3700) at pthread_create.c:308
>         __res = <optimized out>
>         pd = 0x7fb28c0c3700
>         now = <optimized out>
>         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404830516992, -4522449750849005939, 0, 140404830517696, 140404830516992, 140735272359936, 4492375751652759181, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
>         not_first_call = <optimized out>
>         pagesize_m1 = <optimized out>
>         sp = <optimized out>
>         freesize = <optimized out>
> #5  0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
> 
> Thread 2 (Thread 0x7fb28e6ad700 (LWP 29077)):
> #0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
> No locals.
> #1  0x00007fb29e3ba272 in futex_wait (val=4294967295, ev=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:301
> No locals.
> #2  qemu_event_wait (ev=ev@entry=0x7fb29ec84f44 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399
>         value = <optimized out>
> #3  0x00007fb29e3c8526 in call_rcu_thread (opaque=<optimized out>) at util/rcu.c:233
>         tries = 0
>         n = <optimized out>
>         node = <optimized out>
> #4  0x00007fb29cc7edf5 in start_thread (arg=0x7fb28e6ad700) at pthread_create.c:308
>         __res = <optimized out>
>         pd = 0x7fb28e6ad700
>         now = <optimized out>
>         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140404870272768, -4522449750849005939, 0, 140404870273472, 140404870272768, 140405136150080, 4492370572995942029, 4492409237274449549}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
>         not_first_call = <optimized out>
>         pagesize_m1 = <optimized out>
>         sp = <optimized out>
>         freesize = <optimized out>
> #5  0x00007fb29688c1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> No locals.
> 
> Thread 1 (Thread 0x7fb29e07cc00 (LWP 29076)):
> #0  0x00007fb299cbd3ab in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #1  0x00007fb299cbd47b in tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #2  0x00007fb299ccc070 in tc_free () from /lib64/libtcmalloc.so.4
> No symbol table info available.
> #3  0x00007fb29c58d58f in g_free () from /lib64/libglib-2.0.so.0
> No symbol table info available.
> #4  0x00007fb29e3b7721 in parser_context_free (ctxt=0x7fb2a531e0c0) at qobject/json-parser.c:358
>         i = <optimized out>
> #5  json_parser_parse_err (tokens=<optimized out>, ap=ap@entry=0x0, errp=errp@entry=0x0) at qobject/json-parser.c:710
>         result = 0x7fb2a4bdf600
> #6  0x00007fb29e3b7767 in json_parser_parse (tokens=<optimized out>, ap=ap@entry=0x0) at qobject/json-parser.c:694
> No locals.
> #7  0x00007fb29e176e04 in handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5068
>         err = <optimized out>
>         obj = <optimized out>
>         input = 0x0
>         args = 0x0
>         cmd_name = <optimized out>
>         mon = 0x7fb2a153e140
> #8  0x00007fb29e3b64f2 in json_message_process_token (lexer=0x7fb2a1460040, token=0x7fb2a1424880, type=JSON_OPERATOR, x=49, y=104) at qobject/json-streamer.c:87
>         parser = 0x7fb2a1460038
>         dict = 0x7fb2a3e27200
> #9  0x00007fb29e3c891f in json_lexer_feed_char (lexer=lexer@entry=0x7fb2a1460040, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303
>         new_state = 100
> #10 0x00007fb29e3c89ee in json_lexer_feed (lexer=0x7fb2a1460040, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356
>         err = <optimized out>
>         i = <optimized out>
> #11 0x00007fb29e3b6689 in json_message_parser_feed (parser=<optimized out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110
> No locals.
> #12 0x00007fb29e1758cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134
>         old_mon = 0x0
> #13 0x00007fb29e2321b0 in qemu_chr_be_write (len=<optimized out>, buf=0x7fff7bea8a30 "}\212\352{\377\177", s=0x7fb2a14442e0) at qemu-char.c:305
> No locals.
> #14 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7fb2a14442e0) at qemu-char.c:2870
>         chr = 0x7fb2a14442e0
>         s = 0x7fb2a14363f0
>         buf = "}\212\352{\377\177\000\000\360`;\236\262\177\000\000\030\003\000\000\000\000\000\000\205N;\236\262\177\000\000\240LB\241\262\177\000\000\263E;\236\262\177\000\000\240LB\241\262\177", '\000' <repeats 18 times>, "\360\017c\244\262\177\000\000\300\213\352{\377\177\000\000\000\000\000\000\000\000\000\000\060\356t\245\262\177\000\000\000$ᤲ\177\000\000@\232\352{\377\177\000\000H\022\212\226\262\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\220\213\352{\377\177\000\000Њ\352{\377\177\000\000\r\000\000\000\000\000\000\000\340\234\177\000\000\000d\023\245\262\177\000\000`\376\061\245\262\177\000\000Q\000\000\000\000\000\000\000\325b\004\000\000\000\000\000"...
>         len = <optimized out>
>         size = <optimized out>
> #15 0x00007fb29c58799a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
> No symbol table info available.
> #16 0x00007fb29e34b288 in glib_pollfds_poll () at main-loop.c:209
>         context = 0x7fb2a1491140
>         pfds = <optimized out>
> #17 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254
>         ret = 2
>         spin_counter = 0
> #18 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503
>         ret = 2
>         timeout = 4294967295
>         timeout_ns = <optimized out>
> #19 0x00007fb29e14aa4e in main_loop () at vl.c:1818
>         nonblocking = <optimized out>
>         last_io = 2
> #20 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
>         i = <optimized out>
>         snapshot = <optimized out>
>         linux_boot = <optimized out>
>         initrd_filename = <optimized out>
>         kernel_filename = <optimized out>
>         kernel_cmdline = <optimized out>
>         boot_order = 0x7fb29e3dda67 "cad"
>         boot_once = 0x0
>         cyls = <optimized out>
>         heads = <optimized out>
>         secs = <optimized out>
>         translation = <optimized out>
>         hda_opts = <optimized out>
>         opts = <optimized out>
>         machine_opts = <optimized out>
>         icount_opts = <optimized out>
>         olist = <optimized out>
>         optind = 78
>         optarg = 0x7fb2a14ef8c0 "pc-i440fx-rhel7.2.0"
>         loadvm = <optimized out>
>         machine_class = <optimized out>
>         cpu_model = <optimized out>
>         vga_model = 0x0
>         qtest_chrdev = <optimized out>
>         qtest_log = <optimized out>
>         pid_file = <optimized out>
>         incoming = <optimized out>
>         show_vnc_port = <optimized out>
>         defconfig = <optimized out>
>         userconfig = 111
>         log_mask = <optimized out>
>         log_file = <optimized out>
>         mem_trace = {malloc = 0x7fb29e238480 <malloc_and_trace>, realloc = 0x7fb29e238460 <realloc_and_trace>, free = 0x7fb29e238450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
>         trace_events = <optimized out>
>         trace_file = <optimized out>
>         maxram_size = <optimized out>
>         ram_slots = <optimized out>
>         vmstate_dump_file = <optimized out>
>         main_loop_err = 0x0
>         __func__ = "main"
> 
> 
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] Im Auftrag von Paolo Bonzini
> Gesendet: Donnerstag, 19. November 2015 18:02
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>; 'Dr. David Alan Gilbert' <dgilbert@redhat.com>
> Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
> Betreff: Re: WG: [ovirt-users] Segmentation fault in libtcmalloc
> 
> 
> 
> On 19/11/2015 17:00, Grundmann, Christian wrote:
> > Hi, it seems that using virtio-scsi did the trick, But now the VMs are 
> > pausing without an coredump, so the underlying Problem (no storage 
> > Error) is not fixed, As I am using Snapshots (and so the disks have to 
> > grow very fast) I try if tuning "volume_utilization_percent" and 
> > "volume_utilization_chunk_mb" will help 
> > (https://access.redhat.com/solutions/130843)
> 
> The fix for virtio-blk is probably this patch:
> http://article.gmane.org/gmane.comp.emulators.qemu.block/6380/raw
> 
> Paolo
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK