Re: [Qemu-devel] WG: [ovirt-users] Segmentation fault in libtcmalloc

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> Hi,
> 
> qemu-img-ev-2.3.0-29.1.el7.x86_64
> libvirt-daemon-driver-qemu-1.2.8-16.el7_1.4.x86_64
> qemu-kvm-ev-2.3.0-29.1.el7.x86_64
> qemu-kvm-common-ev-2.3.0-29.1.el7.x86_64
> ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
> qemu-kvm-tools-ev-2.3.0-29.1.el7.x86_64
> 
> 
> it seems pc-i440fx-rhel7.2.0 is the default for ovirt 3.6 
> 
> I tried using only virtio-scsi disk but the VM wont boot (not bootable device) so i used IDE for the boot disk.

I think this seg is actually quite different - although it depends where the actual corruption
happened -  looking at the backtrace again the failing thread wasn't the io thread; it
failed in a call from the json parser in the main thread.

Dave


> a
> Thx Christian
> 
> -----Ursprüngliche Nachricht-----
> Von: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] 
> Gesendet: Donnerstag, 03. Dezember 2015 10:04
> An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>
> Cc: 'Paolo Bonzini' <pbonzini@redhat.com>; 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; stefanha@redhat.com
> Betreff: Re: AW: WG: [ovirt-users] Segmentation fault in libtcmalloc
> 
> * Grundmann, Christian (Christian.Grundmann@fabasoft.com) wrote:
> > Hi again,
> > got a Segfault today without virtio :-( (one IDE Disk and one 
> > virtio-scsi)
> > 
> > Core was generated by `/usr/libexec/qemu-kvm -name vmname -S -machine pc-i440fx-rhel7.2.0,accel='.
> 
> Can you confirm the package version you were using; if you're running the pc-i440fx-rhel7.2.0 machine type it must be pretty new.
> 
> Dave
> 
> > Program terminated with signal 11, Segmentation fault.
> > #0  0x00007fb299cbd3ab in 
> > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::Fr
> > eeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4

<deleted>

It looks like it's the main thread in the json parser:

> > Thread 1 (Thread 0x7fb29e07cc00 (LWP 29076)):
> > #0  0x00007fb299cbd3ab in 
> > tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> > #1  0x00007fb299cbd47b in 
> > tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) () from /lib64/libtcmalloc.so.4 No symbol table info available.
> > #2  0x00007fb299ccc070 in tc_free () from /lib64/libtcmalloc.so.4 No 
> > symbol table info available.
> > #3  0x00007fb29c58d58f in g_free () from /lib64/libglib-2.0.so.0 No 
> > symbol table info available.
> > #4  0x00007fb29e3b7721 in parser_context_free (ctxt=0x7fb2a531e0c0) at qobject/json-parser.c:358
> >         i = <optimized out>
> > #5  json_parser_parse_err (tokens=<optimized out>, ap=ap@entry=0x0, errp=errp@entry=0x0) at qobject/json-parser.c:710
> >         result = 0x7fb2a4bdf600
> > #6  0x00007fb29e3b7767 in json_parser_parse (tokens=<optimized out>, 
> > ap=ap@entry=0x0) at qobject/json-parser.c:694 No locals.
> > #7  0x00007fb29e176e04 in handle_qmp_command (parser=<optimized out>, tokens=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5068
> >         err = <optimized out>
> >         obj = <optimized out>
> >         input = 0x0
> >         args = 0x0
> >         cmd_name = <optimized out>
> >         mon = 0x7fb2a153e140
> > #8  0x00007fb29e3b64f2 in json_message_process_token (lexer=0x7fb2a1460040, token=0x7fb2a1424880, type=JSON_OPERATOR, x=49, y=104) at qobject/json-streamer.c:87
> >         parser = 0x7fb2a1460038
> >         dict = 0x7fb2a3e27200
> > #9  0x00007fb29e3c891f in json_lexer_feed_char (lexer=lexer@entry=0x7fb2a1460040, ch=<optimized out>, flush=flush@entry=false) at qobject/json-lexer.c:303
> >         new_state = 100
> > #10 0x00007fb29e3c89ee in json_lexer_feed (lexer=0x7fb2a1460040, buffer=<optimized out>, size=<optimized out>) at qobject/json-lexer.c:356
> >         err = <optimized out>
> >         i = <optimized out>
> > #11 0x00007fb29e3b6689 in json_message_parser_feed (parser=<optimized 
> > out>, buffer=<optimized out>, size=<optimized out>) at qobject/json-streamer.c:110 No locals.
> > #12 0x00007fb29e1758cf in monitor_control_read (opaque=<optimized out>, buf=<optimized out>, size=<optimized out>) at /usr/src/debug/qemu-2.3.0/monitor.c:5134
> >         old_mon = 0x0
> > #13 0x00007fb29e2321b0 in qemu_chr_be_write (len=<optimized out>, 
> > buf=0x7fff7bea8a30 "}\212\352{\377\177", s=0x7fb2a14442e0) at qemu-char.c:305 No locals.
> > #14 tcp_chr_read (chan=<optimized out>, cond=<optimized out>, opaque=0x7fb2a14442e0) at qemu-char.c:2870
> >         chr = 0x7fb2a14442e0
> >         s = 0x7fb2a14363f0
> >         buf = "}\212\352{\377\177\000\000\360`;\236\262\177\000\000\030\003\000\000\000\000\000\000\205N;\236\262\177\000\000\240LB\241\262\177\000\000\263E;\236\262\177\000\000\240LB\241\262\177", '\000' <repeats 18 times>, "\360\017c\244\262\177\000\000\300\213\352{\377\177\000\000\000\000\000\000\000\000\000\000\060\356t\245\262\177\000\000\000$ᤲ\177\000\000@\232\352{\377\177\000\000H\022\212\226\262\177\000\000]\000\000\000\000\000\000\000\060\000\000\000\060\000\000\000\220\213\352{\377\177\000\000Њ\352{\377\177\000\000\r\000\000\000\000\000\000\000\340\234\177\000\000\000d\023\245\262\177\000\000`\376\061\245\262\177\000\000Q\000\000\000\000\000\000\000\325b\004\000\000\000\000\000"...
> >         len = <optimized out>
> >         size = <optimized out>
> > #15 0x00007fb29c58799a in g_main_context_dispatch () from 
> > /lib64/libglib-2.0.so.0 No symbol table info available.
> > #16 0x00007fb29e34b288 in glib_pollfds_poll () at main-loop.c:209
> >         context = 0x7fb2a1491140
> >         pfds = <optimized out>
> > #17 os_host_main_loop_wait (timeout=<optimized out>) at main-loop.c:254
> >         ret = 2
> >         spin_counter = 0
> > #18 main_loop_wait (nonblocking=<optimized out>) at main-loop.c:503
> >         ret = 2
> >         timeout = 4294967295
> >         timeout_ns = <optimized out>
> > #19 0x00007fb29e14aa4e in main_loop () at vl.c:1818
> >         nonblocking = <optimized out>
> >         last_io = 2
> > #20 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4394
> >         i = <optimized out>
> >         snapshot = <optimized out>
> >         linux_boot = <optimized out>
> >         initrd_filename = <optimized out>
> >         kernel_filename = <optimized out>
> >         kernel_cmdline = <optimized out>
> >         boot_order = 0x7fb29e3dda67 "cad"
> >         boot_once = 0x0
> >         cyls = <optimized out>
> >         heads = <optimized out>
> >         secs = <optimized out>
> >         translation = <optimized out>
> >         hda_opts = <optimized out>
> >         opts = <optimized out>
> >         machine_opts = <optimized out>
> >         icount_opts = <optimized out>
> >         olist = <optimized out>
> >         optind = 78
> >         optarg = 0x7fb2a14ef8c0 "pc-i440fx-rhel7.2.0"
> >         loadvm = <optimized out>
> >         machine_class = <optimized out>
> >         cpu_model = <optimized out>
> >         vga_model = 0x0
> >         qtest_chrdev = <optimized out>
> >         qtest_log = <optimized out>
> >         pid_file = <optimized out>
> >         incoming = <optimized out>
> >         show_vnc_port = <optimized out>
> >         defconfig = <optimized out>
> >         userconfig = 111
> >         log_mask = <optimized out>
> >         log_file = <optimized out>
> >         mem_trace = {malloc = 0x7fb29e238480 <malloc_and_trace>, realloc = 0x7fb29e238460 <realloc_and_trace>, free = 0x7fb29e238450 <free_and_trace>, calloc = 0x0, try_malloc = 0x0, try_realloc = 0x0}
> >         trace_events = <optimized out>
> >         trace_file = <optimized out>
> >         maxram_size = <optimized out>
> >         ram_slots = <optimized out>
> >         vmstate_dump_file = <optimized out>
> >         main_loop_err = 0x0
> >         __func__ = "main"
> > 
> > 
> > 
> > 
> > -----Ursprüngliche Nachricht-----
> > Von: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] Im Auftrag von 
> > Paolo Bonzini
> > Gesendet: Donnerstag, 19. November 2015 18:02
> > An: Grundmann, Christian <Christian.Grundmann@fabasoft.com>; 'Dr. 
> > David Alan Gilbert' <dgilbert@redhat.com>
> > Cc: 'qemu-devel@nongnu.org' <qemu-devel@nongnu.org>; 
> > stefanha@redhat.com
> > Betreff: Re: WG: [ovirt-users] Segmentation fault in libtcmalloc
> > 
> > 
> > 
> > On 19/11/2015 17:00, Grundmann, Christian wrote:
> > > Hi, it seems that using virtio-scsi did the trick, But now the VMs 
> > > are pausing without an coredump, so the underlying Problem (no 
> > > storage
> > > Error) is not fixed, As I am using Snapshots (and so the disks have 
> > > to grow very fast) I try if tuning "volume_utilization_percent" and 
> > > "volume_utilization_chunk_mb" will help
> > > (https://access.redhat.com/solutions/130843)
> > 
> > The fix for virtio-blk is probably this patch:
> > http://article.gmane.org/gmane.comp.emulators.qemu.block/6380/raw
> > 
> > Paolo
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK