qemu-kvm crash with "double free or corruption"

qemu-kvm crash with "double free or corruption"

[Conor Murphy]

Hi,

I'm trying to write a vioblk driver for Solaris. I've gotten it to the point
where the devices are visible to Solaris and can create and FDISK partition
table and label it.

However, when I try an use newfs to create a filesystem, the VM crashes with the
following in the log

*** glibc detected *** /usr/bin/qemu-kvm: double free or corruption (!prev):
0x00007fda4c000a00 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3d6da76d63]
/usr/bin/qemu-kvm[0x43d6ce]
/lib64/libpthread.so.0[0x3d6e206ccb]
/lib64/libc.so.6(clone+0x6d)[0x3d6dae0c2d]

I'm using qemu-kvm-0.14.0-4.fc14.x86_64

Any suggestions on how to debug this?

Thanks,
Conor

Re: qemu-kvm crash with "double free or corruption"

[Gleb Natapov]

On Thu, Mar 24, 2011 at 11:55:06AM +0000, Conor Murphy wrote:
> Hi,
> 
> I'm trying to write a vioblk driver for Solaris. I've gotten it to the point
> where the devices are visible to Solaris and can create and FDISK partition
> table and label it.
> 
> However, when I try an use newfs to create a filesystem, the VM crashes with the
> following in the log
> 
> *** glibc detected *** /usr/bin/qemu-kvm: double free or corruption (!prev):
> 0x00007fda4c000a00 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3d6da76d63]
> /usr/bin/qemu-kvm[0x43d6ce]
> /lib64/libpthread.so.0[0x3d6e206ccb]
> /lib64/libc.so.6(clone+0x6d)[0x3d6dae0c2d]
> 
> I'm using qemu-kvm-0.14.0-4.fc14.x86_64
> 
> Any suggestions on how to debug this?
> 
Compile with debug symbols without optimization and run inside gdb to
get better back trace.

--
			Gleb.

Re: qemu-kvm crash with "double free or corruption"

[Conor Murphy]

Forgot to mention that when I attached gdb to the qemu-kvm process before
running the newfs in the guest, the crash does not happen

Some sort of race condition?

Thanks,
Conor

Re: qemu-kvm crash with

[Conor Murphy]

Built with --enable-debug

Running under gdb gives

(gdb) where
#0  0x0000003d6da330c5 in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x0000003d6da34a76 in abort () at abort.c:92
#2  0x0000003d6da6fcfb in __libc_message (do_abort=2, 
fmt=0x3d6db5ea98 "***
glibc detected *** %s: %s: 0x%s ***\n") at
../sysdeps/unix/sysv/linux/libc_fatal.c:186
#3  0x0000003d6da76d63 in malloc_printerr (av=<value optimized out>,
p=0x7fa24c0009f0, have_lock=0) at malloc.c:6283
#4  _int_free (av=<value optimized out>, p=0x7fa24c0009f0, have_lock=0) at
malloc.c:4795
#5  0x00000000004a18fe in qemu_vfree (ptr=0x7fa24c000a00) at oslib-posix.c:76
#6  0x000000000045af3d in handle_aiocb_rw (aiocb=0x7fa2dc034cd0) at
posix-aio-compat.c:301
#7  0x000000000045b09a in aio_thread (unused=0x0) at posix-aio-compat.c:341
#8  0x0000003d6e206ccb in start_thread (arg=0x7fa2e1654700) 
at pthread_create.c:301
#9  0x0000003d6dae0c2d in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb) 


/Conor

Re: qemu-kvm crash with

[Stefan Hajnoczi]

On Thu, Mar 24, 2011 at 1:38 PM, Conor Murphy
<conor_murphy_virt@hotmail.com> wrote:
> #4  _int_free (av=<value optimized out>, p=0x7fa24c0009f0, have_lock=0) at
> malloc.c:4795
> #5  0x00000000004a18fe in qemu_vfree (ptr=0x7fa24c000a00) at oslib-posix.c:76
> #6  0x000000000045af3d in handle_aiocb_rw (aiocb=0x7fa2dc034cd0) at
> posix-aio-compat.c:301

I don't see a way for a double-free to occur so I think something has
overwritten the memory preceeding the allocated buffer.

In gdb you could inspect the aiocb structure to look at its aio_iov[],
aio_niov, and aio_nbytes fields.  They might be invalid or corrupted
somehow.

You could also dump out the memory before 0x7fa24c000a00, specifically
0x7fa24c0009f0, to see if you notice any pattern or printable
characters that give a clue as to what has corrupted the memory here.

Are you running qemu-kvm.git/master?

Stefan

Re: qemu-kvm crash with

[Conor Murphy]

Hi,

The content of aiocb

(gdb) print *aiocb
$1 = {common = {pool = 0x9aced0, bs = 0x1270230, cb = 0x45591f <multiwrite_cb>,
 opaque = 0x7f54b0034f60, next = 0x0}, aio_fildes = 16, 
 {aio_iov = 0x7f54b006cd48, aio_ioctl_buf = 0x7f54b006cd48}, 
 aio_niov = 17, 
  aio_nbytes = 65024, ev_signo = 12, aio_offset = 1081344, 
 node = {tqe_next = 0x0, tqe_prev = 0x9f10a0}, 
  aio_type = 2, ret = -115, active = 1, next = 0x7f54b00409f0, 
 async_context_id = 0}

(gdb) print aiocb->aio_iov[0]
$2 = {iov_base = 0x7f54a9f141f8, iov_len = 3592}
(gdb) print aiocb->aio_iov[1]
$3 = {iov_base = 0x7f54a27d5000, iov_len = 4096}
(gdb) print aiocb->aio_iov[2]
$4 = {iov_base = 0x7f54a30d6000, iov_len = 4096}
(gdb) print aiocb->aio_iov[3]
$5 = {iov_base = 0x7f5433a57000, iov_len = 4096}
(gdb) print aiocb->aio_iov[5]
$6 = {iov_base = 0x7f54a2fd9000, iov_len = 4096}
(gdb) print aiocb->aio_iov[6]
$7 = {iov_base = 0x7f54a275a000, iov_len = 4096}
(gdb) print aiocb->aio_iov[7]
$8 = {iov_base = 0x7f54a2fdb000, iov_len = 4096}
(gdb) print aiocb->aio_iov[8]
$9 = {iov_base = 0x7f54ab55c000, iov_len = 4096}
(gdb) print aiocb->aio_iov[9]
$10 = {iov_base = 0x7f543639d000, iov_len = 4096}
(gdb) print aiocb->aio_iov[10]
$11 = {iov_base = 0x7f543115e000, iov_len = 4096}
(gdb) print aiocb->aio_iov[11]
$12 = {iov_base = 0x7f54361df000, iov_len = 4096}
(gdb) print aiocb->aio_iov[12]
$13 = {iov_base = 0x7f54a9620000, iov_len = 4096}
(gdb) print aiocb->aio_iov[13]
$14 = {iov_base = 0x7f54a23a1000, iov_len = 4096}
(gdb) print aiocb->aio_iov[14]
$15 = {iov_base = 0x7f54ae122000, iov_len = 4096}
(gdb) print aiocb->aio_iov[15]
$16 = {iov_base = 0x7f54312a3000, iov_len = 4096}
(gdb) print aiocb->aio_iov[16]
$17 = {iov_base = 0x7f54a28a4000, iov_len = 503}
(gdb) 

The one thing that seems odd is that the sum of iov_len is 65535 
which is > then aio_nbtyes of 65024

Does this mean the code ends up writing past the end of buf?

/Conor