qemu-kvm-1.2.0: double free or corruption

qemu-kvm-1.2.0: double free or corruption

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 5952 bytes --]

Hi,
on one of our servers, windows 2008 KVM suddenly crashed. I see following
in libvirt log:

*** glibc detected *** /usr/bin/qemu-kvm: double free or corruption (!prev): 0x00007fc634008cd0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75916)[0x7fc9026f4916]
/lib64/libc.so.6(+0x78443)[0x7fc9026f7443]
/usr/bin/qemu-kvm(+0x1faeb1)[0x7fc907187eb1]
/usr/bin/qemu-kvm(+0x1f0e1a)[0x7fc90717de1a]
/usr/bin/qemu-kvm(+0x1fb681)[0x7fc907188681]
/usr/bin/qemu-kvm(+0xed6a7)[0x7fc90707a6a7]
/usr/bin/qemu-kvm(+0x195c31)[0x7fc907122c31]
/usr/bin/qemu-kvm(main+0x106c)[0x7fc90711e5fc]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fc90269dcdd]
/usr/bin/qemu-kvm(+0x749f9)[0x7fc9070019f9]
======= Memory map: ========
7fc5d0000000-7fc5d0021000 rw-p 00000000 00:00 0 
7fc5d0021000-7fc5d4000000 ---p 00000000 00:00 0 
7fc5d4000000-7fc5d4021000 rw-p 00000000 00:00 0 
7fc5d4021000-7fc5d8000000 ---p 00000000 00:00 0 
7fc5d8000000-7fc5d8021000 rw-p 00000000 00:00 0 
7fc5d8021000-7fc5dc000000 ---p 00000000 00:00 0 
7fc5dc000000-7fc5dc021000 rw-p 00000000 00:00 0 
7fc5dc021000-7fc5e0000000 ---p 00000000 00:00 0 
7fc5e0000000-7fc5e0021000 rw-p 00000000 00:00 0 
7fc5e0021000-7fc5e4000000 ---p 00000000 00:00 0 
7fc5e4000000-7fc5e4021000 rw-p 00000000 00:00 0 
7fc5e4021000-7fc5e8000000 ---p 00000000 00:00 0 
7fc5e8000000-7fc5e8021000 rw-p 00000000 00:00 0 
7fc5e8021000-7fc5ec000000 ---p 00000000 00:00 0 
7fc5f0000000-7fc5f0021000 rw-p 00000000 00:00 0 
7fc5f0021000-7fc5f4000000 ---p 00000000 00:00 0 
7fc5f4000000-7fc5f4021000 rw-p 00000000 00:00 0 
7fc5f4021000-7fc5f8000000 ---p 00000000 00:00 0 
7fc5f8000000-7fc5f8021000 rw-p 00000000 00:00 0 
7fc5f8021000-7fc5fc000000 ---p 00000000 00:00 0 
7fc5fc000000-7fc5fc021000 rw-p 00000000 00:00 0 
7fc5fc021000-7fc600000000 ---p 00000000 00:00 0 
7fc600000000-7fc600021000 rw-p 00000000 00:00 0 
7fc600021000-7fc604000000 ---p 00000000 00:00 0 
7fc604000000-7fc604021000 rw-p 00000000 00:00 0 
7fc604021000-7fc608000000 ---p 00000000 00:00 0 
7fc608000000-7fc608021000 rw-p 00000000 00:00 0 
7fc608021000-7fc60c000000 ---p 00000000 00:00 0 
7fc610000000-7fc610021000 rw-p 00000000 00:00 0 
7fc610021000-7fc614000000 ---p 00000000 00:00 0 
7fc614000000-7fc614021000 rw-p 00000000 00:00 0 
7fc614021000-7fc618000000 ---p 00000000 00:00 0 
7fc618000000-7fc618021000 rw-p 00000000 00:00 0 
7fc618021000-7fc61c000000 ---p 00000000 00:00 0 
7fc61c000000-7fc61c021000 rw-p 00000000 00:00 0 
7fc61c021000-7fc620000000 ---p 00000000 00:00 0 
7fc620000000-7fc620021000 rw-p 00000000 00:00 0 
7fc620021000-7fc624000000 ---p 00000000 00:00 0 
7fc624000000-7fc624021000 rw-p 00000000 00:00 0 
7fc624021000-7fc628000000 ---p 00000000 00:00 0 
7fc628000000-7fc628021000 rw-p 00000000 00:00 0 
7fc628021000-7fc62c000000 ---p 00000000 00:00 0 
7fc630000000-7fc630021000 rw-p 00000000 00:00 0 
7fc630021000-7fc634000000 ---p 00000000 00:00 0 
7fc634000000-7fc634219000 rw-p 00000000 00:00 0 
7fc634219000-7fc638000000 ---p 00000000 00:00 0 
7fc638000000-7fc638021000 rw-p 00000000 00:00 0 
7fc638021000-7fc63c000000 ---p 00000000 00:00 0 
7fc63c000000-7fc63c021000 rw-p 00000000 00:00 0 
7fc63c021000-7fc640000000 ---p 00000000 00:00 0 
7fc640000000-7fc640021000 rw-p 00000000 00:00 0 
7fc640021000-7fc644000000 ---p 00000000 00:00 0 
7fc644000000-7fc644021000 rw-p 00000000 00:00 0 
7fc644021000-7fc648000000 ---p 00000000 00:00 0 
7fc648000000-7fc648021000 rw-p 00000000 00:00 0 
7fc648021000-7fc64c000000 ---p 00000000 00:00 0 
7fc650000000-7fc650021000 rw-p 00000000 00:00 0 
7fc650021000-7fc654000000 ---p 00000000 00:00 0 
7fc654000000-7fc654021000 rw-p 00000000 00:00 0 
7fc654021000-7fc658000000 ---p 00000000 00:00 0 
7fc658000000-7fc658021000 rw-p 00000000 00:00 0 
7fc658021000-7fc65c000000 ---p 00000000 00:00 0 
7fc65c000000-7fc65c021000 rw-p 00000000 00:00 0 
7fc65c021000-7fc660000000 ---p 00000000 00:00 0 
7fc660000000-7fc660021000 rw-p 00000000 00:00 0 
7fc660021000-7fc664000000 ---p 00000000 00:00 0 
7fc664000000-7fc664021000 rw-p 00000000 00:00 0 
7fc664021000-7fc668000000 ---p 00000000 00:00 0 
7fc668000000-7fc668021000 rw-p 00000000 00:00 0 
7fc668021000-7fc66c000000 ---p 00000000 00:00 0 
7fc670000000-7fc670021000 rw-p 00000000 00:00 0 
7fc670021000-7fc674000000 ---p 00000000 00:00 0 
7fc674000000-7fc674021000 rw-p 00000000 00:00 0 
7fc674021000-7fc678000000 ---p 00000000 00:00 0 
7fc678000000-7fc678021000 rw-p 00000000 00:00 0 
7fc678021000-7fc67c000000 ---p 00000000 00:00 0 
7fc67c000000-7fc67c021000 rw-p 00000000 00:00 0 
7fc67c021000-7fc680000000 ---p 00000000 00:00 0 
7fc680000000-7fc680021000 rw-p 00000000 00:00 0 
7fc680021000-7fc684000000 ---p 00000000 00:00 0 
7fc684000000-7fc684021000 rw-p 00000000 00:00 0 
7fc684021000-7fc688000000 ---p 00000000 00:00 0 
7fc688000000-7fc688021000 rw-p 00000000 00:00 0 
7fc688021000-7fc68c000000 ---p 00000000 00:00 0 
7fc690000000-7fc690021000 rw-p 00000000 00:00 0 
7fc690021000-7fc694000000 ---p 00000000 00:00 0 
7fc694000000-7fc694021000 rw-p 00000000 00:00 0 
7fc694021000-7fc698000000 ---p 00000000 00:00 0 
7fc698000000-7fc698021000 rw-p 00000000 00:00 0 
.
.
.


I guess this is not of much use, since I didn't have debuginfo package installed
in time of crash. Is it possible to obtain more debuginfo after I installed it?
Is there something else I should check to find where the problem could be?

The system is quad core x86_64 with 32GB RAM, centos 6, running 3.0.51 kernel,
qemu-kvm 1.2.0

I'd be very grateful if somebody could have a look at this.

With best regards

nikola ciprich

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: qemu-kvm-1.2.0: double free or corruption

[Stefan Hajnoczi]

On Mon, Nov 19, 2012 at 8:56 AM, Nikola Ciprich
<nikola.ciprich@linuxbox.cz> wrote:
> on one of our servers, windows 2008 KVM suddenly crashed. I see following
> in libvirt log:
>
> *** glibc detected *** /usr/bin/qemu-kvm: double free or corruption (!prev): 0x00007fc634008cd0 ***
> ======= Backtrace: =========
> /lib64/libc.so.6(+0x75916)[0x7fc9026f4916]
> /lib64/libc.so.6(+0x78443)[0x7fc9026f7443]
> /usr/bin/qemu-kvm(+0x1faeb1)[0x7fc907187eb1]
> /usr/bin/qemu-kvm(+0x1f0e1a)[0x7fc90717de1a]
> /usr/bin/qemu-kvm(+0x1fb681)[0x7fc907188681]
> /usr/bin/qemu-kvm(+0xed6a7)[0x7fc90707a6a7]
> /usr/bin/qemu-kvm(+0x195c31)[0x7fc907122c31]
> /usr/bin/qemu-kvm(main+0x106c)[0x7fc90711e5fc]
> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fc90269dcdd]
> /usr/bin/qemu-kvm(+0x749f9)[0x7fc9070019f9]
[...]
> I guess this is not of much use, since I didn't have debuginfo package installed
> in time of crash. Is it possible to obtain more debuginfo after I installed it?
> Is there something else I should check to find where the problem could be?

No problem, you can still resolve symbols afterwards.  Download the
debuginfo package and use something along the lines of:
$ addr2line -e /path/to/debug-executable 0x1faeb1 0x1f0e1a 0x1fb681
0xed6a7 0x195c31

It's important to fetch the debuginfo package for the exact same
version of the qemu RPM you were running.

Stefan

Re: qemu-kvm-1.2.0: double free or corruption in VNC code

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 1647 bytes --]

Hello Stefan,

thanks! here it goes..

> > *** glibc detected *** /usr/bin/qemu-kvm: double free or corruption (!prev): 0x00007fc634008cd0 ***
> > ======= Backtrace: =========
> > /lib64/libc.so.6(+0x75916)[0x7fc9026f4916]
> > /lib64/libc.so.6(+0x78443)[0x7fc9026f7443]
> > /usr/bin/qemu-kvm(+0x1faeb1)[0x7fc907187eb1]
> > /usr/bin/qemu-kvm(+0x1f0e1a)[0x7fc90717de1a]
> > /usr/bin/qemu-kvm(+0x1fb681)[0x7fc907188681]
> > /usr/bin/qemu-kvm(+0xed6a7)[0x7fc90707a6a7]
> > /usr/bin/qemu-kvm(+0x195c31)[0x7fc907122c31]
> > /usr/bin/qemu-kvm(main+0x106c)[0x7fc90711e5fc]
> > /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fc90269dcdd]
> > /usr/bin/qemu-kvm(+0x749f9)[0x7fc9070019f9]
> [...]

[root@blg qemu-kvm-1.2.0]# addr2line -e /usr/lib/debug/usr/bin/qemu-kvm.debug 0x1faeb1 0x1f0e1a 0x1fb681 0xed6a7 0x195c31 0x106c
/usr/src/debug/qemu-kvm-1.2.0/ui/vnc.c:499
/usr/src/debug/qemu-kvm-1.2.0/ui/vnc-enc-zrle.c:364
/usr/src/debug/qemu-kvm-1.2.0/ui/vnc.c:1037
/usr/src/debug/qemu-kvm-1.2.0/iohandler.c:159
/usr/src/debug/qemu-kvm-1.2.0/main-loop.c:499
??:0

this makes some sense to me, since it crashed while there was VNC
connection active..

> 
> It's important to fetch the debuginfo package for the exact same
> version of the qemu RPM you were running.
sure, it's the same version.

BR

nik

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: qemu-kvm-1.2.0: double free or corruption in VNC code

[Stefan Hajnoczi]

On Wed, Nov 21, 2012 at 07:43:16AM +0100, Nikola Ciprich wrote:
> Hello Stefan,
> 
> thanks! here it goes..
> 
> > > *** glibc detected *** /usr/bin/qemu-kvm: double free or corruption (!prev): 0x00007fc634008cd0 ***
> > > ======= Backtrace: =========
> > > /lib64/libc.so.6(+0x75916)[0x7fc9026f4916]
> > > /lib64/libc.so.6(+0x78443)[0x7fc9026f7443]
> > > /usr/bin/qemu-kvm(+0x1faeb1)[0x7fc907187eb1]
> > > /usr/bin/qemu-kvm(+0x1f0e1a)[0x7fc90717de1a]
> > > /usr/bin/qemu-kvm(+0x1fb681)[0x7fc907188681]
> > > /usr/bin/qemu-kvm(+0xed6a7)[0x7fc90707a6a7]
> > > /usr/bin/qemu-kvm(+0x195c31)[0x7fc907122c31]
> > > /usr/bin/qemu-kvm(main+0x106c)[0x7fc90711e5fc]
> > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fc90269dcdd]
> > > /usr/bin/qemu-kvm(+0x749f9)[0x7fc9070019f9]
> > [...]
> 
> [root@blg qemu-kvm-1.2.0]# addr2line -e /usr/lib/debug/usr/bin/qemu-kvm.debug 0x1faeb1 0x1f0e1a 0x1fb681 0xed6a7 0x195c31 0x106c
> /usr/src/debug/qemu-kvm-1.2.0/ui/vnc.c:499
> /usr/src/debug/qemu-kvm-1.2.0/ui/vnc-enc-zrle.c:364
> /usr/src/debug/qemu-kvm-1.2.0/ui/vnc.c:1037
> /usr/src/debug/qemu-kvm-1.2.0/iohandler.c:159
> /usr/src/debug/qemu-kvm-1.2.0/main-loop.c:499

Please also post the exact package version you are using - the line
numbers change between releases and depend on which patches have been
applied to the source tree.  The distro exact package version allows me
to download the source tree that was used to build this binary and check
the correct line numbers.

Stefan

Re: qemu-kvm-1.2.0: double free or corruption in VNC code

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

> Please also post the exact package version you are using - the line
> numbers change between releases and depend on which patches have been
> applied to the source tree.  The distro exact package version allows me
> to download the source tree that was used to build this binary and check
> the correct line numbers.

Hello Stafan,

it's based on fedora rawhide pkg 2:1.2.0-16 with few minor tweaks to compile
on centos6.
I've uploaded sources used for build here:

http://nik.lbox.cz/download/qemu-kvm-1.2.0.tar.bz2 (after make clean)

or

http://nik.lbox.cz/download/qemu-1.2.0-lb6.01.src.rpm 

will this help?

> 
> Stefan
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: qemu-kvm-1.2.0: double free or corruption in VNC code

[Stefan Hajnoczi]

On Fri, Nov 23, 2012 at 08:24:32PM +0100, Nikola Ciprich wrote:
> > Please also post the exact package version you are using - the line
> > numbers change between releases and depend on which patches have been
> > applied to the source tree.  The distro exact package version allows me
> > to download the source tree that was used to build this binary and check
> > the correct line numbers.
> 
> Hello Stafan,
> 
> it's based on fedora rawhide pkg 2:1.2.0-16 with few minor tweaks to compile
> on centos6.
> I've uploaded sources used for build here:
> 
> http://nik.lbox.cz/download/qemu-kvm-1.2.0.tar.bz2 (after make clean)
> 
> or
> 
> http://nik.lbox.cz/download/qemu-1.2.0-lb6.01.src.rpm 
> 
> will this help?

Thanks, I looked at the backtrace in the source tree.  Unfortunately the
root cause is not obvious to me.  I was looking for a double-free of the
zrle buffers.

If this bug repeatedly bites you, try a different VNC encoding as a
workaround (not ZRLE).

Perhaps someone more familiar with the VNC code will be able to see it.
All the information you have provided is helpful.

Stefan

Re: qemu-kvm-1.2.0: double free or corruption in VNC code

[Nikola Ciprich]

[-- Attachment #1: Type: text/plain, Size: 1448 bytes --]

Hello Stefan,

thanks for Your time looking at it.

> 
> Thanks, I looked at the backtrace in the source tree.  Unfortunately the
> root cause is not obvious to me.  I was looking for a double-free of the
> zrle buffers.
> 
> If this bug repeatedly bites you, try a different VNC encoding as a
> workaround (not ZRLE).
Well, when I reported the problem, it was first time it appeared, so I didn't
consider it big deal, but yesterday we got it again, on different server
(but the backtrace is completely the same) But it was the same person as
before (and quite a new user to our KVM guests), so I suspect it can be
something specific to his client or setup. I'll try to look at it deeper
and let You know if I figure something out...

cheers

nik



> 
> Perhaps someone more familiar with the VNC code will be able to see it.
> All the information you have provided is helpful.
> 
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]