[Qemu-devel] New Qemu Crash found with evidence of memory corruption

[Qemu-devel] New Qemu Crash found with evidence of memory corruption

[Alexey Eremenko]

[-- Attachment #1: Type: text/plain, Size: 7803 bytes --]


Hi Qemu Developers !

Qumranet's Automated testing reveals, that in some cases Qemu double frees memory and crashes.

Tested with both Qemu-CVS-2007-12-10 and KVM-56 (both Userspace-only and
kernelspace/userspace combo).

Error message:
======================================================

*** glibc detected *** /usr/local/bin/qemu-system-x86_64: double free or
corruption (fasttop): 0x0000000002b6cb10 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3dd0270412]
/lib64/libc.so.6(cfree+0x8c)[0x3dd0273b1c]
/usr/local/bin/qemu-system-x86_64[0x4116c1]
/usr/local/bin/qemu-system-x86_64[0x41403d]
/usr/local/bin/qemu-system-x86_64[0x40889e]
/usr/local/bin/qemu-system-x86_64[0x40db72]
/usr/local/bin/qemu-system-x86_64[0x48cf15]
/usr/local/bin/qemu-system-x86_64[0x48cf9b]
/usr/local/bin/qemu-system-x86_64[0x48d381]
/usr/local/bin/qemu-system-x86_64[0x40dd27]
/usr/local/bin/qemu-system-x86_64[0x40fd03]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3dd021daa4]
/usr/local/bin/qemu-system-x86_64[0x4060b9]
======= Memory map: ========
00400000-0055b000 r-xp 00000000 fd:00 1961296
/usr/local/bin/qemu-system-x86_64
0075b000-0076f000 rw-p 0015b000 fd:00 1961296
/usr/local/bin/qemu-system-x86_64
0076f000-01a3a000 rw-p 0076f000 00:00 0
01a3a000-02a3b000 rwxp 01a3a000 00:00 0
02a3b000-02dcb000 rw-p 02a3b000 00:00 0
[heap]
3dcfe00000-3dcfe1a000 r-xp 00000000 fd:00 1267006
/lib64/ld-2.6.so
3dd0019000-3dd001a000 r--p 00019000 fd:00 1267006
/lib64/ld-2.6.so
3dd001a000-3dd001b000 rw-p 0001a000 fd:00 1267006
/lib64/ld-2.6.so
3dd0200000-3dd0347000 r-xp 00000000 fd:00 1267007
/lib64/libc-2.6.so
3dd0347000-3dd0546000 ---p 00147000 fd:00 1267007
/lib64/libc-2.6.so
3dd0546000-3dd054a000 r--p 00146000 fd:00 1267007
/lib64/libc-2.6.so
3dd054a000-3dd054b000 rw-p 0014a000 fd:00 1267007
/lib64/libc-2.6.so
3dd054b000-3dd0550000 rw-p 3dd054b000 00:00 0
3dd0600000-3dd0602000 r-xp 00000000 fd:00 1267010
/lib64/libdl-2.6.so
3dd0602000-3dd0802000 ---p 00002000 fd:00 1267010
/lib64/libdl-2.6.so
3dd0802000-3dd0803000 r--p 00002000 fd:00 1267010
/lib64/libdl-2.6.so
3dd0803000-3dd0804000 rw-p 00003000 fd:00 1267010
/lib64/libdl-2.6.so
3dd0a00000-3dd0a82000 r-xp 00000000 fd:00 1267009
/lib64/libm-2.6.so
3dd0a82000-3dd0c81000 ---p 00082000 fd:00 1267009
/lib64/libm-2.6.so
3dd0c81000-3dd0c82000 r--p 00081000 fd:00 1267009
/lib64/libm-2.6.so
3dd0c82000-3dd0c83000 rw-p 00082000 fd:00 1267009
/lib64/libm-2.6.so
3dd0e00000-3dd0e14000 r-xp 00000000 fd:00 1267008
/lib64/libz.so.1.2.3
3dd0e14000-3dd1013000 ---p 00014000 fd:00 1267008
/lib64/libz.so.1.2.3
3dd1013000-3dd1014000 rw-p 00013000 fd:00 1267008
/lib64/libz.so.1.2.3
3dd1200000-3dd1215000 r-xp 00000000 fd:00 1267012
/lib64/libpthread-2.6.so
3dd1215000-3dd1414000 ---p 00015000 fd:00 1267012
/lib64/libpthread-2.6.so
3dd1414000-3dd1415000 r--p 00014000 fd:00 1267012
/lib64/libpthread-2.6.so
3dd1415000-3dd1416000 rw-p 00015000 fd:00 1267012
/lib64/libpthread-2.6.so
3dd1416000-3dd141a000 rw-p 3dd1416000 00:00 0
3dd1600000-3dd1704000 r-xp 00000000 fd:00 1953728
/usr/lib64/libX11.so.6.2.0
3dd1704000-3dd1904000 ---p 00104000 fd:00 1953728
/usr/lib64/libX11.so.6.2.0
3dd1904000-3dd190b000 rw-p 00104000 fd:00 1953728
/usr/lib64/libX11.so.6.2.0
3dd1a00000-3dd1a02000 r-xp 00000000 fd:00 1952614
/usr/lib64/libXau.so.6.0.0
3dd1a02000-3dd1c01000 ---p 00002000 fd:00 1952614
/usr/lib64/libXau.so.6.0.0
3dd1c01000-3dd1c02000 rw-p 00001000 fd:00 1952614
/usr/lib64/libXau.so.6.0.0
3dd1e00000-3dd1e05000 r-xp 00000000 fd:00 1953727
/usr/lib64/libXdmcp.so.6.0.0
3dd1e05000-3dd2004000 ---p 00005000 fd:00 1953727
/usr/lib64/libXdmcp.so.6.0.0
3dd2004000-3dd2005000 rw-p 00004000 fd:00 1953727
/usr/lib64/libXdmcp.so.6.0.0
3dd2200000-3dd220d000 r-xp 00000000 fd:00 1267013
/lib64/libgcc_s-4.1.2-20070503.so.1
3dd220d000-3dd240d000 ---p 0000d000 fd:00 1267013
/lib64/libgcc_s-4.1.2-20070503.so.1
3dd240d000-3dd240e000 rw-p 0000d000 fd:00 1267013
/lib64/libgcc_s-4.1.2-20070503.so.1
3dd2600000-3dd2610000 r-xp 00000000 fd:00 1953729
/usr/lib64/libXext.so.6.4.0
3dd2610000-3dd2810000 ---p 00010000 fd:00 1953729
/usr/lib64/libXext.so.6.4.0
3dd2810000-3dd2811000 rw-p 00010000 fd:00 1953729
/usr/lib64/libXext.so.6.4.0
3dd4200000-3dd4209000 r-xp 00000000 fd:00 1953339
/usr/lib64/libXrender.so.1.3.0
3dd4209000-3dd4408000 ---p 00009000 fd:00 1953339
/usr/lib64/libXrender.so.1.3.0
3dd4408000-3dd4409000 rw-p 00008000 fd:00 1953339
/usr/lib64/libXrender.so.1.3.0
3dd4e00000-3dd4e11000 r-xp 00000000 fd:00 1267014
/lib64/libresolv-2.6.so
3dd4e11000-3dd5011000 ---p 00011000 fd:00 1267014
/lib64/libresolv-2.6.so
3dd5011000-3dd5012000 r--p 00011000 fd:00 1267014
/lib64/libresolv-2.6.so
3dd5012000-3dd5013000 rw-p 00012000 fd:00 1267014
/lib64/libresolv-2.6.so
3dd5013000-3dd5015000 rw-p 3dd5013000 00:00 0
3dd5200000-3dd5205000 r-xp 00000000 fd:00 1953732
/usr/lib64/libXfixes.so.3.1.0
3dd5205000-3dd5404000 ---p 00005000 fd:00 1953732
/usr/lib64/libXfixes.so.3.1.0
3dd5404000-3dd5405000 rw-p 00004000 fd:00 1953732

======================================================

GDB shows:

(gdb) c
Continuing.

Program received signal SIGABRT, Aborted.
[Switching to Thread 46912496226896 (LWP 8191)]
0x0000003dd02305b5 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x0000003dd02305b5 in raise () from /lib64/libc.so.6
#1 0x0000003dd0232060 in abort () from /lib64/libc.so.6
#2 0x0000003dd0268d0b in __libc_message () from /lib64/libc.so.6
#3 0x0000003dd0270412 in _int_free () from /lib64/libc.so.6
#4 0x0000003dd0273b1c in free () from /lib64/libc.so.6
#5 0x00000000004116c1 in readline_handle_byte (ch=<value optimized out>)
at /root/Linstall/kvm-56/qemu/readline.c:280
#6 0x000000000041403d in term_read (opaque=<value optimized out>,
buf=0x7fff4089e12d "", size=6) at
/root/Linstall/kvm-56/qemu/monitor.c:2592
#7 0x000000000040889e in tcp_chr_read (opaque=<value optimized out>)
at /root/Linstall/kvm-56/qemu/vl.c:3080
#8 0x000000000040db72 in main_loop_wait (timeout=<value optimized out>)
at /root/Linstall/kvm-56/qemu/vl.c:7178
#9 0x000000000048cf15 in kvm_eat_signals (env=0x2ac75b0, timeout=0)
at /root/Linstall/kvm-56/qemu/qemu-kvm.c:210
#10 0x000000000048cf9b in kvm_main_loop_wait (env=0x2ac75b0, timeout=0)
at /root/Linstall/kvm-56/qemu/qemu-kvm.c:218
#11 0x000000000048d381 in kvm_main_loop_cpu (env=0x2ac75b0)
at /root/Linstall/kvm-56/qemu/qemu-kvm.c:337
#12 0x000000000040dd27 in main_loop () at
/root/Linstall/kvm-56/qemu/vl.c:7238
#13 0x000000000040fd03 in main (argc=<value optimized out>,
argv=<value optimized out>) at /root/Linstall/kvm-56/qemu/vl.c:8978
(gdb)

======================================================
The error seems to be in Qemu's readline.c:

if (idx == TERM_MAX_CMDS) {
/* Need to get one free slot */
free(term_history[0]); <-- Here is the error.
memcpy(term_history, &term_history[1],
&term_history[TERM_MAX_CMDS] - &term_history[1]);
term_history[TERM_MAX_CMDS - 1] = NULL;
idx = TERM_MAX_CMDS - 1;
}


======================================================
Possible workaround:

changing in readline.c from:
#define TERM_MAX_CMDS 64
-to-
#define TERM_MAX_CMDS 4096

======================================================

This bug affects stability of testing, and at least two guest OSes are affected: SUSE Linux 9.1 and OpenBSD 4.1. (Automated setup crashes).

NOTE: I'we been unable to reproduce this crash scenario manually. Perhaps it requires sending a *lot* of commands into Qemu Monitor. Some commands must be illegal, such as "-" sign.

NOTE2: Same bug in KVM bugzilla: http://sourceforge.net/tracker/index.php?func=detail&aid=1851814&group_id=180599&atid=893831

Any ideas?

-Technologov, QA Team Member, Qumranet.

[-- Attachment #2: Type: text/html, Size: 9181 bytes --]

Re: [Qemu-devel] New Qemu Crash found with evidence of memory corruption

[Andreas Schwab]

"Alexey Eremenko" <alexey.eremenko@qumranet.com> writes:

> ======================================================
> The error seems to be in Qemu's readline.c:
>
> if (idx == TERM_MAX_CMDS) {
> /* Need to get one free slot */
> free(term_history[0]); <-- Here is the error.
> memcpy(term_history, &term_history[1],
> &term_history[TERM_MAX_CMDS] - &term_history[1]);
> term_history[TERM_MAX_CMDS - 1] = NULL;
> idx = TERM_MAX_CMDS - 1;
> }
>

Please try this:

--- readline.c	09 Dez 2007 19:27:48 +0100	1.7
+++ readline.c	16 Dez 2007 18:22:43 +0100	
@@ -267,7 +267,7 @@ static void term_hist_add(const char *cm
 	    new_entry = hist_entry;
 	    /* Put this entry at the end of history */
 	    memmove(&term_history[idx], &term_history[idx + 1],
-		    &term_history[TERM_MAX_CMDS] - &term_history[idx + 1]);
+		    (TERM_MAX_CMDS - idx + 1) * sizeof(char *));
 	    term_history[TERM_MAX_CMDS - 1] = NULL;
 	    for (; idx < TERM_MAX_CMDS; idx++) {
 		if (term_history[idx] == NULL)
@@ -280,7 +280,7 @@ static void term_hist_add(const char *cm
 	/* Need to get one free slot */
 	free(term_history[0]);
 	memcpy(term_history, &term_history[1],
-	       &term_history[TERM_MAX_CMDS] - &term_history[1]);
+	       (TERM_MAX_CMDS - 1) * sizeof(char *));
 	term_history[TERM_MAX_CMDS - 1] = NULL;
 	idx = TERM_MAX_CMDS - 1;
     }

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

RE: [Qemu-devel] New Qemu Crash found with evidence of memorycorruption

[Alexey Eremenko]

[-- Attachment #1: Type: text/plain, Size: 1365 bytes --]




-----Original Message-----
From: qemu-devel-bounces+alexeye=qumranet.com@nongnu.org on behalf of Andreas Schwab
Sent: Sun 12/16/2007 9:24 AM
To: qemu-devel@nongnu.org
Cc: Yaniv Kaul
Subject: Re: [Qemu-devel] New Qemu Crash found with evidence of memorycorruption

Please try this:

--- readline.c	09 Dez 2007 19:27:48 +0100	1.7
+++ readline.c	16 Dez 2007 18:22:43 +0100	
@@ -267,7 +267,7 @@ static void term_hist_add(const char *cm
 	    new_entry = hist_entry;
 	    /* Put this entry at the end of history */
 	    memmove(&term_history[idx], &term_history[idx + 1],
-		    &term_history[TERM_MAX_CMDS] - &term_history[idx + 1]);
+		    (TERM_MAX_CMDS - idx + 1) * sizeof(char *));
 	    term_history[TERM_MAX_CMDS - 1] = NULL;
 	    for (; idx < TERM_MAX_CMDS; idx++) {
 		if (term_history[idx] == NULL)
@@ -280,7 +280,7 @@ static void term_hist_add(const char *cm
 	/* Need to get one free slot */
 	free(term_history[0]);
 	memcpy(term_history, &term_history[1],
-	       &term_history[TERM_MAX_CMDS] - &term_history[1]);
+	       (TERM_MAX_CMDS - 1) * sizeof(char *));
 	term_history[TERM_MAX_CMDS - 1] = NULL;
 	idx = TERM_MAX_CMDS - 1;
     }

================================================================================

Hi Andreas,

Yes, this works ! Please submit this code to Qemu.

-Technologov, Qumranet

[-- Attachment #2: Type: text/html, Size: 2941 bytes --]

Re: [Qemu-devel] New Qemu Crash found with evidence of memory corruption

[andrzej zaborowski]

On 16/12/2007, Andreas Schwab <schwab@suse.de> wrote:
> "Alexey Eremenko" <alexey.eremenko@qumranet.com> writes:
>
> > ======================================================
> > The error seems to be in Qemu's readline.c:
> >
> > if (idx == TERM_MAX_CMDS) {
> > /* Need to get one free slot */
> > free(term_history[0]); <-- Here is the error.
> > memcpy(term_history, &term_history[1],
> > &term_history[TERM_MAX_CMDS] - &term_history[1]);
> > term_history[TERM_MAX_CMDS - 1] = NULL;
> > idx = TERM_MAX_CMDS - 1;
> > }
> >
>
> Please try this:
>
> --- readline.c  09 Dez 2007 19:27:48 +0100      1.7
> +++ readline.c  16 Dez 2007 18:22:43 +0100
> @@ -267,7 +267,7 @@ static void term_hist_add(const char *cm
>             new_entry = hist_entry;
>             /* Put this entry at the end of history */
>             memmove(&term_history[idx], &term_history[idx + 1],
> -                   &term_history[TERM_MAX_CMDS] - &term_history[idx + 1]);
> +                   (TERM_MAX_CMDS - idx + 1) * sizeof(char *));
>             term_history[TERM_MAX_CMDS - 1] = NULL;
>             for (; idx < TERM_MAX_CMDS; idx++) {
>                 if (term_history[idx] == NULL)
> @@ -280,7 +280,7 @@ static void term_hist_add(const char *cm
>         /* Need to get one free slot */
>         free(term_history[0]);
>         memcpy(term_history, &term_history[1],
> -              &term_history[TERM_MAX_CMDS] - &term_history[1]);
> +              (TERM_MAX_CMDS - 1) * sizeof(char *));
>         term_history[TERM_MAX_CMDS - 1] = NULL;
>         idx = TERM_MAX_CMDS - 1;
>      }

This is correct. I remember submitting the exact same fix about a year
and a half ago in the patch to save/restore monitor history between
sessions. By the way would there be interest to have such feature in
mainline cvs?

Regards