From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=34643 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Q1amh-0000gO-DH for qemu-devel@nongnu.org; Mon, 21 Mar 2011 04:48:20 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Q1ame-0002Vw-Ia for qemu-devel@nongnu.org; Mon, 21 Mar 2011 04:48:17 -0400 Received: from mx3-phx2.redhat.com ([209.132.183.24]:34350) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Q1ame-0002Vg-2l for qemu-devel@nongnu.org; Mon, 21 Mar 2011 04:48:16 -0400 Received: from mail03.corp.redhat.com (zmail07.collab.prod.int.phx2.redhat.com [10.5.5.47]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id p2L8mESx002301 for ; Mon, 21 Mar 2011 04:48:14 -0400 Date: Mon, 21 Mar 2011 04:48:10 -0400 (EDT) From: Ulrich Obergfell Message-ID: <1368889216.435910.1300697290562.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> In-Reply-To: <1753565547.435765.1300696715736.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Subject: [Qemu-devel] [PATCH] vnc: segmentation fault caused by incorrect 'bytes' count calculated in tight_compress_data() List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org tight_compress_data() calculates an incorrect 'bytes' count if 'zstream->total_out' is greater than 0x7fffffff, because the type of the variable 'previous_out' is 'int'. 852 int previous_out; : 872 previous_out = zstream->total_out; : 881 bytes = zstream->total_out - previous_out; 882 883 tight_send_compact_size(vs, bytes); 884 vnc_write(vs, vs->tight.zlib.buffer, bytes); The incorrect 'bytes' count causes segmentation faults in functions called by tight_compress_data(). For example: (gdb) bt #0 0x000000396ab3d2b6 in __memcpy_ssse3_back () from /lib64/libc.so.6 #1 0x00000000004b3b91 in buffer_append (buffer=0x322d660, data=, len=) at /usr/include/bits/string3.h:52 #2 0x00000000004bbf02 in tight_compress_data (vs=0x32215b0, stream_id=, bytes=0x100024881, level=, strategy=) at ui/vnc-enc-tight.c:884 ... (gdb) x/i $rip => 0x396ab3d2b6 <__memcpy_ssse3_back+6710>: movdqu -0x10(%rsi),%xmm0 (gdb) print $rsi+0x10 $1 = 0x103f2ab21 (gdb) x/xb 0x103f2ab21 0x103f2ab21: Cannot access memory at address 0x103f2ab21 The following program illustrates the problem. $ cat t.c #include main() { int previous_out; unsigned long total_out = 0x80000000; size_t bytes; previous_out = total_out; total_out += 0x10000; bytes = total_out - previous_out; printf("%lx\n", bytes); } $ cc t.c -o t $ ./t 100010000 The patch changes the type of 'previous_out' to 'uLong' which is the same as the type of 'zstream->total_out'. Signed-off-by: Ulrich Obergfell diff -up ./ui/vnc-enc-tight.c.orig0 ./ui/vnc-enc-tight.c --- ./ui/vnc-enc-tight.c.orig0 2011-03-15 03:53:22.000000000 +0100 +++ ./ui/vnc-enc-tight.c 2011-03-20 12:14:48.013560009 +0100 @@ -849,7 +849,7 @@ static int tight_compress_data(VncState int level, int strategy) { z_streamp zstream = &vs->tight.stream[stream_id]; - int previous_out; + uLong previous_out; if (bytes < VNC_TIGHT_MIN_TO_COMPRESS) { vnc_write(vs, vs->tight.tight.buffer, vs->tight.tight.offset);