[Qemu-devel] [Patch] VNC: Fix crash with non-resizing clients

["Eduardo Felipe" <edusaper@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 510 bytes --]

Hi,

2007/9/25, GUERRAZ Francois <francois.guerraz@alyotech.fr>:
>
>
> About your VNC problems : I have had problems w/ vnc too (see
> http://qemu-forum.ipi.fi/viewtopic.php?p=10468) but had no answer as
> well...


This problem happens when the VNC client doesn't support the DesktopSize
pseudo-encoding. Qemu crashes when the guest resizes down its display and
the VNC client sends a SetPixelFormat afterwards.

Attached patch should fix this. It also forces a full buffer update after
resize.

Regards,
Edu

[-- Attachment #1.2: Type: text/html, Size: 862 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: vnc-crash-on-resize.patch --]
[-- Type: text/x-patch; name="vnc-crash-on-resize.patch", Size: 1052 bytes --]

*** vnc.c	30 Sep 2007 13:01:15 -0000	1.25
--- vnc.c	3 Oct 2007 18:44:17 -0000
@@ -289,15 +289,19 @@
     ds->width = w;
     ds->height = h;
     ds->linesize = w * vs->depth;
-    if (vs->csock != -1 && vs->has_resize && size_changed) {
-	vnc_write_u8(vs, 0);  /* msg id */
-	vnc_write_u8(vs, 0);
-	vnc_write_u16(vs, 1); /* number of rects */
-	vnc_framebuffer_update(vs, 0, 0, ds->width, ds->height, -223);
-	vnc_flush(vs);
-	vs->width = ds->width;
-	vs->height = ds->height;
+    if (size_changed) {
+        if (vs->csock != -1 && vs->has_resize) {
+            vnc_write_u8(vs, 0);  /* msg id */
+            vnc_write_u8(vs, 0);
+            vnc_write_u16(vs, 1); /* number of rects */
+            vnc_framebuffer_update(vs, 0, 0, ds->width, ds->height, -223);
+            vnc_flush(vs);
+        }
+        memset(vs->dirty_row, 0xFF, sizeof(vs->dirty_row));
+        memset(vs->old_data, 42, ds->linesize * ds->height);
     }
+    vs->width = ds->width;
+    vs->height = ds->height;
 }
 
 /* fastest code */