[Qemu-devel] [PATCH 1/3] vnc: fix server surface pixel format.

[Qemu-devel] [PATCH 1/3] vnc: fix server surface pixel format.

[Gerd Hoffmann]

Format must be identical to the guest surface, we can't work with
the 32 bpp used by the default surface allocator.

Without this patch vnc doesn't get the conversions right when sending
pixel data to the client.  The bug triggers if
  (a) the client doesn't support WMVi, and
  (b) the guest screen depth is != 32 bpp.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 vnc.c |   21 +++++++++------------
 1 files changed, 9 insertions(+), 12 deletions(-)

diff --git a/vnc.c b/vnc.c
index ab1f044..8b381c9 100644
--- a/vnc.c
+++ b/vnc.c
@@ -366,17 +366,13 @@ static void vnc_resize(VncState *vs)
     memset(vs->guest.dirty, 0xFF, sizeof(vs->guest.dirty));
 
     /* server surface */
-    if (!vs->server.ds) {
-        vs->server.ds = default_allocator.create_displaysurface(ds_get_width(ds),
-                                                                ds_get_height(ds));
-    } else {
-        default_allocator.resize_displaysurface(vs->server.ds,
-                                                ds_get_width(ds), ds_get_height(ds));
-    }
-    if (vs->server.ds->data == NULL) {
-        fprintf(stderr, "vnc: memory allocation failed\n");
-        exit(1);
-    }
+    if (!vs->server.ds)
+        vs->server.ds = qemu_mallocz(sizeof(*vs->server.ds));
+    if (vs->server.ds->data)
+        qemu_free(vs->server.ds->data);
+    *(vs->server.ds) = *(ds->surface);
+    vs->server.ds->data = qemu_mallocz(vs->server.ds->linesize *
+                                       vs->server.ds->height);
     memset(vs->server.dirty, 0xFF, sizeof(vs->guest.dirty));
 }
 
@@ -918,7 +914,8 @@ int vnc_client_io_error(VncState *vs, int ret, int last_errno)
         if (!vs->vd->clients)
             dcl->idle = 1;
 
-        default_allocator.free_displaysurface(vs->server.ds);
+        qemu_free(vs->server.ds->data);
+        qemu_free(vs->server.ds);
         qemu_free(vs->guest.ds);
         qemu_free(vs);
 
-- 
1.6.2.2

[Qemu-devel] [PATCH 2/3] vnc: no need to set force_update for incremental update requests.

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 vnc.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/vnc.c b/vnc.c
index 8b381c9..3a25e1d 100644
--- a/vnc.c
+++ b/vnc.c
@@ -1412,8 +1412,8 @@ static void framebuffer_update_request(VncState *vs, int incremental,
 
     int i;
     vs->need_update = 1;
-    vs->force_update = 1;
     if (!incremental) {
+        vs->force_update = 1;
         for (i = 0; i < h; i++) {
             vnc_set_bits(vs->guest.dirty[y_position + i],
                          (ds_get_width(vs->ds) / 16), VNC_DIRTY_WORDS);
-- 
1.6.2.2

[Qemu-devel] [PATCH 3/3] vnc: kill leftover debug statement.

[Gerd Hoffmann]

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 vnc.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/vnc.c b/vnc.c
index 3a25e1d..8e8acc2 100644
--- a/vnc.c
+++ b/vnc.c
@@ -683,7 +683,7 @@ static int find_and_clear_dirty_height(struct VncSurface *s,
 {
     int h;
 
-    for (h = 1; h < (s->ds->height - y) && h < 1; h++) {
+    for (h = 1; h < (s->ds->height - y); h++) {
         int tmp_x;
         if (!vnc_get_bit(s->dirty[y + h], last_x))
             break;
-- 
1.6.2.2

Re: [Qemu-devel] [PATCH 1/3] vnc: fix server surface pixel format.

[Daniel Gollub]

On Monday 27 April 2009 04:39:51 pm Gerd Hoffmann wrote:
> Format must be identical to the guest surface, we can't work with
> the 32 bpp used by the default surface allocator.
>
> Without this patch vnc doesn't get the conversions right when sending
> pixel data to the client.  The bug triggers if
>   (a) the client doesn't support WMVi, and
>   (b) the guest screen depth is != 32 bpp.

Tried your patches - qemu crashs now sometimes with:
*** glibc detected *** ./x86_64-softmmu/qemu-system-x86_64: malloc(): memory corruption: 0x0000000000e28e50 ***


(When it doesn't crash it just works perfectly fine again with vncviewer.)

Backtrace looks like this:

Core was generated by `./x86_64-softmmu/qemu-system-x86_64 -hda /home/dgollub/VMs/sles11_x86_64.qcow2'.
Program terminated with signal 6, Aborted.
#0  0x00007f7189867645 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007f7189867645 in raise () from /lib64/libc.so.6
#1  0x00007f7189868c33 in abort () from /lib64/libc.so.6
#2  0x00007f71898a38e8 in ?? () from /lib64/libc.so.6
#3  0x00007f71898a9118 in ?? () from /lib64/libc.so.6
#4  0x00007f71898aba5c in ?? () from /lib64/libc.so.6
#5  0x00007f71898ace64 in ?? () from /lib64/libc.so.6
#6  0x00007f71898ade38 in realloc () from /lib64/libc.so.6
#7  0x000000000046fd3e in qemu_realloc (ptr=0x4195, size=16789) at qemu-malloc.c:52
#8  0x000000000049a10c in buffer_reserve (buffer=0xd3d090, len=<value optimized out>) at vnc.c:313
#9  0x000000000049a16b in vnc_write (vs=0xd2d010, data=0x7fff93b52ca0, len=6) at vnc.c:1135
#10 0x000000000049a29a in vnc_write_pixels_generic (vs=0xd2d010, pixels1=0xe09a40, size=<value optimized out>)
    at vnc.c:457
#11 0x000000000049e0f4 in send_framebuffer_update (vs=0xd2d010, x=0, y=0, w=800, h=600) at vnc.c:479
#12 0x000000000049e6f2 in vnc_update_client (opaque=0xd2d010) at vnc.c:788
#13 0x0000000000409dbc in main_loop_wait (timeout=<value optimized out>) at /home/dgollub/projects/qemu/vl.c:1227
#14 0x000000000040d80d in main (argc=5, argv=0x7fff93b538d8, envp=<value optimized out>)
    at /home/dgollub/projects/qemu/vl.c:4448
(gdb) 

Hope that helps.

Thanks for fixing the vncviewer regression!

Best Regards,
Daniel

-- 
Daniel Gollub                        Geschaeftsfuehrer: Ralph Dehner
FOSS Developer                       Unternehmenssitz:  Vohburg
B1 Systems GmbH                      Amtsgericht:       Ingolstadt
Mobil: +49-(0)-160 47 73 970         Handelsregister:   HRB 3537
EMail: gollub@b1-systems.de          http://www.b1-systems.de

Adresse: B1 Systems GmbH, Osterfeldstraße 7, 85088 Vohburg
http://pgpkeys.pca.dfn.de/pks/lookup?op=get&search=0xED14B95C2F8CA78D