[Qemu-devel] [PATCH] Add JPEG encoding to VNC server

[Qemu-devel] [PATCH] Add JPEG encoding to VNC server

[Alexander Graf]

We know most of the tight protocol already, so implementing JPEG is rather
easy, especially considering that I had this implementation lying around
still anyways.

The big concern Anthony raised about JPEG compression is that as soon as you
use JPEG, CopyRect looks ugly for most use cases and most users probably don't
want it anyways.

So the road I went for this patch was to only enable JPEG encoding when it's
the only available choice. Allow any other protocols? You don't get JPEG then.

While this might sound like it renders the whole implementation useless, it
does make sense to implement it nevertheless. I have some ideas to implement
progressive encodings for video.

So when we'd detect that one region is updated a lot in a short about of time
with content that zlib can't really handle well, we'd just send a really low
quality JPEG first and then send the update after a timer if the region wasn't
updated within that timeframe.

But this is all ideas so far. For now the JPEG implementation is stand alone,
but would enable either me or someone else who'd like to do it worlds of
encoding fun :-).

Also, if you're daring, you can always see if JPEG performs good for your
specific VNC workload and at least have the chance to use it.

Signed-off-by: Alexander Graf <agraf@suse.de>
---
 Makefile.target |    4 +
 configure       |   24 ++++++++
 vnc.c           |  178 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 vnc.h           |    9 +++
 4 files changed, 214 insertions(+), 1 deletions(-)

diff --git a/Makefile.target b/Makefile.target
index 49ba08d..f1dd54d 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -303,6 +303,10 @@ CPPFLAGS += $(VNC_SASL_CFLAGS)
 LIBS += $(VNC_SASL_LIBS)
 endif
 
+ifdef CONFIG_VNC_JPEG
+LIBS += $(CONFIG_VNC_JPEG_LIBS)
+endif
+
 ifdef CONFIG_BLUEZ
 LIBS += $(BLUEZ_LIBS)
 endif
diff --git a/configure b/configure
index 8160bed..67fe51a 100755
--- a/configure
+++ b/configure
@@ -175,6 +175,7 @@ fmod_inc=""
 oss_lib=""
 vnc_tls="yes"
 vnc_sasl="yes"
+vnc_jpeg="yes"
 bsd="no"
 linux="no"
 solaris="no"
@@ -428,6 +429,8 @@ for opt do
   ;;
   --disable-vnc-sasl) vnc_sasl="no"
   ;;
+  --disable-jpeg) vnc_jpeg="no"
+  ;;
   --disable-slirp) slirp="no"
   ;;
   --disable-vde) vde="no"
@@ -649,6 +652,7 @@ echo "  --disable-xen            disable xen backend driver support"
 echo "  --disable-brlapi         disable BrlAPI"
 echo "  --disable-vnc-tls        disable TLS encryption for VNC server"
 echo "  --disable-vnc-sasl       disable SASL encryption for VNC server"
+echo "  --disable-jpeg           disable JPEG compression for VNC server"
 echo "  --disable-curses         disable curses output"
 echo "  --disable-curl           disable curl connectivity"
 echo "  --disable-bluez          disable bluez stack connectivity"
@@ -994,6 +998,21 @@ if $cc $ARCH_CFLAGS -o $TMPE $TMPC > /dev/null 2> /dev/null ; then
 fi
 
 ##########################################
+# VNC JPEG detection
+if test "$vnc_jpeg" = "yes" ; then
+cat > $TMPC <<EOF
+#include <jpeglib.h>
+int main(void) { jpeg_compress_struct s; jpeg_create_compress(&s); return 0; }
+EOF
+    vnc_jpeg_libs="-ljpeg"
+    if $cc $ARCH_CFLAGS -o $TMPE ${OS_CFLAGS} $TMPC $vnc_jpeg_libs > /dev/null 2> /dev/null ; then
+	:
+    else
+	vnc_tls="no"
+    fi
+fi
+
+##########################################
 # vde libraries probe
 if test "$vde" = "yes" ; then
   vde=no
@@ -1452,6 +1471,7 @@ if test "$vnc_sasl" = "yes" ; then
     echo "    SASL CFLAGS    $vnc_sasl_cflags"
     echo "    SASL LIBS      $vnc_sasl_libs"
 fi
+echo "VNC JPEG support  $vnc_jpeg"
 if test -n "$sparc_cpu"; then
     echo "Target Sparc Arch $sparc_cpu"
 fi
@@ -1604,6 +1624,10 @@ fi
 if test "$fnmatch" = "yes" ; then
   echo "CONFIG_FNMATCH=y" >> $config_host_mak
 fi
+if test "$vnc_jpeg" = "yes" ; then
+  echo "CONFIG_VNC_JPEG=y" >> $config_host_mak
+  echo "CONFIG_VNC_JPEG_LIBS=$vnc_jpeg_libs" >> $config_host_mak
+fi
 qemu_version=`head $source_path/VERSION`
 echo "VERSION=$qemu_version" >>$config_host_mak
 echo "PKGVERSION=$pkgversion" >>$config_host_mak
diff --git a/vnc.c b/vnc.c
index 903dd95..4e0c967 100644
--- a/vnc.c
+++ b/vnc.c
@@ -635,9 +635,171 @@ static void send_framebuffer_update_zlib(VncState *vs, int x, int y, int w, int
     vs->output.offset = new_offset;
 }
 
+#ifdef CONFIG_VNC_JPEG
+/* This is called once per encoding */
+static void jpeg_init_destination(j_compress_ptr cinfo)
+{
+    VncState *vs = (VncState*)cinfo->client_data;
+    Buffer *buffer = &vs->jpeg_buffer;
+
+    cinfo->dest->next_output_byte = (JOCTET *)buffer->buffer + buffer->offset;
+    cinfo->dest->free_in_buffer = (size_t)(buffer->capacity - buffer->offset);
+}
+
+/* This is called when we ran out of buffer (shouldn't happen!) */
+static boolean jpeg_empty_output_buffer(j_compress_ptr cinfo)
+{
+    VncState *vs = (VncState*)cinfo->client_data;
+    Buffer *buffer = &vs->jpeg_buffer;
+
+    buffer->offset = buffer->capacity;
+    buffer_reserve(buffer, 2048);
+    jpeg_init_destination(cinfo);
+    return TRUE;
+}
+
+/* This is called when we are done processing data */
+static void jpeg_term_destination(j_compress_ptr cinfo)
+{
+    VncState *vs = (VncState*)cinfo->client_data;
+    Buffer *buffer = &vs->jpeg_buffer;
+
+    buffer->offset = buffer->capacity - cinfo->dest->free_in_buffer;
+}
+
+
+static void vnc_send_compact_size(VncState *vs, int len) 
+{
+    char buf[3];
+    int lpc = 0;
+    int bytes = 0;
+
+    /* Adapted from SendCompressedData() in Xvnc/programs/Xserver/hw/vnc/tight.c */
+    buf[bytes++] = len & 0x7F;
+    if (len > 0x7F) {
+	buf[bytes-1] |= 0x80;
+	buf[bytes++] = len >> 7 & 0x7F;
+	if (len > 0x3FFF) {
+	    buf[bytes-1] |= 0x80;
+	    buf[bytes++] = len >> 14 & 0xFF;
+	}
+    }
+
+    for(lpc = 0; lpc < bytes; lpc++) {
+	vnc_write_u8(vs, buf[lpc]);
+    }
+}
+
+static void jpeg_row2pixel(VncState *vs, char *in, char *out, int len)
+{
+    char *pi = in;
+    char *po = out;
+    int depth = vs->server.ds->pf.bytes_per_pixel;
+    int i;
+
+    for (i = 0; i < len; i++) {
+        uint32_t v;
+        uint8_t r, g, b;
+        switch (depth) {
+            case 1:
+                po[0] = pi[0];
+                po[1] = pi[0];
+                po[2] = pi[0];
+                continue;
+                break;
+            case 2:
+                v = *((uint16_t*)pi);
+                break;
+            case 4:
+                v = *((uint32_t*)pi);
+                break;
+        }
+        r = ((((v & vs->server.ds->pf.rmask) >> vs->server.ds->pf.rshift)
+            << vs->clientds.pf.rbits) >> vs->server.ds->pf.rbits);
+        g = ((((v & vs->server.ds->pf.gmask) >> vs->server.ds->pf.gshift)
+            << vs->clientds.pf.gbits) >> vs->server.ds->pf.gbits);
+        b = ((((v & vs->server.ds->pf.bmask) >> vs->server.ds->pf.bshift)
+            << vs->clientds.pf.bbits) >> vs->server.ds->pf.bbits);
+
+        po[0] = r;
+        po[1] = g;
+        po[2] = b;
+
+        pi += depth;
+        po += 3; // RGB
+    }
+}
+
+static void send_framebuffer_update_tight(VncState *vs, int x, int y, int w, int h)
+{
+    struct jpeg_compress_struct cinfo;
+    struct jpeg_error_mgr jerr;
+    uint8_t *row = ds_get_data(vs->ds) +
+                   y * ds_get_linesize(vs->ds) +
+                   x * ds_get_bytes_per_pixel(vs->ds);
+    int dy;
+    JSAMPROW row_pointer[1];
+
+    if(vnc_has_feature(vs, VNC_FEATURE_HEXTILE) && (w * h) < 300) {
+	/* Below a certain size its actually more efficient to send hextiles
+	 * Take a rough stab in the dark at 300 for text-based displays */
+	 send_framebuffer_update_hextile(vs, x, y, w, h);
+	 return;
+    }
+    
+    vnc_framebuffer_update(vs, x, y, w, h, VNC_ENCODING_TIGHT);
+
+    // XXX For now let's be stupid and always send JPEG data. Tight can do a lot more!
+
+    // Indicate its a Jpeg data stream
+    vnc_write_u8(vs, VNC_TIGHT_CCB_TYPE_JPEG);
+
+    // Compress data
+    cinfo.client_data = vs;
+    cinfo.err = jpeg_std_error(&jerr);
+    jpeg_create_compress(&cinfo);
+    cinfo.image_width = w;
+    cinfo.image_height = h;
+    cinfo.input_components = 3;
+    cinfo.in_color_space = JCS_RGB;
+
+    jpeg_set_defaults(&cinfo);
+    jpeg_set_quality(&cinfo, (vs->tight_quality+1) * 10, TRUE);
+
+    buffer_reserve(&vs->jpeg_buffer, 1024);
+    vs->jpeg_dst_manager.init_destination = jpeg_init_destination;
+    vs->jpeg_dst_manager.empty_output_buffer = jpeg_empty_output_buffer;
+    vs->jpeg_dst_manager.term_destination = jpeg_term_destination;
+    cinfo.dest = &vs->jpeg_dst_manager;
+
+    jpeg_start_compress(&cinfo, TRUE);
+
+    row_pointer[0] = qemu_malloc(3 * w);
+    for (dy = 0; dy < h; dy++) {
+        jpeg_row2pixel(vs, (char*)row, (char*)row_pointer[0], w);
+        jpeg_write_scanlines(&cinfo, row_pointer, 1);
+        row += ds_get_linesize(vs->ds);
+    }
+    qemu_free(row_pointer[0]);
+
+    jpeg_finish_compress(&cinfo);
+    jpeg_destroy_compress(&cinfo);
+
+    VNC_DEBUG("JPEG: Sending %d bytes of jpeg data\n", (int)vs->jpeg_buffer.offset);
+    vnc_send_compact_size(vs, vs->jpeg_buffer.offset);
+    vnc_write(vs, vs->jpeg_buffer.buffer, vs->jpeg_buffer.offset);
+    buffer_reset(&vs->jpeg_buffer);
+}
+#endif /* CONFIG_VNC_JPEG */
+
 static void send_framebuffer_update(VncState *vs, int x, int y, int w, int h)
 {
     switch(vs->vnc_encoding) {
+#ifdef CONFIG_VNC_JPEG
+        case VNC_ENCODING_TIGHT:
+	    send_framebuffer_update_tight(vs, x, y, w, h);
+	    break;
+#endif /* CONFIG_VNC_JPEG */
         case VNC_ENCODING_ZLIB:
             send_framebuffer_update_zlib(vs, x, y, w, h);
             break;
@@ -1552,7 +1714,7 @@ static void set_encodings(VncState *vs, int32_t *encodings, size_t n_encodings)
 
     vnc_zlib_init(vs);
     vs->features = 0;
-    vs->vnc_encoding = 0;
+    vs->vnc_encoding = -1;
     vs->tight_compression = 9;
     vs->tight_quality = 9;
     vs->absolute = -1;
@@ -1574,6 +1736,17 @@ static void set_encodings(VncState *vs, int32_t *encodings, size_t n_encodings)
             vs->features |= VNC_FEATURE_ZLIB_MASK;
             vs->vnc_encoding = enc;
             break;
+        case VNC_ENCODING_TIGHT:
+#ifdef CONFIG_VNC_JPEG
+            buffer_reset(&vs->jpeg_buffer);
+#endif
+            vs->features |= VNC_FEATURE_TIGHT_MASK;
+
+            /* We don't want to do JPEG encoding by accident, so only
+             * enable it when it's the only choice. */
+            if (vs->vnc_encoding == -1)
+                vs->vnc_encoding = enc;
+            break;
         case VNC_ENCODING_DESKTOPRESIZE:
             vs->features |= VNC_FEATURE_RESIZE_MASK;
             break;
@@ -1601,6 +1774,9 @@ static void set_encodings(VncState *vs, int32_t *encodings, size_t n_encodings)
         }
     }
 
+    if (vs->vnc_encoding == -1)
+        vs->vnc_encoding = VNC_ENCODING_RAW;
+
     check_pointer_type_change(vs, kbd_mouse_is_absolute());
 }
 
diff --git a/vnc.h b/vnc.h
index 3ae95f3..b57081f 100644
--- a/vnc.h
+++ b/vnc.h
@@ -33,6 +33,10 @@
 #include "audio/audio.h"
 #include <zlib.h>
 
+#ifdef CONFIG_VNC_JPEG
+#include <jpeglib.h>
+#endif /* CONFIG_VNC_JPEG */
+
 #include "keymaps.h"
 
 // #define _VNC_DEBUG 1
@@ -161,6 +165,11 @@ struct VncState
     Buffer zlib_tmp;
     z_stream zlib_stream[4];
 
+#ifdef CONFIG_VNC_JPEG
+    Buffer jpeg_buffer;
+    struct jpeg_destination_mgr jpeg_dst_manager;
+#endif /* CONFIG_VNC_JPEG */
+
     VncState *next;
 };
 
-- 
1.6.0.2

Re: [Qemu-devel] [PATCH] Add JPEG encoding to VNC server

[David Turner]

[-- Attachment #1: Type: text/plain, Size: 12429 bytes --]

the feature check seems to use "vnc_tls=no" if -ljpeg linking doesn't work.
shouldn't this be "vnc_jpeg=no" instead ?

On Fri, Jul 31, 2009 at 9:21 AM, Alexander Graf <agraf@suse.de> wrote:

> We know most of the tight protocol already, so implementing JPEG is rather
> easy, especially considering that I had this implementation lying around
> still anyways.
>
> The big concern Anthony raised about JPEG compression is that as soon as
> you
> use JPEG, CopyRect looks ugly for most use cases and most users probably
> don't
> want it anyways.
>
> So the road I went for this patch was to only enable JPEG encoding when
> it's
> the only available choice. Allow any other protocols? You don't get JPEG
> then.
>
> While this might sound like it renders the whole implementation useless, it
> does make sense to implement it nevertheless. I have some ideas to
> implement
> progressive encodings for video.
>
> So when we'd detect that one region is updated a lot in a short about of
> time
> with content that zlib can't really handle well, we'd just send a really
> low
> quality JPEG first and then send the update after a timer if the region
> wasn't
> updated within that timeframe.
>
> But this is all ideas so far. For now the JPEG implementation is stand
> alone,
> but would enable either me or someone else who'd like to do it worlds of
> encoding fun :-).
>
> Also, if you're daring, you can always see if JPEG performs good for your
> specific VNC workload and at least have the chance to use it.
>
> Signed-off-by: Alexander Graf <agraf@suse.de>
> ---
>  Makefile.target |    4 +
>  configure       |   24 ++++++++
>  vnc.c           |  178
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  vnc.h           |    9 +++
>  4 files changed, 214 insertions(+), 1 deletions(-)
>
> diff --git a/Makefile.target b/Makefile.target
> index 49ba08d..f1dd54d 100644
> --- a/Makefile.target
> +++ b/Makefile.target
> @@ -303,6 +303,10 @@ CPPFLAGS += $(VNC_SASL_CFLAGS)
>  LIBS += $(VNC_SASL_LIBS)
>  endif
>
> +ifdef CONFIG_VNC_JPEG
> +LIBS += $(CONFIG_VNC_JPEG_LIBS)
> +endif
> +
>  ifdef CONFIG_BLUEZ
>  LIBS += $(BLUEZ_LIBS)
>  endif
> diff --git a/configure b/configure
> index 8160bed..67fe51a 100755
> --- a/configure
> +++ b/configure
> @@ -175,6 +175,7 @@ fmod_inc=""
>  oss_lib=""
>  vnc_tls="yes"
>  vnc_sasl="yes"
> +vnc_jpeg="yes"
>  bsd="no"
>  linux="no"
>  solaris="no"
> @@ -428,6 +429,8 @@ for opt do
>   ;;
>   --disable-vnc-sasl) vnc_sasl="no"
>   ;;
> +  --disable-jpeg) vnc_jpeg="no"
> +  ;;
>   --disable-slirp) slirp="no"
>   ;;
>   --disable-vde) vde="no"
> @@ -649,6 +652,7 @@ echo "  --disable-xen            disable xen backend
> driver support"
>  echo "  --disable-brlapi         disable BrlAPI"
>  echo "  --disable-vnc-tls        disable TLS encryption for VNC server"
>  echo "  --disable-vnc-sasl       disable SASL encryption for VNC server"
> +echo "  --disable-jpeg           disable JPEG compression for VNC server"
>  echo "  --disable-curses         disable curses output"
>  echo "  --disable-curl           disable curl connectivity"
>  echo "  --disable-bluez          disable bluez stack connectivity"
> @@ -994,6 +998,21 @@ if $cc $ARCH_CFLAGS -o $TMPE $TMPC > /dev/null 2>
> /dev/null ; then
>  fi
>
>  ##########################################
> +# VNC JPEG detection
> +if test "$vnc_jpeg" = "yes" ; then
> +cat > $TMPC <<EOF
> +#include <jpeglib.h>
> +int main(void) { jpeg_compress_struct s; jpeg_create_compress(&s); return
> 0; }
> +EOF
> +    vnc_jpeg_libs="-ljpeg"
> +    if $cc $ARCH_CFLAGS -o $TMPE ${OS_CFLAGS} $TMPC $vnc_jpeg_libs >
> /dev/null 2> /dev/null ; then
> +       :
> +    else
> +       vnc_tls="no"
> +    fi
> +fi
> +
> +##########################################
>  # vde libraries probe
>  if test "$vde" = "yes" ; then
>   vde=no
> @@ -1452,6 +1471,7 @@ if test "$vnc_sasl" = "yes" ; then
>     echo "    SASL CFLAGS    $vnc_sasl_cflags"
>     echo "    SASL LIBS      $vnc_sasl_libs"
>  fi
> +echo "VNC JPEG support  $vnc_jpeg"
>  if test -n "$sparc_cpu"; then
>     echo "Target Sparc Arch $sparc_cpu"
>  fi
> @@ -1604,6 +1624,10 @@ fi
>  if test "$fnmatch" = "yes" ; then
>   echo "CONFIG_FNMATCH=y" >> $config_host_mak
>  fi
> +if test "$vnc_jpeg" = "yes" ; then
> +  echo "CONFIG_VNC_JPEG=y" >> $config_host_mak
> +  echo "CONFIG_VNC_JPEG_LIBS=$vnc_jpeg_libs" >> $config_host_mak
> +fi
>  qemu_version=`head $source_path/VERSION`
>  echo "VERSION=$qemu_version" >>$config_host_mak
>  echo "PKGVERSION=$pkgversion" >>$config_host_mak
> diff --git a/vnc.c b/vnc.c
> index 903dd95..4e0c967 100644
> --- a/vnc.c
> +++ b/vnc.c
> @@ -635,9 +635,171 @@ static void send_framebuffer_update_zlib(VncState
> *vs, int x, int y, int w, int
>     vs->output.offset = new_offset;
>  }
>
> +#ifdef CONFIG_VNC_JPEG
> +/* This is called once per encoding */
> +static void jpeg_init_destination(j_compress_ptr cinfo)
> +{
> +    VncState *vs = (VncState*)cinfo->client_data;
> +    Buffer *buffer = &vs->jpeg_buffer;
> +
> +    cinfo->dest->next_output_byte = (JOCTET *)buffer->buffer +
> buffer->offset;
> +    cinfo->dest->free_in_buffer = (size_t)(buffer->capacity -
> buffer->offset);
> +}
> +
> +/* This is called when we ran out of buffer (shouldn't happen!) */
> +static boolean jpeg_empty_output_buffer(j_compress_ptr cinfo)
> +{
> +    VncState *vs = (VncState*)cinfo->client_data;
> +    Buffer *buffer = &vs->jpeg_buffer;
> +
> +    buffer->offset = buffer->capacity;
> +    buffer_reserve(buffer, 2048);
> +    jpeg_init_destination(cinfo);
> +    return TRUE;
> +}
> +
> +/* This is called when we are done processing data */
> +static void jpeg_term_destination(j_compress_ptr cinfo)
> +{
> +    VncState *vs = (VncState*)cinfo->client_data;
> +    Buffer *buffer = &vs->jpeg_buffer;
> +
> +    buffer->offset = buffer->capacity - cinfo->dest->free_in_buffer;
> +}
> +
> +
> +static void vnc_send_compact_size(VncState *vs, int len)
> +{
> +    char buf[3];
> +    int lpc = 0;
> +    int bytes = 0;
> +
> +    /* Adapted from SendCompressedData() in
> Xvnc/programs/Xserver/hw/vnc/tight.c */
> +    buf[bytes++] = len & 0x7F;
> +    if (len > 0x7F) {
> +       buf[bytes-1] |= 0x80;
> +       buf[bytes++] = len >> 7 & 0x7F;
> +       if (len > 0x3FFF) {
> +           buf[bytes-1] |= 0x80;
> +           buf[bytes++] = len >> 14 & 0xFF;
> +       }
> +    }
> +
> +    for(lpc = 0; lpc < bytes; lpc++) {
> +       vnc_write_u8(vs, buf[lpc]);
> +    }
> +}
> +
> +static void jpeg_row2pixel(VncState *vs, char *in, char *out, int len)
> +{
> +    char *pi = in;
> +    char *po = out;
> +    int depth = vs->server.ds->pf.bytes_per_pixel;
> +    int i;
> +
> +    for (i = 0; i < len; i++) {
> +        uint32_t v;
> +        uint8_t r, g, b;
> +        switch (depth) {
> +            case 1:
> +                po[0] = pi[0];
> +                po[1] = pi[0];
> +                po[2] = pi[0];
> +                continue;
> +                break;
> +            case 2:
> +                v = *((uint16_t*)pi);
> +                break;
> +            case 4:
> +                v = *((uint32_t*)pi);
> +                break;
> +        }
> +        r = ((((v & vs->server.ds->pf.rmask) >> vs->server.ds->pf.rshift)
> +            << vs->clientds.pf.rbits) >> vs->server.ds->pf.rbits);
> +        g = ((((v & vs->server.ds->pf.gmask) >> vs->server.ds->pf.gshift)
> +            << vs->clientds.pf.gbits) >> vs->server.ds->pf.gbits);
> +        b = ((((v & vs->server.ds->pf.bmask) >> vs->server.ds->pf.bshift)
> +            << vs->clientds.pf.bbits) >> vs->server.ds->pf.bbits);
> +
> +        po[0] = r;
> +        po[1] = g;
> +        po[2] = b;
> +
> +        pi += depth;
> +        po += 3; // RGB
> +    }
> +}
> +
> +static void send_framebuffer_update_tight(VncState *vs, int x, int y, int
> w, int h)
> +{
> +    struct jpeg_compress_struct cinfo;
> +    struct jpeg_error_mgr jerr;
> +    uint8_t *row = ds_get_data(vs->ds) +
> +                   y * ds_get_linesize(vs->ds) +
> +                   x * ds_get_bytes_per_pixel(vs->ds);
> +    int dy;
> +    JSAMPROW row_pointer[1];
> +
> +    if(vnc_has_feature(vs, VNC_FEATURE_HEXTILE) && (w * h) < 300) {
> +       /* Below a certain size its actually more efficient to send
> hextiles
> +        * Take a rough stab in the dark at 300 for text-based displays */
> +        send_framebuffer_update_hextile(vs, x, y, w, h);
> +        return;
> +    }
> +
> +    vnc_framebuffer_update(vs, x, y, w, h, VNC_ENCODING_TIGHT);
> +
> +    // XXX For now let's be stupid and always send JPEG data. Tight can do
> a lot more!
> +
> +    // Indicate its a Jpeg data stream
> +    vnc_write_u8(vs, VNC_TIGHT_CCB_TYPE_JPEG);
> +
> +    // Compress data
> +    cinfo.client_data = vs;
> +    cinfo.err = jpeg_std_error(&jerr);
> +    jpeg_create_compress(&cinfo);
> +    cinfo.image_width = w;
> +    cinfo.image_height = h;
> +    cinfo.input_components = 3;
> +    cinfo.in_color_space = JCS_RGB;
> +
> +    jpeg_set_defaults(&cinfo);
> +    jpeg_set_quality(&cinfo, (vs->tight_quality+1) * 10, TRUE);
> +
> +    buffer_reserve(&vs->jpeg_buffer, 1024);
> +    vs->jpeg_dst_manager.init_destination = jpeg_init_destination;
> +    vs->jpeg_dst_manager.empty_output_buffer = jpeg_empty_output_buffer;
> +    vs->jpeg_dst_manager.term_destination = jpeg_term_destination;
> +    cinfo.dest = &vs->jpeg_dst_manager;
> +
> +    jpeg_start_compress(&cinfo, TRUE);
> +
> +    row_pointer[0] = qemu_malloc(3 * w);
> +    for (dy = 0; dy < h; dy++) {
> +        jpeg_row2pixel(vs, (char*)row, (char*)row_pointer[0], w);
> +        jpeg_write_scanlines(&cinfo, row_pointer, 1);
> +        row += ds_get_linesize(vs->ds);
> +    }
> +    qemu_free(row_pointer[0]);
> +
> +    jpeg_finish_compress(&cinfo);
> +    jpeg_destroy_compress(&cinfo);
> +
> +    VNC_DEBUG("JPEG: Sending %d bytes of jpeg data\n",
> (int)vs->jpeg_buffer.offset);
> +    vnc_send_compact_size(vs, vs->jpeg_buffer.offset);
> +    vnc_write(vs, vs->jpeg_buffer.buffer, vs->jpeg_buffer.offset);
> +    buffer_reset(&vs->jpeg_buffer);
> +}
> +#endif /* CONFIG_VNC_JPEG */
> +
>  static void send_framebuffer_update(VncState *vs, int x, int y, int w, int
> h)
>  {
>     switch(vs->vnc_encoding) {
> +#ifdef CONFIG_VNC_JPEG
> +        case VNC_ENCODING_TIGHT:
> +           send_framebuffer_update_tight(vs, x, y, w, h);
> +           break;
> +#endif /* CONFIG_VNC_JPEG */
>         case VNC_ENCODING_ZLIB:
>             send_framebuffer_update_zlib(vs, x, y, w, h);
>             break;
> @@ -1552,7 +1714,7 @@ static void set_encodings(VncState *vs, int32_t
> *encodings, size_t n_encodings)
>
>     vnc_zlib_init(vs);
>     vs->features = 0;
> -    vs->vnc_encoding = 0;
> +    vs->vnc_encoding = -1;
>     vs->tight_compression = 9;
>     vs->tight_quality = 9;
>     vs->absolute = -1;
> @@ -1574,6 +1736,17 @@ static void set_encodings(VncState *vs, int32_t
> *encodings, size_t n_encodings)
>             vs->features |= VNC_FEATURE_ZLIB_MASK;
>             vs->vnc_encoding = enc;
>             break;
> +        case VNC_ENCODING_TIGHT:
> +#ifdef CONFIG_VNC_JPEG
> +            buffer_reset(&vs->jpeg_buffer);
> +#endif
> +            vs->features |= VNC_FEATURE_TIGHT_MASK;
> +
> +            /* We don't want to do JPEG encoding by accident, so only
> +             * enable it when it's the only choice. */
> +            if (vs->vnc_encoding == -1)
> +                vs->vnc_encoding = enc;
> +            break;
>         case VNC_ENCODING_DESKTOPRESIZE:
>             vs->features |= VNC_FEATURE_RESIZE_MASK;
>             break;
> @@ -1601,6 +1774,9 @@ static void set_encodings(VncState *vs, int32_t
> *encodings, size_t n_encodings)
>         }
>     }
>
> +    if (vs->vnc_encoding == -1)
> +        vs->vnc_encoding = VNC_ENCODING_RAW;
> +
>     check_pointer_type_change(vs, kbd_mouse_is_absolute());
>  }
>
> diff --git a/vnc.h b/vnc.h
> index 3ae95f3..b57081f 100644
> --- a/vnc.h
> +++ b/vnc.h
> @@ -33,6 +33,10 @@
>  #include "audio/audio.h"
>  #include <zlib.h>
>
> +#ifdef CONFIG_VNC_JPEG
> +#include <jpeglib.h>
> +#endif /* CONFIG_VNC_JPEG */
> +
>  #include "keymaps.h"
>
>  // #define _VNC_DEBUG 1
> @@ -161,6 +165,11 @@ struct VncState
>     Buffer zlib_tmp;
>     z_stream zlib_stream[4];
>
> +#ifdef CONFIG_VNC_JPEG
> +    Buffer jpeg_buffer;
> +    struct jpeg_destination_mgr jpeg_dst_manager;
> +#endif /* CONFIG_VNC_JPEG */
> +
>     VncState *next;
>  };
>
> --
> 1.6.0.2
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 14854 bytes --]

Re: [Qemu-devel] [PATCH] Add JPEG encoding to VNC server

[Alexander Graf]

On 31.07.2009, at 14:52, David Turner wrote:

> the feature check seems to use "vnc_tls=no" if -ljpeg linking  
> doesn't work.
> shouldn't this be "vnc_jpeg=no" instead ?

Nice catch. Very true indeed.

Alex

[Qemu-devel] Re: [PATCH] Add JPEG encoding to VNC server

[Anthony Liguori]

Alexander Graf wrote:
> We know most of the tight protocol already, so implementing JPEG is rather
> easy, especially considering that I had this implementation lying around
> still anyways.
>
> The big concern Anthony raised about JPEG compression is that as soon as you
> use JPEG, CopyRect looks ugly for most use cases and most users probably don't
> want it anyways.
>
> So the road I went for this patch was to only enable JPEG encoding when it's
> the only available choice. Allow any other protocols? You don't get JPEG then.
>
> While this might sound like it renders the whole implementation useless, it
> does make sense to implement it nevertheless. I have some ideas to implement
> progressive encodings for video.
>
> So when we'd detect that one region is updated a lot in a short about of time
> with content that zlib can't really handle well, we'd just send a really low
> quality JPEG first and then send the update after a timer if the region wasn't
> updated within that timeframe.
>
> But this is all ideas so far. For now the JPEG implementation is stand alone,
> but would enable either me or someone else who'd like to do it worlds of
> encoding fun :-).
>
> Also, if you're daring, you can always see if JPEG performs good for your
> specific VNC workload and at least have the chance to use it.
>   

Most clients selectively enable the jpeg part of tight by specifying a 
jpeg quality encoding.  They usually don't do this by default.

What I would like to see, is that we respect the quality encoding and if 
tight is requested without a jpeg quality, we send another type of tight 
update (even the compressed raw tiles would be fine).

It's important for this to be useful with normal clients and for general 
assumptions about how servers deal with tight to be maintained.

> Signed-off-by: Alexander Graf <agraf@suse.de>
> ---
>  Makefile.target |    4 +
>  configure       |   24 ++++++++
>  vnc.c           |  178 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  vnc.h           |    9 +++
>  4 files changed, 214 insertions(+), 1 deletions(-)
>
> diff --git a/Makefile.target b/Makefile.target
> index 49ba08d..f1dd54d 100644
> --- a/Makefile.target
> +++ b/Makefile.target
> @@ -303,6 +303,10 @@ CPPFLAGS += $(VNC_SASL_CFLAGS)
>  LIBS += $(VNC_SASL_LIBS)
>  endif
>  
> +ifdef CONFIG_VNC_JPEG
>   

just CONFIG_JPEG please.

> +vnc_jpeg="yes"

just jpeg.

>  ##########################################
> +# VNC JPEG detection
> +if test "$vnc_jpeg" = "yes" ; then
> +cat > $TMPC <<EOF
> +#include <jpeglib.h>
> +int main(void) { jpeg_compress_struct s; jpeg_create_compress(&s); return 0; }
> +EOF
> +    vnc_jpeg_libs="-ljpeg"
> +    if $cc $ARCH_CFLAGS -o $TMPE ${OS_CFLAGS} $TMPC $vnc_jpeg_libs > /dev/null 2> /dev/null ; then
> +	:
> +    else
> +	vnc_tls="no"
>   

That's a typeo.

> +#ifdef CONFIG_VNC_JPEG
>   

How difficult would it be to introduce a vnc-tight.c?  vnc.h has all the 
necessary structure definitions.

That way, we can just avoid compiling vnc-tight.c if we don't have 
libjpeg instead of having tons of ifdefs.

> +/* This is called once per encoding */
> +static void jpeg_init_destination(j_compress_ptr cinfo)
> +{
> +    VncState *vs = (VncState*)cinfo->client_data;
> +    Buffer *buffer = &vs->jpeg_buffer;
> +
> +    cinfo->dest->next_output_byte = (JOCTET *)buffer->buffer + buffer->offset;
> +    cinfo->dest->free_in_buffer = (size_t)(buffer->capacity - buffer->offset);
> +}
> +
> +/* This is called when we ran out of buffer (shouldn't happen!) */
> +static boolean jpeg_empty_output_buffer(j_compress_ptr cinfo)
> +{
> +    VncState *vs = (VncState*)cinfo->client_data;
> +    Buffer *buffer = &vs->jpeg_buffer;
> +
> +    buffer->offset = buffer->capacity;
> +    buffer_reserve(buffer, 2048);
> +    jpeg_init_destination(cinfo);
> +    return TRUE;
> +}
> +
> +/* This is called when we are done processing data */
> +static void jpeg_term_destination(j_compress_ptr cinfo)
> +{
> +    VncState *vs = (VncState*)cinfo->client_data;
> +    Buffer *buffer = &vs->jpeg_buffer;
> +
> +    buffer->offset = buffer->capacity - cinfo->dest->free_in_buffer;
> +}
> +
> +
> +static void vnc_send_compact_size(VncState *vs, int len) 
> +{
> +    char buf[3];
> +    int lpc = 0;
> +    int bytes = 0;
> +
> +    /* Adapted from SendCompressedData() in Xvnc/programs/Xserver/hw/vnc/tight.c */
> +    buf[bytes++] = len & 0x7F;
> +    if (len > 0x7F) {
> +	buf[bytes-1] |= 0x80;
> +	buf[bytes++] = len >> 7 & 0x7F;
> +	if (len > 0x3FFF) {
> +	    buf[bytes-1] |= 0x80;
> +	    buf[bytes++] = len >> 14 & 0xFF;
> +	}
> +    }
> +
> +    for(lpc = 0; lpc < bytes; lpc++) {
> +	vnc_write_u8(vs, buf[lpc]);
> +    }
> +}
> +
> +static void jpeg_row2pixel(VncState *vs, char *in, char *out, int len)
> +{
> +    char *pi = in;
> +    char *po = out;
> +    int depth = vs->server.ds->pf.bytes_per_pixel;
> +    int i;
> +
> +    for (i = 0; i < len; i++) {
> +        uint32_t v;
> +        uint8_t r, g, b;
> +        switch (depth) {
> +            case 1:
> +                po[0] = pi[0];
> +                po[1] = pi[0];
> +                po[2] = pi[0];
> +                continue;
> +                break;
> +            case 2:
> +                v = *((uint16_t*)pi);
> +                break;
> +            case 4:
> +                v = *((uint32_t*)pi);
> +                break;
> +        }
> +        r = ((((v & vs->server.ds->pf.rmask) >> vs->server.ds->pf.rshift)
> +            << vs->clientds.pf.rbits) >> vs->server.ds->pf.rbits);
> +        g = ((((v & vs->server.ds->pf.gmask) >> vs->server.ds->pf.gshift)
> +            << vs->clientds.pf.gbits) >> vs->server.ds->pf.gbits);
> +        b = ((((v & vs->server.ds->pf.bmask) >> vs->server.ds->pf.bshift)
> +            << vs->clientds.pf.bbits) >> vs->server.ds->pf.bbits);
> +
> +        po[0] = r;
> +        po[1] = g;
> +        po[2] = b;
> +
> +        pi += depth;
> +        po += 3; // RGB
>   

No C99 comments please.

> +    }
> +}
> +
> +static void send_framebuffer_update_tight(VncState *vs, int x, int y, int w, int h)
> +{
> +    struct jpeg_compress_struct cinfo;
> +    struct jpeg_error_mgr jerr;
> +    uint8_t *row = ds_get_data(vs->ds) +
> +                   y * ds_get_linesize(vs->ds) +
> +                   x * ds_get_bytes_per_pixel(vs->ds);
> +    int dy;
> +    JSAMPROW row_pointer[1];
> +
> +    if(vnc_has_feature(vs, VNC_FEATURE_HEXTILE) && (w * h) < 300) {
>   

Watch the formatting.

After Stefano's series, this patch needs an update because vs->server no 
longer exists.

Regards,

Anthony Liguori

[Qemu-devel] Re: [PATCH] Add JPEG encoding to VNC server

[Alexander Graf]

On 31.07.2009, at 16:11, Anthony Liguori wrote:

> Alexander Graf wrote:
>> We know most of the tight protocol already, so implementing JPEG is  
>> rather
>> easy, especially considering that I had this implementation lying  
>> around
>> still anyways.
>>
>> The big concern Anthony raised about JPEG compression is that as  
>> soon as you
>> use JPEG, CopyRect looks ugly for most use cases and most users  
>> probably don't
>> want it anyways.
>>
>> So the road I went for this patch was to only enable JPEG encoding  
>> when it's
>> the only available choice. Allow any other protocols? You don't get  
>> JPEG then.
>>
>> While this might sound like it renders the whole implementation  
>> useless, it
>> does make sense to implement it nevertheless. I have some ideas to  
>> implement
>> progressive encodings for video.
>>
>> So when we'd detect that one region is updated a lot in a short  
>> about of time
>> with content that zlib can't really handle well, we'd just send a  
>> really low
>> quality JPEG first and then send the update after a timer if the  
>> region wasn't
>> updated within that timeframe.
>>
>> But this is all ideas so far. For now the JPEG implementation is  
>> stand alone,
>> but would enable either me or someone else who'd like to do it  
>> worlds of
>> encoding fun :-).
>>
>> Also, if you're daring, you can always see if JPEG performs good  
>> for your
>> specific VNC workload and at least have the chance to use it.
>>
>
> Most clients selectively enable the jpeg part of tight by specifying  
> a jpeg quality encoding.  They usually don't do this by default.

I know you said this before and then I checked some of my clients and  
they all do set quality encoding.

> What I would like to see, is that we respect the quality encoding  
> and if tight is requested without a jpeg quality, we send another  
> type of tight update (even the compressed raw tiles would be fine).
>
> It's important for this to be useful with normal clients and for  
> general assumptions about how servers deal with tight to be  
> maintained.

I agree, but this is the only "safe" way I could think of. If the  
client disables all but tight (jpeg), take it - otherwise don't.

We could of course define thresholds as in "tight quality > 5 means no  
jpeg" - but IMHO that's not exactly a good solution either.

What I'd like to see is:

-- compress several times, send only once --

   Go through the different encodings the client supports and figure  
which one is the smallest. Only send that one.

-- progressive updates --

   When playing video in my guest, I want the video data to be  
transferred as JPEG, the rest as zlib.
   That means for regions where we sent JPEG, we'd have to remember to

   a ) send lossless before doing CopyRect
   b ) send lossless after a certain amount of time the region didn't  
change

So far all of these are ideas, but it might help others or me too at  
one point to start implementing it :-).

And the good news is I had the patch around anyways and the way it's  
implemented now shouldn't hurt anyone. Admittedly, it probably doesn't  
benefit most people either. But for most workloads zlib should  
actually perform better than jpeg anyways, so there's little use in  
taking jpeg for the full desktop.

>
>> Signed-off-by: Alexander Graf <agraf@suse.de>
>> ---
>> Makefile.target |    4 +
>> configure       |   24 ++++++++
>> vnc.c           |  178 +++++++++++++++++++++++++++++++++++++++++++++ 
>> +++++++++-
>> vnc.h           |    9 +++
>> 4 files changed, 214 insertions(+), 1 deletions(-)
>>
>> diff --git a/Makefile.target b/Makefile.target
>> index 49ba08d..f1dd54d 100644
>> --- a/Makefile.target
>> +++ b/Makefile.target
>> @@ -303,6 +303,10 @@ CPPFLAGS += $(VNC_SASL_CFLAGS)
>> LIBS += $(VNC_SASL_LIBS)
>> endif
>> +ifdef CONFIG_VNC_JPEG
>>
>
> just CONFIG_JPEG please.

Ok.

>
>> +vnc_jpeg="yes"
>
> just jpeg.
>
>> ##########################################
>> +# VNC JPEG detection
>> +if test "$vnc_jpeg" = "yes" ; then
>> +cat > $TMPC <<EOF
>> +#include <jpeglib.h>
>> +int main(void) { jpeg_compress_struct s; jpeg_create_compress(&s);  
>> return 0; }
>> +EOF
>> +    vnc_jpeg_libs="-ljpeg"
>> +    if $cc $ARCH_CFLAGS -o $TMPE ${OS_CFLAGS} $TMPC $vnc_jpeg_libs  
>> > /dev/null 2> /dev/null ; then
>> +	:
>> +    else
>> +	vnc_tls="no"
>>
>
> That's a typeo.

typeo :)

>
>> +#ifdef CONFIG_VNC_JPEG
>>
>
> How difficult would it be to introduce a vnc-tight.c?  vnc.h has all  
> the necessary structure definitions.
>
> That way, we can just avoid compiling vnc-tight.c if we don't have  
> libjpeg instead of having tons of ifdefs.

Is it that many ifdefs?

We could probably move them to vnc-tight.c and create stubs for when  
it's not there - the question is if it's worth it.

>> +/* This is called once per encoding */
>> +static void jpeg_init_destination(j_compress_ptr cinfo)
>> +{
>> +    VncState *vs = (VncState*)cinfo->client_data;
>> +    Buffer *buffer = &vs->jpeg_buffer;
>> +
>> +    cinfo->dest->next_output_byte = (JOCTET *)buffer->buffer +  
>> buffer->offset;
>> +    cinfo->dest->free_in_buffer = (size_t)(buffer->capacity -  
>> buffer->offset);
>> +}
>> +
>> +/* This is called when we ran out of buffer (shouldn't happen!) */
>> +static boolean jpeg_empty_output_buffer(j_compress_ptr cinfo)
>> +{
>> +    VncState *vs = (VncState*)cinfo->client_data;
>> +    Buffer *buffer = &vs->jpeg_buffer;
>> +
>> +    buffer->offset = buffer->capacity;
>> +    buffer_reserve(buffer, 2048);
>> +    jpeg_init_destination(cinfo);
>> +    return TRUE;
>> +}
>> +
>> +/* This is called when we are done processing data */
>> +static void jpeg_term_destination(j_compress_ptr cinfo)
>> +{
>> +    VncState *vs = (VncState*)cinfo->client_data;
>> +    Buffer *buffer = &vs->jpeg_buffer;
>> +
>> +    buffer->offset = buffer->capacity - cinfo->dest->free_in_buffer;
>> +}
>> +
>> +
>> +static void vnc_send_compact_size(VncState *vs, int len) +{
>> +    char buf[3];
>> +    int lpc = 0;
>> +    int bytes = 0;
>> +
>> +    /* Adapted from SendCompressedData() in Xvnc/programs/Xserver/ 
>> hw/vnc/tight.c */
>> +    buf[bytes++] = len & 0x7F;
>> +    if (len > 0x7F) {
>> +	buf[bytes-1] |= 0x80;
>> +	buf[bytes++] = len >> 7 & 0x7F;
>> +	if (len > 0x3FFF) {
>> +	    buf[bytes-1] |= 0x80;
>> +	    buf[bytes++] = len >> 14 & 0xFF;
>> +	}
>> +    }
>> +
>> +    for(lpc = 0; lpc < bytes; lpc++) {
>> +	vnc_write_u8(vs, buf[lpc]);
>> +    }
>> +}
>> +
>> +static void jpeg_row2pixel(VncState *vs, char *in, char *out, int  
>> len)
>> +{
>> +    char *pi = in;
>> +    char *po = out;
>> +    int depth = vs->server.ds->pf.bytes_per_pixel;
>> +    int i;
>> +
>> +    for (i = 0; i < len; i++) {
>> +        uint32_t v;
>> +        uint8_t r, g, b;
>> +        switch (depth) {
>> +            case 1:
>> +                po[0] = pi[0];
>> +                po[1] = pi[0];
>> +                po[2] = pi[0];
>> +                continue;
>> +                break;
>> +            case 2:
>> +                v = *((uint16_t*)pi);
>> +                break;
>> +            case 4:
>> +                v = *((uint32_t*)pi);
>> +                break;
>> +        }
>> +        r = ((((v & vs->server.ds->pf.rmask) >> vs->server.ds- 
>> >pf.rshift)
>> +            << vs->clientds.pf.rbits) >> vs->server.ds->pf.rbits);
>> +        g = ((((v & vs->server.ds->pf.gmask) >> vs->server.ds- 
>> >pf.gshift)
>> +            << vs->clientds.pf.gbits) >> vs->server.ds->pf.gbits);
>> +        b = ((((v & vs->server.ds->pf.bmask) >> vs->server.ds- 
>> >pf.bshift)
>> +            << vs->clientds.pf.bbits) >> vs->server.ds->pf.bbits);
>> +
>> +        po[0] = r;
>> +        po[1] = g;
>> +        po[2] = b;
>> +
>> +        pi += depth;
>> +        po += 3; // RGB
>>
>
> No C99 comments please.
>
>> +    }
>> +}
>> +
>> +static void send_framebuffer_update_tight(VncState *vs, int x, int  
>> y, int w, int h)
>> +{
>> +    struct jpeg_compress_struct cinfo;
>> +    struct jpeg_error_mgr jerr;
>> +    uint8_t *row = ds_get_data(vs->ds) +
>> +                   y * ds_get_linesize(vs->ds) +
>> +                   x * ds_get_bytes_per_pixel(vs->ds);
>> +    int dy;
>> +    JSAMPROW row_pointer[1];
>> +
>> +    if(vnc_has_feature(vs, VNC_FEATURE_HEXTILE) && (w * h) < 300) {
>>
>
> Watch the formatting.
>
> After Stefano's series, this patch needs an update because vs- 
> >server no longer exists.

I always hit the timing when someone else is pushing something on the  
same codebase :-)

Alex

Re: [Qemu-devel] [PATCH] Add JPEG encoding to VNC server

[Jamie Lokier]

Alexander Graf wrote:
> While this might sound like it renders the whole implementation useless, it
> does make sense to implement it nevertheless. I have some ideas to implement
> progressive encodings for video.
> 
> So when we'd detect that one region is updated a lot in a short
> about of time with content that zlib can't really handle well, we'd
> just send a really low quality JPEG first and then send the update
> after a timer if the region wasn't updated within that timeframe.

Detecting video regions may also be possible if virtual video overlay
hardware can be offered.  It would probably require a special guest
driver.  virtio-video :-) Then again the trend, if you believe in the
<video> element demos in web browsers, is likely towards OpenGL-driven
video frame rendering because it's more versatile.

Sending JPEG quality first and a lossless update after is an
interesting idea, even for non-video.  On a slow link, seeing boxes
and buttons appear quickly might be quicker to click through a GUI, as
you can see where to click even if the text and decorations are a bit
fuzzy for a moment.  Especially if you already know what to expect.

Is it possible to use any of the lossless or "near-lossless" JPEG
encodings, and would that be any more efficient than VNC's usual
lossless encoding?

-- Jamie

Re: [Qemu-devel] [PATCH] Add JPEG encoding to VNC server

[Anthony Liguori]

Jamie Lokier wrote:
> Alexander Graf wrote:
>   
>> While this might sound like it renders the whole implementation useless, it
>> does make sense to implement it nevertheless. I have some ideas to implement
>> progressive encodings for video.
>>
>> So when we'd detect that one region is updated a lot in a short
>> about of time with content that zlib can't really handle well, we'd
>> just send a really low quality JPEG first and then send the update
>> after a timer if the region wasn't updated within that timeframe.
>>     
>
> Detecting video regions may also be possible if virtual video overlay
> hardware can be offered.  It would probably require a special guest
> driver.  virtio-video :-)

vmware-vga can do it already.

I don't want to send JPEG, I'd rather send YUV.  The nice thing about an 
overlay is that you get the yuv data directly and you get small update 
regions (which is a feature of vmware-vga).  The nice thing is that you 
get pretty close to mpeg encoding with that.  If a blit is used for tile 
motion, that's even better.

Also, hardware scaling could be coded effectively too.

>  Then again the trend, if you believe in the
> <video> element demos in web browsers, is likely towards OpenGL-driven
> video frame rendering because it's more versatile.
>
> Sending JPEG quality first and a lossless update after is an
> interesting idea, even for non-video.  On a slow link, seeing boxes
> and buttons appear quickly might be quicker to click through a GUI, as
> you can see where to click even if the text and decorations are a bit
> fuzzy for a moment.  Especially if you already know what to expect.
>
> Is it possible to use any of the lossless or "near-lossless" JPEG
> encodings, and would that be any more efficient than VNC's usual
> lossless encoding?
>   

near-lossless is going to result in very large images.  I'm quite sure 
it's worse than even hextile.

OTOH, zlib compressed raw tiles is probably pretty close to as good as 
hextile and it's quite easy to do.  Alex has already done it once for 
the zlib encoding :-)

Regards,

Anthony Liguori