[Qemu-devel] [PATCH 0/3] migration: add sztd compression

[Qemu-devel] [PATCH 0/3] migration: add sztd compression

[Denis Plotnikov]

zstd date compression algorithm shows better performance on data compression.
It might be useful to employ the algorithm in VM migration to reduce CPU usage.
A user will be able to choose between those algorithms, therefor compress-type
migration parameter is added.

Here are some results of performance comparison zstd vs gzip:

host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
migration to the same host
VM: 2xVCPU, 8G RAM total
5G RAM used, memory populated with postgreqsl data
produced by pgbench performance benchmark


Threads: 1 compress – 1 decompress

zstd provides slightly less compression ratio with almost the same
CPU usage but copes with RAM  compression roghly 2 times faster

compression type              zlib       |      zstd
---------------------------------------------------------
compression level          1       5     |   1       5
compression ratio          6.92    7.05  |   6.69    6.89
cpu idle, %                82      83    |   86      80
time, sec                  49      71    |   26      31
time diff to zlib, sec                      -25     -41


Threads: 8 compress – 2 decompress

zstd provides the same migration time with less cpu consumption

compression type         none  |        gzip(zlib)    |          zstd
------------------------------------------------------------------------------
compression level        -     |  1      5       9    |   1       5       15
compression ratio        -     |  6.94   6.99    7.14 |   6.64    6.89    6.93
time, sec                154   |  22     23      27   |   23      23      25
cpu idle, %              99    |  45     30      12   |   70      52      23
cpu idle diff to zlib          |                      |  -25%    -22%    -11%


Denis Plotnikov (3):
  migration: rework compression code for adding more data compressors
  hmp: add compress-type parameter to migration parameters
  migration: add zstd compression

 configure             |  26 ++++
 hmp.c                 |   8 ++
 migration/migration.c |  45 ++++++-
 migration/migration.h |   1 +
 migration/qemu-file.c |  39 ++----
 migration/qemu-file.h |  18 ++-
 migration/ram.c       | 291 ++++++++++++++++++++++++++++++++++--------
 qapi/migration.json   |  26 +++-
 8 files changed, 369 insertions(+), 85 deletions(-)

-- 
2.17.0

[Qemu-devel] [PATCH 1/3] migration: rework compression code for adding more data compressors

[Denis Plotnikov]

Also, the patch adds new migration parameter parameter: compress-type
to be able choose between data compressors available.

By the moment, the only available data compressor is gzip (zlib)

Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
 migration/migration.c |  42 ++++++++-
 migration/migration.h |   1 +
 migration/qemu-file.c |  39 +++------
 migration/qemu-file.h |  17 +++-
 migration/ram.c       | 196 +++++++++++++++++++++++++++++++-----------
 qapi/migration.json   |  26 ++++--
 6 files changed, 236 insertions(+), 85 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 37e06b76dc..10cecb0eeb 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -739,6 +739,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
     params->max_postcopy_bandwidth = s->parameters.max_postcopy_bandwidth;
     params->has_max_cpu_throttle = true;
     params->max_cpu_throttle = s->parameters.max_cpu_throttle;
+    params->has_compress_type = true;
+    params->compress_type = s->parameters.compress_type;
 
     return params;
 }
@@ -1027,10 +1029,27 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
  */
 static bool migrate_params_check(MigrationParameters *params, Error **errp)
 {
+    int max_compress_level = -1;
+
+    if (params->has_compress_type) {
+        switch (params->compress_type) {
+        case COMPRESSION_TYPE_ZLIB:
+            max_compress_level = 9;
+            break;
+        default:
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "compress_type",
+                       "values: 0 - gzip");
+            return false;
+        }
+    }
+
     if (params->has_compress_level &&
-        (params->compress_level > 9)) {
+        (params->compress_level > max_compress_level)) {
+        char level_range_msg[30];
+        snprintf(level_range_msg, 30, "values from 0 to %d",
+                 max_compress_level);
         error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "compress_level",
-                   "is invalid, it should be in the range of 0 to 9");
+                   level_range_msg);
         return false;
     }
 
@@ -1125,6 +1144,9 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
     *dest = migrate_get_current()->parameters;
 
     /* TODO use QAPI_CLONE() instead of duplicating it inline */
+    if (params->has_compress_type) {
+        dest->compress_type = params->compress_type;
+    }
 
     if (params->has_compress_level) {
         dest->compress_level = params->compress_level;
@@ -1272,6 +1294,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
     if (params->has_max_cpu_throttle) {
         s->parameters.max_cpu_throttle = params->max_cpu_throttle;
     }
+    if (params->has_compress_type) {
+        s->parameters.compress_type = params->compress_type;
+    }
 }
 
 void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
@@ -1938,6 +1963,15 @@ bool migrate_use_compression(void)
     return s->enabled_capabilities[MIGRATION_CAPABILITY_COMPRESS];
 }
 
+int migrate_compress_type(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->parameters.compress_type;
+}
+
 int migrate_compress_level(void)
 {
     MigrationState *s;
@@ -3234,6 +3268,9 @@ static Property migration_properties[] = {
                       decompress_error_check, true),
 
     /* Migration parameters */
+    DEFINE_PROP_UINT8("x-compress-type", MigrationState,
+                      parameters.compress_type,
+                      COMPRESSION_TYPE_ZLIB),
     DEFINE_PROP_UINT8("x-compress-level", MigrationState,
                       parameters.compress_level,
                       DEFAULT_MIGRATE_COMPRESS_LEVEL),
@@ -3346,6 +3383,7 @@ static void migration_instance_init(Object *obj)
     params->has_xbzrle_cache_size = true;
     params->has_max_postcopy_bandwidth = true;
     params->has_max_cpu_throttle = true;
+    params->has_compress_type = true;
 
     qemu_sem_init(&ms->postcopy_pause_sem, 0);
     qemu_sem_init(&ms->postcopy_pause_rp_sem, 0);
diff --git a/migration/migration.h b/migration/migration.h
index dcd05d9f87..ddb9efec86 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -280,6 +280,7 @@ bool migrate_use_return_path(void);
 uint64_t ram_get_total_transferred_pages(void);
 
 bool migrate_use_compression(void);
+int migrate_compress_type(void);
 int migrate_compress_level(void);
 int migrate_compress_threads(void);
 int migrate_compress_wait_thread(void);
diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 977b9ae07c..cd95749aa6 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -662,28 +662,10 @@ uint64_t qemu_get_be64(QEMUFile *f)
     return v;
 }
 
-/* return the size after compression, or negative value on error */
-static int qemu_compress_data(z_stream *stream, uint8_t *dest, size_t dest_len,
+static int qemu_compress_data(Compression *comp, uint8_t *dest, size_t dest_len,
                               const uint8_t *source, size_t source_len)
 {
-    int err;
-
-    err = deflateReset(stream);
-    if (err != Z_OK) {
-        return -1;
-    }
-
-    stream->avail_in = source_len;
-    stream->next_in = (uint8_t *)source;
-    stream->avail_out = dest_len;
-    stream->next_out = dest;
-
-    err = deflate(stream, Z_FINISH);
-    if (err != Z_STREAM_END) {
-        return -1;
-    }
-
-    return stream->next_out - dest;
+    return comp->process(comp, dest, dest_len, source, source_len);
 }
 
 /* Compress size bytes of data start at p and store the compressed
@@ -695,23 +677,30 @@ static int qemu_compress_data(z_stream *stream, uint8_t *dest, size_t dest_len,
  * do fflush first, if f still has no space to save the compressed
  * data, return -1.
  */
-ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
+ssize_t qemu_put_compression_data(QEMUFile *f, Compression *comp,
                                   const uint8_t *p, size_t size)
 {
-    ssize_t blen = IO_BUF_SIZE - f->buf_index - sizeof(int32_t);
+    int blen = IO_BUF_SIZE - f->buf_index - sizeof(int32_t);
+    unsigned long bound;
 
-    if (blen < compressBound(size)) {
+    bound = comp->get_bound(size);
+
+    if (blen < bound) {
         if (!qemu_file_is_writable(f)) {
+            error_report("compression: qemu file is not writable");
             return -1;
         }
+
         qemu_fflush(f);
         blen = IO_BUF_SIZE - sizeof(int32_t);
-        if (blen < compressBound(size)) {
+        if (blen < bound) {
+            error_report("compression: io buffer is too small:%d needed: %lu",
+                         IO_BUF_SIZE, bound);
             return -1;
         }
     }
 
-    blen = qemu_compress_data(stream, f->buf + f->buf_index + sizeof(int32_t),
+    blen = qemu_compress_data(comp, f->buf + f->buf_index + sizeof(int32_t),
                               blen, p, size);
     if (blen < 0) {
         return -1;
diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index 2ccfcfb2a8..24cf0d7e25 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -115,6 +115,21 @@ typedef struct QEMUFileHooks {
     QEMURamSaveFunc *save_page;
 } QEMUFileHooks;
 
+typedef enum CompressionType {
+    COMPRESSION_TYPE_ZLIB = 0,
+} CompressionType;
+
+struct Compression {
+    CompressionType type;
+    bool is_decompression;
+    void *stream;
+    int (*process)(struct Compression *comp, uint8_t *dest, size_t dest_len,
+                    const uint8_t *source, size_t source_len);
+    unsigned long (*get_bound)(unsigned long);
+};
+
+typedef struct Compression Compression;
+
 QEMUFile *qemu_fopen_ops(void *opaque, const QEMUFileOps *ops);
 void qemu_file_set_hooks(QEMUFile *f, const QEMUFileHooks *hooks);
 int qemu_get_fd(QEMUFile *f);
@@ -134,7 +149,7 @@ bool qemu_file_is_writable(QEMUFile *f);
 
 size_t qemu_peek_buffer(QEMUFile *f, uint8_t **buf, size_t size, size_t offset);
 size_t qemu_get_buffer_in_place(QEMUFile *f, uint8_t **buf, size_t size);
-ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
+ssize_t qemu_put_compression_data(QEMUFile *f, Compression *comp,
                                   const uint8_t *p, size_t size);
 int qemu_put_qemu_file(QEMUFile *f_des, QEMUFile *f_src);
 
diff --git a/migration/ram.c b/migration/ram.c
index 59191c1ed2..9ff154ed7b 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -360,8 +360,8 @@ struct CompressParam {
     ram_addr_t offset;
 
     /* internally used fields */
-    z_stream stream;
     uint8_t *originbuf;
+    Compression comp;
 };
 typedef struct CompressParam CompressParam;
 
@@ -373,7 +373,7 @@ struct DecompressParam {
     void *des;
     uint8_t *compbuf;
     int len;
-    z_stream stream;
+    Compression comp;
 };
 typedef struct DecompressParam DecompressParam;
 
@@ -394,8 +394,114 @@ static QemuThread *decompress_threads;
 static QemuMutex decomp_done_lock;
 static QemuCond decomp_done_cond;
 
-static bool do_compress_ram_page(QEMUFile *f, z_stream *stream, RAMBlock *block,
-                                 ram_addr_t offset, uint8_t *source_buf);
+static bool do_compress_ram_page(QEMUFile *f, Compression *comp,
+                                 RAMBlock *block, ram_addr_t offset,
+                                 uint8_t *source_buf);
+
+static int zlib_compress(Compression *comp, uint8_t *dest, size_t dest_len,
+                         const uint8_t *source, size_t source_len)
+{
+    int err;
+    z_stream *stream = comp->stream;
+
+    err = deflateReset(comp->stream);
+    if (err != Z_OK) {
+        return -1;
+    }
+
+    stream->avail_in = source_len;
+    stream->next_in = (uint8_t *)source;
+    stream->avail_out = dest_len;
+    stream->next_out = dest;
+
+    err = deflate(stream, Z_FINISH);
+    if (err != Z_STREAM_END) {
+        return -1;
+    }
+
+    return stream->next_out - dest;
+}
+
+static int zlib_decompress(Compression *comp, uint8_t *dest, size_t dest_len,
+                         const uint8_t *source, size_t source_len)
+{
+    int err;
+    z_stream *stream = comp->stream;
+
+    err = inflateReset(stream);
+    if (err != Z_OK) {
+        return -1;
+    }
+
+    stream->avail_in = source_len;
+    stream->next_in = (uint8_t *)source;
+    stream->avail_out = dest_len;
+    stream->next_out = dest;
+
+    err = inflate(stream, Z_NO_FLUSH);
+    if (err != Z_STREAM_END) {
+        return -1;
+    }
+
+    return stream->total_out;
+}
+
+static int init_compression(Compression *comp, CompressionType type,
+                            bool is_decompression)
+{
+    int res;
+
+    switch (type) {
+    case COMPRESSION_TYPE_ZLIB:
+        comp->stream = g_new0(z_stream, 1);
+
+        if (is_decompression) {
+            res = inflateInit(comp->stream);
+        } else {
+            res = deflateInit(comp->stream, migrate_compress_level());
+        }
+
+        if (res != Z_OK) {
+            g_free(comp->stream);
+            return 1;
+        }
+
+        if (is_decompression) {
+            comp->process = zlib_decompress;
+        } else {
+            comp->process = zlib_compress;
+        }
+
+        comp->get_bound = compressBound;
+        break;
+    default:
+        return 1;
+    }
+
+    comp->type = type;
+    comp->is_decompression = is_decompression;
+    return 0;
+}
+
+static void destroy_compression(Compression *comp)
+{
+    assert(comp);
+
+    switch (comp->type) {
+    case COMPRESSION_TYPE_ZLIB:
+        if (comp->is_decompression) {
+            inflateEnd(comp->stream);
+        } else {
+            deflateEnd(comp->stream);
+        }
+        g_free(comp->stream);
+        break;
+    default:
+        assert(false);
+    }
+
+    memset(comp, 0, sizeof(Compression));
+}
 
 static void *do_data_compress(void *opaque)
 {
@@ -412,7 +518,7 @@ static void *do_data_compress(void *opaque)
             param->block = NULL;
             qemu_mutex_unlock(&param->mutex);
 
-            zero_page = do_compress_ram_page(param->file, &param->stream,
+            zero_page = do_compress_ram_page(param->file, &param->comp,
                                              block, offset, param->originbuf);
 
             qemu_mutex_lock(&comp_done_lock);
@@ -457,7 +563,7 @@ static void compress_threads_save_cleanup(void)
         qemu_thread_join(compress_threads + i);
         qemu_mutex_destroy(&comp_param[i].mutex);
         qemu_cond_destroy(&comp_param[i].cond);
-        deflateEnd(&comp_param[i].stream);
+        destroy_compression(&comp_param->comp);
         g_free(comp_param[i].originbuf);
         qemu_fclose(comp_param[i].file);
         comp_param[i].file = NULL;
@@ -480,31 +586,32 @@ static int compress_threads_save_setup(void)
     thread_count = migrate_compress_threads();
     compress_threads = g_new0(QemuThread, thread_count);
     comp_param = g_new0(CompressParam, thread_count);
+
     qemu_cond_init(&comp_done_cond);
     qemu_mutex_init(&comp_done_lock);
     for (i = 0; i < thread_count; i++) {
-        comp_param[i].originbuf = g_try_malloc(TARGET_PAGE_SIZE);
-        if (!comp_param[i].originbuf) {
+        CompressParam *comp = &comp_param[i];
+
+        comp->originbuf = g_try_malloc(TARGET_PAGE_SIZE);
+        if (!comp->originbuf) {
             goto exit;
         }
 
-        if (deflateInit(&comp_param[i].stream,
-                        migrate_compress_level()) != Z_OK) {
-            g_free(comp_param[i].originbuf);
+        if (init_compression(&comp->comp, migrate_compress_type(), false)) {
+            g_free(comp->originbuf);
             goto exit;
         }
 
         /* comp_param[i].file is just used as a dummy buffer to save data,
          * set its ops to empty.
          */
-        comp_param[i].file = qemu_fopen_ops(NULL, &empty_ops);
-        comp_param[i].done = true;
-        comp_param[i].quit = false;
-        qemu_mutex_init(&comp_param[i].mutex);
-        qemu_cond_init(&comp_param[i].cond);
-        qemu_thread_create(compress_threads + i, "compress",
-                           do_data_compress, comp_param + i,
-                           QEMU_THREAD_JOINABLE);
+        comp->file = qemu_fopen_ops(NULL, &empty_ops);
+        comp->done = true;
+        comp->quit = false;
+        qemu_mutex_init(&comp->mutex);
+        qemu_cond_init(&comp->cond);
+        qemu_thread_create(compress_threads + i, "compress", do_data_compress,
+                           comp, QEMU_THREAD_JOINABLE);
     }
     return 0;
 
@@ -1890,8 +1997,9 @@ static int ram_save_multifd_page(RAMState *rs, RAMBlock *block,
     return 1;
 }
 
-static bool do_compress_ram_page(QEMUFile *f, z_stream *stream, RAMBlock *block,
-                                 ram_addr_t offset, uint8_t *source_buf)
+static bool do_compress_ram_page(QEMUFile *f, Compression *comp,
+                                 RAMBlock *block, ram_addr_t offset,
+                                 uint8_t *source_buf)
 {
     RAMState *rs = ram_state;
     uint8_t *p = block->host + (offset & TARGET_PAGE_MASK);
@@ -1911,7 +2019,7 @@ static bool do_compress_ram_page(QEMUFile *f, z_stream *stream, RAMBlock *block,
      * decompression
      */
     memcpy(source_buf, p, TARGET_PAGE_SIZE);
-    ret = qemu_put_compression_data(f, stream, source_buf, TARGET_PAGE_SIZE);
+    ret = qemu_put_compression_data(f, comp, source_buf, TARGET_PAGE_SIZE);
     if (ret < 0) {
         qemu_file_set_error(migrate_get_current()->to_dst_file, ret);
         error_report("compressed data failed!");
@@ -3502,28 +3610,14 @@ void ram_handle_compressed(void *host, uint8_t ch, uint64_t size)
 }
 
 /* return the size after decompression, or negative value on error */
-static int
-qemu_uncompress_data(z_stream *stream, uint8_t *dest, size_t dest_len,
-                     const uint8_t *source, size_t source_len)
+static int qemu_uncompress_data(Compression *comp, uint8_t *dest,
+                                size_t dest_len, const uint8_t *source,
+                                size_t source_len)
 {
-    int err;
-
-    err = inflateReset(stream);
-    if (err != Z_OK) {
+    if (source_len > comp->get_bound(TARGET_PAGE_SIZE)) {
         return -1;
     }
-
-    stream->avail_in = source_len;
-    stream->next_in = (uint8_t *)source;
-    stream->avail_out = dest_len;
-    stream->next_out = dest;
-
-    err = inflate(stream, Z_NO_FLUSH);
-    if (err != Z_STREAM_END) {
-        return -1;
-    }
-
-    return stream->total_out;
+    return comp->process(comp, dest, dest_len, source, source_len);
 }
 
 static void *do_data_decompress(void *opaque)
@@ -3543,7 +3637,7 @@ static void *do_data_decompress(void *opaque)
 
             pagesize = TARGET_PAGE_SIZE;
 
-            ret = qemu_uncompress_data(&param->stream, des, pagesize,
+            ret = qemu_uncompress_data(&param->comp, des, pagesize,
                                        param->compbuf, len);
             if (ret < 0 && migrate_get_current()->decompress_error_check) {
                 error_report("decompress data failed");
@@ -3614,7 +3708,7 @@ static void compress_threads_load_cleanup(void)
         qemu_thread_join(decompress_threads + i);
         qemu_mutex_destroy(&decomp_param[i].mutex);
         qemu_cond_destroy(&decomp_param[i].cond);
-        inflateEnd(&decomp_param[i].stream);
+        destroy_compression(&decomp_param[i].comp);
         g_free(decomp_param[i].compbuf);
         decomp_param[i].compbuf = NULL;
     }
@@ -3640,15 +3734,17 @@ static int compress_threads_load_setup(QEMUFile *f)
     qemu_cond_init(&decomp_done_cond);
     decomp_file = f;
     for (i = 0; i < thread_count; i++) {
-        if (inflateInit(&decomp_param[i].stream) != Z_OK) {
+        DecompressParam *decomp = &decomp_param[i];
+
+        if (init_compression(&decomp->comp, migrate_compress_type(), true)) {
             goto exit;
         }
 
-        decomp_param[i].compbuf = g_malloc0(compressBound(TARGET_PAGE_SIZE));
-        qemu_mutex_init(&decomp_param[i].mutex);
-        qemu_cond_init(&decomp_param[i].cond);
-        decomp_param[i].done = true;
-        decomp_param[i].quit = false;
+        decomp->compbuf = g_malloc0(decomp->comp.get_bound(TARGET_PAGE_SIZE));
+        qemu_mutex_init(&decomp->mutex);
+        qemu_cond_init(&decomp->cond);
+        decomp->done = true;
+        decomp->quit = false;
         qemu_thread_create(decompress_threads + i, "decompress",
                            do_data_decompress, decomp_param + i,
                            QEMU_THREAD_JOINABLE);
@@ -4169,7 +4265,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
 
         case RAM_SAVE_FLAG_COMPRESS_PAGE:
             len = qemu_get_be32(f);
-            if (len < 0 || len > compressBound(TARGET_PAGE_SIZE)) {
+            if (len < 0) {
                 error_report("Invalid compressed data length: %d", len);
                 ret = -EINVAL;
                 break;
diff --git a/qapi/migration.json b/qapi/migration.json
index 7a795ecc16..9a3110e383 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -480,10 +480,15 @@
 #
 # Migration parameters enumeration
 #
+# @compress-type: Set the compression type to be used in live migration,
+#          the compression type is an integer from the list:
+#          0 - gzip
+#
 # @compress-level: Set the compression level to be used in live migration,
-#          the compression level is an integer between 0 and 9, where 0 means
-#          no compression, 1 means the best compression speed, and 9 means best
-#          compression ratio which will consume more CPU.
+#          the compression level is an integer between 0 and 9,
+#          where 0 means no compression, 1 means the best compression speed,
+#          and the highest value depending on the compression type means
+#          the best compression ratio which will consume more CPU.
 #
 # @compress-threads: Set compression thread count to be used in live migration,
 #          the compression thread count is an integer between 1 and 255.
@@ -560,8 +565,8 @@
 # Since: 2.4
 ##
 { 'enum': 'MigrationParameter',
-  'data': ['compress-level', 'compress-threads', 'decompress-threads',
-           'compress-wait-thread',
+  'data': ['compress-type', 'compress-level', 'compress-threads',
+           'decompress-threads', 'compress-wait-thread',
            'cpu-throttle-initial', 'cpu-throttle-increment',
            'tls-creds', 'tls-hostname', 'max-bandwidth',
            'downtime-limit', 'x-checkpoint-delay', 'block-incremental',
@@ -572,6 +577,9 @@
 ##
 # @MigrateSetParameters:
 #
+# @compress-type: Compression type is used for migration.
+#                 Available types:  0 - gzip
+#
 # @compress-level: compression level
 #
 # @compress-threads: compression thread count
@@ -653,7 +661,8 @@
 # TODO either fuse back into MigrationParameters, or make
 # MigrationParameters members mandatory
 { 'struct': 'MigrateSetParameters',
-  'data': { '*compress-level': 'int',
+  'data': { '*compress-type': 'int',
+            '*compress-level': 'int',
             '*compress-threads': 'int',
             '*compress-wait-thread': 'bool',
             '*decompress-threads': 'int',
@@ -692,6 +701,8 @@
 #
 # The optional members aren't actually optional.
 #
+# @compress-type: compression type
+#
 # @compress-level: compression level
 #
 # @compress-threads: compression thread count
@@ -769,7 +780,8 @@
 # Since: 2.4
 ##
 { 'struct': 'MigrationParameters',
-  'data': { '*compress-level': 'uint8',
+  'data': { '*compress-type': 'uint8',
+            '*compress-level': 'uint8',
             '*compress-threads': 'uint8',
             '*compress-wait-thread': 'bool',
             '*decompress-threads': 'uint8',
-- 
2.17.0

[Qemu-devel] [PATCH 2/3] hmp: add compress-type parameter to migration parameters

[Denis Plotnikov]

Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
 hmp.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/hmp.c b/hmp.c
index b2a2b1f84e..5f105b816f 100644
--- a/hmp.c
+++ b/hmp.c
@@ -334,6 +334,10 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
     params = qmp_query_migrate_parameters(NULL);
 
     if (params) {
+        assert(params->has_compress_type);
+        monitor_printf(mon, "%s: %u\n",
+            MigrationParameter_str(MIGRATION_PARAMETER_COMPRESS_TYPE),
+            params->compress_type);
         assert(params->has_compress_level);
         monitor_printf(mon, "%s: %u\n",
             MigrationParameter_str(MIGRATION_PARAMETER_COMPRESS_LEVEL),
@@ -1757,6 +1761,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
         p->has_max_postcopy_bandwidth = true;
         visit_type_size(v, param, &p->max_postcopy_bandwidth, &err);
         break;
+    case MIGRATION_PARAMETER_COMPRESS_TYPE:
+        p->has_compress_type = true;
+        visit_type_int(v, param, &p->compress_type, &err);
+        break;
     default:
         assert(0);
     }
-- 
2.17.0

[Qemu-devel] [PATCH 3/3] migration: add zstd compression

[Denis Plotnikov]

zstd allows to migrate with less cpu consumption maintaining the
the same level of data compression as qzip (zlib).

Compression level for zstd is set with migration parameter "compress-level"
in the range 1 - 22. 1 - the best speed, 22 - the best compression.

Levels in the range of 20-22 should be used with care because they lead
to significant growth of CPU and memory usage.

Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
 configure             | 26 ++++++++++++
 migration/migration.c |  5 ++-
 migration/qemu-file.h |  1 +
 migration/ram.c       | 95 +++++++++++++++++++++++++++++++++++++++++++
 qapi/migration.json   |  6 +--
 5 files changed, 129 insertions(+), 4 deletions(-)

diff --git a/configure b/configure
index f8176b3c40..9dd1c18650 100755
--- a/configure
+++ b/configure
@@ -432,6 +432,7 @@ opengl_dmabuf="no"
 cpuid_h="no"
 avx2_opt=""
 zlib="yes"
+zstd="yes"
 capstone=""
 lzo=""
 snappy=""
@@ -1301,6 +1302,8 @@ for opt do
   ;;
   --disable-zlib-test) zlib="no"
   ;;
+  --disable-zstd-test) zstd="no"
+  ;;
   --disable-lzo) lzo="no"
   ;;
   --enable-lzo) lzo="yes"
@@ -3586,6 +3589,29 @@ EOF
     fi
 fi
 
+#########################################
+# zstd check
+
+if test "$zstd" != "no" ; then
+    if $pkg_config --exists libzstd; then
+        zstd_cflags=$($pkg_config --cflags libzstd)
+        zstd_libs=$($pkg_config --libs libzstd)
+        QEMU_CFLAGS="$zstd_cflags $QEMU_CFLAGS"
+        LIBS="$zstd_libs $LIBS"
+    else
+        cat > $TMPC << EOF
+#include <zstd.h>
+int main(void) { ZSTD_versionNumber(); return 0; }
+EOF
+        if compile_prog "" "-lzstd" ; then
+            LIBS="$LIBS -lzstd"
+        else
+            error_exit "zstd check failed" \
+                "Make sure to have the zstd libs and headers installed."
+        fi
+    fi
+fi
+
 ##########################################
 # SHA command probe for modules
 if test "$modules" = yes; then
diff --git a/migration/migration.c b/migration/migration.c
index 10cecb0eeb..a7875bbb47 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1036,9 +1036,12 @@ static bool migrate_params_check(MigrationParameters *params, Error **errp)
         case COMPRESSION_TYPE_ZLIB:
             max_compress_level = 9;
             break;
+        case COMPRESSION_TYPE_ZSTD:
+            max_compress_level = 22;
+            break;
         default:
             error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "compress_type",
-                       "values: 0 - gzip");
+                       "values: 0 - gzip, 1 - zstd");
             return false;
         }
     }
diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index 24cf0d7e25..7cd054f73e 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -117,6 +117,7 @@ typedef struct QEMUFileHooks {
 
 typedef enum CompressionType {
     COMPRESSION_TYPE_ZLIB = 0,
+    COMPRESSION_TYPE_ZSTD = 1,
 } CompressionType;
 
 struct Compression {
diff --git a/migration/ram.c b/migration/ram.c
index 9ff154ed7b..4be5d100df 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -57,6 +57,7 @@
 #include "qemu/uuid.h"
 #include "savevm.h"
 #include "qemu/iov.h"
+#include <zstd.h>
 
 /***********************************************************/
 /* ram save/restore */
@@ -446,6 +447,59 @@ static int zlib_decompress(Compression *comp, uint8_t *dest, size_t dest_len,
     return stream->total_out;
 }
 
+static int zstd_compress(Compression *comp, uint8_t *dest, size_t dest_len,
+                         const uint8_t *source, size_t source_len)
+{
+    int res;
+    ZSTD_inBuffer input = {source, source_len, 0};
+    ZSTD_outBuffer output = {dest, dest_len, 0};
+
+    res = ZSTD_initCStream(comp->stream, migrate_compress_level());
+
+    if (ZSTD_isError(res)) {
+        error_report("zstd: compression stream initialization error: %s",
+                     ZSTD_getErrorName(res));
+        return -1;
+    }
+
+    res = ZSTD_compressStream(comp->stream, &output, &input);
+
+    if (ZSTD_isError(res)) {
+        error_report("zstd: compression error: %s",
+                     ZSTD_getErrorName(res));
+        return -1;
+    }
+
+    res = ZSTD_endStream(comp->stream, &output);
+
+    if (ZSTD_isError(res)) {
+        error_report("zstd: end stream error: %s",
+                     ZSTD_getErrorName(res));
+        return -1;
+    }
+
+    return output.pos;
+}
+
+static int zstd_decompress(Compression *comp, uint8_t *dest, size_t dest_len,
+                         const uint8_t *source, size_t source_len)
+{
+    int res;
+    ZSTD_inBuffer input = {source, source_len, 0};
+    ZSTD_outBuffer output = {dest, dest_len, 0};
+
+    res = ZSTD_decompressStream(comp->stream, &output, &input);
+
+    if (ZSTD_isError(res)) {
+        error_report("zstd: decompression error: %s",
+                      ZSTD_getErrorName(res));
+         return -1;
+    }
+
+    return output.pos;
+}
+
+
 static int init_compression(Compression *comp, CompressionType type,
                             bool is_decompression)
 {
@@ -474,6 +528,40 @@ static int init_compression(Compression *comp, CompressionType type,
 
         comp->get_bound = compressBound;
         break;
+    case COMPRESSION_TYPE_ZSTD:
+        if (is_decompression) {
+            int res;
+
+            comp->stream = ZSTD_createDStream();
+
+            if (comp->stream == NULL) {
+                error_report("zstd: can't create decompression stream");
+                return 1;
+            }
+
+            res = ZSTD_initDStream(comp->stream);
+
+            if (ZSTD_isError(res)) {
+                error_report("zstd: can't initialzie decompression: %s",
+                             ZSTD_getErrorName(res));
+                ZSTD_freeDStream(comp->stream);
+                return 1;
+            }
+
+            comp->process = zstd_decompress;
+        } else {
+            comp->stream = ZSTD_createCStream();
+
+            if (comp->stream == NULL) {
+                error_report("zstd: can't create compression stream");
+                return 1;
+            }
+
+            comp->process = zstd_compress;
+        }
+
+        comp->get_bound = ZSTD_compressBound;
+        break;
     default:
         return 1;
     }
@@ -496,6 +584,13 @@ static void destroy_compression(Compression *comp)
         }
         g_free(comp->stream);
         break;
+    case COMPRESSION_TYPE_ZSTD:
+        if (comp->is_decompression) {
+            ZSTD_freeDStream(comp->stream);
+        } else {
+            ZSTD_freeCStream(comp->stream);
+        }
+        break;
     default:
         assert(false);
     }
diff --git a/qapi/migration.json b/qapi/migration.json
index 9a3110e383..c0d48d21d4 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -482,10 +482,10 @@
 #
 # @compress-type: Set the compression type to be used in live migration,
 #          the compression type is an integer from the list:
-#          0 - gzip
+#          0 - gzip, 1 -zstd
 #
 # @compress-level: Set the compression level to be used in live migration,
-#          the compression level is an integer between 0 and 9,
+#          the compression level is an integer between 0 and gzip:9, zstd:22
 #          where 0 means no compression, 1 means the best compression speed,
 #          and the highest value depending on the compression type means
 #          the best compression ratio which will consume more CPU.
@@ -578,7 +578,7 @@
 # @MigrateSetParameters:
 #
 # @compress-type: Compression type is used for migration.
-#                 Available types:  0 - gzip
+#                 Available types:  0 - gzip, 1 - zstd
 #
 # @compress-level: compression level
 #
-- 
2.17.0

Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression

[Denis Plotnikov]

ping!

On 26.02.2019 16:15, Denis Plotnikov wrote:
> zstd date compression algorithm shows better performance on data compression.
> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
> A user will be able to choose between those algorithms, therefor compress-type
> migration parameter is added.
> 
> Here are some results of performance comparison zstd vs gzip:
> 
> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
> migration to the same host
> VM: 2xVCPU, 8G RAM total
> 5G RAM used, memory populated with postgreqsl data
> produced by pgbench performance benchmark
> 
> 
> Threads: 1 compress – 1 decompress
> 
> zstd provides slightly less compression ratio with almost the same
> CPU usage but copes with RAM  compression roghly 2 times faster
> 
> compression type              zlib       |      zstd
> ---------------------------------------------------------
> compression level          1       5     |   1       5
> compression ratio          6.92    7.05  |   6.69    6.89
> cpu idle, %                82      83    |   86      80
> time, sec                  49      71    |   26      31
> time diff to zlib, sec                      -25     -41
> 
> 
> Threads: 8 compress – 2 decompress
> 
> zstd provides the same migration time with less cpu consumption
> 
> compression type         none  |        gzip(zlib)    |          zstd
> ------------------------------------------------------------------------------
> compression level        -     |  1      5       9    |   1       5       15
> compression ratio        -     |  6.94   6.99    7.14 |   6.64    6.89    6.93
> time, sec                154   |  22     23      27   |   23      23      25
> cpu idle, %              99    |  45     30      12   |   70      52      23
> cpu idle diff to zlib          |                      |  -25%    -22%    -11%
> 
> 
> Denis Plotnikov (3):
>    migration: rework compression code for adding more data compressors
>    hmp: add compress-type parameter to migration parameters
>    migration: add zstd compression
> 
>   configure             |  26 ++++
>   hmp.c                 |   8 ++
>   migration/migration.c |  45 ++++++-
>   migration/migration.h |   1 +
>   migration/qemu-file.c |  39 ++----
>   migration/qemu-file.h |  18 ++-
>   migration/ram.c       | 291 ++++++++++++++++++++++++++++++++++--------
>   qapi/migration.json   |  26 +++-
>   8 files changed, 369 insertions(+), 85 deletions(-)
> 

-- 
Best,
Denis

Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression

[Denis Plotnikov]

ping ping

On 04.03.2019 18:10, Denis Plotnikov wrote:
> ping!
> 
> On 26.02.2019 16:15, Denis Plotnikov wrote:
>> zstd date compression algorithm shows better performance on data compression.
>> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
>> A user will be able to choose between those algorithms, therefor compress-type
>> migration parameter is added.
>>
>> Here are some results of performance comparison zstd vs gzip:
>>
>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>> migration to the same host
>> VM: 2xVCPU, 8G RAM total
>> 5G RAM used, memory populated with postgreqsl data
>> produced by pgbench performance benchmark
>>
>>
>> Threads: 1 compress – 1 decompress
>>
>> zstd provides slightly less compression ratio with almost the same
>> CPU usage but copes with RAM  compression roghly 2 times faster
>>
>> compression type              zlib       |      zstd
>> ---------------------------------------------------------
>> compression level          1       5     |   1       5
>> compression ratio          6.92    7.05  |   6.69    6.89
>> cpu idle, %                82      83    |   86      80
>> time, sec                  49      71    |   26      31
>> time diff to zlib, sec                      -25     -41
>>
>>
>> Threads: 8 compress – 2 decompress
>>
>> zstd provides the same migration time with less cpu consumption
>>
>> compression type         none  |        gzip(zlib)    |          zstd
>> ------------------------------------------------------------------------------
>> compression level        -     |  1      5       9    |   1       5       15
>> compression ratio        -     |  6.94   6.99    7.14 |   6.64    6.89    6.93
>> time, sec                154   |  22     23      27   |   23      23      25
>> cpu idle, %              99    |  45     30      12   |   70      52      23
>> cpu idle diff to zlib          |                      |  -25%    -22%    -11%
>>
>>
>> Denis Plotnikov (3):
>>     migration: rework compression code for adding more data compressors
>>     hmp: add compress-type parameter to migration parameters
>>     migration: add zstd compression
>>
>>    configure             |  26 ++++
>>    hmp.c                 |   8 ++
>>    migration/migration.c |  45 ++++++-
>>    migration/migration.h |   1 +
>>    migration/qemu-file.c |  39 ++----
>>    migration/qemu-file.h |  18 ++-
>>    migration/ram.c       | 291 ++++++++++++++++++++++++++++++++++--------
>>    qapi/migration.json   |  26 +++-
>>    8 files changed, 369 insertions(+), 85 deletions(-)
>>
> 

-- 
Best,
Denis

Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression

[Denis Plotnikov]

ping ping ping!

On 04.03.2019 18:10, Denis Plotnikov wrote:
> ping!
> 
> On 26.02.2019 16:15, Denis Plotnikov wrote:
>> zstd date compression algorithm shows better performance on data compression.
>> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
>> A user will be able to choose between those algorithms, therefor compress-type
>> migration parameter is added.
>>
>> Here are some results of performance comparison zstd vs gzip:
>>
>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>> migration to the same host
>> VM: 2xVCPU, 8G RAM total
>> 5G RAM used, memory populated with postgreqsl data
>> produced by pgbench performance benchmark
>>
>>
>> Threads: 1 compress – 1 decompress
>>
>> zstd provides slightly less compression ratio with almost the same
>> CPU usage but copes with RAM  compression roghly 2 times faster
>>
>> compression type              zlib       |      zstd
>> ---------------------------------------------------------
>> compression level          1       5     |   1       5
>> compression ratio          6.92    7.05  |   6.69    6.89
>> cpu idle, %                82      83    |   86      80
>> time, sec                  49      71    |   26      31
>> time diff to zlib, sec                      -25     -41
>>
>>
>> Threads: 8 compress – 2 decompress
>>
>> zstd provides the same migration time with less cpu consumption
>>
>> compression type         none  |        gzip(zlib)    |          zstd
>> ------------------------------------------------------------------------------
>> compression level        -     |  1      5       9    |   1       5       15
>> compression ratio        -     |  6.94   6.99    7.14 |   6.64    6.89    6.93
>> time, sec                154   |  22     23      27   |   23      23      25
>> cpu idle, %              99    |  45     30      12   |   70      52      23
>> cpu idle diff to zlib          |                      |  -25%    -22%    -11%
>>
>>
>> Denis Plotnikov (3):
>>     migration: rework compression code for adding more data compressors
>>     hmp: add compress-type parameter to migration parameters
>>     migration: add zstd compression
>>
>>    configure             |  26 ++++
>>    hmp.c                 |   8 ++
>>    migration/migration.c |  45 ++++++-
>>    migration/migration.h |   1 +
>>    migration/qemu-file.c |  39 ++----
>>    migration/qemu-file.h |  18 ++-
>>    migration/ram.c       | 291 ++++++++++++++++++++++++++++++++++--------
>>    qapi/migration.json   |  26 +++-
>>    8 files changed, 369 insertions(+), 85 deletions(-)
>>
> 

-- 
Best,
Denis

[Qemu-devel] [PINGl] [PATCH 0/3] migration: add sztd compression

[Denis Plotnikov]

ping!

On 26.03.2019 18:51, Denis Plotnikov wrote:
> ping ping ping ping ping!
> 
> On 18.03.2019 10:53, Denis Plotnikov wrote:
>> ping ping ping ping!
>>
>> On 11.03.2019 11:20, Denis Plotnikov wrote:
>>> ping ping ping!
>>>
>>> On 04.03.2019 18:10, Denis Plotnikov wrote:
>>>> ping!
>>>>
>>>> On 26.02.2019 16:15, Denis Plotnikov wrote:
>>>>> zstd date compression algorithm shows better performance on data compression.
>>>>> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
>>>>> A user will be able to choose between those algorithms, therefor compress-type
>>>>> migration parameter is added.
>>>>>
>>>>> Here are some results of performance comparison zstd vs gzip:
>>>>>
>>>>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>>>>> migration to the same host
>>>>> VM: 2xVCPU, 8G RAM total
>>>>> 5G RAM used, memory populated with postgreqsl data
>>>>> produced by pgbench performance benchmark
>>>>>
>>>>>
>>>>> Threads: 1 compress – 1 decompress
>>>>>
>>>>> zstd provides slightly less compression ratio with almost the same
>>>>> CPU usage but copes with RAM  compression roghly 2 times faster
>>>>>
>>>>> compression type              zlib       |      zstd
>>>>> ---------------------------------------------------------
>>>>> compression level          1       5     |   1       5
>>>>> compression ratio          6.92    7.05  |   6.69    6.89
>>>>> cpu idle, %                82      83    |   86      80
>>>>> time, sec                  49      71    |   26      31
>>>>> time diff to zlib, sec                      -25     -41
>>>>>
>>>>>
>>>>> Threads: 8 compress – 2 decompress
>>>>>
>>>>> zstd provides the same migration time with less cpu consumption
>>>>>
>>>>> compression type         none  |        gzip(zlib)    |          zstd
>>>>> ------------------------------------------------------------------------------
>>>>> compression level        -     |  1      5       9    |   1       5       15
>>>>> compression ratio        -     |  6.94   6.99    7.14 |   6.64    6.89    6.93
>>>>> time, sec                154   |  22     23      27   |   23      23      25
>>>>> cpu idle, %              99    |  45     30      12   |   70      52      23
>>>>> cpu idle diff to zlib          |                      |  -25%    -22%    -11%
>>>>>
>>>>>
>>>>> Denis Plotnikov (3):
>>>>>        migration: rework compression code for adding more data compressors
>>>>>        hmp: add compress-type parameter to migration parameters
>>>>>        migration: add zstd compression
>>>>>
>>>>>       configure             |  26 ++++
>>>>>       hmp.c                 |   8 ++
>>>>>       migration/migration.c |  45 ++++++-
>>>>>       migration/migration.h |   1 +
>>>>>       migration/qemu-file.c |  39 ++----
>>>>>       migration/qemu-file.h |  18 ++-
>>>>>       migration/ram.c       | 291 ++++++++++++++++++++++++++++++++++--------
>>>>>       qapi/migration.json   |  26 +++-
>>>>>       8 files changed, 369 insertions(+), 85 deletions(-)
>>>>>
>>>>
>>>
>>
> 

-- 
Best,
Denis

[Qemu-devel] [PINGl] [PATCH 0/3] migration: add sztd compression

[Denis Plotnikov]

ping!

On 26.03.2019 18:51, Denis Plotnikov wrote:
> ping ping ping ping ping!
> 
> On 18.03.2019 10:53, Denis Plotnikov wrote:
>> ping ping ping ping!
>>
>> On 11.03.2019 11:20, Denis Plotnikov wrote:
>>> ping ping ping!
>>>
>>> On 04.03.2019 18:10, Denis Plotnikov wrote:
>>>> ping!
>>>>
>>>> On 26.02.2019 16:15, Denis Plotnikov wrote:
>>>>> zstd date compression algorithm shows better performance on data compression.
>>>>> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
>>>>> A user will be able to choose between those algorithms, therefor compress-type
>>>>> migration parameter is added.
>>>>>
>>>>> Here are some results of performance comparison zstd vs gzip:
>>>>>
>>>>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>>>>> migration to the same host
>>>>> VM: 2xVCPU, 8G RAM total
>>>>> 5G RAM used, memory populated with postgreqsl data
>>>>> produced by pgbench performance benchmark
>>>>>
>>>>>
>>>>> Threads: 1 compress – 1 decompress
>>>>>
>>>>> zstd provides slightly less compression ratio with almost the same
>>>>> CPU usage but copes with RAM  compression roghly 2 times faster
>>>>>
>>>>> compression type              zlib       |      zstd
>>>>> ---------------------------------------------------------
>>>>> compression level          1       5     |   1       5
>>>>> compression ratio          6.92    7.05  |   6.69    6.89
>>>>> cpu idle, %                82      83    |   86      80
>>>>> time, sec                  49      71    |   26      31
>>>>> time diff to zlib, sec                      -25     -41
>>>>>
>>>>>
>>>>> Threads: 8 compress – 2 decompress
>>>>>
>>>>> zstd provides the same migration time with less cpu consumption
>>>>>
>>>>> compression type         none  |        gzip(zlib)    |          zstd
>>>>> ------------------------------------------------------------------------------
>>>>> compression level        -     |  1      5       9    |   1       5       15
>>>>> compression ratio        -     |  6.94   6.99    7.14 |   6.64    6.89    6.93
>>>>> time, sec                154   |  22     23      27   |   23      23      25
>>>>> cpu idle, %              99    |  45     30      12   |   70      52      23
>>>>> cpu idle diff to zlib          |                      |  -25%    -22%    -11%
>>>>>
>>>>>
>>>>> Denis Plotnikov (3):
>>>>>        migration: rework compression code for adding more data compressors
>>>>>        hmp: add compress-type parameter to migration parameters
>>>>>        migration: add zstd compression
>>>>>
>>>>>       configure             |  26 ++++
>>>>>       hmp.c                 |   8 ++
>>>>>       migration/migration.c |  45 ++++++-
>>>>>       migration/migration.h |   1 +
>>>>>       migration/qemu-file.c |  39 ++----
>>>>>       migration/qemu-file.h |  18 ++-
>>>>>       migration/ram.c       | 291 ++++++++++++++++++++++++++++++++++--------
>>>>>       qapi/migration.json   |  26 +++-
>>>>>       8 files changed, 369 insertions(+), 85 deletions(-)
>>>>>
>>>>
>>>
>>
> 

-- 
Best,
Denis

[Qemu-devel] [PING] [PATCH 0/3] migration: add sztd compression

[Denis Plotnikov]

ping!

On 26.03.2019 18:51, Denis Plotnikov wrote:
> ping ping ping ping ping!
> 
> On 18.03.2019 10:53, Denis Plotnikov wrote:
>> ping ping ping ping!
>>
>> On 11.03.2019 11:20, Denis Plotnikov wrote:
>>> ping ping ping!
>>>
>>> On 04.03.2019 18:10, Denis Plotnikov wrote:
>>>> ping!
>>>>
>>>> On 26.02.2019 16:15, Denis Plotnikov wrote:
>>>>> zstd date compression algorithm shows better performance on data compression.
>>>>> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
>>>>> A user will be able to choose between those algorithms, therefor compress-type
>>>>> migration parameter is added.
>>>>>
>>>>> Here are some results of performance comparison zstd vs gzip:
>>>>>
>>>>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>>>>> migration to the same host
>>>>> VM: 2xVCPU, 8G RAM total
>>>>> 5G RAM used, memory populated with postgreqsl data
>>>>> produced by pgbench performance benchmark
>>>>>
>>>>>
>>>>> Threads: 1 compress – 1 decompress
>>>>>
>>>>> zstd provides slightly less compression ratio with almost the same
>>>>> CPU usage but copes with RAM  compression roghly 2 times faster
>>>>>
>>>>> compression type              zlib       |      zstd
>>>>> ---------------------------------------------------------
>>>>> compression level          1       5     |   1       5
>>>>> compression ratio          6.92    7.05  |   6.69    6.89
>>>>> cpu idle, %                82      83    |   86      80
>>>>> time, sec                  49      71    |   26      31
>>>>> time diff to zlib, sec                      -25     -41
>>>>>
>>>>>
>>>>> Threads: 8 compress – 2 decompress
>>>>>
>>>>> zstd provides the same migration time with less cpu consumption
>>>>>
>>>>> compression type         none  |        gzip(zlib)    |          zstd
>>>>> ------------------------------------------------------------------------------
>>>>> compression level        -     |  1      5       9    |   1       5       15
>>>>> compression ratio        -     |  6.94   6.99    7.14 |   6.64    6.89    6.93
>>>>> time, sec                154   |  22     23      27   |   23      23      25
>>>>> cpu idle, %              99    |  45     30      12   |   70      52      23
>>>>> cpu idle diff to zlib          |                      |  -25%    -22%    -11%
>>>>>
>>>>>
>>>>> Denis Plotnikov (3):
>>>>>        migration: rework compression code for adding more data compressors
>>>>>        hmp: add compress-type parameter to migration parameters
>>>>>        migration: add zstd compression
>>>>>
>>>>>       configure             |  26 ++++
>>>>>       hmp.c                 |   8 ++
>>>>>       migration/migration.c |  45 ++++++-
>>>>>       migration/migration.h |   1 +
>>>>>       migration/qemu-file.c |  39 ++----
>>>>>       migration/qemu-file.h |  18 ++-
>>>>>       migration/ram.c       | 291 ++++++++++++++++++++++++++++++++++--------
>>>>>       qapi/migration.json   |  26 +++-
>>>>>       8 files changed, 369 insertions(+), 85 deletions(-)
>>>>>
>>>>
>>>
>>
> 

-- 
Best,
Denis

[Qemu-devel] [PING] [PATCH 0/3] migration: add sztd compression

[Denis Plotnikov]

ping!

On 26.03.2019 18:51, Denis Plotnikov wrote:
> ping ping ping ping ping!
> 
> On 18.03.2019 10:53, Denis Plotnikov wrote:
>> ping ping ping ping!
>>
>> On 11.03.2019 11:20, Denis Plotnikov wrote:
>>> ping ping ping!
>>>
>>> On 04.03.2019 18:10, Denis Plotnikov wrote:
>>>> ping!
>>>>
>>>> On 26.02.2019 16:15, Denis Plotnikov wrote:
>>>>> zstd date compression algorithm shows better performance on data compression.
>>>>> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
>>>>> A user will be able to choose between those algorithms, therefor compress-type
>>>>> migration parameter is added.
>>>>>
>>>>> Here are some results of performance comparison zstd vs gzip:
>>>>>
>>>>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>>>>> migration to the same host
>>>>> VM: 2xVCPU, 8G RAM total
>>>>> 5G RAM used, memory populated with postgreqsl data
>>>>> produced by pgbench performance benchmark
>>>>>
>>>>>
>>>>> Threads: 1 compress – 1 decompress
>>>>>
>>>>> zstd provides slightly less compression ratio with almost the same
>>>>> CPU usage but copes with RAM  compression roghly 2 times faster
>>>>>
>>>>> compression type              zlib       |      zstd
>>>>> ---------------------------------------------------------
>>>>> compression level          1       5     |   1       5
>>>>> compression ratio          6.92    7.05  |   6.69    6.89
>>>>> cpu idle, %                82      83    |   86      80
>>>>> time, sec                  49      71    |   26      31
>>>>> time diff to zlib, sec                      -25     -41
>>>>>
>>>>>
>>>>> Threads: 8 compress – 2 decompress
>>>>>
>>>>> zstd provides the same migration time with less cpu consumption
>>>>>
>>>>> compression type         none  |        gzip(zlib)    |          zstd
>>>>> ------------------------------------------------------------------------------
>>>>> compression level        -     |  1      5       9    |   1       5       15
>>>>> compression ratio        -     |  6.94   6.99    7.14 |   6.64    6.89    6.93
>>>>> time, sec                154   |  22     23      27   |   23      23      25
>>>>> cpu idle, %              99    |  45     30      12   |   70      52      23
>>>>> cpu idle diff to zlib          |                      |  -25%    -22%    -11%
>>>>>
>>>>>
>>>>> Denis Plotnikov (3):
>>>>>        migration: rework compression code for adding more data compressors
>>>>>        hmp: add compress-type parameter to migration parameters
>>>>>        migration: add zstd compression
>>>>>
>>>>>       configure             |  26 ++++
>>>>>       hmp.c                 |   8 ++
>>>>>       migration/migration.c |  45 ++++++-
>>>>>       migration/migration.h |   1 +
>>>>>       migration/qemu-file.c |  39 ++----
>>>>>       migration/qemu-file.h |  18 ++-
>>>>>       migration/ram.c       | 291 ++++++++++++++++++++++++++++++++++--------
>>>>>       qapi/migration.json   |  26 +++-
>>>>>       8 files changed, 369 insertions(+), 85 deletions(-)
>>>>>
>>>>
>>>
>>
> 

-- 
Best,
Denis

Re: [Qemu-devel] [PING PING PING] [PATCH 0/3] migration: add sztd compression

[Denis Plotnikov]

On 05.04.2019 14:40, Denis Plotnikov wrote:
> ping!
> 
> On 26.03.2019 18:51, Denis Plotnikov wrote:
>> ping ping ping ping ping!
>>
>> On 18.03.2019 10:53, Denis Plotnikov wrote:
>>> ping ping ping ping!
>>>
>>> On 11.03.2019 11:20, Denis Plotnikov wrote:
>>>> ping ping ping!
>>>>
>>>> On 04.03.2019 18:10, Denis Plotnikov wrote:
>>>>> ping!
>>>>>
>>>>> On 26.02.2019 16:15, Denis Plotnikov wrote:
>>>>>> zstd date compression algorithm shows better performance on data 
>>>>>> compression.
>>>>>> It might be useful to employ the algorithm in VM migration to 
>>>>>> reduce CPU usage.
>>>>>> A user will be able to choose between those algorithms, therefor 
>>>>>> compress-type
>>>>>> migration parameter is added.
>>>>>>
>>>>>> Here are some results of performance comparison zstd vs gzip:
>>>>>>
>>>>>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>>>>>> migration to the same host
>>>>>> VM: 2xVCPU, 8G RAM total
>>>>>> 5G RAM used, memory populated with postgreqsl data
>>>>>> produced by pgbench performance benchmark
>>>>>>
>>>>>>
>>>>>> Threads: 1 compress – 1 decompress
>>>>>>
>>>>>> zstd provides slightly less compression ratio with almost the same
>>>>>> CPU usage but copes with RAM  compression roghly 2 times faster
>>>>>>
>>>>>> compression type              zlib       |      zstd
>>>>>> ---------------------------------------------------------
>>>>>> compression level          1       5     |   1       5
>>>>>> compression ratio          6.92    7.05  |   6.69    6.89
>>>>>> cpu idle, %                82      83    |   86      80
>>>>>> time, sec                  49      71    |   26      31
>>>>>> time diff to zlib, sec                      -25     -41
>>>>>>
>>>>>>
>>>>>> Threads: 8 compress – 2 decompress
>>>>>>
>>>>>> zstd provides the same migration time with less cpu consumption
>>>>>>
>>>>>> compression type         none  |        gzip(zlib)    |          zstd
>>>>>> ------------------------------------------------------------------------------ 
>>>>>>
>>>>>> compression level        -     |  1      5       9    |   1       
>>>>>> 5       15
>>>>>> compression ratio        -     |  6.94   6.99    7.14 |   6.64    
>>>>>> 6.89    6.93
>>>>>> time, sec                154   |  22     23      27   |   23      
>>>>>> 23      25
>>>>>> cpu idle, %              99    |  45     30      12   |   70      
>>>>>> 52      23
>>>>>> cpu idle diff to zlib          |                      |  -25%    
>>>>>> -22%    -11%
>>>>>>
>>>>>>
>>>>>> Denis Plotnikov (3):
>>>>>>        migration: rework compression code for adding more data 
>>>>>> compressors
>>>>>>        hmp: add compress-type parameter to migration parameters
>>>>>>        migration: add zstd compression
>>>>>>
>>>>>>       configure             |  26 ++++
>>>>>>       hmp.c                 |   8 ++
>>>>>>       migration/migration.c |  45 ++++++-
>>>>>>       migration/migration.h |   1 +
>>>>>>       migration/qemu-file.c |  39 ++----
>>>>>>       migration/qemu-file.h |  18 ++-
>>>>>>       migration/ram.c       | 291 
>>>>>> ++++++++++++++++++++++++++++++++++--------
>>>>>>       qapi/migration.json   |  26 +++-
>>>>>>       8 files changed, 369 insertions(+), 85 deletions(-)
>>>>>>
>>>>>
>>>>
>>>
>>
> 

-- 
Best,
Denis

Re: [Qemu-devel] [PING PING PING] [PATCH 0/3] migration: add sztd compression

[Denis Plotnikov]

On 05.04.2019 14:40, Denis Plotnikov wrote:
> ping!
> 
> On 26.03.2019 18:51, Denis Plotnikov wrote:
>> ping ping ping ping ping!
>>
>> On 18.03.2019 10:53, Denis Plotnikov wrote:
>>> ping ping ping ping!
>>>
>>> On 11.03.2019 11:20, Denis Plotnikov wrote:
>>>> ping ping ping!
>>>>
>>>> On 04.03.2019 18:10, Denis Plotnikov wrote:
>>>>> ping!
>>>>>
>>>>> On 26.02.2019 16:15, Denis Plotnikov wrote:
>>>>>> zstd date compression algorithm shows better performance on data 
>>>>>> compression.
>>>>>> It might be useful to employ the algorithm in VM migration to 
>>>>>> reduce CPU usage.
>>>>>> A user will be able to choose between those algorithms, therefor 
>>>>>> compress-type
>>>>>> migration parameter is added.
>>>>>>
>>>>>> Here are some results of performance comparison zstd vs gzip:
>>>>>>
>>>>>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>>>>>> migration to the same host
>>>>>> VM: 2xVCPU, 8G RAM total
>>>>>> 5G RAM used, memory populated with postgreqsl data
>>>>>> produced by pgbench performance benchmark
>>>>>>
>>>>>>
>>>>>> Threads: 1 compress – 1 decompress
>>>>>>
>>>>>> zstd provides slightly less compression ratio with almost the same
>>>>>> CPU usage but copes with RAM  compression roghly 2 times faster
>>>>>>
>>>>>> compression type              zlib       |      zstd
>>>>>> ---------------------------------------------------------
>>>>>> compression level          1       5     |   1       5
>>>>>> compression ratio          6.92    7.05  |   6.69    6.89
>>>>>> cpu idle, %                82      83    |   86      80
>>>>>> time, sec                  49      71    |   26      31
>>>>>> time diff to zlib, sec                      -25     -41
>>>>>>
>>>>>>
>>>>>> Threads: 8 compress – 2 decompress
>>>>>>
>>>>>> zstd provides the same migration time with less cpu consumption
>>>>>>
>>>>>> compression type         none  |        gzip(zlib)    |          zstd
>>>>>> ------------------------------------------------------------------------------ 
>>>>>>
>>>>>> compression level        -     |  1      5       9    |   1       
>>>>>> 5       15
>>>>>> compression ratio        -     |  6.94   6.99    7.14 |   6.64    
>>>>>> 6.89    6.93
>>>>>> time, sec                154   |  22     23      27   |   23      
>>>>>> 23      25
>>>>>> cpu idle, %              99    |  45     30      12   |   70      
>>>>>> 52      23
>>>>>> cpu idle diff to zlib          |                      |  -25%    
>>>>>> -22%    -11%
>>>>>>
>>>>>>
>>>>>> Denis Plotnikov (3):
>>>>>>        migration: rework compression code for adding more data 
>>>>>> compressors
>>>>>>        hmp: add compress-type parameter to migration parameters
>>>>>>        migration: add zstd compression
>>>>>>
>>>>>>       configure             |  26 ++++
>>>>>>       hmp.c                 |   8 ++
>>>>>>       migration/migration.c |  45 ++++++-
>>>>>>       migration/migration.h |   1 +
>>>>>>       migration/qemu-file.c |  39 ++----
>>>>>>       migration/qemu-file.h |  18 ++-
>>>>>>       migration/ram.c       | 291 
>>>>>> ++++++++++++++++++++++++++++++++++--------
>>>>>>       qapi/migration.json   |  26 +++-
>>>>>>       8 files changed, 369 insertions(+), 85 deletions(-)
>>>>>>
>>>>>
>>>>
>>>
>>
> 

-- 
Best,
Denis

Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression

[no-reply]

Patchew URL: https://patchew.org/QEMU/20190226131535.30361-1-dplotnikov@virtuozzo.com/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-mingw@fedora SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/install --python=/usr/bin/python3 --cross-prefix=x86_64-w64-mingw32- --enable-trace-backends=simple --enable-gnutls --enable-nettle --enable-curl --enable-vnc --enable-bzip2 --enable-guest-agent

ERROR: zstd check failed
       Make sure to have the zstd libs and headers installed.

# QEMU configure log Wed Mar  6 11:31:58 UTC 2019
---
funcs: do_compiler do_cc compile_object check_define main
lines: 92 122 619 636 0
x86_64-w64-mingw32-gcc -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c:2:2: error: #error __linux__ not defined
 #error __linux__ not defined
  ^~~~~

---
funcs: do_compiler do_cc compile_object check_define main
lines: 92 122 619 688 0
x86_64-w64-mingw32-gcc -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c:2:2: error: #error __i386__ not defined
 #error __i386__ not defined
  ^~~~~

---
funcs: do_compiler do_cc compile_object check_define main
lines: 92 122 619 691 0
x86_64-w64-mingw32-gcc -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c:2:2: error: #error __ILP32__ not defined
 #error __ILP32__ not defined
  ^~~~~

---
lines: 92 128 923 0
x86_64-w64-mingw32-gcc -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -g -liberty
/usr/lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld: cannot find -liberty
collect2: error: ld returned 1 exit status

funcs: do_compiler do_cc compile_object main
lines: 92 122 1825 0
---
funcs: do_compiler do_cc compile_prog cc_has_warning_flag main
lines: 92 128 1906 1910 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Werror -Wstring-plus-int -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
x86_64-w64-mingw32-gcc: error: unrecognized command line option '-Wstring-plus-int'; did you mean '-Wstrict-aliasing'?

funcs: do_compiler do_cc compile_prog cc_has_warning_flag main
lines: 92 128 1906 1910 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Werror -Winitializer-overrides -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
x86_64-w64-mingw32-gcc: error: unrecognized command line option '-Winitializer-overrides'; did you mean '-Wno-suggest-override'?

funcs: do_compiler do_cc compile_prog cc_has_warning_flag main
lines: 92 128 1906 1910 0
---
lines: 92 122 2165 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -Werror -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c: In function 'main':
config-temp/qemu-conf.c:4:21: error: unknown conversion type character 'z' in format [-Werror=format=]
     return printf("%zu", SIZE_MAX);
                     ^
config-temp/qemu-conf.c:4:19: error: too many arguments for format [-Werror=format-extra-args]
     return printf("%zu", SIZE_MAX);
                   ^~~~~
config-temp/qemu-conf.c:4:21: error: unknown conversion type character 'z' in format [-Werror=format=]
     return printf("%zu", SIZE_MAX);
                     ^
config-temp/qemu-conf.c:4:19: error: too many arguments for format [-Werror=format-extra-args]
     return printf("%zu", SIZE_MAX);
                   ^~~~~
cc1: all warnings being treated as errors
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2177 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
config-temp/qemu-conf.c:1:10: fatal error: sys/socket.h: No such file or directory
 #include <sys/socket.h>
          ^~~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2248 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -llzo2
config-temp/qemu-conf.c:1:10: fatal error: lzo/lzo1x.h: No such file or directory
 #include <lzo/lzo1x.h>
          ^~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2267 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lsnappy
config-temp/qemu-conf.c:1:10: fatal error: snappy-c.h: No such file or directory
 #include <snappy-c.h>
          ^~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2304 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -llzfse
config-temp/qemu-conf.c:1:10: fatal error: lzfse.h: No such file or directory
 #include <lzfse.h>
          ^~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2387 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lxenstore -lxenctrl -lxenguest
config-temp/qemu-conf.c:1:10: fatal error: xenctrl.h: No such file or directory
 #include <xenctrl.h>
          ^~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2892 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lpam
config-temp/qemu-conf.c:1:10: fatal error: security/pam_appl.h: No such file or directory
 #include <security/pam_appl.h>
          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_object check_include main
lines: 92 122 627 2907 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c:1:10: fatal error: ifaddrs.h: No such file or directory
 #include <ifaddrs.h>
          ^~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3045 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -Dmain=SDL_main -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/SDL2 -Wno-undef -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -L/usr/x86_64-w64-mingw32/sys-root/mingw/lib -lmingw32 -lSDL2main -lSDL2 -mwindows
config-temp/qemu-conf.c:5:2: error: #error No x11 support
 #error No x11 support
  ^~~~~
In file included from /usr/x86_64-w64-mingw32/sys-root/mingw/include/SDL2/SDL.h:32,
                 from config-temp/qemu-conf.c:1:
/usr/x86_64-w64-mingw32/sys-root/mingw/include/SDL2/SDL_main.h:111:17: error: conflicting types for 'SDL_main'
 #define main    SDL_main
                 ^~~~~~~~
config-temp/qemu-conf.c:7:5: note: in expansion of macro 'main'
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3060 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lrdmacm -libverbs -libumad
config-temp/qemu-conf.c:1:10: fatal error: rdma/rdma_cma.h: No such file or directory
 #include <rdma/rdma_cma.h>
          ^~~~~~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3130 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lsasl2
config-temp/qemu-conf.c:1:10: fatal error: sasl/sasl.h: No such file or directory
 #include <sasl/sasl.h>
          ^~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3224 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
config-temp/qemu-conf.c:2:10: fatal error: xfs/xfs.h: No such file or directory
 #include <xfs/xfs.h>
          ^~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3248 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lvdeplug
config-temp/qemu-conf.c:1:10: fatal error: libvdeplug.h: No such file or directory
 #include <libvdeplug.h>
          ^~~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3298 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lcap-ng
config-temp/qemu-conf.c:1:10: fatal error: cap-ng.h: No such file or directory
 #include <cap-ng.h>
          ^~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3394 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lbrlapi
config-temp/qemu-conf.c:1:10: fatal error: brlapi.h: No such file or directory
 #include <brlapi.h>
          ^~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3436 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -DNCURSES_WIDECHAR -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
config-temp/qemu-conf.c:2:10: fatal error: curses.h: No such file or directory
 #include <curses.h>
          ^~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3436 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -DNCURSES_WIDECHAR -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lpdcurses
config-temp/qemu-conf.c:2:10: fatal error: curses.h: No such file or directory
 #include <curses.h>
          ^~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3489 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
config-temp/qemu-conf.c:1:10: fatal error: bluetooth/bluetooth.h: No such file or directory
 #include <bluetooth/bluetooth.h>
          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3609 0
x86_64-w64-mingw32-gcc -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -mms-bitfields -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/glib-2.0 -I/usr/x86_64-w64-mingw32/sys-root/mingw/lib/glib-2.0/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lzstd
config-temp/qemu-conf.c:1:10: fatal error: zstd.h: No such file or directory
 #include <zstd.h>
          ^~~~~~~~
compilation terminated.


The full log is available at
http://patchew.org/logs/20190226131535.30361-1-dplotnikov@virtuozzo.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [PATCH 0/3] migration: add sztd compression

[Juan Quintela]

Denis Plotnikov <dplotnikov@virtuozzo.com> wrote:
> zstd date compression algorithm shows better performance on data compression.
> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
> A user will be able to choose between those algorithms, therefor compress-type
> migration parameter is added.
>
> Here are some results of performance comparison zstd vs gzip:

Please, could you comment on the series:

[PATCH v3 00/21] Multifd Migration Compression

That series integrated zstd/zlib compression on top of multifd,
advantages over "old" compression code are:
- We don't have to copy data back and forth
- The unit of compression is 512KB instead of 4kb
- We "conserve" the compression state between packets (this is specially
  interesting for zstd, that uses dictionaries)

> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
> migration to the same host
> VM: 2xVCPU, 8G RAM total
> 5G RAM used, memory populated with postgreqsl data
> produced by pgbench performance benchmark
>
>
> Threads: 1 compress – 1 decompress
>
> zstd provides slightly less compression ratio with almost the same
> CPU usage but copes with RAM  compression roghly 2 times faster
>
> compression type              zlib       |      zstd
> ---------------------------------------------------------
> compression level          1       5     |   1       5
> compression ratio          6.92    7.05  |   6.69    6.89
> cpu idle, %                82      83    |   86      80
> time, sec                  49      71    |   26      31
> time diff to zlib, sec                      -25     -41
>
>
> Threads: 8 compress – 2 decompress
>
> zstd provides the same migration time with less cpu consumption
>
> compression type         none  |        gzip(zlib)    |          zstd
> ------------------------------------------------------------------------------
> compression level        -     |  1      5       9    |   1       5       15
> compression ratio        -     |  6.94   6.99    7.14 |   6.64    6.89    6.93
> time, sec                154   |  22     23      27   |   23      23      25
> cpu idle, %              99    |  45     30      12   |   70      52      23
> cpu idle diff to zlib          |                      |  -25%    -22%    -11%

I don't have handy results, but it looked for me like you:
- zstd has a way better latency than zlib (i.e. the packet cames sooner)
- And it compress much better

On the migration test (best possible case for a compressor, as we are
writting just one byte of each page, and we write the same value in all
pages):

- zlib: compress 512KB -> 2500 bytes
- zstd: compess 512KB -> 52 bytes (yeap, I tested several times, it
  looked too small)

See that I posted another patch to "delete" the old compression code.
Why?
- I have been unable to modify migration-test to test it and work
  reliablely (only way was to allow a really huge downtime)
- Even with slow networking (1Gigabit) I got really mixed results,
  because as it is so slow, the guest continue dirtying memory, and in
  my tests it was never a winner

Thanks, Juan.

Re: [PATCH 0/3] migration: add sztd compression

[Denis Plotnikov]

Hi, Juan

I'll read the series soon. Thanks for sending that to me!

Denis

On 24.01.2020 15:43, Juan Quintela wrote:
> Denis Plotnikov <dplotnikov@virtuozzo.com> wrote:
>> zstd date compression algorithm shows better performance on data compression.
>> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
>> A user will be able to choose between those algorithms, therefor compress-type
>> migration parameter is added.
>>
>> Here are some results of performance comparison zstd vs gzip:
> Please, could you comment on the series:
>
> [PATCH v3 00/21] Multifd Migration Compression
>
> That series integrated zstd/zlib compression on top of multifd,
> advantages over "old" compression code are:
> - We don't have to copy data back and forth
> - The unit of compression is 512KB instead of 4kb
> - We "conserve" the compression state between packets (this is specially
>    interesting for zstd, that uses dictionaries)
>
>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>> migration to the same host
>> VM: 2xVCPU, 8G RAM total
>> 5G RAM used, memory populated with postgreqsl data
>> produced by pgbench performance benchmark
>>
>>
>> Threads: 1 compress – 1 decompress
>>
>> zstd provides slightly less compression ratio with almost the same
>> CPU usage but copes with RAM  compression roghly 2 times faster
>>
>> compression type              zlib       |      zstd
>> ---------------------------------------------------------
>> compression level          1       5     |   1       5
>> compression ratio          6.92    7.05  |   6.69    6.89
>> cpu idle, %                82      83    |   86      80
>> time, sec                  49      71    |   26      31
>> time diff to zlib, sec                      -25     -41
>>
>>
>> Threads: 8 compress – 2 decompress
>>
>> zstd provides the same migration time with less cpu consumption
>>
>> compression type         none  |        gzip(zlib)    |          zstd
>> ------------------------------------------------------------------------------
>> compression level        -     |  1      5       9    |   1       5       15
>> compression ratio        -     |  6.94   6.99    7.14 |   6.64    6.89    6.93
>> time, sec                154   |  22     23      27   |   23      23      25
>> cpu idle, %              99    |  45     30      12   |   70      52      23
>> cpu idle diff to zlib          |                      |  -25%    -22%    -11%
> I don't have handy results, but it looked for me like you:
> - zstd has a way better latency than zlib (i.e. the packet cames sooner)
> - And it compress much better
>
> On the migration test (best possible case for a compressor, as we are
> writting just one byte of each page, and we write the same value in all
> pages):
>
> - zlib: compress 512KB -> 2500 bytes
> - zstd: compess 512KB -> 52 bytes (yeap, I tested several times, it
>    looked too small)
>
> See that I posted another patch to "delete" the old compression code.
> Why?
> - I have been unable to modify migration-test to test it and work
>    reliablely (only way was to allow a really huge downtime)
> - Even with slow networking (1Gigabit) I got really mixed results,
>    because as it is so slow, the guest continue dirtying memory, and in
>    my tests it was never a winner
>
> Thanks, Juan.
>