[Qemu-devel] [RFC 0/8] block: Live backup prototype

[Qemu-devel] [RFC 0/8] block: Live backup prototype

[Stefan Hajnoczi]

Dietmar Maurer's recent "Efficient VM backup for qemu" patch series has spawned
a lot of discussion.  I'm afraid we are close to an impasse so I decided to
prototype the approach that I'm advocating - just make sure it actually works
:).

This series implements vmstate and disk backup for running guests.  It's just a
quick hack, especially the Python code is missing error handling.  My goal is
to show by example how the 'block-backup' block job should work and how a VMA
backup archive writer can be implemented as an external program.

The code is available in my public repo:

  git://github.com/stefanha/qemu.git backup-block-job

The key feature is the new 'block-backup' QMP command.  It takes a
point-in-time copy of a block device and writes it to a target device.  This is
related to block-stream and drive-mirror but the snapshot is copied out while
the guest continues writing to the block device.

Later patches are Python scripts that use the new 'block-backup' QMP command to
implement live backup.  The scripts implement Dietmar's VMA backup archive
format in order to prove that VMA can be done with 'block-backup'.

  $ python backup.py /tmp/monitor.sock backup-20130301.vma
  Running migration...
  Running block-backup...
  Finished device "virtio-drive0"
  Backup complete, terminating writer process

Behind the scenes backup.py spawns a script called vma-writer.py.  The
vma-writer.py process receives migration vmstate and 'block-backup' data.  The
'block-backup' block jobs send data to vma-writer.py using the NBD protocol.

The difference between this approach and Dietmar's series is that the backup
archive format is implemented outside QEMU and runs as a separate program.

This way, management tools like proxmox, oVirt, OpenStack, and others can
provide their preferred backup archive formats without modifying QEMU.  This
has many advantages:

 * 'block-backup' composes with 'migration' and other commands, unlike the
   monolithic 'backup' command designed just for writing backup archives

 * Backup code can be updated or added outside the QEMU release cycle

 * Choice of language, coding style, and license for backup code

 * Less QEMU code to test and maintain

The objection to this approach has been performance.  Exporting vmstate and
disk data over UNIX domain sockets to an external process incurs IPC overhead.
This prototype shows that even Python code hacked up in a day achieves decent
performance.

I'm leaving benchmarking as an exercise for the reader.  I tested a single
scenario with a 16 GB raw disk and 1 GB RAM, backup time was 28% longer (+30
seconds) than Dietmar's series.  Below are a few starting points:

 * I moved the buffer_is_zero() check from the VMA writer into the block job.
   We now skip writing zero clusters and the file contains no extents for them.

 * Migration data uses no fixed block size, there are many small writes that
   could be optimized by batching into a buffer.

 * The block job issues 64 KB writes, it should be possible to use a larger
   buffer size like the 512 KB buffer in block/stream.c.

 * Another trick in block/mirror.c is to use asynchronous I/O so that there can
   be multiple requests pending.

Dietmar Maurer (1):
  add basic backup support to block driver

Stefan Hajnoczi (7):
  block: add virtual_size to query-block QMP output
  backup: write to BlockDriverState instead of BackupDumpFunc
  block: add block_backup QMP command
  Add nbd server Python module
  Add VMA backup archive writer Python module
  Add vma-writer.py tool
  Add backup.py tool

 Makefile.objs             |   1 +
 backup.c                  | 321 ++++++++++++++++++++++++++++++++++++++++++++++
 backup.py                 |  84 ++++++++++++
 block.c                   |  72 ++++++++++-
 blockdev.c                |  92 +++++++++++++
 include/block/block.h     |   2 +
 include/block/block_int.h |  16 +++
 include/block/blockjob.h  |  10 ++
 nbd.py                    | 124 ++++++++++++++++++
 qapi-schema.json          |  33 ++++-
 qmp-commands.hx           |   6 +
 vma-writer.py             | 126 ++++++++++++++++++
 vma.py                    | 236 ++++++++++++++++++++++++++++++++++
 13 files changed, 1116 insertions(+), 7 deletions(-)
 create mode 100644 backup.c
 create mode 100644 backup.py
 create mode 100644 nbd.py
 create mode 100644 vma-writer.py
 create mode 100644 vma.py

-- 
1.8.1.4

[Qemu-devel] [RFC 1/8] block: add virtual_size to query-block QMP output

[Stefan Hajnoczi]

There is currently no way to query the size of a drive.  Add a
'virtual_size' field to the 'query-block' QMP output.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 block.c          | 1 +
 qapi-schema.json | 5 ++++-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/block.c b/block.c
index 124a9eb..0128e27 100644
--- a/block.c
+++ b/block.c
@@ -2908,6 +2908,7 @@ BlockInfo *bdrv_query_info(BlockDriverState *bs)
         info->has_inserted = true;
         info->inserted = g_malloc0(sizeof(*info->inserted));
         info->inserted->file = g_strdup(bs->filename);
+        info->inserted->virtual_size = bdrv_getlength(bs);
         info->inserted->ro = bs->read_only;
         info->inserted->drv = g_strdup(bs->drv->format_name);
         info->inserted->encrypted = bs->encrypted;
diff --git a/qapi-schema.json b/qapi-schema.json
index 28b070f..6b64aec 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -751,6 +751,8 @@
 #
 # @iops_wr: write I/O operations per second is specified
 #
+# @virtual_size: size of block device, in bytes
+#
 # Since: 0.14.0
 #
 # Notes: This interface is only found in @BlockInfo.
@@ -760,7 +762,8 @@
             '*backing_file': 'str', 'backing_file_depth': 'int',
             'encrypted': 'bool', 'encryption_key_missing': 'bool',
             'bps': 'int', 'bps_rd': 'int', 'bps_wr': 'int',
-            'iops': 'int', 'iops_rd': 'int', 'iops_wr': 'int'} }
+            'iops': 'int', 'iops_rd': 'int', 'iops_wr': 'int',
+            'virtual_size': 'int' } }
 
 ##
 # @BlockDeviceIoStatus:
-- 
1.8.1.4

[Qemu-devel] [RFC 2/8] add basic backup support to block driver

[Stefan Hajnoczi]

From: Dietmar Maurer <dietmar@proxmox.com>

Function backup_job_create() creates a block job to backup a block device.
The coroutine is started with backup_job_start().

We call backup_do_cow() for each write during backup. That function
reads the original data and pass it to backup_dump_cb().

The tracked_request infrastructure is used to serialize access.

Currently backup cluster size is hardcoded to 65536 bytes.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 Makefile.objs            |   1 +
 backup.c                 | 355 +++++++++++++++++++++++++++++++++++++++++++++++
 backup.h                 |  30 ++++
 block.c                  |  71 +++++++++-
 include/block/block.h    |   2 +
 include/block/blockjob.h |  10 ++
 6 files changed, 463 insertions(+), 6 deletions(-)
 create mode 100644 backup.c
 create mode 100644 backup.h

diff --git a/Makefile.objs b/Makefile.objs
index a68cdac..df64f70 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -13,6 +13,7 @@ block-obj-$(CONFIG_POSIX) += aio-posix.o
 block-obj-$(CONFIG_WIN32) += aio-win32.o
 block-obj-y += block/
 block-obj-y += qapi-types.o qapi-visit.o
+block-obj-y += backup.o
 
 block-obj-y += qemu-coroutine.o qemu-coroutine-lock.o qemu-coroutine-io.o
 block-obj-y += qemu-coroutine-sleep.o
diff --git a/backup.c b/backup.c
new file mode 100644
index 0000000..8955e1a
--- /dev/null
+++ b/backup.c
@@ -0,0 +1,355 @@
+/*
+ * QEMU backup
+ *
+ * Copyright (C) 2013 Proxmox Server Solutions
+ *
+ * Authors:
+ *  Dietmar Maurer (dietmar@proxmox.com)
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include <stdio.h>
+#include <errno.h>
+#include <unistd.h>
+
+#include "block/block.h"
+#include "block/block_int.h"
+#include "block/blockjob.h"
+#include "qemu/ratelimit.h"
+#include "backup.h"
+
+#define DEBUG_BACKUP 0
+
+#define USE_ALLOCATION_CHECK 0
+
+#define DPRINTF(fmt, ...) \
+    do { if (DEBUG_BACKUP) { printf("backup: " fmt, ## __VA_ARGS__); } } \
+    while (0)
+
+
+#define SLICE_TIME 100000000ULL /* ns */
+
+typedef struct BackupBlockJob {
+    BlockJob common;
+    RateLimit limit;
+    CoRwlock rwlock;
+    uint64_t sectors_read;
+    unsigned long *bitmap;
+    int bitmap_size;
+    BackupDumpFunc *backup_dump_cb;
+    BlockDriverCompletionFunc *backup_complete_cb;
+    void *opaque;
+} BackupBlockJob;
+
+static bool backup_get_bitmap(BackupBlockJob *job, int64_t cluster_num)
+{
+    assert(job);
+    assert(job->bitmap);
+
+    unsigned long val, idx, bit;
+
+    idx = cluster_num / BITS_PER_LONG;
+
+    assert(job->bitmap_size > idx);
+
+    bit = cluster_num % BITS_PER_LONG;
+    val = job->bitmap[idx];
+
+    return !!(val & (1UL << bit));
+}
+
+static void backup_set_bitmap(BackupBlockJob *job, int64_t cluster_num,
+                              bool dirty)
+{
+    assert(job);
+    assert(job->bitmap);
+
+    unsigned long val, idx, bit;
+
+    idx = cluster_num / BITS_PER_LONG;
+
+    assert(job->bitmap_size > idx);
+
+    bit = cluster_num % BITS_PER_LONG;
+    val = job->bitmap[idx];
+    if (dirty) {
+        val |= 1UL << bit;
+    } else {
+        val &= ~(1UL << bit);
+    }
+    job->bitmap[idx] = val;
+}
+
+static int coroutine_fn backup_do_cow(BlockDriverState *bs,
+                                      int64_t sector_num, int nb_sectors)
+{
+    assert(bs);
+    BackupBlockJob *job = (BackupBlockJob *)bs->job;
+    assert(job);
+
+    BlockDriver *drv = bs->drv;
+    struct iovec iov;
+    QEMUIOVector bounce_qiov;
+    void *bounce_buffer = NULL;
+    int ret = 0;
+
+    qemu_co_rwlock_rdlock(&job->rwlock);
+
+    int64_t start, end;
+
+    start = sector_num / BACKUP_BLOCKS_PER_CLUSTER;
+    end = (sector_num + nb_sectors + BACKUP_BLOCKS_PER_CLUSTER - 1) /
+        BACKUP_BLOCKS_PER_CLUSTER;
+
+    DPRINTF("brdv_co_backup_cow enter %s C%" PRId64 " %" PRId64 " %d\n",
+            bdrv_get_device_name(bs), start, sector_num, nb_sectors);
+
+    for (; start < end; start++) {
+        bool zero = 0;
+
+        if (backup_get_bitmap(job, start)) {
+            DPRINTF("brdv_co_backup_cow skip C%" PRId64 "\n", start);
+            continue; /* already copied */
+        }
+
+        /* immediately set bitmap (avoid coroutine race) */
+        backup_set_bitmap(job, start, 1);
+
+        DPRINTF("brdv_co_backup_cow C%" PRId64 "\n", start);
+
+        if (!bounce_buffer) {
+            iov.iov_len = BACKUP_CLUSTER_SIZE;
+            iov.iov_base = bounce_buffer = qemu_blockalign(bs, iov.iov_len);
+            qemu_iovec_init_external(&bounce_qiov, &iov, 1);
+        }
+
+#if USE_ALLOCATION_CHECK
+        int n = 0;
+        ret = bdrv_co_is_allocated_above(bs, NULL,
+                                         start * BACKUP_BLOCKS_PER_CLUSTER,
+                                         BACKUP_BLOCKS_PER_CLUSTER, &n);
+        if (ret < 0) {
+            DPRINTF("brdv_co_backup_cow is_allocated C%" PRId64 " failed\n",
+                    start);
+            goto out;
+        }
+
+        zero = (ret == 0) && (n == BACKUP_BLOCKS_PER_CLUSTER);
+
+        if (!zero) {
+#endif
+            ret = drv->bdrv_co_readv(bs, start * BACKUP_BLOCKS_PER_CLUSTER,
+                                     BACKUP_BLOCKS_PER_CLUSTER,
+                                     &bounce_qiov);
+            if (ret < 0) {
+                DPRINTF("brdv_co_backup_cow bdrv_read C%" PRId64 " failed\n",
+                        start);
+                goto out;
+            }
+#if USE_ALLOCATION_CHECK
+        }
+#endif
+        job->sectors_read += BACKUP_BLOCKS_PER_CLUSTER;
+
+        ret = job->backup_dump_cb(job->opaque, bs, start,
+                                  zero ? NULL : bounce_buffer);
+        if (ret < 0) {
+            DPRINTF("brdv_co_backup_cow dump_cluster_cb C%" PRId64 " failed\n",
+                    start);
+            goto out;
+        }
+
+        DPRINTF("brdv_co_backup_cow done C%" PRId64 "\n", start);
+    }
+
+out:
+    if (bounce_buffer) {
+        qemu_vfree(bounce_buffer);
+    }
+
+    qemu_co_rwlock_unlock(&job->rwlock);
+
+    return ret;
+}
+
+static int coroutine_fn backup_before_read(BlockDriverState *bs,
+                                           int64_t sector_num,
+                                           int nb_sectors, QEMUIOVector *qiov)
+{
+    return backup_do_cow(bs, sector_num, nb_sectors);
+}
+
+static int coroutine_fn backup_before_write(BlockDriverState *bs,
+                                            int64_t sector_num,
+                                            int nb_sectors, QEMUIOVector *qiov)
+{
+    return backup_do_cow(bs, sector_num, nb_sectors);
+}
+
+static void backup_set_speed(BlockJob *job, int64_t speed, Error **errp)
+{
+    BackupBlockJob *s = container_of(job, BackupBlockJob, common);
+
+    if (speed < 0) {
+        error_set(errp, QERR_INVALID_PARAMETER, "speed");
+        return;
+    }
+    ratelimit_set_speed(&s->limit, speed / BDRV_SECTOR_SIZE, SLICE_TIME);
+}
+
+static BlockJobType backup_job_type = {
+    .instance_size = sizeof(BackupBlockJob),
+    .before_read = backup_before_read,
+    .before_write = backup_before_write,
+    .job_type = "backup",
+    .set_speed = backup_set_speed,
+};
+
+static void coroutine_fn backup_run(void *opaque)
+{
+    BackupBlockJob *job = opaque;
+    BlockDriverState *bs = job->common.bs;
+    assert(bs);
+
+    int64_t start, end;
+
+    start = 0;
+    end = (bs->total_sectors + BACKUP_BLOCKS_PER_CLUSTER - 1) /
+        BACKUP_BLOCKS_PER_CLUSTER;
+
+    DPRINTF("backup_run start %s %" PRId64 " %" PRId64 "\n",
+            bdrv_get_device_name(bs), start, end);
+
+    int ret = 0;
+
+    for (; start < end; start++) {
+        if (block_job_is_cancelled(&job->common)) {
+            ret = -1;
+            break;
+        }
+
+        /* we need to yield so that qemu_aio_flush() returns.
+         * (without, VM does not reboot)
+         * Note: use 1000 instead of 0 (0 prioritize this task too much)
+         */
+        if (job->common.speed) {
+            uint64_t delay_ns = ratelimit_calculate_delay(
+                &job->limit, job->sectors_read);
+            job->sectors_read = 0;
+            block_job_sleep_ns(&job->common, rt_clock, delay_ns);
+        } else {
+            block_job_sleep_ns(&job->common, rt_clock, 1000);
+        }
+
+        if (block_job_is_cancelled(&job->common)) {
+            ret = -1;
+            break;
+        }
+
+        if (backup_get_bitmap(job, start)) {
+            continue; /* already copied */
+        }
+
+        DPRINTF("backup_run loop C%" PRId64 "\n", start);
+
+        /**
+         * This triggers a cluster copy
+         * Note: avoid direct call to brdv_co_backup_cow, because
+         * this does not call tracked_request_begin()
+         */
+        ret = bdrv_co_backup(bs, start*BACKUP_BLOCKS_PER_CLUSTER, 1);
+        if (ret < 0) {
+            break;
+        }
+        /* Publish progress */
+        job->common.offset += BACKUP_CLUSTER_SIZE;
+    }
+
+    /* wait until pending backup_do_cow()calls have completed */
+    qemu_co_rwlock_wrlock(&job->rwlock);
+    qemu_co_rwlock_unlock(&job->rwlock);
+
+    DPRINTF("backup_run complete %d\n", ret);
+    block_job_completed(&job->common, ret);
+}
+
+static void backup_job_cleanup_cb(void *opaque, int ret)
+{
+    BlockDriverState *bs = opaque;
+    assert(bs);
+    BackupBlockJob *job = (BackupBlockJob *)bs->job;
+    assert(job);
+
+    DPRINTF("backup_job_cleanup_cb start %d\n", ret);
+
+    job->backup_complete_cb(job->opaque, ret);
+
+    DPRINTF("backup_job_cleanup_cb end\n");
+
+    g_free(job->bitmap);
+}
+
+void
+backup_job_start(BlockDriverState *bs, bool cancel)
+{
+    assert(bs);
+    assert(bs->job);
+    assert(bs->job->co == NULL);
+
+    if (cancel) {
+        block_job_cancel(bs->job); /* set cancel flag */
+    }
+
+    bs->job->co = qemu_coroutine_create(backup_run);
+    qemu_coroutine_enter(bs->job->co, bs->job);
+}
+
+int
+backup_job_create(BlockDriverState *bs, BackupDumpFunc *backup_dump_cb,
+                  BlockDriverCompletionFunc *backup_complete_cb,
+                  void *opaque, int64_t speed)
+{
+    assert(bs);
+    assert(backup_dump_cb);
+    assert(backup_complete_cb);
+
+    if (bs->job) {
+        DPRINTF("bdrv_backup_init failed - running job on %s\n",
+                bdrv_get_device_name(bs));
+        return -1;
+    }
+
+    int64_t bitmap_size;
+    const char *devname = bdrv_get_device_name(bs);
+
+    if (!devname || !devname[0]) {
+        return -1;
+    }
+
+    DPRINTF("bdrv_backup_init %s\n", bdrv_get_device_name(bs));
+
+    Error *errp;
+    BackupBlockJob *job = block_job_create(&backup_job_type, bs, speed,
+                                           backup_job_cleanup_cb, bs, &errp);
+
+    qemu_co_rwlock_init(&job->rwlock);
+
+    job->common.cluster_size = BACKUP_CLUSTER_SIZE;
+
+    bitmap_size = bs->total_sectors +
+        BACKUP_BLOCKS_PER_CLUSTER * BITS_PER_LONG - 1;
+    bitmap_size /= BACKUP_BLOCKS_PER_CLUSTER * BITS_PER_LONG;
+
+    job->backup_dump_cb = backup_dump_cb;
+    job->backup_complete_cb = backup_complete_cb;
+    job->opaque = opaque;
+    job->bitmap_size = bitmap_size;
+    job->bitmap = g_new0(unsigned long, bitmap_size);
+
+    job->common.len = bs->total_sectors*BDRV_SECTOR_SIZE;
+
+    return 0;
+}
diff --git a/backup.h b/backup.h
new file mode 100644
index 0000000..9b1ea1c
--- /dev/null
+++ b/backup.h
@@ -0,0 +1,30 @@
+/*
+ * QEMU backup related definitions
+ *
+ * Copyright (C) 2013 Proxmox Server Solutions
+ *
+ * Authors:
+ *  Dietmar Maurer (dietmar@proxmox.com)
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef QEMU_BACKUP_H
+#define QEMU_BACKUP_H
+
+#define BACKUP_CLUSTER_BITS 16
+#define BACKUP_CLUSTER_SIZE (1<<BACKUP_CLUSTER_BITS)
+#define BACKUP_BLOCKS_PER_CLUSTER (BACKUP_CLUSTER_SIZE/BDRV_SECTOR_SIZE)
+
+typedef int BackupDumpFunc(void *opaque, BlockDriverState *bs,
+                           int64_t cluster_num, unsigned char *buf);
+
+void backup_job_start(BlockDriverState *bs, bool cancel);
+
+int backup_job_create(BlockDriverState *bs, BackupDumpFunc *backup_dump_cb,
+                      BlockDriverCompletionFunc *backup_complete_cb,
+                      void *opaque, int64_t speed);
+
+#endif /* QEMU_BACKUP_H */
diff --git a/block.c b/block.c
index 0128e27..fbb51e9 100644
--- a/block.c
+++ b/block.c
@@ -54,6 +54,7 @@
 typedef enum {
     BDRV_REQ_COPY_ON_READ = 0x1,
     BDRV_REQ_ZERO_WRITE   = 0x2,
+    BDRV_REQ_BACKUP_ONLY  = 0x4,
 } BdrvRequestFlags;
 
 static void bdrv_dev_change_media_cb(BlockDriverState *bs, bool load);
@@ -1574,7 +1575,7 @@ int bdrv_commit(BlockDriverState *bs)
 
     if (!drv)
         return -ENOMEDIUM;
-    
+
     if (!bs->backing_hd) {
         return -ENOTSUP;
     }
@@ -1713,6 +1714,22 @@ void bdrv_round_to_clusters(BlockDriverState *bs,
     }
 }
 
+/**
+ * Round a region to job cluster boundaries
+ */
+static void round_to_job_clusters(BlockDriverState *bs,
+                                  int64_t sector_num, int nb_sectors,
+                                  int job_cluster_size,
+                                  int64_t *cluster_sector_num,
+                                  int *cluster_nb_sectors)
+{
+    int64_t c = job_cluster_size/BDRV_SECTOR_SIZE;
+
+    *cluster_sector_num = QEMU_ALIGN_DOWN(sector_num, c);
+    *cluster_nb_sectors = QEMU_ALIGN_UP(sector_num - *cluster_sector_num +
+                                        nb_sectors, c);
+}
+
 static bool tracked_request_overlaps(BdrvTrackedRequest *req,
                                      int64_t sector_num, int nb_sectors) {
     /*        aaaa   bbbb */
@@ -1727,7 +1744,9 @@ static bool tracked_request_overlaps(BdrvTrackedRequest *req,
 }
 
 static void coroutine_fn wait_for_overlapping_requests(BlockDriverState *bs,
-        int64_t sector_num, int nb_sectors)
+                                                       int64_t sector_num,
+                                                       int nb_sectors,
+                                                       int job_cluster_size)
 {
     BdrvTrackedRequest *req;
     int64_t cluster_sector_num;
@@ -1743,6 +1762,11 @@ static void coroutine_fn wait_for_overlapping_requests(BlockDriverState *bs,
     bdrv_round_to_clusters(bs, sector_num, nb_sectors,
                            &cluster_sector_num, &cluster_nb_sectors);
 
+    if (job_cluster_size) {
+        round_to_job_clusters(bs, sector_num, nb_sectors, job_cluster_size,
+                              &cluster_sector_num, &cluster_nb_sectors);
+    }
+
     do {
         retry = false;
         QLIST_FOREACH(req, &bs->tracked_requests, list) {
@@ -2282,12 +2306,24 @@ static int coroutine_fn bdrv_co_do_readv(BlockDriverState *bs,
         bs->copy_on_read_in_flight++;
     }
 
-    if (bs->copy_on_read_in_flight) {
-        wait_for_overlapping_requests(bs, sector_num, nb_sectors);
+    int job_cluster_size = bs->job && bs->job->cluster_size ?
+        bs->job->cluster_size : 0;
+
+    if (bs->copy_on_read_in_flight || job_cluster_size) {
+        wait_for_overlapping_requests(bs, sector_num, nb_sectors,
+                                      job_cluster_size);
     }
 
     tracked_request_begin(&req, bs, sector_num, nb_sectors, false);
 
+    if (bs->job && bs->job->job_type->before_read) {
+        ret = bs->job->job_type->before_read(bs, sector_num, nb_sectors, qiov);
+        if ((ret < 0) || (flags & BDRV_REQ_BACKUP_ONLY)) {
+            /* Note: We do not return any data to the caller */
+            goto out;
+        }
+    }
+
     if (flags & BDRV_REQ_COPY_ON_READ) {
         int pnum;
 
@@ -2331,6 +2367,17 @@ int coroutine_fn bdrv_co_copy_on_readv(BlockDriverState *bs,
                             BDRV_REQ_COPY_ON_READ);
 }
 
+int coroutine_fn bdrv_co_backup(BlockDriverState *bs,
+    int64_t sector_num, int nb_sectors)
+{
+    if (!bs->job) {
+        return -ENOTSUP;
+    }
+
+    return bdrv_co_do_readv(bs, sector_num, nb_sectors, NULL,
+                            BDRV_REQ_BACKUP_ONLY);
+}
+
 static int coroutine_fn bdrv_co_do_write_zeroes(BlockDriverState *bs,
     int64_t sector_num, int nb_sectors)
 {
@@ -2388,12 +2435,23 @@ static int coroutine_fn bdrv_co_do_writev(BlockDriverState *bs,
         bdrv_io_limits_intercept(bs, true, nb_sectors);
     }
 
-    if (bs->copy_on_read_in_flight) {
-        wait_for_overlapping_requests(bs, sector_num, nb_sectors);
+    int job_cluster_size = bs->job && bs->job->cluster_size ?
+        bs->job->cluster_size : 0;
+
+    if (bs->copy_on_read_in_flight || job_cluster_size) {
+        wait_for_overlapping_requests(bs, sector_num, nb_sectors,
+                                      job_cluster_size);
     }
 
     tracked_request_begin(&req, bs, sector_num, nb_sectors, true);
 
+    if (bs->job && bs->job->job_type->before_write) {
+        ret = bs->job->job_type->before_write(bs, sector_num, nb_sectors, qiov);
+        if (ret < 0) {
+            goto out;
+        }
+    }
+
     if (flags & BDRV_REQ_ZERO_WRITE) {
         ret = bdrv_co_do_write_zeroes(bs, sector_num, nb_sectors);
     } else {
@@ -2412,6 +2470,7 @@ static int coroutine_fn bdrv_co_do_writev(BlockDriverState *bs,
         bs->wr_highest_sector = sector_num + nb_sectors - 1;
     }
 
+out:
     tracked_request_end(&req);
 
     return ret;
diff --git a/include/block/block.h b/include/block/block.h
index 0f750d7..7095aeb 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -175,6 +175,8 @@ int coroutine_fn bdrv_co_readv(BlockDriverState *bs, int64_t sector_num,
     int nb_sectors, QEMUIOVector *qiov);
 int coroutine_fn bdrv_co_copy_on_readv(BlockDriverState *bs,
     int64_t sector_num, int nb_sectors, QEMUIOVector *qiov);
+int coroutine_fn bdrv_co_backup(BlockDriverState *bs,
+    int64_t sector_num, int nb_sectors);
 int coroutine_fn bdrv_co_writev(BlockDriverState *bs, int64_t sector_num,
     int nb_sectors, QEMUIOVector *qiov);
 /*
diff --git a/include/block/blockjob.h b/include/block/blockjob.h
index c290d07..6f42495 100644
--- a/include/block/blockjob.h
+++ b/include/block/blockjob.h
@@ -50,6 +50,13 @@ typedef struct BlockJobType {
      * manually.
      */
     void (*complete)(BlockJob *job, Error **errp);
+
+    /** tracked requests */
+    int coroutine_fn (*before_read)(BlockDriverState *bs, int64_t sector_num,
+                                    int nb_sectors, QEMUIOVector *qiov);
+    int coroutine_fn (*before_write)(BlockDriverState *bs, int64_t sector_num,
+                                     int nb_sectors, QEMUIOVector *qiov);
+
 } BlockJobType;
 
 /**
@@ -103,6 +110,9 @@ struct BlockJob {
     /** Speed that was set with @block_job_set_speed.  */
     int64_t speed;
 
+    /** tracked requests */
+    int cluster_size;
+
     /** The completion function that will be called when the job completes.  */
     BlockDriverCompletionFunc *cb;
 
-- 
1.8.1.4

[Qemu-devel] [RFC 3/8] backup: write to BlockDriverState instead of BackupDumpFunc

[Stefan Hajnoczi]

Remove the BackupDumpFunc function pointer and write directly to a
BlockDriverState.  This allows the 'backup' command to copy out into a
fresh qcow2 or raw image.  If no built-in image format is suitable, use
NBD to export the data to an external process.

A few other things in this commit:
 * Detect zero clusters with buffer_is_zero()
 * Skip zero clusters, don't write them to the target
 * Use 0 delay instead of 1us, like other block jobs
 * Delete the backup.h header file, it is no longer necessary
 * Unify creation/start functions into backup_start()
 * Simplify cleanup, free bitmap in backup_run() instead of cb function
 * Use bdrv_getlength() instead of accessing ->total_sectors directly

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 backup.c                  | 110 ++++++++++++++++------------------------------
 backup.h                  |  30 -------------
 include/block/block_int.h |  16 +++++++
 3 files changed, 54 insertions(+), 102 deletions(-)
 delete mode 100644 backup.h

diff --git a/backup.c b/backup.c
index 8955e1a..8ee3450 100644
--- a/backup.c
+++ b/backup.c
@@ -19,7 +19,6 @@
 #include "block/block_int.h"
 #include "block/blockjob.h"
 #include "qemu/ratelimit.h"
-#include "backup.h"
 
 #define DEBUG_BACKUP 0
 
@@ -29,19 +28,20 @@
     do { if (DEBUG_BACKUP) { printf("backup: " fmt, ## __VA_ARGS__); } } \
     while (0)
 
+#define BACKUP_CLUSTER_BITS 16
+#define BACKUP_CLUSTER_SIZE (1<<BACKUP_CLUSTER_BITS)
+#define BACKUP_BLOCKS_PER_CLUSTER (BACKUP_CLUSTER_SIZE/BDRV_SECTOR_SIZE)
 
 #define SLICE_TIME 100000000ULL /* ns */
 
 typedef struct BackupBlockJob {
     BlockJob common;
+    BlockDriverState *target;
     RateLimit limit;
     CoRwlock rwlock;
     uint64_t sectors_read;
     unsigned long *bitmap;
     int bitmap_size;
-    BackupDumpFunc *backup_dump_cb;
-    BlockDriverCompletionFunc *backup_complete_cb;
-    void *opaque;
 } BackupBlockJob;
 
 static bool backup_get_bitmap(BackupBlockJob *job, int64_t cluster_num)
@@ -149,17 +149,22 @@ static int coroutine_fn backup_do_cow(BlockDriverState *bs,
                         start);
                 goto out;
             }
+
+            zero = buffer_is_zero(bounce_buffer, BACKUP_CLUSTER_SIZE);
 #if USE_ALLOCATION_CHECK
         }
 #endif
         job->sectors_read += BACKUP_BLOCKS_PER_CLUSTER;
 
-        ret = job->backup_dump_cb(job->opaque, bs, start,
-                                  zero ? NULL : bounce_buffer);
-        if (ret < 0) {
-            DPRINTF("brdv_co_backup_cow dump_cluster_cb C%" PRId64 " failed\n",
-                    start);
-            goto out;
+        if (!zero) {
+            ret = bdrv_co_writev(job->target, start * BACKUP_BLOCKS_PER_CLUSTER,
+                                 BACKUP_BLOCKS_PER_CLUSTER,
+                                 &bounce_qiov);
+            if (ret < 0) {
+                DPRINTF("brdv_co_backup_cow dump_cluster_cb C%" PRId64
+                        " failed\n", start);
+                goto out;
+            }
         }
 
         DPRINTF("brdv_co_backup_cow done C%" PRId64 "\n", start);
@@ -217,8 +222,8 @@ static void coroutine_fn backup_run(void *opaque)
     int64_t start, end;
 
     start = 0;
-    end = (bs->total_sectors + BACKUP_BLOCKS_PER_CLUSTER - 1) /
-        BACKUP_BLOCKS_PER_CLUSTER;
+    end = (bdrv_getlength(bs) / BDRV_SECTOR_SIZE +
+           BACKUP_BLOCKS_PER_CLUSTER - 1) / BACKUP_BLOCKS_PER_CLUSTER;
 
     DPRINTF("backup_run start %s %" PRId64 " %" PRId64 "\n",
             bdrv_get_device_name(bs), start, end);
@@ -233,7 +238,6 @@ static void coroutine_fn backup_run(void *opaque)
 
         /* we need to yield so that qemu_aio_flush() returns.
          * (without, VM does not reboot)
-         * Note: use 1000 instead of 0 (0 prioritize this task too much)
          */
         if (job->common.speed) {
             uint64_t delay_ns = ratelimit_calculate_delay(
@@ -241,7 +245,7 @@ static void coroutine_fn backup_run(void *opaque)
             job->sectors_read = 0;
             block_job_sleep_ns(&job->common, rt_clock, delay_ns);
         } else {
-            block_job_sleep_ns(&job->common, rt_clock, 1000);
+            block_job_sleep_ns(&job->common, rt_clock, 0);
         }
 
         if (block_job_is_cancelled(&job->common)) {
@@ -272,84 +276,46 @@ static void coroutine_fn backup_run(void *opaque)
     qemu_co_rwlock_wrlock(&job->rwlock);
     qemu_co_rwlock_unlock(&job->rwlock);
 
-    DPRINTF("backup_run complete %d\n", ret);
-    block_job_completed(&job->common, ret);
-}
-
-static void backup_job_cleanup_cb(void *opaque, int ret)
-{
-    BlockDriverState *bs = opaque;
-    assert(bs);
-    BackupBlockJob *job = (BackupBlockJob *)bs->job;
-    assert(job);
-
-    DPRINTF("backup_job_cleanup_cb start %d\n", ret);
-
-    job->backup_complete_cb(job->opaque, ret);
-
-    DPRINTF("backup_job_cleanup_cb end\n");
-
     g_free(job->bitmap);
-}
 
-void
-backup_job_start(BlockDriverState *bs, bool cancel)
-{
-    assert(bs);
-    assert(bs->job);
-    assert(bs->job->co == NULL);
+    bdrv_delete(job->target);
 
-    if (cancel) {
-        block_job_cancel(bs->job); /* set cancel flag */
-    }
-
-    bs->job->co = qemu_coroutine_create(backup_run);
-    qemu_coroutine_enter(bs->job->co, bs->job);
+    DPRINTF("backup_run complete %d\n", ret);
+    block_job_completed(&job->common, ret);
 }
 
-int
-backup_job_create(BlockDriverState *bs, BackupDumpFunc *backup_dump_cb,
-                  BlockDriverCompletionFunc *backup_complete_cb,
-                  void *opaque, int64_t speed)
+void backup_start(BlockDriverState *bs, BlockDriverState *target,
+                  int64_t speed,
+                  BlockDriverCompletionFunc *cb, void *opaque,
+                  Error **errp)
 {
-    assert(bs);
-    assert(backup_dump_cb);
-    assert(backup_complete_cb);
-
-    if (bs->job) {
-        DPRINTF("bdrv_backup_init failed - running job on %s\n",
-                bdrv_get_device_name(bs));
-        return -1;
-    }
-
     int64_t bitmap_size;
-    const char *devname = bdrv_get_device_name(bs);
 
-    if (!devname || !devname[0]) {
-        return -1;
-    }
+    assert(bs);
+    assert(target);
+    assert(cb);
 
-    DPRINTF("bdrv_backup_init %s\n", bdrv_get_device_name(bs));
+    DPRINTF("backup_start %s\n", bdrv_get_device_name(bs));
 
-    Error *errp;
     BackupBlockJob *job = block_job_create(&backup_job_type, bs, speed,
-                                           backup_job_cleanup_cb, bs, &errp);
+                                           cb, opaque, errp);
+    if (!job) {
+        return;
+    }
 
     qemu_co_rwlock_init(&job->rwlock);
 
+    job->target = target;
     job->common.cluster_size = BACKUP_CLUSTER_SIZE;
+    job->common.len = bdrv_getlength(bs);
 
-    bitmap_size = bs->total_sectors +
+    bitmap_size = job->common.len / BDRV_SECTOR_SIZE +
         BACKUP_BLOCKS_PER_CLUSTER * BITS_PER_LONG - 1;
     bitmap_size /= BACKUP_BLOCKS_PER_CLUSTER * BITS_PER_LONG;
 
-    job->backup_dump_cb = backup_dump_cb;
-    job->backup_complete_cb = backup_complete_cb;
-    job->opaque = opaque;
     job->bitmap_size = bitmap_size;
     job->bitmap = g_new0(unsigned long, bitmap_size);
 
-    job->common.len = bs->total_sectors*BDRV_SECTOR_SIZE;
-
-    return 0;
+    job->common.co = qemu_coroutine_create(backup_run);
+    qemu_coroutine_enter(job->common.co, job);
 }
diff --git a/backup.h b/backup.h
deleted file mode 100644
index 9b1ea1c..0000000
--- a/backup.h
+++ /dev/null
@@ -1,30 +0,0 @@
-/*
- * QEMU backup related definitions
- *
- * Copyright (C) 2013 Proxmox Server Solutions
- *
- * Authors:
- *  Dietmar Maurer (dietmar@proxmox.com)
- *
- * This work is licensed under the terms of the GNU GPL, version 2 or later.
- * See the COPYING file in the top-level directory.
- *
- */
-
-#ifndef QEMU_BACKUP_H
-#define QEMU_BACKUP_H
-
-#define BACKUP_CLUSTER_BITS 16
-#define BACKUP_CLUSTER_SIZE (1<<BACKUP_CLUSTER_BITS)
-#define BACKUP_BLOCKS_PER_CLUSTER (BACKUP_CLUSTER_SIZE/BDRV_SECTOR_SIZE)
-
-typedef int BackupDumpFunc(void *opaque, BlockDriverState *bs,
-                           int64_t cluster_num, unsigned char *buf);
-
-void backup_job_start(BlockDriverState *bs, bool cancel);
-
-int backup_job_create(BlockDriverState *bs, BackupDumpFunc *backup_dump_cb,
-                      BlockDriverCompletionFunc *backup_complete_cb,
-                      void *opaque, int64_t speed);
-
-#endif /* QEMU_BACKUP_H */
diff --git a/include/block/block_int.h b/include/block/block_int.h
index eaad53e..ab8af77 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -366,4 +366,20 @@ void mirror_start(BlockDriverState *bs, BlockDriverState *target,
                   BlockDriverCompletionFunc *cb,
                   void *opaque, Error **errp);
 
+/*
+ * backup_start:
+ * @bs: Block device to operate on.
+ * @target: Block device to write to.
+ * @speed: The maximum speed, in bytes per second, or 0 for unlimited.
+ * @cb: Completion function for the job.
+ * @opaque: Opaque pointer value passed to @cb.
+ *
+ * Start a backup operation on @bs.  Clusters in @bs are written to @target
+ * until the job is cancelled or manually completed.
+ */
+void backup_start(BlockDriverState *bs, BlockDriverState *target,
+                  int64_t speed,
+                  BlockDriverCompletionFunc *cb, void *opaque,
+                  Error **errp);
+
 #endif /* BLOCK_INT_H */
-- 
1.8.1.4

[Qemu-devel] [RFC 4/8] block: add block_backup QMP command

[Stefan Hajnoczi]

@block-backup

Start a point-in-time copy of a block device to a new destination.

@device:  the name of the device whose writes should be mirrored.

@target: the target of the new image. If the file exists, or if it
         is a device, the existing file/device will be used as the new
         destination.  If it does not exist, a new file will be created.

@format: #optional the format of the new destination, default is to
         probe if @mode is 'existing', else the format of the source

@mode: #optional whether and how QEMU should create a new image, default is
       'absolute-paths'.

@speed:  #optional the maximum speed, in bytes per second

Returns: nothing on success
         If @device is not a valid block device, DeviceNotFound

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 blockdev.c       | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 qapi-schema.json | 28 +++++++++++++++++
 qmp-commands.hx  |  6 ++++
 3 files changed, 126 insertions(+)

diff --git a/blockdev.c b/blockdev.c
index 0e67d06..95a72de 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1208,6 +1208,98 @@ void qmp_block_commit(const char *device,
     drive_get_ref(drive_get_by_blockdev(bs));
 }
 
+void qmp_block_backup(const char *device, const char *target,
+                      bool has_format, const char *format,
+                      bool has_mode, enum NewImageMode mode,
+                      bool has_speed, int64_t speed,
+                      Error **errp)
+{
+    BlockDriverState *bs;
+    BlockDriverState *target_bs;
+    BlockDriver *proto_drv;
+    BlockDriver *drv = NULL;
+    Error *local_err = NULL;
+    int flags;
+    uint64_t size;
+    int ret;
+
+    if (!has_speed) {
+        speed = 0;
+    }
+    if (!has_mode) {
+        mode = NEW_IMAGE_MODE_ABSOLUTE_PATHS;
+    }
+
+    bs = bdrv_find(device);
+    if (!bs) {
+        error_set(errp, QERR_DEVICE_NOT_FOUND, device);
+        return;
+    }
+
+    if (!bdrv_is_inserted(bs)) {
+        error_set(errp, QERR_DEVICE_HAS_NO_MEDIUM, device);
+        return;
+    }
+
+    if (!has_format) {
+        format = mode == NEW_IMAGE_MODE_EXISTING ? NULL : bs->drv->format_name;
+    }
+    if (format) {
+        drv = bdrv_find_format(format);
+        if (!drv) {
+            error_set(errp, QERR_INVALID_BLOCK_FORMAT, format);
+            return;
+        }
+    }
+
+    if (bdrv_in_use(bs)) {
+        error_set(errp, QERR_DEVICE_IN_USE, device);
+        return;
+    }
+
+    flags = bs->open_flags | BDRV_O_RDWR;
+
+    proto_drv = bdrv_find_protocol(target);
+    if (!proto_drv) {
+        error_set(errp, QERR_INVALID_BLOCK_FORMAT, format);
+        return;
+    }
+
+    bdrv_get_geometry(bs, &size);
+    size *= 512;
+    if (mode != NEW_IMAGE_MODE_EXISTING) {
+        assert(format && drv);
+        bdrv_img_create(target, format,
+                        NULL, NULL, NULL, size, flags, &local_err, false);
+    }
+
+    if (error_is_set(&local_err)) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    target_bs = bdrv_new("");
+    ret = bdrv_open(target_bs, target, flags, drv);
+
+    if (ret < 0) {
+        bdrv_delete(target_bs);
+        error_set(errp, QERR_OPEN_FILE_FAILED, target);
+        return;
+    }
+
+    backup_start(bs, target_bs, speed, block_job_cb, bs, &local_err);
+    if (local_err != NULL) {
+        bdrv_delete(target_bs);
+        error_propagate(errp, local_err);
+        return;
+    }
+
+    /* Grab a reference so hotplug does not delete the BlockDriverState from
+     * underneath us.
+     */
+    drive_get_ref(drive_get_by_blockdev(bs));
+}
+
 #define DEFAULT_MIRROR_BUF_SIZE   (10 << 20)
 
 void qmp_drive_mirror(const char *device, const char *target,
diff --git a/qapi-schema.json b/qapi-schema.json
index 6b64aec..1dbf7b5 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -1716,6 +1716,34 @@
             '*speed': 'int' } }
 
 ##
+# @block-backup
+#
+# Start a point-in-time copy of a block device to a new destination.
+#
+# @device:  the name of the device whose writes should be mirrored.
+#
+# @target: the target of the new image. If the file exists, or if it
+#          is a device, the existing file/device will be used as the new
+#          destination.  If it does not exist, a new file will be created.
+#
+# @format: #optional the format of the new destination, default is to
+#          probe if @mode is 'existing', else the format of the source
+#
+# @mode: #optional whether and how QEMU should create a new image, default is
+#        'absolute-paths'.
+#
+# @speed:  #optional the maximum speed, in bytes per second
+#
+# Returns: nothing on success
+#          If @device is not a valid block device, DeviceNotFound
+#
+# Since 1.5
+##
+{ 'command': 'block-backup',
+  'data': { 'device': 'str', 'target': 'str', '*format': 'str',
+            '*mode': 'NewImageMode', '*speed': 'int' } }
+
+##
 # @drive-mirror
 #
 # Start mirroring a block device's writes to a new destination.
diff --git a/qmp-commands.hx b/qmp-commands.hx
index 95022e2..bda8eb4 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -889,6 +889,12 @@ EQMP
     },
 
     {
+        .name       = "block-backup",
+        .args_type  = "device:B,target:s,speed:i?,mode:s?,format:s?",
+        .mhandler.cmd_new = qmp_marshal_input_block_backup,
+    },
+
+    {
         .name       = "block-job-set-speed",
         .args_type  = "device:B,speed:o",
         .mhandler.cmd_new = qmp_marshal_input_block_job_set_speed,
-- 
1.8.1.4

[Qemu-devel] [RFC 5/8] Add nbd server Python module

[Stefan Hajnoczi]

The nbd module works like this:

  server = nbd.Server(sock)
  server.add_export('drive0', handler0)
  server.add_export('drive1', handler1)
  server.run()

The user must provide a handler object which defines the behavior of an
export:

  class MyNBDHandler(nbd.ExportHandler):
      def write(self, offset, data):
          pass # do something

      def size(self):
          return 10 * 1024 * 1024 * 1024

Note that the handler is invoked from a thread.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 nbd.py | 124 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 124 insertions(+)
 create mode 100644 nbd.py

diff --git a/nbd.py b/nbd.py
new file mode 100644
index 0000000..2528531
--- /dev/null
+++ b/nbd.py
@@ -0,0 +1,124 @@
+# NBD server module
+#
+# Copyright 2013 Red Hat, Inc. and/or its affiliates
+#
+# Authors:
+#   Stefan Hajnoczi <stefanha@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+import struct
+import collections
+import threading
+
+__all__ = ['ExportHandler', 'Server']
+
+NBD_CMD_WRITE = 1
+NBD_CMD_DISC = 2
+NBD_REQUEST_MAGIC = 0x25609513
+NBD_REPLY_MAGIC = 0x67446698
+NBD_PASSWD = 0x4e42444d41474943
+NBD_OPTS_MAGIC = 0x49484156454F5054
+NBD_OPT_EXPORT_NAME = 1 << 0
+
+neg1_struct = struct.Struct('>QQH')
+export_tuple = collections.namedtuple('Export', 'reserved magic opt len')
+export_struct = struct.Struct('>IQII')
+neg2_struct = struct.Struct('>QH124x')
+request_tuple = collections.namedtuple('Request', 'magic type handle from_ len')
+request_struct = struct.Struct('>IIQQI')
+reply_struct = struct.Struct('>IIQ')
+
+def recvall(sock, bufsize):
+    received = 0
+    chunks = []
+    while received < bufsize:
+        chunk = sock.recv(bufsize - received)
+        if len(chunk) == 0:
+            raise Exception('unexpected disconnect')
+        chunks.append(chunk)
+        received += len(chunk)
+    return ''.join(chunks)
+
+class ExportHandler(object):
+    def write(self, offset, data):
+        pass
+
+    def size(self):
+        return 0
+
+def negotiate(conn, exports):
+    '''Negotiate export with client'''
+    # Send negotiation part 1
+    buf = neg1_struct.pack(NBD_PASSWD, NBD_OPTS_MAGIC, 0)
+    conn.sendall(buf)
+
+    # Receive export option
+    buf = recvall(conn, export_struct.size)
+    export = export_tuple._make(export_struct.unpack(buf))
+    assert export.magic == NBD_OPTS_MAGIC
+    assert export.opt == NBD_OPT_EXPORT_NAME
+    name = recvall(conn, export.len)
+
+    if name not in exports:
+        print 'name "%s" not in exports' % name
+        return None
+    handler = exports[name]
+
+    # Send negotiation part 2
+    buf = neg2_struct.pack(handler.size(), 0)
+    conn.sendall(buf)
+    return handler
+
+def read_request(conn):
+    '''Parse NBD request from client'''
+    buf = recvall(conn, request_struct.size)
+    req = request_tuple._make(request_struct.unpack(buf))
+    assert req.magic == NBD_REQUEST_MAGIC
+    return req
+
+def write_reply(conn, error, handle):
+    buf = reply_struct.pack(NBD_REPLY_MAGIC, error, handle)
+    conn.sendall(buf)
+
+def server_connection_thread(conn, exports):
+    handler = negotiate(conn, exports)
+    if handler is None:
+        conn.close()
+        return
+
+    while True:
+        req = read_request(conn)
+        if req.type == NBD_CMD_WRITE:
+            # Reply immediately, don't propagate internal errors to client
+            write_reply(conn, 0, req.handle)
+
+            data = recvall(conn, req.len)
+            handler.write(req.from_, data)
+        elif req.type == NBD_CMD_DISC:
+            break
+        else:
+            print 'unrecognized command type %#02x' % req.type
+            break
+    conn.close()
+
+class Server(object):
+    def __init__(self, sock):
+        self.sock = sock
+        self.exports = {}
+
+    def add_export(self, name, handler):
+        self.exports[name] = handler
+
+    def run(self):
+        threads = []
+        for i in range(len(self.exports)):
+            conn, _ = self.sock.accept()
+            t = threading.Thread(target=server_connection_thread,
+                                 args=(conn, self.exports))
+            t.daemon = True
+            t.start()
+            threads.append(t)
+        for t in threads:
+            t.join()
-- 
1.8.1.4

[Qemu-devel] [RFC 6/8] Add VMA backup archive writer Python module

[Stefan Hajnoczi]

The vma module provides an interface for writing VMA backup archives:

  writer = vma.Writer(open('test.vma', 'wb))
  writer.add_config('guest.xml', '<guest></guest>')
  stream_id = writer.add_stream('foo',  # name
                                65536)  # size
  writer.write(stream_id, 0, '\0' * 32768)
  writer.write(stream_id, 32768, '\1' * 32768)
  writer.close()

The Writer handles sequential writes that are not cluster-aligned.  This
is typically only the vmstate.  Disk writes are 64 KB aligned in
practice.

VMA supports zero regions within a 64 KB cluster.  The vma module does
not implement this, the full cluster is written.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 vma.py | 236 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 236 insertions(+)
 create mode 100644 vma.py

diff --git a/vma.py b/vma.py
new file mode 100644
index 0000000..236ba14
--- /dev/null
+++ b/vma.py
@@ -0,0 +1,236 @@
+# VMA writer module
+#
+# Copyright 2013 Red Hat, Inc. and/or its affiliates
+#
+# Authors:
+#   Stefan Hajnoczi <stefanha@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+import array
+import struct
+import hashlib
+import uuid
+import time
+
+__all__ = ['Writer']
+
+VMA_MAGIC = 0x564d4100
+VMA_VERSION = 1
+VMA_MAX_CONFIGS = 256
+VMA_CLUSTER_SIZE = 65536
+VMA_BLOCKS_PER_EXTENT = 59
+VMA_EXTENT_MAGIC = 0x564d4145
+
+header_struct = struct.Struct('>II16cQ16cIII')
+dev_info_struct = struct.Struct('>IIQQQ')
+extent_struct = struct.Struct('>I2xH16c16c')
+le16_struct = struct.Struct('<H')
+be32_struct = struct.Struct('>I')
+be64_struct = struct.Struct('>Q')
+
+class Writer(object):
+    def __init__(self, fobj):
+        self.fobj = fobj
+        self.uuid = uuid.uuid4().bytes
+        self.streams = []
+        self.blobs = ['\0']
+        self.blob_offset = 1
+        self.config_names = []
+        self.config_data = []
+        self.header_written = False
+        self.align_bufs = {}
+        self.extent = []
+
+    def alloc_blob(self, blob):
+        '''Return allocated blob buffer offset'''
+        offset = self.blob_offset
+        self.blobs.append(le16_struct.pack(len(blob)))
+        self.blobs.append(blob)
+        self.blob_offset += le16_struct.size + len(blob)
+        return offset
+
+    def alloc_blob_str(self, s):
+        '''Return allocated blob buffer offset for string'''
+        return self.alloc_blob(s + '\0')
+
+    def build_dev_info(self):
+        '''Return a buffer with device infos'''
+        bufs = ['\0' * dev_info_struct.size]
+        for name, size in self.streams:
+            name_ptr = self.alloc_blob_str(name)
+            buf = dev_info_struct.pack(name_ptr, 0, size, 0, 0)
+            bufs.append(buf)
+        padding = (255 - len(self.streams)) * dev_info_struct.size
+        bufs.append('\0' * padding)
+        return ''.join(bufs)
+
+    def build_blob_buffer(self):
+        '''Return a buffer with blob data'''
+        return ''.join(self.blobs)
+
+    def build_config(self):
+        '''Return a buffer with config names and data'''
+        bufs = []
+
+        for ptr in self.config_names:
+            bufs.append(be32_struct.pack(ptr))
+        padding = (VMA_MAX_CONFIGS - len(self.config_names)) * be32_struct.size
+        bufs.append('\0' * padding)
+
+        for ptr in self.config_data:
+            bufs.append(be32_struct.pack(ptr))
+        padding = (VMA_MAX_CONFIGS - len(self.config_data)) * be32_struct.size
+        bufs.append('\0' * padding)
+
+        return ''.join(bufs)
+
+    def write_header(self):
+        # Build header pieces
+        config = self.build_config()
+        dev_info = self.build_dev_info()
+        blob_buffer = self.build_blob_buffer()
+
+        # Size the header
+        blob_buffer_offset = header_struct.size + 1984 + \
+                             len(config) + 4 + len(dev_info)
+        header_size = blob_buffer_offset + len(blob_buffer)
+
+        # Build header without checksum
+        fields = (VMA_MAGIC,
+                  VMA_VERSION) + \
+                 tuple(self.uuid) + \
+                 (int(time.mktime(time.gmtime())),) + \
+                 tuple('\0' * 16) + \
+                 (blob_buffer_offset,
+                  len(blob_buffer),
+                  header_size)
+        header = header_struct.pack(*fields)
+
+        # Checksum header
+        buf = ''.join([header,
+                       '\0' * 1984,
+                       config,
+                       '\0' * 4, # VMAHeader.dev_info is unaligned (vma.h bug)
+                       dev_info,
+                       blob_buffer])
+        digest = hashlib.md5(buf).digest()
+        buf = array.array('c', buf) # string does not support assignment
+        buf[32:32 + 16] = array.array('c', digest)
+
+        self.fobj.write(buf)
+
+    def add_config(self, name, data):
+        name_ptr = self.alloc_blob_str(name)
+        data_ptr = self.alloc_blob(data)
+        self.config_names.append(name_ptr)
+        self.config_data.append(name_ptr)
+
+    def add_stream(self, name, size):
+        self.streams.append((name, size))
+        return len(self.streams)
+
+    def build_blockinfo(self):
+        '''Return a blockinfo buffer for the current extent'''
+        bufs = []
+        for stream_id, offset, _ in self.extent:
+            buf = be64_struct.pack(0xffff000000000000 | \
+                                   (stream_id << 32)  | \
+                                   offset // VMA_CLUSTER_SIZE)
+            bufs.append(buf)
+        padding = (VMA_BLOCKS_PER_EXTENT - len(self.extent)) * be64_struct.size
+        bufs.append('\0' * padding)
+        return ''.join(bufs)
+
+    def write_extent(self):
+        blockinfo = self.build_blockinfo()
+        block_count = len(self.extent) * (VMA_CLUSTER_SIZE // 4096)
+
+        # Build header without checksum
+        fields = (VMA_EXTENT_MAGIC,
+                  block_count) + \
+                 tuple(self.uuid) + \
+                 tuple('\0' * 16)
+        header = extent_struct.pack(*fields)
+
+        # Checksum header
+        buf = ''.join([header, blockinfo])
+        digest = hashlib.md5(buf).digest()
+        buf = array.array('c', buf) # string does not support assignment
+        buf[24:24 + 16] = array.array('c', digest)
+
+        self.fobj.write(buf)
+        for _, _, data in self.extent:
+            self.fobj.write(data)
+
+        self.extent = []
+
+    def append_cluster(self, stream_id, offset, data):
+        '''Append one cluster to the current extent'''
+        self.extent.append((stream_id, offset, data))
+        if len(self.extent) == VMA_BLOCKS_PER_EXTENT:
+            self.write_extent()
+
+    def align_write(self, stream_id, offset, data):
+        '''Buffer writes whose length is not cluster-aligned (vmstate)'''
+        # Fast path for aligned writes
+        mod = len(data) % VMA_CLUSTER_SIZE
+        if stream_id not in self.align_bufs and mod == 0:
+            return False, offset, data
+
+        # Add data to buffer
+        bufs, start, total = self.align_bufs.get(stream_id, ([], offset, 0))
+        assert start + total == offset # must be sequential
+        bufs.append(data)
+        total += len(data)
+        self.align_bufs[stream_id] = (bufs, start, total)
+
+        # Stop if we don't have a cluster yet
+        if total < VMA_CLUSTER_SIZE:
+            return True, None, None
+
+        # Take as many clusters as possible
+        end = (total // VMA_CLUSTER_SIZE) * VMA_CLUSTER_SIZE
+        aligned = []
+        nbytes = 0
+        while nbytes < end:
+            buf = bufs.pop(0)
+            aligned.append(buf)
+            nbytes += len(buf)
+        if nbytes > end:
+            buf = aligned[-1]
+            keep = end - (nbytes - len(buf))
+            left, right = buf[:keep], buf[keep:]
+            aligned[-1] = left
+            bufs.insert(0, right)
+        self.align_bufs[stream_id] = (bufs, start + end, total - end)
+        return False, start, ''.join(aligned)
+
+    def write(self, stream_id, offset, data):
+        if not self.header_written:
+            self.write_header()
+            self.header_written = True
+
+        need_more, offset, data = self.align_write(stream_id, offset, data)
+        if need_more:
+            return
+
+        for i in range(len(data) // VMA_CLUSTER_SIZE):
+            self.append_cluster(stream_id, offset, data[:VMA_CLUSTER_SIZE])
+            data = data[VMA_CLUSTER_SIZE:]
+            offset += VMA_CLUSTER_SIZE
+
+    def close(self):
+        # Flush unaligned data
+        for stream_id in self.align_bufs.keys():
+            bufs, start, total = self.align_bufs[stream_id]
+            assert total < VMA_CLUSTER_SIZE
+            padding = VMA_CLUSTER_SIZE - total
+            bufs.append('\0' * padding)
+            self.append_cluster(stream_id, start, ''.join(bufs))
+        self.align_bufs = {}
+
+        # Write final extent, if necessary
+        if self.extent:
+            self.write_extent()
-- 
1.8.1.4

[Qemu-devel] [RFC 7/8] Add vma-writer.py tool

[Stefan Hajnoczi]

The VMA writer bundles vmstate and disk backups into a backup archive
file.  It is normally invoked by a management tool like this:

  vma-writer.py --incoming /tmp/migrate.sock \
                --nbd /tmp/nbd.sock \
                --drive name=drive0,size=10737418240 \
                --drive name=drive1,size=34359738368 \
                --output backup-20130301.vma

The basic flow is:

1. Set up UNIX domain listen sockets.
2. Print 'Ready' so parent process knows sockets are listening.
3. Stream migration data into VMA.
4. Stream NBD disk data into VMA.

The VMA writer runs in its own thread.  Other threads can send commands
via a thread-safe Queue.  This way multiple NBD export threads can queue
data in parallel, the data will be serialized and written out by the VMA
writer thread.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 vma-writer.py | 126 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 126 insertions(+)
 create mode 100644 vma-writer.py

diff --git a/vma-writer.py b/vma-writer.py
new file mode 100644
index 0000000..dc7f828
--- /dev/null
+++ b/vma-writer.py
@@ -0,0 +1,126 @@
+#!/usr/bin/env python
+#
+# VMA backup archive writer
+#
+# Copyright 2013 Red Hat, Inc. and/or its affiliates
+#
+# Authors:
+#   Stefan Hajnoczi <stefanha@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+import argparse
+import threading
+import Queue
+import socket
+import sys
+import os
+import nbd
+import vma
+
+WRITER_OP_WRITE = 0
+WRITER_OP_STOP = 1
+
+def setup_listen_sockets(paths):
+    '''Return list of listening UNIX domain sockets'''
+    socks = []
+    for path in paths:
+        s = socket.socket(socket.AF_UNIX)
+        try:
+            os.unlink(path)
+        except OSError:
+            pass
+        s.bind(path)
+        s.listen(0)
+        socks.append(s)
+    return socks
+
+def vma_writer_thread(writer, queue):
+    while True:
+        cmd = queue.get()
+        if cmd[0] == WRITER_OP_STOP:
+            break
+        assert cmd[0] == WRITER_OP_WRITE
+        _, stream_id, offset, data = cmd
+        writer.write(stream_id, offset, data)
+    writer.close()
+
+def setup_vma_writer(filename, drives):
+    vma_file = open(filename, 'wb')
+    writer = vma.Writer(vma_file)
+
+    vmstate_id = writer.add_stream('vmstate', 1)
+    for drive in drives:
+        drive['stream_id'] = writer.add_stream(drive['name'], int(drive['size']))
+
+    queue = Queue.Queue()
+    t = threading.Thread(target=vma_writer_thread, args=(writer, queue))
+    t.start()
+    return queue, vmstate_id
+
+def consume_migration(sock, queue, stream_id):
+    '''Write vmstate data into archive'''
+    conn, _ = sock.accept()
+    sock.close()
+
+    offset = 0
+    while True:
+        buf = conn.recv(256 * 1024)
+        if len(buf) == 0:
+            break
+        queue.put((WRITER_OP_WRITE, stream_id, offset, buf))
+        offset += len(buf)
+
+    conn.close()
+
+def parse_option_list(s):
+    return dict(kv.split('=') for kv in s.split(','))
+
+class NBDHandler(nbd.ExportHandler):
+    def __init__(self, size, queue, stream_id):
+        self._size = size
+        self.queue = queue
+        self.stream_id = stream_id
+
+    def write(self, offset, data):
+        self.queue.put((WRITER_OP_WRITE, self.stream_id, offset, data))
+
+    def size(self):
+        return self._size
+
+def consume_nbd(sock, drives):
+    server = nbd.Server(sock)
+    for drive in drives:
+        server.add_export(drive['name'],
+                NBDHandler(int(drive['size']), queue, drive['stream_id']))
+    server.run()
+
+parser = argparse.ArgumentParser(description='VMA backup archive writer')
+parser.add_argument('--incoming',
+                    help='UNIX domain socket for incoming migration',
+                    required=True)
+parser.add_argument('--nbd',
+                    help='UNIX domain socket for NBD server',
+                    required=True)
+parser.add_argument('--drive',
+                    help='Device name of drive to back up',
+                    action='append',
+                    default=[])
+parser.add_argument('--output',
+                    help='Backup archive filename',
+                    required=True)
+
+args = parser.parse_args()
+drives = [parse_option_list(opts) for opts in args.drive]
+queue, vmstate_id = setup_vma_writer(args.output, drives)
+socks = setup_listen_sockets((args.incoming, args.nbd))
+
+# Let parent process know the sockets are listening
+sys.stdout.write('Ready\n')
+sys.stdout.flush()
+
+consume_migration(socks[0], queue, vmstate_id)
+consume_nbd(socks[1], drives)
+
+queue.put((WRITER_OP_STOP,))
-- 
1.8.1.4

[Qemu-devel] [RFC 8/8] Add backup.py tool

[Stefan Hajnoczi]

The backup.py tool connects to a running guest's QMP monitor and
orchestrates a live backup.

To invoke a backup:

  qemu -qmp unix:/tmp/monitor.sock,server,nowait ...
  backup.py /tmp/monitor.sock backup-20130301.vma

The script uses vma-writer.py to bundle vmstate and disk backups into a
single backup archive file.  vma-writer.py is launched as a separate
process.

First a 'migrate' command is used to send the vmstate to the writer
process and pause the guest.

Then the 'block-backup' command is used to send a point-in-time copy of
all disks to the writer process before resuming the guest.

The guest continues running while disks are being backed up over NBD.

Finally, the backup completes when the 'block-backup' block jobs finish.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 backup.py | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 84 insertions(+)
 create mode 100644 backup.py

diff --git a/backup.py b/backup.py
new file mode 100644
index 0000000..2e0b179
--- /dev/null
+++ b/backup.py
@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+#
+# Live backup tool for QEMU
+#
+# Copyright 2013 Red Hat, Inc. and/or its affiliates
+#
+# Authors:
+#   Stefan Hajnoczi <stefanha@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+
+import sys; sys.path.append('QMP')
+import qmp
+import subprocess
+
+MIGRATE_PATH = '/tmp/migrate.sock'
+NBD_PATH = '/tmp/nbd.sock'
+
+def find_backup_drives(mon):
+    '''Return list of devices that should be backed up'''
+    result = []
+    for info in mon.command('query-block'):
+        # Skip empty drives
+        if 'inserted' not in info:
+            continue
+
+        # Skip read-only drives
+        if info['inserted']['ro']:
+            continue
+
+        result.append((info['device'], info['inserted']['virtual_size']))
+    return result
+
+def spawn_writer_process(filename, drives):
+    '''Return Popen instance for vma-writer.py process'''
+    args = ['python', 'vma-writer.py',
+            '--output', filename,
+            '--incoming', MIGRATE_PATH,
+            '--nbd', NBD_PATH]
+    for name, size in drives:
+        args.extend(('--drive', 'name=%s,size=%s' % (name, size)))
+    writer = subprocess.Popen(args, stdout=subprocess.PIPE)
+    writer.stdout.readline() # Wait for "Ready"
+    return writer
+
+def main(args):
+    mon = qmp.QEMUMonitorProtocol(args[1])
+    mon.connect()
+
+    drives = find_backup_drives(mon)
+    writer = spawn_writer_process(args[2], drives)
+
+    sys.stderr.write('Running migration...\n')
+    mon.command('migrate', uri='unix:' + MIGRATE_PATH)
+    while True:
+        evt = mon.pull_event(wait=True)
+        if evt['event'] == 'STOP':
+            break
+
+    sys.stderr.write('Running block-backup...\n')
+    for name, _ in drives:
+        mon.command('block-backup',
+                    device=name,
+                    target='nbd+unix:///%s?socket=%s' % (name, NBD_PATH),
+                    format='raw',
+                    mode='existing')
+    mon.command('cont')
+    pending_drives = set(name for name, _ in drives)
+    while pending_drives:
+        evt = mon.pull_event(wait=True)
+        if evt['event'] == 'BLOCK_JOB_COMPLETED':
+            name = evt['data']['device']
+            sys.stderr.write('Finished device "%s"\n' % name)
+            pending_drives.discard(name)
+    sys.stderr.write('Backup complete, terminating writer process\n')
+
+    # Wait for writer process to terminate and print its output
+    sys.stdout.write(writer.communicate()[0])
+
+    mon.close()
+
+if __name__ == '__main__':
+    main(sys.argv)
-- 
1.8.1.4

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Dietmar Maurer]

> The objection to this approach has been performance.  Exporting vmstate and
> disk data over UNIX domain sockets to an external process incurs IPC overhead.
> This prototype shows that even Python code hacked up in a day achieves decent
> performance.
> 
> I'm leaving benchmarking as an exercise for the reader.  I tested a single scenario
> with a 16 GB raw disk and 1 GB RAM, backup time was 28% longer (+30
> seconds) than Dietmar's series.  Below are a few starting points:

I already have an implementation based on nbd.c, and that also shows considerable overhead.
A Backup task is extremely performance critical, so any additional overhead is bad.
I can see the advantage to move the code out of qemu, but I want to avoid that overhead
by all means. So are there any other ideas to avoid the overhead of a socket based IPC?

>  * I moved the buffer_is_zero() check from the VMA writer into the block job.
>    We now skip writing zero clusters and the file contains no extents for them.

With VMA we track zero blocks at 4KB level. If you move that code, you need to
test for zero regions two times, because NBD offers no way to pass that information (sending
multiple discard message at 4KB level is no option because that adds to much overhead).

NBD does not allow to pass additional infos, and it is quite slow. So it would be even faster
to write to a pipe and define our own backup protocol instead (less syscalls). But I have not done
any performance measurement for that.

Many thanks for your efforts on that topic. It would be great if we can get at least the first
patch "add basic backup support to block driver" into qemu. This is the basic framework, and
a starting point for anyone who want to play around with backup.

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Dietmar Maurer]

> The difference between this approach and Dietmar's series is that the backup
> archive format is implemented outside QEMU and runs as a separate program.
> 
> This way, management tools like proxmox, oVirt, OpenStack, and others can
> provide their preferred backup archive formats without modifying QEMU.  

So you propose that everyone should use another backup format - this is a bad thing because
it lead to interoperability problems (vendor lock-in?)

> This has many advantages:
> 
>  * 'block-backup' composes with 'migration' and other commands, unlike the
>    monolithic 'backup' command designed just for writing backup archives

Yes, but you can add that easily on top of my patches.

> 
>  * Backup code can be updated or added outside the QEMU release cycle
> 
>  * Choice of language, coding style, and license for backup code
> 
>  * Less QEMU code to test and maintain

The question is why you would want to write and maintain your own backup solution?
This is an disadvantage, not an advantage. If we have working code inside qemu, everybody 
can use it. This will lead to better testing, more stable code, and finally we can even try 
to make backup/restore work across different management frameworks.
 
So while you have "Less QEMU code to test and maintain", it will lead to more code
you have to test and maintain - not inside qemu, but outside qemu ;-)




 

Re: [Qemu-devel] [RFC 3/8] backup: write to BlockDriverState instead of BackupDumpFunc

[Dietmar Maurer]

> Remove the BackupDumpFunc function pointer and write directly to a
> BlockDriverState.  

My callback approach is the generic one. You can easily implement the
BlockDriverState approach using the BackupDumpFunc? 

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Stefan Hajnoczi]

On Sun, Mar 10, 2013 at 10:14 AM, Dietmar Maurer <dietmar@proxmox.com> wrote:
>> The objection to this approach has been performance.  Exporting vmstate and
>> disk data over UNIX domain sockets to an external process incurs IPC overhead.
>> This prototype shows that even Python code hacked up in a day achieves decent
>> performance.
>>
>> I'm leaving benchmarking as an exercise for the reader.  I tested a single scenario
>> with a 16 GB raw disk and 1 GB RAM, backup time was 28% longer (+30
>> seconds) than Dietmar's series.  Below are a few starting points:
>
> I already have an implementation based on nbd.c, and that also shows considerable overhead.
> A Backup task is extremely performance critical, so any additional overhead is bad.
> I can see the advantage to move the code out of qemu, but I want to avoid that overhead
> by all means. So are there any other ideas to avoid the overhead of a socket based IPC?

There are enough avenues to investigate better performance that I
think the solution is to just profile and optimize a little.  More
ideas below...

>>  * I moved the buffer_is_zero() check from the VMA writer into the block job.
>>    We now skip writing zero clusters and the file contains no extents for them.
>
> With VMA we track zero blocks at 4KB level. If you move that code, you need to
> test for zero regions two times, because NBD offers no way to pass that information (sending
> multiple discard message at 4KB level is no option because that adds to much overhead).

About 4KB zero tracking:

The vma.py module does not check for zeros, the mask is always 0xffff.
 There is a pathalogical case of a disk with every 2nd 4 KB block
zeroed, here vma.py would create a fully-allocated file.But the more
common case is that the whole 64 KB cluster is zero - think about the
fact that qcow2/qed only do discard (zero clusters) at cluster level
(64 KB).

Also, if you try hard to zero 4 KB blocks then two things can happen:
1. You fragment the file, saving space now but turning sequential
access into random access once those zero regions get filled in.
2. The underlying file system ignores the holes to reduce
fragmentation and all the effort was wasted - I guess this can be
checked with ioctl(FIEMAP/FIBMAP) to see how ext4/xfs/btrfs handle it.

For these reasons I figure it is better to call buffer_is_zero() in
the block job and not make use of the VMA 4 KB zero feature.  We'll
catch the big zero regions anyway.

About NBD performance:

The trick to good NBD performance is to pipeline commands instead of
doing them synchronously.  The protocol supports it, there is nothing
stopping us from sending a single buffer with 16 discard commands and
getting a single buffer back with 16 replies.  This is like
setsockopt(TCP_CORK) or sendmsg(MSG_MORE).

> NBD does not allow to pass additional infos, and it is quite slow. So it would be even faster
> to write to a pipe and define our own backup protocol instead (less syscalls). But I have not done
> any performance measurement for that.

We're not talking about 10x or even 2x overhead, this is within the
range of incremental optimization.

There are two performance factors that can be controlled:

1. Frequency of writes - use a larger block job buffer like I
suggested to reduce back and forth.

2. Asynchronous I/O like block/mirror.c.  Don't wait for the NBD write
to complete.

I'm suspect that implementing either or both of these will reduce
overhead to <10%.

> Many thanks for your efforts on that topic. It would be great if we can get at least the first
> patch "add basic backup support to block driver" into qemu. This is the basic framework, and
> a starting point for anyone who want to play around with backup.

Feel free to take patches that are useful when you create your patch series.

For your Patch 2/6, you could even squash all or part of my backup.c changes.

Stefan

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Dietmar Maurer]

> > I can see the advantage to move the code out of qemu, but I want to
> > avoid that overhead by all means. So are there any other ideas to avoid the
> overhead of a socket based IPC?
> 
> There are enough avenues to investigate better performance that I think the
> solution is to just profile and optimize a little.

I know that I can optimize, but I always thought it is better to avoid overhead
it is possible. Socket based IPC involves copying large amount of data, and you
cannot optimize that away.

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Stefan Hajnoczi]

On Sun, Mar 10, 2013 at 10:57 AM, Dietmar Maurer <dietmar@proxmox.com> wrote:
>> The difference between this approach and Dietmar's series is that the backup
>> archive format is implemented outside QEMU and runs as a separate program.
>>
>> This way, management tools like proxmox, oVirt, OpenStack, and others can
>> provide their preferred backup archive formats without modifying QEMU.
>
> So you propose that everyone should use another backup format - this is a bad thing because
> it lead to interoperability problems (vendor lock-in?)

No, they can use a common format (OVF?).  It's up to them.

>> This has many advantages:
>>
>>  * 'block-backup' composes with 'migration' and other commands, unlike the
>>    monolithic 'backup' command designed just for writing backup archives
>
> Yes, but you can add that easily on top of my patches.

True, it can be added on top but it's simpler to have just the key
primitive rather than adding it on top.

>>  * Backup code can be updated or added outside the QEMU release cycle
>>
>>  * Choice of language, coding style, and license for backup code
>>
>>  * Less QEMU code to test and maintain
>
> The question is why you would want to write and maintain your own backup solution?
> This is an disadvantage, not an advantage. If we have working code inside qemu, everybody
> can use it. This will lead to better testing, more stable code, and finally we can even try
> to make backup/restore work across different management frameworks.
>
> So while you have "Less QEMU code to test and maintain", it will lead to more code
> you have to test and maintain - not inside qemu, but outside qemu ;-)

QEMU is not the only component that can be shared.  If you want to put
VMA into a common component, try libvirt.

QEMU does not have enough information to create a full backup archive
or restore it.  Therefore QEMU is the wrong place for backup archive
code.

Stefan

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Dietmar Maurer]

> About 4KB zero tracking:
> 
> The vma.py module does not check for zeros, the mask is always 0xffff.
>  There is a pathalogical case of a disk with every 2nd 4 KB block zeroed, here
> vma.py would create a fully-allocated file.But the more common case is that the
> whole 64 KB cluster is zero - think about the fact that qcow2/qed only do discard
> (zero clusters) at cluster level
> (64 KB).
> 
> Also, if you try hard to zero 4 KB blocks then two things can happen:
> 1. You fragment the file, saving space now but turning sequential access into
> random access once those zero regions get filled in.
> 2. The underlying file system ignores the holes to reduce fragmentation and all
> the effort was wasted - I guess this can be checked with ioctl(FIEMAP/FIBMAP)
> to see how ext4/xfs/btrfs handle it.

We track zero blocks at 4KB level to get small backup files. Restore can handle those
blocks differently, either pre-allocate or create holes. That can even be a restore option.

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

On Sun, Mar 10, 2013 at 11:38 AM, Dietmar Maurer <dietmar@proxmox.com> wrote:
>> > I can see the advantage to move the code out of qemu, but I want to
>> > avoid that overhead by all means. So are there any other ideas to avoid the
>> overhead of a socket based IPC?
>>
>> There are enough avenues to investigate better performance that I think the
>> solution is to just profile and optimize a little.
>
> I know that I can optimize, but I always thought it is better to avoid overhead
> it is possible. Socket based IPC involves copying large amount of data, and you
> cannot optimize that away.

I have attached the top of the perf -a -g profile for my RFC series.
Observations:

 * Disk encryption is the biggest CPU hog (19%)
 * QEMU memmove takes 4.64%, haven't figured out where it gets called from
 * buffer_is_zero() takes 1.47%, that's less than cpuidle (system has 4 cores).
 * Python and QEMU together only take 1.57% for
copy_user_enhanced_fast_string() (used for sendmsg/recvmsg)

Socket memcpy overhead is lower than the unidentified memmove() in QEMU.

The vma-writer.c code you posted has several memcpy(), I don't think
the socket memcpy makes up for a big percentage.  Rather, lowering the
frequency of messages and employing pipelining will let the system get
more work done with less communication.

Stefan

[-- Attachment #2: report.txt --]
[-- Type: text/plain, Size: 134476 bytes --]

# ========
# captured on: Sun Mar 10 11:49:39 2013
# hostname : stefanha-thinkpad
# os release : 3.8.1-201.fc18.x86_64
# perf version : 3.8.1-201.fc18.x86_64
# arch : x86_64
# nrcpus online : 4
# nrcpus avail : 4
# cpudesc : Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
# cpuid : GenuineIntel,6,58,9
# total memory : 7896012 kB
# cmdline : /usr/bin/perf record -a -g 
# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_guest = 1, precise_ip = 0, id = { 87425, 87426, 87427, 87428 }
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# pmu mappings: cpu = 4, software = 1, tracepoint = 2, breakpoint = 5
# ========
#
# Samples: 924K of event 'cycles'
# Event count (approx.): 196971251707
#
# Overhead          Command                        Shared Object                                                                                                                                                                                                                                                                                Symbol
# ........  ...............  ...................................  ....................................................................................................................................................................................................................................................................................
#
     5.24%      kworker/0:0  [kernel.kallsyms]                    [k] _aesni_dec4                                                                                                                                                                                                                                                                     
                |
                --- _aesni_dec4
                    xts_crypt
                    xts_decrypt
                    ablk_decrypt
                    crypt_convert
                    kcryptd_crypt
                    process_one_work
                    worker_thread
                    kthread
                    ret_from_fork

     4.64%  qemu-system-x86  libc-2.16.so                         [.] __memmove_ssse3_back                                                                                                                                                                                                                                                            
            |
            --- __memmove_ssse3_back

     4.37%      kworker/0:0  [kernel.kallsyms]                    [k] xts_crypt                                                                                                                                                                                                                                                                       
                |
                --- xts_crypt
                   |          
                   |--90.57%-- xts_decrypt
                   |          ablk_decrypt
                   |          crypt_convert
                   |          kcryptd_crypt
                   |          process_one_work
                   |          worker_thread
                   |          kthread
                   |          ret_from_fork
                   |          
                   |--9.26%-- xts_encrypt
                   |          __ablk_encrypt
                   |          ablk_encrypt
                   |          crypt_convert
                   |          kcryptd_crypt
                   |          process_one_work
                   |          worker_thread
                   |          kthread
                   |          ret_from_fork
                    --0.17%-- [...]

     3.76%      kworker/0:2  [kernel.kallsyms]                    [k] _aesni_dec4                                                                                                                                                                                                                                                                     
                |
                --- _aesni_dec4
                    xts_crypt
                    xts_decrypt
                    ablk_decrypt
                    crypt_convert
                    kcryptd_crypt
                    process_one_work
                    worker_thread
                    kthread
                    ret_from_fork

     3.14%          swapper  [kernel.kallsyms]                    [k] intel_idle                                                                                                                                                                                                                                                                      
                    |
                    --- intel_idle
                       |          
                       |--99.17%-- cpuidle_enter
                       |          cpuidle_wrap_enter
                       |          cpuidle_enter_tk
                       |          cpuidle_idle_call
                       |          cpu_idle
                       |          |          
                       |          |--68.09%-- start_secondary
                       |          |          
                       |           --31.91%-- rest_init
                       |                     start_kernel
                       |                     x86_64_start_reservations
                       |                     x86_64_start_kernel
                       |          
                        --0.83%-- cpuidle_wrap_enter
                                  cpuidle_enter_tk
                                  cpuidle_idle_call
                                  cpu_idle
                                  |          
                                  |--71.61%-- start_secondary
                                  |          
                                   --28.39%-- rest_init
                                             start_kernel
                                             x86_64_start_reservations
                                             x86_64_start_kernel

     2.97%      kworker/0:2  [kernel.kallsyms]                    [k] xts_crypt                                                                                                                                                                                                                                                                       
                |
                --- xts_crypt
                   |          
                   |--96.01%-- xts_decrypt
                   |          ablk_decrypt
                   |          crypt_convert
                   |          kcryptd_crypt
                   |          process_one_work
                   |          worker_thread
                   |          kthread
                   |          ret_from_fork
                   |          
                   |--3.83%-- xts_encrypt
                   |          __ablk_encrypt
                   |          ablk_encrypt
                   |          crypt_convert
                   |          kcryptd_crypt
                   |          process_one_work
                   |          worker_thread
                   |          kthread
                   |          ret_from_fork
                    --0.17%-- [...]

     1.47%  qemu-system-x86  qemu-system-x86_64                   [.] buffer_is_zero                                                                                                                                                                                                                                                                  
            |
            --- buffer_is_zero
                0x7f1e2da33640
                0x7f1e2be94bee

     1.29%      kworker/0:0  [kernel.kallsyms]                    [k] gf128mul_x_ble                                                                                                                                                                                                                                                                  
                |
                --- gf128mul_x_ble
                   |          
                   |--57.06%-- xts_crypt
                   |          |          
                   |          |--89.43%-- xts_decrypt
                   |          |          ablk_decrypt
                   |          |          crypt_convert
                   |          |          kcryptd_crypt
                   |          |          process_one_work
                   |          |          worker_thread
                   |          |          kthread
                   |          |          ret_from_fork
                   |          |          
                   |           --10.57%-- xts_encrypt
                   |                     __ablk_encrypt
                   |                     ablk_encrypt
                   |                     crypt_convert
                   |                     kcryptd_crypt
                   |                     process_one_work
                   |                     worker_thread
                   |                     kthread
                   |                     ret_from_fork
                   |          
                   |--39.21%-- xts_decrypt
                   |          ablk_decrypt
                   |          crypt_convert
                   |          kcryptd_crypt
                   |          process_one_work
                   |          worker_thread
                   |          kthread
                   |          ret_from_fork
                   |          
                    --3.72%-- xts_encrypt
                              __ablk_encrypt
                              ablk_encrypt
                              crypt_convert
                              kcryptd_crypt
                              process_one_work
                              worker_thread
                              kthread
                              ret_from_fork

     1.01%           python  [kernel.kallsyms]                    [k] copy_user_enhanced_fast_string                                                                                                                                                                                                                                                  
                     |
                     --- copy_user_enhanced_fast_string
                        |          
                        |--58.36%-- generic_file_buffered_write
                        |          __generic_file_aio_write
                        |          generic_file_aio_write
                        |          ext4_file_write
                        |          do_sync_write
                        |          vfs_write
                        |          sys_write
                        |          system_call_fastpath
                        |          __GI___libc_write
                        |          |          
                        |          |--14.05%-- 0x0
                        |          |          |          
                        |          |           --100.00%-- 0x0
                        |           --85.95%-- [...]
                        |          
                        |--41.10%-- unix_stream_recvmsg
                        |          sock_recvmsg
                        |          sys_recvfrom
                        |          system_call_fastpath
                        |          __libc_recv
                        |          |          
                        |          |--36.53%-- 0xbb6c80
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.66%-- 0xc72a20
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.53%-- 0xc72300
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.26%-- 0xc725b8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.20%-- 0xbec470
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.17%-- 0xc724f8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.13%-- 0xc72960
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.13%-- 0xc69860
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.09%-- 0xc72a80
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.08%-- 0xc72510
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.07%-- 0xc728e8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.07%-- 0xc72a50
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.07%-- 0xc72318
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.06%-- 0xc72a68
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.05%-- 0xc69908
                        |          |          0x306f1835e0
                        |          |          
                        |          |--1.03%-- 0xc6e078
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.99%-- 0xc698a8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.96%-- 0xc72918
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.95%-- 0xc72a08
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.95%-- 0xc72540
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.95%-- 0xc723d8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.94%-- 0xc725e8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.94%-- 0xbec4b8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.91%-- 0xbec4e8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.90%-- 0xc72678
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.88%-- 0xc725a0
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.88%-- 0xc72420
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.87%-- 0xc72570
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.87%-- 0xc723c0
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.87%-- 0xc72378
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.87%-- 0xc72330
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.85%-- 0xc6e210
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.84%-- 0xc724c8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.83%-- 0xbec440
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.82%-- 0xaef530
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.81%-- 0xc729a8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.78%-- 0xc698c0
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.78%-- 0xc6dfe8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.77%-- 0xc6e1e0
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.77%-- 0xc698d8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.76%-- 0xc72360
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.75%-- 0xc6e000
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.73%-- 0xc6e060
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.73%-- 0xc6e0c0
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.72%-- 0xc72468
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.71%-- 0xc6e390
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.68%-- 0xbec518
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.68%-- 0xc6e258
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.65%-- 0xc72a38
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.65%-- 0xc6e180
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.65%-- 0xc72348
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.65%-- 0xc72618
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.63%-- 0xc697e8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.63%-- 0xc6e2a0
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.62%-- 0xc6e2e8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.62%-- 0xc69950
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.61%-- 0xc69938
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.59%-- 0xc729d8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.59%-- 0xc722e8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.58%-- 0xc6e240
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.56%-- 0xc72558
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.56%-- 0xbec488
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.54%-- 0xc6e1f8
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.54%-- 0xc69830
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.51%-- 0xc69818
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.51%-- 0xc729c0
                        |          |          0x306f1835e0
                        |          |          
                        |          |--0.51%-- 0xc72630
                        |          |          0x306f1835e0
                        |           --7.91%-- [...]
                         --0.53%-- [...]

     0.91%           chrome  chrome                               [.] 0x0000000000d54920                                                                                                                                                                                                                                                              
                     |          
                     |--4.50%-- 0x7f68deda92ea
                     |          |          
                     |          |--48.81%-- 0x490000187de8f041
                     |          |          
                     |          |--20.41%-- 0x187de8f041
                     |          |          
                     |          |--2.26%-- 0xaa00000001000000
                     |          |          
                     |          |--2.08%-- 0x390000187de8f041
                     |          |          
                     |          |--1.10%-- 0x200000001000000
                     |          |          
                     |          |--1.06%-- 0x8900003042d7d1f2
                     |          |          
                     |          |--1.06%-- 0xb200000002000000
                     |          |          
                     |          |--1.05%-- 0xf600000000000000
                     |          |          
                     |          |--1.05%-- 0x390000357751a426
                     |          |          
                     |          |--1.05%-- 0x3900000ca1445ebb
                     |          |          
                     |          |--1.05%-- 0x3100000044000000
                     |          |          
                     |          |--1.05%-- 0x30000000d000000
                     |          |          
                     |          |--1.05%-- 0x6a00000002000000
                     |          |          
                     |          |--1.05%-- 0x390000132b445546
                     |          |          
                     |          |--1.05%-- 0xa10000187de8f041
                     |          |          
                     |          |--1.04%-- 0x9600000011000000
                     |          |          
                     |          |--1.04%-- 0x7600000002000000
                     |          |          
                     |          |--1.04%-- 0x620000000e000000
                     |          |          
                     |          |--1.04%-- 0x300000009000000
                     |          |          
                     |          |--1.04%-- 0x5a00000003000000
                     |          |          
                     |          |--1.02%-- 0x30000000a000000
                     |          |          
                     |          |--0.98%-- 0x5600000002000000
                     |          |          
                     |          |--0.83%-- 0x890000187de8f041
                     |          |          
                     |          |--0.77%-- 0x3600000010000000
                     |          |          
                     |          |--0.66%-- 0x30000000b000000
                     |          |          
                     |          |--0.59%-- 0x6600000002000000
                     |           --4.77%-- [...]
                     |          
                     |--4.42%-- 0x7f68deda0428
                     |          0x0
                     |          
                     |--1.67%-- 0x7f68deda04f4
                     |          0x0
                     |          
                     |--1.39%-- 0x7f68dedac7a1
                     |          0x0
                     |          
                     |--1.34%-- 0x7f68deda057e
                     |          0x0
                     |          
                     |--1.23%-- 0x7f68deda0615
                     |          0x0
                     |          
                     |--1.09%-- 0x7f68deda05b9
                     |          0x0
                     |          
                     |--1.04%-- 0x7f68dede05ac
                     |          0x40003000000
                     |          
                     |--1.01%-- 0x7f68deda149e
                     |          |          
                     |          |--33.99%-- 0x490000187de8f041
                     |          |          
                     |          |--17.77%-- 0x7de8f0a869000000
                     |          |          
                     |          |--12.84%-- 0x6900000000000000
                     |          |          
                     |          |--7.28%-- 0xb200000006000000
                     |          |          
                     |          |--4.67%-- 0xd900001afa660f31
                     |          |          
                     |          |--4.66%-- 0x7de8f0a719000001
                     |          |          
                     |          |--4.64%-- 0x2100000884a5b0af
                     |          |          
                     |          |--4.46%-- 0x3200000016000000
                     |          |          
                     |          |--4.11%-- 0x5900001fbf980e53
                     |          |          
                     |          |--2.53%-- 0x300000004000000
                     |          |          
                     |          |--1.64%-- 0xa200000006000000
                     |          |          
                     |           --1.41%-- 0x4100002c86ccbf9a
                     |          
                     |--0.78%-- 0x7f68dedae976
                     |          0x2577cd04101
                     |          0x7f68dd4d4c60
                     |          
                     |--0.78%-- 0x7f68deda984a
                     |          |          
                     |          |--22.10%-- 0x187de8f04139
                     |          |          |          
                     |          |          |--83.05%-- 0x3100000044000000
                     |          |          |          
                     |          |          |--9.53%-- 0xe900000006000000
                     |          |          |          
                     |          |           --7.42%-- 0xc100000009000000
                     |          |          
                     |          |--19.96%-- 0x5978fe04121
                     |          |          |          
                     |          |          |--69.93%-- 0x6900000000000000
                     |          |          |          
                     |          |           --30.07%-- 0xd100000002000000
                     |          |          
                     |          |--14.18%-- 0xca144504101
                     |          |          |          
                     |          |          |--42.82%-- 0xc100002d7c9487da
                     |          |          |          
                     |          |          |--42.25%-- 0xc1000022f374a1f8
                     |          |          |          
                     |          |           --14.93%-- 0xa10000187de8f0b6
                     |          |          
                     |          |--7.42%-- 0x299480004101
                     |          |          |          
                     |          |          |--71.46%-- 0xc100000edf3481ef
                     |          |          |          
                     |          |           --28.54%-- 0xc10000187de8f09c
                     |          |          
                     |          |--6.33%-- 0x345d68860a19
                     |          |          0x7900000005000000
                     |          |          
                     |          |--6.07%-- 0x2f667713d109
                     |          |          0xa10000187de8f09c
                     |          |          
                     |          |--6.05%-- 0x39681b430cc1
                     |          |          0x187de8f041
                     |          |          
                     |          |--6.03%-- 0x5978fe2be41
                     |          |          0x187de8f041
                     |          |          
                     |          |--5.99%-- 0x5c35e19a3a9
                     |          |          0x1119000aad19000a
                     |          |          
                     |          |--2.85%-- 0x345d688532e9
                     |          |          0x3100000044000000
                     |          |          
                     |          |--2.05%-- 0x1b40628834f1
                     |          |          0xc10000187de8f0d2
                     |          |          
                     |           --0.96%-- 0x5c35e19bbc1
                     |                     0xc100001408d4eba8
                     |          
                     |--0.75%-- 0x7f68dede099f
                     |          |          
                     |          |--68.75%-- 0x5978fe04121
                     |          |          0x490000187de8f041
                     |          |          
                     |          |--6.25%-- 0x22f374a141a9
                     |          |          0x187de8f041
                     |          |          
                     |          |--6.25%-- 0x3bb394864b91
                     |          |          0xe0000000b000000
                     |          |          
                     |          |--6.25%-- 0xedf34866c69
                     |          |          0x490000187de8f041
                     |          |          
                     |          |--6.25%-- 0x5978fe04141
                     |          |          0x920000001c000000
                     |          |          
                     |           --6.25%-- 0x10d8e8c76979
                     |                     0x520000002e000000
                     |          
                     |--0.67%-- 0x7f68deda9317
                     |          |          
                     |          |--7.15%-- 0x3600000004000000
                     |          |          
                     |          |--7.15%-- 0xd200000003000000
                     |          |          
                     |          |--7.13%-- 0xe200000005000000
                     |          |          
                     |          |--6.99%-- 0x620000000a000000
                     |          |          
                     |          |--6.97%-- 0x9a00000002000000
                     |          |          
                     |          |--6.96%-- 0x200000002000000
                     |          |          
                     |          |--6.96%-- 0xbe00000003000000
                     |          |          
                     |          |--6.94%-- 0xa200000002000000
                     |          |          
                     |          |--6.92%-- 0xf200000002000000
                     |          |          
                     |          |--6.76%-- 0x1e00000014000000
                     |          |          
                     |          |--6.16%-- 0x8200000002000000
                     |          |          
                     |          |--4.80%-- 0xbe0000000a000000
                     |          |          
                     |          |--2.47%-- 0x3200000002000000
                     |          |          
                     |          |--2.46%-- 0xee00000002000000
                     |          |          
                     |          |--2.45%-- 0xda00000002000000
                     |          |          
                     |          |--2.44%-- 0x9200000010000000
                     |          |          
                     |          |--2.40%-- 0x2e00000003000000
                     |          |          
                     |          |--2.39%-- 0x4a00000003000000
                     |          |          
                     |          |--2.34%-- 0x7e00000002000000
                     |          |          
                     |           --2.18%-- 0x300000019000000
                     |          
                     |--0.67%-- 0x7f68deda06a3
                     |          0x0
                     |          
                     |--0.62%-- 0x7f68deda0438
                     |          0x0
                     |          
                     |--0.62%-- 0x7f68dedac57b
                     |          0x0
                     |          
                     |--0.55%-- 0x7f68dedad338
                     |          0x0
                     |          
                     |--0.53%-- 0x7f68dedaceb4
                     |          |          
                     |          |--16.88%-- 0x9100000002000000
                     |          |          
                     |          |--8.85%-- 0xd1190009ad190009
                     |          |          
                     |          |--8.83%-- 0x100000001000000
                     |          |          
                     |          |--8.82%-- 0xf1190001ad190007
                     |          |          
                     |          |--8.81%-- 0xb1190000ad130001
                     |          |          
                     |          |--8.78%-- 0x7100000002000000
                     |          |          
                     |          |--8.77%-- 0xe1190000ad130001
                     |          |          
                     |          |--8.67%-- 0x5100000002000000
                     |          |          
                     |          |--8.65%-- 0xb1190009ad19000b
                     |          |          
                     |          |--5.38%-- 0x490000187de8f041
                     |          |          
                     |          |--4.99%-- 0x31190009ad19000a
                     |          |          
                     |           --2.58%-- 0x100000002000000
                      --76.35%-- [...]

     0.88%      kworker/0:2  [kernel.kallsyms]                    [k] gf128mul_x_ble                                                                                                                                                                                                                                                                  
                |
                --- gf128mul_x_ble
                   |          
                   |--55.48%-- xts_crypt
                   |          |          
                   |          |--96.65%-- xts_decrypt
                   |          |          ablk_decrypt
                   |          |          crypt_convert
                   |          |          kcryptd_crypt
                   |          |          process_one_work
                   |          |          worker_thread
                   |          |          kthread
                   |          |          ret_from_fork
                   |          |          
                   |           --3.35%-- xts_encrypt
                   |                     __ablk_encrypt
                   |                     ablk_encrypt
                   |                     crypt_convert
                   |                     kcryptd_crypt
                   |                     process_one_work
                   |                     worker_thread
                   |                     kthread
                   |                     ret_from_fork
                   |          
                   |--43.24%-- xts_decrypt
                   |          ablk_decrypt
                   |          crypt_convert
                   |          kcryptd_crypt
                   |          process_one_work
                   |          worker_thread
                   |          kthread
                   |          ret_from_fork
                   |          
                    --1.29%-- xts_encrypt
                              __ablk_encrypt
                              ablk_encrypt
                              crypt_convert
                              kcryptd_crypt
                              process_one_work
                              worker_thread
                              kthread
                              ret_from_fork

     0.83%           python  libpython2.7.so.1.0                  [.] PyEval_EvalFrameEx                                                                                                                                                                                                                                                              
                     |
                     --- PyEval_EvalFrameEx
                        |          
                        |--11.78%-- 0x7c00617800037d00
                        |          
                        |--10.54%-- 0x64000174001b7300
                        |          
                        |--4.39%-- 0x8300016a00007c00
                        |          
                        |--3.62%-- 0x306f183ea0
                        |          |          
                        |          |--32.64%-- 0x306f1835e0
                        |          |          
                        |          |--30.91%-- 0xc244f0
                        |          |          
                        |          |--13.62%-- 0x306f1730a0
                        |          |          
                        |          |--11.35%-- 0x306f172f00
                        |          |          
                        |          |--9.81%-- 0x306f16fb20
                        |          |          
                        |          |--0.52%-- 0x306f183b60
                        |           --1.15%-- [...]
                        |          
                        |--3.60%-- 0x8300016a00017c00
                        |          
                        |--3.58%-- 0x1147a0100008300
                        |          
                        |--3.47%-- 0x47d1600017400
                        |          
                        |--3.30%-- 0xdb7a0100008300
                        |          
                        |--3.17%-- 0x306f17f340
                        |          
                        |--2.97%-- 0x7c00027d00028300
                        |          
                        |--2.36%-- 0x7f4bc864b3d0
                        |          |          
                        |          |--98.04%-- 0x306f17eb60
                        |          |          
                        |          |--1.50%-- 0x306f188280
                        |           --0.46%-- [...]
                        |          
                        |--2.27%-- 0x7200018300016400
                        |          
                        |--2.23%-- 0x7c00017400017d00
                        |          
                        |--2.17%-- 0x306f17de00
                        |          
                        |--2.07%-- 0x7d00028300026a00
                        |          
                        |--1.93%-- 0x1600027c00017c00
                        |          
                        |--1.78%-- 0x4400006a00007c00
                        |          
                        |--1.58%-- 0x7c00027c00017c00
                        |          
                        |--1.49%-- 0xaef7a0
                        |          |          
                        |          |--66.78%-- 0xcad870
                        |          |          
                        |          |--19.57%-- 0x306f1730a0
                        |          |          
                        |           --13.64%-- 0x306f172f00
                        |          
                        |--1.46%-- 0xaef530
                        |          |          
                        |          |--72.45%-- 0x306f1835e0
                        |          |          
                        |           --27.55%-- 0x306f188280
                        |          
                        |--1.30%-- 0x7c00047d00028300
                        |          
                        |--1.23%-- 0x6a00007c00027400
                        |          
                        |--1.15%-- 0x8300027c00017c00
                        |          
                        |--1.12%-- 0x100018300017c00
                        |          
                        |--0.93%-- 0x640100008300
                        |          
                        |--0.86%-- 0x7f4bc972f0b0
                        |          |          
                        |          |--86.69%-- 0xc244f0
                        |          |          
                        |          |--9.92%-- 0x306f172f00
                        |          |          
                        |          |--1.85%-- 0x306f16fb20
                        |          |          
                        |           --1.54%-- 0x306f183b60
                        |          
                        |--0.84%-- 0xaef7b8
                        |          |          
                        |          |--34.65%-- 0x306f1835e0
                        |          |          
                        |          |--26.33%-- 0xcad870
                        |          |          
                        |          |--19.23%-- 0x306f172f00
                        |          |          
                        |          |--14.95%-- 0x306f1730a0
                        |          |          
                        |          |--2.00%-- 0x306f189760
                        |          |          
                        |          |--1.67%-- 0x306f183b60
                        |          |          
                        |           --1.16%-- 0x306f180420
                        |          
                        |--0.81%-- 0x7c00028300007c00
                        |          
                        |--0.79%-- 0x5300018300
                        |          
                        |--0.68%-- 0x7f4bc865b780
                        |          |          
                        |          |--99.27%-- 0x306f1835e0
                        |          |          
                        |           --0.73%-- 0x306f189760
                        |          
                        |--0.64%-- 0x7f4bc971de68
                        |          |          
                        |          |--54.43%-- 0xc244f0
                        |          |          
                        |          |--34.88%-- 0x306f172f00
                        |          |          
                        |           --10.69%-- 0x306f183b60
                        |          
                        |--0.63%-- 0xaef7d0
                        |          |          
                        |          |--99.07%-- 0x306f1835e0
                        |          |          
                        |           --0.93%-- 0x306f16fb20
                        |          
                        |--0.61%-- 0x7f4bd064e7d0
                        |          |          
                        |          |--39.48%-- 0x306f1835e0
                        |          |          
                        |          |--33.99%-- 0xc244f0
                        |          |          
                        |          |--17.57%-- 0x306f183b60
                        |          |          
                        |           --8.96%-- 0x306f172f00
                        |          
                        |--0.54%-- 0x7f4bc9724738
                        |          |          
                        |          |--64.76%-- 0xc244f0
                        |          |          
                        |          |--29.68%-- 0x306f172f00
                        |          |          
                        |           --5.56%-- 0x306f183b60
                        |          
                        |--0.53%-- 0x7d00037d00025c00
                         --17.59%-- [...]

     0.78%      kworker/0:0  [kernel.kallsyms]                    [k] _aesni_enc1                                                                                                                                                                                                                                                                     
                |
                --- _aesni_enc1
                    xts_crypt
                   |          
                   |--91.62%-- xts_decrypt
                   |          ablk_decrypt
                   |          crypt_convert
                   |          kcryptd_crypt
                   |          process_one_work
                   |          worker_thread
                   |          kthread
                   |          ret_from_fork
                   |          
                    --8.38%-- xts_encrypt
                              __ablk_encrypt
                              ablk_encrypt
                              crypt_convert
                              kcryptd_crypt
                              process_one_work
                              worker_thread
                              kthread
                              ret_from_fork

     0.76%  qemu-system-x86  [kernel.kallsyms]                    [k] fget_light                                                                                                                                                                                                                                                                      
            |
            --- fget_light
               |          
               |--56.26%-- do_sys_poll
               |          sys_poll
               |          system_call_fastpath
               |          __GI___libc_poll
               |          |          
               |           --100.00%-- 0x7f1e2daf5080
               |          
               |--16.78%-- sys_write
               |          system_call_fastpath
               |          0x7f1e29fc60cd
               |          |          
               |          |--99.14%-- 0x0
               |           --0.86%-- [...]
               |          
               |--8.82%-- sockfd_lookup_light
               |          |          
               |          |--96.66%-- sys_recvfrom
               |          |          system_call_fastpath
               |          |          __libc_recv
               |          |          
               |          |--2.94%-- sys_sendmsg
               |          |          system_call_fastpath
               |          |          __libc_sendmsg
               |          |          |          
               |          |          |--73.99%-- 0x10000
               |          |          |          |          
               |          |          |          |--24.01%-- 0x6156100000000
               |          |          |          |          
               |          |          |          |--10.16%-- 0x3e04ccf29e9192db
               |          |          |          |          
               |          |          |          |--10.07%-- 0xef01
               |          |          |          |          
               |          |          |          |--5.93%-- 0x203d206574616470
               |          |          |          |          
               |          |          |          |--5.93%-- 0x7269406c69617469
               |          |          |          |          
               |          |          |          |--5.49%-- 0x650a252072616863
               |          |          |          |          
               |          |          |          |--5.23%-- 0x32762f646e69622f
               |          |          |          |          
               |          |          |          |--5.22%-- 0x2064657669726544
               |          |          |          |          
               |          |          |          |--5.19%-- 0x4ee500004ee4
               |          |          |          |          
               |          |          |          |--5.12%-- 0xba3c000000000000
               |          |          |          |          
               |          |          |          |--5.08%-- 0xc3f40d87a8e73963
               |          |          |          |          
               |          |          |          |--4.52%-- 0x525e00000001
               |          |          |          |          
               |          |          |          |--4.47%-- 0x67207473756a2065
               |          |          |          |          
               |          |          |           --3.57%-- 0x63
               |          |          |          
               |          |           --26.01%-- 0x1c
               |          |                     0x4009a82d1e7f0000
               |           --0.39%-- [...]
               |          
               |--5.47%-- sys_read
               |          system_call_fastpath
               |          0x7f1e29fc612d
               |          |          
               |          |--54.66%-- 0x0
               |          |          
               |           --45.34%-- 0x2da2f23000000000
               |          
               |--3.90%-- sys_pread64
               |          system_call_fastpath
               |          0x7f1e29fc6993
               |          |          
               |          |--99.14%-- 0xa
               |          |          0x206200000000
               |          |          
               |           --0.86%-- 0xb
               |                     0x4000000001
               |          
               |--3.87%-- sys_poll
               |          system_call_fastpath
               |          __GI___libc_poll
               |          
               |--2.91%-- system_call_fastpath
               |          |          
               |          |--42.30%-- 0x7f1e29fc60cd
               |          |          |          
               |          |          |--98.39%-- 0x0
               |          |          |          
               |          |           --1.61%-- 0x12e640000
               |          |                     0x400000000
               |          |          
               |          |--32.24%-- 0x7f1e29fc612d
               |          |          |          
               |          |          |--64.74%-- 0x2da2f23000000000
               |          |          |          
               |          |           --35.26%-- 0x0
               |          |          
               |          |--23.58%-- 0x7f1e29fc6993
               |          |          |          
               |          |          |--99.08%-- 0xa
               |          |          |          0x206200000000
               |          |          |          
               |          |           --0.92%-- 0xb
               |          |                     0x4000000001
               |          |          
               |           --1.89%-- __GI___ioctl
               |                     |          
               |                     |--62.28%-- 0x170
               |                     |          memory_region_iorange_write
               |                     |          0x6c894cfd8948d824
               |                     |          
               |                      --37.72%-- 0x100000002
               |          
               |--1.22%-- sys_recvfrom
               |          system_call_fastpath
               |          __libc_recv
               |          
               |--0.72%-- sys_ioctl
               |          system_call_fastpath
               |          __GI___ioctl
               |          |          
               |          |--57.26%-- 0x100000002
               |          |          
               |          |--33.78%-- 0x170
               |          |          memory_region_iorange_write
               |          |          0x6c894cfd8948d824
               |          |          
               |           --8.95%-- 0x100000006
                --0.06%-- [...]

     0.76%      kworker/0:0  [kernel.kallsyms]                    [k] math_state_restore                                                                                                                                                                                                                                                              
                |
                --- math_state_restore
                   |          
                   |--97.35%-- __kernel_fpu_end
                   |          |          
                   |          |--90.54%-- xts_decrypt
                   |          |          ablk_decrypt
                   |          |          crypt_convert
                   |          |          kcryptd_crypt
                   |          |          process_one_work
                   |          |          worker_thread
                   |          |          kthread
                   |          |          ret_from_fork
                   |          |          
                   |           --9.46%-- xts_encrypt
                   |                     __ablk_encrypt
                   |                     ablk_encrypt
                   |                     crypt_convert
                   |                     kcryptd_crypt
                   |                     process_one_work
                   |                     worker_thread
                   |                     kthread
                   |                     ret_from_fork
                   |          
                   |--2.42%-- xts_decrypt
                   |          ablk_decrypt
                   |          crypt_convert
                   |          kcryptd_crypt
                   |          process_one_work
                   |          worker_thread
                   |          kthread
                   |          ret_from_fork
                    --0.23%-- [...]

     0.75%  qemu-system-x86  [kernel.kallsyms]                    [k] _raw_spin_lock_irqsave                                                                                                                                                                                                                                                          
            |
            --- _raw_spin_lock_irqsave
               |          
               |--29.57%-- add_wait_queue
               |          __pollwait
               |          |          
               |          |--48.83%-- eventfd_poll
               |          |          do_sys_poll
               |          |          sys_poll
               |          |          system_call_fastpath
               |          |          __GI___libc_poll
               |          |          
               |          |--23.73%-- unix_poll
               |          |          sock_poll
               |          |          do_sys_poll
               |          |          sys_poll
               |          |          system_call_fastpath
               |          |          __GI___libc_poll
               |          |          
               |          |--13.93%-- signalfd_poll
               |          |          do_sys_poll
               |          |          sys_poll
               |          |          system_call_fastpath
               |          |          __GI___libc_poll
               |          |          
               |          |--11.49%-- tcp_poll
               |          |          sock_poll
               |          |          do_sys_poll
               |          |          sys_poll
               |          |          system_call_fastpath
               |          |          __GI___libc_poll
               |          |          
               |           --2.02%-- datagram_poll
               |                     udp_poll
               |                     sock_poll
               |                     do_sys_poll
               |                     sys_poll
               |                     system_call_fastpath
               |                     __GI___libc_poll
               |          
               |--24.54%-- remove_wait_queue
               |          poll_freewait
               |          do_sys_poll
               |          sys_poll
               |          system_call_fastpath
               |          __GI___libc_poll
               |          
               |--22.32%-- eventfd_poll
               |          do_sys_poll
               |          sys_poll
               |          system_call_fastpath
               |          __GI___libc_poll
               |          
               |--5.63%-- lock_hrtimer_base.isra.21
               |          |          
               |          |--58.87%-- __hrtimer_start_range_ns
               |          |          |          
               |          |          |--98.59%-- hrtimer_start_range_ns
               |          |          |          |          
               |          |          |          |--43.22%-- schedule_hrtimeout_range_clock
               |          |          |          |          schedule_hrtimeout_range
               |          |          |          |          poll_schedule_timeout
               |          |          |          |          do_sys_poll
               |          |          |          |          sys_poll
               |          |          |          |          system_call_fastpath
               |          |          |          |          __GI___libc_poll
               |          |          |          |          
               |          |          |          |--30.98%-- futex_wait_queue_me
               |          |          |          |          futex_wait
               |          |          |          |          do_futex
               |          |          |          |          sys_futex
               |          |          |          |          system_call_fastpath
               |          |          |          |          sem_timedwait
               |          |          |          |          
               |          |          |           --25.80%-- common_timer_set
               |          |          |                     sys_timer_settime
               |          |          |                     system_call_fastpath
               |          |          |                     timer_settime@@GLIBC_2.3.3
               |          |          |                     0x0
               |          |          |          
               |          |           --1.41%-- hrtimer_start
               |          |                     start_apic_timer
               |          |                     apic_reg_write
               |          |                     apic_mmio_write
               |          |                     write_mmio
               |          |                     emulator_read_write_onepage
               |          |                     emulator_read_write
               |          |                     emulator_write_emulated
               |          |                     segmented_write.isra.23
               |          |                     x86_emulate_insn
               |          |                     x86_emulate_instruction
               |          |                     handle_apic_access
               |          |                     vmx_handle_exit
               |          |                     kvm_arch_vcpu_ioctl_run
               |          |                     kvm_vcpu_ioctl
               |          |                     do_vfs_ioctl
               |          |                     sys_ioctl
               |          |                     system_call_fastpath
               |          |                     __GI___ioctl
               |          |                     |          
               |          |                     |--52.26%-- 0x100000002
               |          |                     |          
               |          |                      --47.74%-- 0x10100000006
               |          |          
               |           --41.13%-- hrtimer_try_to_cancel
               |                     |          
               |                     |--62.98%-- hrtimer_cancel
               |                     |          |          
               |                     |          |--66.47%-- schedule_hrtimeout_range_clock
               |                     |          |          schedule_hrtimeout_range
               |                     |          |          poll_schedule_timeout
               |                     |          |          do_sys_poll
               |                     |          |          sys_poll
               |                     |          |          system_call_fastpath
               |                     |          |          __GI___libc_poll
               |                     |          |          
               |                     |          |--31.71%-- futex_wait
               |                     |          |          do_futex
               |                     |          |          sys_futex
               |                     |          |          system_call_fastpath
               |                     |          |          sem_timedwait
               |                     |          |          
               |                     |          |--1.05%-- apic_reg_write
               |                     |          |          apic_mmio_write
               |                     |          |          write_mmio
               |                     |          |          emulator_read_write_onepage
               |                     |          |          emulator_read_write
               |                     |          |          emulator_write_emulated
               |                     |          |          segmented_write.isra.23
               |                     |          |          x86_emulate_insn
               |                     |          |          x86_emulate_instruction
               |                     |          |          handle_apic_access
               |                     |          |          vmx_handle_exit
               |                     |          |          kvm_arch_vcpu_ioctl_run
               |                     |          |          kvm_vcpu_ioctl
               |                     |          |          do_vfs_ioctl
               |                     |          |          sys_ioctl
               |                     |          |          system_call_fastpath
               |                     |          |          __GI___ioctl
               |                     |          |          0x100000002
               |                     |          |          
               |                     |           --0.78%-- __kvm_migrate_apic_timer
               |                     |                     __kvm_migrate_timers
               |                     |                     kvm_arch_vcpu_ioctl_run
               |                     |                     kvm_vcpu_ioctl
               |                     |                     do_vfs_ioctl
               |                     |                     sys_ioctl
               |                     |                     system_call_fastpath
               |                     |                     __GI___ioctl
               |                     |                     0x100000002
               |                     |          
               |                      --37.02%-- common_timer_set
               |                                sys_timer_settime
               |                                system_call_fastpath
               |                                timer_settime@@GLIBC_2.3.3
               |                                0x0
               |          
               |--4.22%-- try_to_wake_up
               |          |          
               |          |--53.28%-- default_wake_function
               |          |          |          
               |          |          |--93.39%-- pollwake
               |          |          |          __wake_up_common
               |          |          |          __wake_up_locked_key
               |          |          |          eventfd_write
               |          |          |          vfs_write
               |          |          |          sys_write
               |          |          |          system_call_fastpath
               |          |          |          0x7f1e29fc60cd
               |          |          |          
               |          |           --6.61%-- autoremove_wake_function
               |          |                     __wake_up_common
               |          |                     |          
               |          |                     |--97.21%-- __wake_up_sync_key
               |          |                     |          sock_def_readable
               |          |                     |          unix_stream_sendmsg
               |          |                     |          sock_sendmsg
               |          |                     |          __sys_sendmsg
               |          |                     |          sys_sendmsg
               |          |                     |          system_call_fastpath
               |          |                     |          __libc_sendmsg
               |          |                     |          0x1c
               |          |                     |          0x4009a82d1e7f0000
               |          |                     |          
               |          |                      --2.79%-- __wake_up
               |          |                                apic_timer_fn
               |          |                                __run_hrtimer
               |          |                                hrtimer_interrupt
               |          |                                smp_apic_timer_interrupt
               |          |                                apic_timer_interrupt
               |          |                                vfs_write
               |          |                                sys_write
               |          |                                system_call_fastpath
               |          |                                0x7f1e29fc60cd
               |          |          
               |          |--43.56%-- wake_up_state
               |          |          wake_futex
               |          |          futex_wake
               |          |          do_futex
               |          |          sys_futex
               |          |          system_call_fastpath
               |          |          |          
               |          |          |--95.98%-- sem_post
               |          |          |          |          
               |          |          |          |--99.21%-- 0xa
               |          |          |          |          0x206200000000
               |          |          |          |          
               |          |          |           --0.79%-- 0xb
               |          |          |                     0x4000000001
               |          |          |          
               |          |           --4.02%-- __lll_unlock_wake
               |          |                     |          
               |          |                     |--54.43%-- 0x100000002
               |          |                     |          
               |          |                      --45.57%-- 0x100000006
               |          |          
               |           --3.16%-- wake_up_process
               |                     wake_up_worker
               |                     insert_work
               |                     __queue_work
               |                     delayed_work_timer_fn
               |                     call_timer_fn
               |                     run_timer_softirq
               |                     __do_softirq
               |                     call_softirq
               |                     do_softirq
               |                     irq_exit
               |                     smp_apic_timer_interrupt
               |                     apic_timer_interrupt
               |                     |          
               |                     |--18.33%-- bmdma_rw_buf
               |                     |          
               |                     |--13.84%-- add_wait_queue
               |                     |          __pollwait
               |                     |          eventfd_poll
               |                     |          do_sys_poll
               |                     |          sys_poll
               |                     |          system_call_fastpath
               |                     |          __GI___libc_poll
               |                     |          
               |                     |--10.31%-- 0x7f1e27398550
               |                     |          
               |                     |--9.72%-- elv_set_request
               |                     |          get_request
               |                     |          blk_queue_bio
               |                     |          generic_make_request
               |                     |          submit_bio
               |                     |          do_blockdev_direct_IO
               |                     |          __blockdev_direct_IO
               |                     |          ext4_ind_direct_IO
               |                     |          ext4_direct_IO
               |                     |          generic_file_aio_read
               |                     |          do_sync_read
               |                     |          vfs_read
               |                     |          sys_pread64
               |                     |          system_call_fastpath
               |                     |          0x7f1e29fc6993
               |                     |          0xa
               |                     |          0x206200000000
               |                     |          
               |                     |--8.83%-- fsnotify
               |                     |          vfs_write
               |                     |          sys_write
               |                     |          system_call_fastpath
               |                     |          0x7f1e29fc60cd
               |                     |          0x0
               |                     |          
               |                     |--7.71%-- dm_get_live_table
               |                     |          |          
               |                     |          |--63.20%-- dm_merge_bvec
               |                     |          |          __bio_add_page.part.13
               |                     |          |          bio_add_page
               |                     |          |          do_blockdev_direct_IO
               |                     |          |          __blockdev_direct_IO
               |                     |          |          ext4_ind_direct_IO
               |                     |          |          ext4_direct_IO
               |                     |          |          generic_file_aio_read
               |                     |          |          do_sync_read
               |                     |          |          vfs_read
               |                     |          |          sys_pread64
               |                     |          |          system_call_fastpath
               |                     |          |          0x7f1e29fc6993
               |                     |          |          0xa
               |                     |          |          0x206200000000
               |                     |          |          
               |                     |           --36.80%-- __split_and_process_bio
               |                     |                     dm_request
               |                     |                     generic_make_request
               |                     |                     submit_bio
               |                     |                     do_blockdev_direct_IO
               |                     |                     __blockdev_direct_IO
               |                     |                     ext4_ind_direct_IO
               |                     |                     ext4_direct_IO
               |                     |                     generic_file_aio_read
               |                     |                     do_sync_read
               |                     |                     vfs_read
               |                     |                     sys_pread64
               |                     |                     system_call_fastpath
               |                     |                     0x7f1e29fc6993
               |                     |                     0xa
               |                     |                     0x206200000000
               |                     |          
               |                     |--7.57%-- __blockdev_direct_IO
               |                     |          ext4_ind_direct_IO
               |                     |          ext4_direct_IO
               |                     |          generic_file_aio_read
               |                     |          do_sync_read
               |                     |          vfs_read
               |                     |          sys_pread64
               |                     |          system_call_fastpath
               |                     |          0x7f1e29fc6993
               |                     |          0xa
               |                     |          0x206200000000
               |                     |          
               |                     |--6.38%-- __split_and_process_bio
               |                     |          dm_request
               |                     |          generic_make_request
               |                     |          submit_bio
               |                     |          do_blockdev_direct_IO
               |                     |          __blockdev_direct_IO
               |                     |          ext4_ind_direct_IO
               |                     |          ext4_direct_IO
               |                     |          generic_file_aio_read
               |                     |          do_sync_read
               |                     |          vfs_read
               |                     |          sys_pread64
               |                     |          system_call_fastpath
               |                     |          0x7f1e29fc6993
               |                     |          0xa
               |                     |          0x206200000000
               |                     |          
               |                     |--5.84%-- ata_scsi_queuecmd
               |                     |          scsi_dispatch_cmd
               |                     |          scsi_request_fn
               |                     |          __blk_run_queue
               |                     |          queue_unplugged
               |                     |          blk_flush_plug_list
               |                     |          blk_finish_plug
               |                     |          do_blockdev_direct_IO
               |                     |          __blockdev_direct_IO
               |                     |          ext4_ind_direct_IO
               |                     |          ext4_direct_IO
               |                     |          generic_file_aio_read
               |                     |          do_sync_read
               |                     |          vfs_read
               |                     |          sys_pread64
               |                     |          system_call_fastpath
               |                     |          0x7f1e29fc6993
               |                     |          0xa
               |                     |          0x206200000000
               |                     |          
               |                     |--4.89%-- mempool_alloc_slab
               |                     |          mempool_alloc
               |                     |          crypt_map
               |                     |          __map_bio
               |                     |          __split_and_process_bio
               |                     |          dm_request
               |                     |          generic_make_request
               |                     |          submit_bio
               |                     |          do_blockdev_direct_IO
               |                     |          __blockdev_direct_IO
               |                     |          ext4_ind_direct_IO
               |                     |          ext4_direct_IO
               |                     |          generic_file_aio_read
               |                     |          do_sync_read
               |                     |          vfs_read
               |                     |          sys_pread64
               |                     |          system_call_fastpath
               |                     |          0x7f1e29fc6993
               |                     |          0xa
               |                     |          0x206200000000
               |                     |          
               |                     |--4.73%-- pthread_mutex_unlock
               |                     |          
               |                      --1.83%-- kvm_vcpu_ioctl
               |                                do_vfs_ioctl
               |                                sys_ioctl
               |                                system_call_fastpath
               |                                __GI___ioctl
               |                                0x100000002
               |          
               |--3.03%-- do_blockdev_direct_IO
               |          __blockdev_direct_IO
               |          ext4_ind_direct_IO
               |          ext4_direct_IO
               |          generic_file_aio_read
               |          do_sync_read
               |          vfs_read
               |          sys_pread64
               |          system_call_fastpath
               |          0x7f1e29fc6993
               |          0xa
               |          0x206200000000
               |          
               |--2.19%-- __lock_timer
               |          |          
               |          |--70.88%-- sys_timer_gettime
               |          |          system_call_fastpath
               |          |          timer_gettime@@GLIBC_2.3.3
               |          |          0x0
               |          |          
               |           --29.12%-- sys_timer_settime
               |                     system_call_fastpath
               |                     timer_settime@@GLIBC_2.3.3
               |                     0x0
               |          
               |--1.33%-- do_sys_poll
               |          sys_poll
               |          system_call_fastpath
               |          __GI___libc_poll
               |          
               |--0.89%-- __pollwait
               |          |          
               |          |--36.02%-- eventfd_poll
               |          |          do_sys_poll
               |          |          sys_poll
               |          |          system_call_fastpath
               |          |          __GI___libc_poll
               |          |          
               |          |--26.54%-- tcp_poll
               |          |          sock_poll
               |          |          do_sys_poll
               |          |          sys_poll
               |          |          system_call_fastpath
               |          |          __GI___libc_poll
               |          |          
               |          |--22.54%-- signalfd_poll
               |          |          do_sys_poll
               |          |          sys_poll
               |          |          system_call_fastpath
               |          |          __GI___libc_poll
               |          |          
               |          |--11.63%-- unix_poll
               |          |          sock_poll
               |          |          do_sys_poll
               |          |          sys_poll
               |          |          system_call_fastpath
               |          |          __GI___libc_poll
               |          |          
               |           --3.29%-- datagram_poll
               |                     udp_poll
               |                     sock_poll
               |                     do_sys_poll
               |                     sys_poll
               |                     system_call_fastpath
               |                     __GI___libc_poll
               |          
               |--0.84%-- get_page_from_freelist
               |          __alloc_pages_nodemask
               |          kmalloc_large_node
               |          __kmalloc_node_track_caller
               |          __kmalloc_reserve
               |          __alloc_skb
               |          sock_alloc_send_pskb
               |          sock_alloc_send_skb
               |          unix_stream_sendmsg
               |          |          
               |          |--55.13%-- sock_aio_write
               |          |          do_sync_write
               |          |          vfs_write
               |          |          sys_write
               |          |          system_call_fastpath
               |          |          0x7f1e29fc60cd
               |          |          
               |           --44.87%-- sock_sendmsg
               |                     __sys_sendmsg
               |                     sys_sendmsg
               |                     system_call_fastpath
               |                     __libc_sendmsg
               |                     0x10000
               |                     |          
               |                     |--5.71%-- 0x21940000334f
               |                     |          
               |                     |--5.13%-- 0x0
               |                     |          
               |                     |--5.04%-- 0x2020202020202020
               |                     |          
               |                     |--4.40%-- 0x894100011005bfff
               |                     |          
               |                     |--4.23%-- 0x8
               |                     |          
               |                     |--4.12%-- 0x3e414641383c093e
               |                     |          
               |                     |--4.05%-- 0x50ab5bf04b2110ba
               |                     |          
               |                     |--3.83%-- 0x7830092c37646465
               |                     |          
               |                     |--3.82%-- 0x5eb1aa993fc78bcc
               |                     |          
               |                     |--3.82%-- 0x101a2b90101a2b2
               |                     |          
               |                     |--3.59%-- 0x7c000800c6030000
               |                     |          
               |                     |--3.31%-- 0x9e0185016b015001
               |                     |          
               |                     |--3.22%-- 0xc07100000002e
               |                     |          
               |                     |--3.05%-- 0x725f7064755f7469
               |                     |          
               |                     |--3.02%-- 0xc00003c7f0000
               |                     |          
               |                     |--2.87%-- 0x2853893424548b20
               |                     |          
               |                     |--2.87%-- 0x50ab5bb15112553f
               |                     |          
               |                     |--2.87%-- 0x81aa
               |                     |          
               |                     |--2.83%-- 0xfc212bba4cd963fb
               |                     |          
               |                     |--2.79%-- 0xc6ade0de8834a6bd
               |                     |          
               |                     |--2.76%-- 0x9f2110000002e
               |                     |          
               |                     |--2.72%-- 0x8597e30300000084
               |                     |          
               |                     |--2.63%-- 0x446c74100
               |                     |          
               |                     |--2.58%-- 0x100000000
               |                     |          
               |                     |--2.56%-- 0x7265636f794b0300
               |                     |          
               |                     |--2.52%-- 0x42280e41300e1466
               |                     |          
               |                     |--2.51%-- 0x9af000009b0
               |                     |          
               |                     |--2.46%-- 0x697270273d646920
               |                     |          
               |                     |--2.35%-- 0x14ef000014f0
               |                     |          
               |                      --2.33%-- 0x50ab5bf34b2110ba
               |          
               |--0.83%-- ata_scsi_queuecmd
               |          scsi_dispatch_cmd
               |          scsi_request_fn
               |          __blk_run_queue
               |          |          
               |          |--97.70%-- queue_unplugged
               |          |          blk_flush_plug_list
               |          |          blk_finish_plug
               |          |          do_blockdev_direct_IO
               |          |          __blockdev_direct_IO
               |          |          ext4_ind_direct_IO
               |          |          ext4_direct_IO
               |          |          generic_file_aio_read
               |          |          do_sync_read
               |          |          vfs_read
               |          |          sys_pread64
               |          |          system_call_fastpath
               |          |          0x7f1e29fc6993
               |          |          0xa
               |          |          0x206200000000
               |          |          
               |           --2.30%-- cfq_insert_request
               |                     __elv_add_request
               |                     blk_flush_plug_list
               |                     blk_finish_plug
               |                     do_blockdev_direct_IO
               |                     __blockdev_direct_IO
               |                     ext4_ind_direct_IO
               |                     ext4_direct_IO
               |                     generic_file_aio_read
               |                     do_sync_read
               |                     vfs_read
               |                     sys_pread64
               |                     system_call_fastpath
               |                     0x7f1e29fc6993
               |                     0xa
               |                     0x206200000000
               |          
               |--0.75%-- poll_freewait
               |          do_sys_poll
               |          sys_poll
               |          system_call_fastpath
               |          __GI___libc_poll
               |          
               |--0.71%-- skb_queue_tail
               |          unix_stream_sendmsg
               |          |          
               |          |--83.31%-- sock_sendmsg
               |          |          __sys_sendmsg
               |          |          sys_sendmsg
               |          |          system_call_fastpath
               |          |          __libc_sendmsg
               |          |          |          
               |          |          |--88.07%-- 0x10000
               |          |          |          |          
               |          |          |          |--5.85%-- 0x0
               |          |          |          |          
               |          |          |          |--4.74%-- 0x1c000000e9
               |          |          |          |          
               |          |          |          |--3.76%-- 0x50ab5c2050c5dcfc
               |          |          |          |          
               |          |          |          |--3.56%-- 0x894c01f88328ec83
               |          |          |          |          
               |          |          |          |--3.07%-- 0x6c6966207861746e
               |          |          |          |          
               |          |          |          |--2.62%-- 0x5a6400005a63
               |          |          |          |          
               |          |          |          |--2.42%-- 0x459f0000
               |          |          |          |          
               |          |          |          |--2.37%-- 0xa4a260
               |          |          |          |          
               |          |          |          |--2.34%-- 0x4dc857ce2bd30302
               |          |          |          |          
               |          |          |          |--2.33%-- 0x73747261700a7265
               |          |          |          |          
               |          |          |          |--2.31%-- 0x656c75627275740a
               |          |          |          |          
               |          |          |          |--2.26%-- 0x1d0
               |          |          |          |          
               |          |          |          |--2.20%-- 0x293b4b5428220001
               |          |          |          |          
               |          |          |          |--2.20%-- 0x16e80000000000
               |          |          |          |          
               |          |          |          |--2.17%-- 0x2d5980c2f07c3d9b
               |          |          |          |          
               |          |          |          |--2.11%-- 0x841f0f
               |          |          |          |          
               |          |          |          |--2.08%-- 0x101010101010101
               |          |          |          |          
               |          |          |          |--2.07%-- 0xdf24bc8000212f38
               |          |          |          |          
               |          |          |          |--2.06%-- 0x7f1a8b5d1030
               |          |          |          |          
               |          |          |          |--2.06%-- 0x50000000000014c
               |          |          |          |          
               |          |          |          |--2.04%-- 0x503020b79840100
               |          |          |          |          
               |          |          |          |--1.98%-- 0x17ab2
               |          |          |          |          
               |          |          |          |--1.96%-- 0x65745f0000000874
               |          |          |          |          
               |          |          |          |--1.94%-- 0x50ab5bd64fa022eb
               |          |          |          |          
               |          |          |          |--1.94%-- 0xf000251024448b11
               |          |          |          |          
               |          |          |          |--1.93%-- 0xf0
               |          |          |          |          
               |          |          |          |--1.91%-- 0x480000076cc38130
               |          |          |          |          
               |          |          |          |--1.91%-- 0x100e410052bc8704
               |          |          |          |          
               |          |          |          |--1.89%-- 0x4b10000000
               |          |          |          |          
               |          |          |          |--1.87%-- 0x1c227370f21896d9
               |          |          |          |          
               |          |          |          |--1.84%-- 0x246c8b48d88948ff
               |          |          |          |          
               |          |          |          |--1.80%-- 0x748bef8948000000
               |          |          |          |          
               |          |          |          |--1.76%-- 0x4f0139004d013200
               |          |          |          |          
               |          |          |          |--1.75%-- 0x1b00000013
               |          |          |          |          
               |          |          |          |--1.74%-- 0x5dfbdb23d2f99a26
               |          |          |          |          
               |          |          |          |--1.74%-- 0x8b4cfffe6672e8e7
               |          |          |          |          
               |          |          |          |--1.71%-- 0xb5058b481c7d0033
               |          |          |          |          
               |          |          |          |--1.69%-- 0x50ab5bd74f61e467
               |          |          |          |          
               |          |          |          |--1.65%-- 0x36e36f5156850302
               |          |          |          |          
               |          |          |          |--1.65%-- 0x4e204f4420202021
               |          |          |          |          
               |          |          |          |--1.56%-- 0x6e614d6e6f697461
               |          |          |          |          
               |          |          |          |--1.56%-- 0xa963202401000000
               |          |          |          |          
               |          |          |          |--1.51%-- 0x48da894800002ae4
               |          |          |          |          
               |          |          |          |--1.40%-- 0x6feb938201aa0b8
               |          |          |          |          
               |          |          |          |--1.36%-- 0x63fd
               |          |          |          |          
               |          |          |           --1.35%-- 0x2073612079706d75
               |          |          |          
               |          |           --11.93%-- 0x1c
               |          |                     0x4009a82d1e7f0000
               |          |          
               |           --16.69%-- sock_aio_write
               |                     do_sync_write
               |                     vfs_write
               |                     sys_write
               |                     system_call_fastpath
               |                     0x7f1e29fc60cd
                --3.14%-- [...]

     0.73%           python  libpython2.7.so.1.0                  [.] 0x000000000008d1a5                                                                                                                                                                                                                                                              
                     |          
                     |--6.45%-- 0x306ee7d893
                     |          
                     |--2.22%-- 0x306ee7d815
                     |          |          
                     |           --100.00%-- 0x306f188280
                     |          
                     |--1.67%-- 0x306ee971e8
                     |          |          
                     |          |--95.87%-- 0x306f189760
                     |          |          
                     |           --4.13%-- 0xc741b0
                     |          
                     |--1.24%-- 0x306eeea37f
                     |          0x306f189760
                     |          
                     |--1.05%-- 0x306ee6c210
                     |          |          
                     |          |--41.69%-- 0x306f17de00
                     |          |          
                     |          |--35.16%-- 0x7f4bc864b3d0
                     |          |          0x306f17eb60
                     |          |          
                     |          |--7.80%-- 0xaef7a0
                     |          |          0xcad870
                     |          |          
                     |          |--4.34%-- 0x306f183ea0
                     |          |          |          
                     |          |          |--86.86%-- 0xc244f0
                     |          |          |          
                     |          |           --13.14%-- 0x306f1730a0
                     |          |          
                     |          |--2.41%-- 0x7f4bc865ad60
                     |          |          0x306f1730a0
                     |          |          
                     |          |--2.09%-- 0x7f4bc865a940
                     |          |          0x306f1730a0
                     |          |          
                     |          |--1.54%-- 0x7f4bc865ae10
                     |          |          0x306f1730a0
                     |          |          
                     |          |--1.45%-- 0xc69908
                     |          |          0xcae240
                     |          |          
                     |          |--1.41%-- 0x7f4bc865ad08
                     |          |          0x306f1730a0
                     |          |          
                     |          |--1.12%-- 0x7f4bc865a890
                     |          |          0x306f1730a0
                     |          |          
                     |           --0.99%-- 0x7f4bd064e7d0
                     |                     0xc244f0
                     |          
                     |--0.95%-- 0x306eee7f6e
                     |          |          
                     |          |--24.87%-- 0x7f4bc7736940
                     |          |          
                     |          |--11.93%-- 0x7f4bc6f35a40
                     |          |          
                     |          |--11.88%-- 0x7f4bc6f35b60
                     |          |          
                     |          |--10.90%-- 0x7f4bc6f35eb0
                     |          |          
                     |          |--10.37%-- 0x7f4bc7736a20
                     |          |          
                     |          |--7.78%-- 0x7f4bc6f35740
                     |          |          
                     |          |--7.50%-- 0x7f4bc7736e90
                     |          |          
                     |          |--4.34%-- 0x7fff9b3719d0
                     |          |          
                     |          |--3.86%-- 0x7fff9b372140
                     |          |          
                     |          |--2.64%-- 0x7f4bc6f35cb0
                     |          |          
                     |          |--2.15%-- 0x7f4bc7736cb0
                     |          |          
                     |           --1.76%-- 0x7f4bc7736a30
                     |          
                     |--0.94%-- 0x306ee6c26c
                     |          0x306f17de00
                     |          
                     |--0.94%-- 0x306ee7d810
                     |          |          
                     |           --100.00%-- 0x306f188280
                     |          
                     |--0.94%-- 0x306ee97190
                     |          |          
                     |          |--22.26%-- 0x306f189760
                     |          |          
                     |          |--19.16%-- 0xaef530
                     |          |          0x306f1835e0
                     |          |          
                     |          |--13.90%-- 0xaef7d0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--11.63%-- 0x306f17de00
                     |          |          
                     |          |--6.13%-- 0x8300016a00017c00
                     |          |          
                     |          |--5.23%-- 0x306f1835e0
                     |          |          
                     |          |--4.32%-- 0x8300016a00007c00
                     |          |          
                     |          |--3.18%-- 0x7f4bc8ccf060
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.21%-- 0xc741b0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.14%-- 0x7f4bb8162460
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.06%-- 0x7f4bb80e22e0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.77%-- 0x7f4bc8caf770
                     |          |          0x306f18a0a0
                     |          |          
                     |          |--1.61%-- 0xc723d8
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.54%-- 0x7f4bc9f5e560
                     |          |          
                     |          |--1.46%-- 0xc6e1f8
                     |          |          0x306f1835e0
                     |          |          
                     |           --1.41%-- 0xbb6c80
                     |                     0x306f1835e0
                     |          
                     |--0.92%-- 0x306ee81840
                     |          |          
                     |          |--30.95%-- 0x306f183ea0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--15.18%-- 0x7f4bc865b780
                     |          |          0x306f1835e0
                     |          |          
                     |          |--7.27%-- 0xbec578
                     |          |          0x306f1835e0
                     |          |          
                     |          |--5.81%-- 0xaef7d0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--4.63%-- 0x7f4bc865b680
                     |          |          0x306f1835e0
                     |          |          
                     |          |--3.91%-- 0x7f4bc8ccd050
                     |          |          0x306f1835e0
                     |          |          
                     |          |--3.84%-- 0x306f16fb20
                     |          |          
                     |          |--2.74%-- 0x7f4bc8cbd8d0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.72%-- 0x7f4bc86517d0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.18%-- 0x7f4bc0281760
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.15%-- 0x7f4bb8112370
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.10%-- 0x7f4bc8cc45d0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.03%-- 0x7f4bc865b5f0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.02%-- 0x7f4bc8653c20
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.99%-- 0x7f4bb8392af0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.96%-- 0x7f4bc8cbe050
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.74%-- 0x7f4bb80e22e0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.73%-- 0x7f4bc865adb8
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.10%-- 0x7f4bc0040230
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.05%-- 0x7f4bc865c640
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.01%-- 0x7f4bb8012070
                     |          |          0x306f1835e0
                     |          |          
                     |          |--0.64%-- 0x7f4bb81c2580
                     |          |          0x306f1835e0
                     |          |          
                     |          |--0.62%-- 0x282034312e6f732e
                     |          |          0x306f1835e0
                     |          |          
                     |           --0.60%-- 0x7f4bd064e7d0
                     |                     0x306f1835e0
                     |          
                     |--0.90%-- 0x306ee5e885
                     |          |          
                     |          |--94.87%-- 0x306f175020
                     |          |          
                     |           --5.13%-- 0x306f18a0a0
                     |          
                     |--0.89%-- 0x306eeea334
                     |          0x306f189760
                     |          
                     |--0.82%-- 0x306ee97192
                     |          |          
                     |          |--32.57%-- 0xaef530
                     |          |          0x306f1835e0
                     |          |          
                     |          |--15.73%-- 0x306f189760
                     |          |          
                     |          |--7.25%-- 0x306f17f340
                     |          |          
                     |          |--5.21%-- 0xaef7d0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--4.90%-- 0xbb6c80
                     |          |          0x306f1835e0
                     |          |          
                     |          |--4.52%-- 0x8300016a00017c00
                     |          |          
                     |          |--3.78%-- 0x8300016a00007c00
                     |          |          
                     |          |--3.33%-- 0x7f4bc9f5e560
                     |          |          
                     |          |--2.77%-- 0x306f17de00
                     |          |          
                     |          |--2.50%-- 0x7f4bc01264e0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.45%-- 0xaef7b8
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.37%-- 0xc72540
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.33%-- 0x7f4bb80c2280
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.04%-- 0xc72a68
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.99%-- 0x7f4bb8412ca0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.98%-- 0x7f4bc8caf770
                     |          |          0x306f18a0a0
                     |          |          
                     |          |--1.48%-- 0x7f4bc864e298
                     |          |          0xc5fe20
                     |          |          
                     |          |--1.44%-- 0xc69938
                     |          |          0x306f1835e0
                     |          |          
                     |           --1.38%-- 0x306f1835e0
                     |          
                     |--0.79%-- 0x306ee547d1
                     |          |          
                     |          |--38.84%-- 0x7f4bc8caf770
                     |          |          0x306f172f00
                     |          |          
                     |          |--19.69%-- 0x7f4bc971de68
                     |          |          0x306f172f00
                     |          |          
                     |          |--10.14%-- 0x306f183ea0
                     |          |          0x306f172f00
                     |          |          
                     |          |--9.63%-- 0xaef7b8
                     |          |          0x306f172f00
                     |          |          
                     |          |--4.14%-- 0x7f4bc864e420
                     |          |          0x306f172f00
                     |          |          
                     |          |--2.56%-- 0x7f4bc865a100
                     |          |          0x306f172f00
                     |          |          
                     |          |--2.31%-- 0x7f4bc865aba8
                     |          |          0x306f172f00
                     |          |          
                     |          |--2.23%-- 0x7f4bc865a4c8
                     |          |          0x306f172f00
                     |          |          
                     |          |--2.18%-- 0x7f4bc864e298
                     |          |          0x306f172f00
                     |          |          
                     |          |--1.88%-- 0xc728e8
                     |          |          0x306f172f00
                     |          |          
                     |          |--1.70%-- 0xaef7a0
                     |          |          0x306f172f00
                     |          |          
                     |          |--1.52%-- 0x7f4bc972f0b0
                     |          |          0x306f172f00
                     |          |          
                     |          |--1.12%-- 0xc698c0
                     |          |          0x306f172f00
                     |          |          
                     |          |--1.10%-- 0xc6e3a8
                     |          |          0x306f172f00
                     |          |          
                     |           --0.96%-- 0x7f4bc865a208
                     |                     0x306f172f00
                     |          
                     |--0.73%-- 0x306ee6c212
                     |          |          
                     |          |--47.69%-- 0x7f4bc864b3d0
                     |          |          0x306f17eb60
                     |          |          
                     |          |--37.20%-- 0x306f17de00
                     |          |          
                     |          |--10.14%-- 0x306f183ea0
                     |          |          0xc244f0
                     |          |          
                     |          |--2.71%-- 0xaefa10
                     |          |          0xcad870
                     |          |          
                     |          |--1.23%-- 0xaef7a0
                     |          |          0xcad870
                     |          |          
                     |           --1.03%-- 0x7f4bc8651790
                     |                     0xcad870
                     |          
                     |--0.72%-- 0x306ee6d857
                     |          |          
                     |          |--81.77%-- 0x306f17eb60
                     |          |          
                     |           --18.23%-- 0x306f1730a0
                     |          
                     |--0.68%-- 0x306ee55dae
                     |          0x306f1730a0
                     |          
                     |--0.65%-- 0x306ee6e890
                     |          |          
                     |          |--31.60%-- 0x306f17f340
                     |          |          
                     |          |--20.42%-- 0xc741b0
                     |          |          
                     |          |--14.30%-- 0x306f188280
                     |          |          
                     |          |--12.10%-- 0x306f17de00
                     |          |          
                     |          |--9.68%-- 0x306f189760
                     |          |          
                     |          |--6.30%-- 0x7d00037d00025c00
                     |          |          
                     |           --5.60%-- 0x7c00617800037d00
                     |          
                     |--0.60%-- 0x306eed3914
                     |          |          
                     |          |--42.80%-- 0x7f4bc97377c0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--19.84%-- 0x7f4bc865b780
                     |          |          0x306f1835e0
                     |          |          
                     |          |--4.12%-- 0x7f4bb8042100
                     |          |          0x306f1835e0
                     |          |          
                     |          |--3.82%-- 0xe57bf0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--3.32%-- 0x7f4bb8001e50
                     |          |          0x306f1835e0
                     |          |          
                     |          |--3.29%-- 0xd29e00
                     |          |          0x306f1835e0
                     |          |          
                     |          |--3.12%-- 0x7f4bb83f2c10
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.86%-- 0x7f4bb81d25b0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.72%-- 0x7f4bb82a2820
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.66%-- 0x7f4bb8202640
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.52%-- 0x7f4bb8212670
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.48%-- 0x7f4bb83229a0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.40%-- 0x7f4bb83625f0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.06%-- 0x7f4bb82f6800
                     |          |          0x306f1835e0
                     |          |          
                     |           --2.00%-- 0x7f4bb82e28e0
                     |                     0x306f1835e0
                     |          
                     |--0.58%-- 0x306ee547d0
                     |          |          
                     |          |--29.69%-- 0x7f4bc8caf770
                     |          |          0x306f172f00
                     |          |          
                     |          |--25.61%-- 0x7f4bc971de68
                     |          |          0x306f172f00
                     |          |          
                     |          |--13.60%-- 0xaef7b8
                     |          |          0x306f172f00
                     |          |          
                     |          |--9.53%-- 0xaef7a0
                     |          |          0x306f172f00
                     |          |          
                     |          |--5.08%-- 0x306f183ea0
                     |          |          0x306f172f00
                     |          |          
                     |          |--3.97%-- 0x7f4bc865ad08
                     |          |          0x306f172f00
                     |          |          
                     |          |--2.86%-- 0xc723c0
                     |          |          0x306f172f00
                     |          |          
                     |          |--2.76%-- 0xc697b8
                     |          |          0x306f172f00
                     |          |          
                     |          |--2.73%-- 0xbec530
                     |          |          0x306f172f00
                     |          |          
                     |          |--2.32%-- 0x7f4bc864e3e8
                     |          |          0x306f172f00
                     |          |          
                     |           --1.84%-- 0xc72930
                     |                     0x306f172f00
                     |          
                     |--0.57%-- 0x306ee55d85
                     |          0x306f188280
                     |          
                     |--0.56%-- 0x306ee55de5
                     |          0x306f1730a0
                     |          
                     |--0.56%-- 0x306ee7d97f
                     |          
                     |--0.56%-- 0x306ee70dcb
                     |          |          
                     |          |--42.51%-- 0x7c00017400017d00
                     |          |          
                     |          |--27.93%-- 0x8300016a00007c00
                     |          |          
                     |          |--15.62%-- 0x4400006a00007c00
                     |          |          
                     |           --13.93%-- 0x64000174001b7300
                     |          
                     |--0.53%-- 0x306ee8b854
                     |          |          
                     |          |--39.85%-- 0x7f4bc865b780
                     |          |          0x306f1835e0
                     |          |          
                     |          |--6.49%-- 0x306f188280
                     |          |          
                     |          |--5.43%-- 0xd173a0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--4.82%-- 0xcc90e0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--4.47%-- 0xd09110
                     |          |          0x306f1835e0
                     |          |          
                     |          |--4.45%-- 0x7f4bb8152430
                     |          |          0x306f1835e0
                     |          |          
                     |          |--3.59%-- 0x7f4bb8012070
                     |          |          0x306f1835e0
                     |          |          
                     |          |--3.49%-- 0x7f4bb82827c0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--3.04%-- 0x7f4bb8412c70
                     |          |          0x306f1835e0
                     |          |          
                     |          |--3.02%-- 0x7f4bb82e28e0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--3.01%-- 0x7f4bb82326d0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.74%-- 0x7f4bb8352a30
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.59%-- 0x7f4bb8342a00
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.37%-- 0x7f4bb81f2610
                     |          |          0x306f1835e0
                     |          |          
                     |          |--2.24%-- 0x7f4bb8212670
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.97%-- 0x7f4bb81a2520
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.47%-- 0x7f4bb8102340
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.08%-- 0x7f4bb8312970
                     |          |          0x306f1835e0
                     |          |          
                     |          |--1.01%-- 0x7f4bb8162460
                     |          |          0x306f1835e0
                     |          |          
                     |          |--0.99%-- 0x7f4bb80220a0
                     |          |          0x306f1835e0
                     |          |          
                     |          |--0.99%-- 0x7f4bb8202640
                     |          |          0x306f1835e0
                     |          |          
                     |           --0.89%-- 0x7f4bb81223a0
                     |                     0x306f1835e0
                     |          
                     |--0.52%-- 0x306ee6c2d5
                     |          0x306f17de00
                     |          
                     |--0.51%-- 0x306ee6c387
                     |          0x306f17de00
                      --71.51%-- [...]

     0.70%      kworker/0:0  [dm_crypt]                           [k] crypt_convert                                                                                                                                                                                                                                                                   
                |
                --- crypt_convert
                   |          
                   |--99.31%-- kcryptd_crypt
                   |          process_one_work
                   |          worker_thread
                   |          kthread
                   |          ret_from_fork
                   |          
                    --0.69%-- process_one_work
                              worker_thread
                              kthread
                              ret_from_fork

     0.59%           python  libc-2.16.so                         [.] __memcpy_ssse3_back                                                                                                                                                                                                                                                             
                     |
                     --- __memcpy_ssse3_back
                        |          
                        |--0.21%-- 0x306f181280
                        |          
                        |--0.09%-- 0x0
                        |          0x10
                        |          
                        |--0.07%-- 0x300000001
                        |          
                        |--0.03%-- 0x7f4bc863c500
                        |          
                        |--0.03%-- 0x400000001
                         --99.57%-- [...]

     0.57%  qemu-system-x86  [kernel.kallsyms]                    [k] copy_user_enhanced_fast_string                                                                                                                                                                                                                                                  
            |
            --- copy_user_enhanced_fast_string
               |          
               |--68.83%-- unix_stream_sendmsg
               |          |          
               |          |--63.11%-- sock_sendmsg
               |          |          __sys_sendmsg
               |          |          sys_sendmsg
               |          |          system_call_fastpath
               |          |          __libc_sendmsg
               |          |          |          
               |          |          |--99.80%-- 0x10000
               |          |          |          |          
               |          |          |          |--9.24%-- 0x0
               |          |          |          |          
               |          |          |          |--0.66%-- 0x8
               |          |          |          |          
               |          |          |          |--0.65%-- 0x31223d6e6f697372
               |          |          |          |          
               |          |          |          |--0.60%-- 0x63
               |          |          |           --88.85%-- [...]
               |          |           --0.20%-- [...]
               |          |          
               |           --36.89%-- sock_aio_write
               |                     do_sync_write
               |                     vfs_write
               |                     sys_write
               |                     system_call_fastpath
               |                     0x7f1e29fc60cd
               |          
               |--12.58%-- vfs_write
               |          sys_write
               |          system_call_fastpath
               |          0x7f1e29fc60cd
               |          |          
               |          |--97.60%-- 0x0
               |           --2.40%-- [...]
               |          
               |--8.29%-- sys_poll
               |          system_call_fastpath
               |          __GI___libc_poll
               |          
               |--4.46%-- system_call_fastpath
               |          |          
               |          |--36.20%-- timer_gettime@@GLIBC_2.3.3
               |          |          0x0
               |          |          
               |          |--25.62%-- timer_settime@@GLIBC_2.3.3
               |          |          0x0
               |          |          
               |          |--22.07%-- __libc_sigaction
               |          |          
               |           --16.11%-- sem_timedwait
               |          
               |--2.33%-- generic_file_aio_read
               |          do_sync_read
               |          vfs_read
               |          sys_pread64
               |          system_call_fastpath
               |          0x7f1e29fc6993
               |          0xb
               |          0x4000000001
               |          
               |--0.74%-- kvm_arch_vcpu_ioctl_run
               |          kvm_vcpu_ioctl
               |          do_vfs_ioctl
               |          sys_ioctl
               |          system_call_fastpath
               |          __GI___ioctl
               |          |          
               |          |--59.92%-- 0x100000002
               |          |          
               |          |--23.46%-- 0x10100000006
               |          |          
               |          |--12.93%-- 0x100000006
               |          |          
               |           --3.69%-- 0x10100000002
               |          
               |--0.68%-- futex_wait_setup
               |          futex_wait
               |          do_futex
               |          sys_futex
               |          system_call_fastpath
               |          sem_timedwait
               |          
               |--0.53%-- kvm_read_guest
               |          |          
               |          |--88.57%-- kvm_read_guest_virt_helper
               |          |          kvm_fetch_guest_virt
               |          |          do_insn_fetch
               |          |          x86_decode_insn
               |          |          x86_emulate_instruction
               |          |          |          
               |          |          |--50.47%-- handle_apic_access
               |          |          |          vmx_handle_exit
               |          |          |          kvm_arch_vcpu_ioctl_run
               |          |          |          kvm_vcpu_ioctl
               |          |          |          do_vfs_ioctl
               |          |          |          sys_ioctl
               |          |          |          system_call_fastpath
               |          |          |          __GI___ioctl
               |          |          |          |          
               |          |          |          |--60.21%-- 0x10100000006
               |          |          |          |          
               |          |          |           --39.79%-- 0x100000002
               |          |          |          
               |          |          |--46.34%-- handle_io
               |          |          |          vmx_handle_exit
               |          |          |          kvm_arch_vcpu_ioctl_run
               |          |          |          kvm_vcpu_ioctl
               |          |          |          do_vfs_ioctl
               |          |          |          sys_ioctl
               |          |          |          system_call_fastpath
               |          |          |          __GI___ioctl
               |          |          |          0x100000002
               |          |          |          
               |          |           --3.19%-- handle_ept_misconfig
               |          |                     vmx_handle_exit
               |          |                     kvm_arch_vcpu_ioctl_run
               |          |                     kvm_vcpu_ioctl
               |          |                     do_vfs_ioctl
               |          |                     sys_ioctl
               |          |                     system_call_fastpath
               |          |                     __GI___ioctl
               |          |                     0x100000006
               |          |          
               |           --11.43%-- read_emulate
               |                     emulator_read_write_onepage
               |                     emulator_read_write
               |                     emulator_read_emulated
               |                     segmented_read.isra.29
               |                     x86_emulate_insn
               |                     x86_emulate_instruction
               |                     handle_io
               |                     vmx_handle_exit
               |                     kvm_arch_vcpu_ioctl_run
               |                     kvm_vcpu_ioctl
               |                     do_vfs_ioctl
               |                     sys_ioctl
               |                     system_call_fastpath
               |                     __GI___ioctl
               |                     0x100000002
                --1.56%-- [...]

     0.56%  qemu-system-x86  [kernel.kallsyms]                    [k] do_blockdev_direct_IO                                                                                                                                                                                                                                                           
            |
            --- do_blockdev_direct_IO
               |          
               |--99.85%-- __blockdev_direct_IO
               |          ext4_ind_direct_IO
               |          ext4_direct_IO
               |          generic_file_aio_read
               |          do_sync_read
               |          vfs_read
               |          sys_pread64
               |          system_call_fastpath
               |          0x7f1e29fc6993
               |          0xa
               |          0x206200000000
                --0.15%-- [...]

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Stefan Hajnoczi]

On Sun, Mar 10, 2013 at 11:50 AM, Dietmar Maurer <dietmar@proxmox.com> wrote:
>> About 4KB zero tracking:
>>
>> The vma.py module does not check for zeros, the mask is always 0xffff.
>>  There is a pathalogical case of a disk with every 2nd 4 KB block zeroed, here
>> vma.py would create a fully-allocated file.But the more common case is that the
>> whole 64 KB cluster is zero - think about the fact that qcow2/qed only do discard
>> (zero clusters) at cluster level
>> (64 KB).
>>
>> Also, if you try hard to zero 4 KB blocks then two things can happen:
>> 1. You fragment the file, saving space now but turning sequential access into
>> random access once those zero regions get filled in.
>> 2. The underlying file system ignores the holes to reduce fragmentation and all
>> the effort was wasted - I guess this can be checked with ioctl(FIEMAP/FIBMAP)
>> to see how ext4/xfs/btrfs handle it.
>
> We track zero blocks at 4KB level to get small backup files. Restore can handle those
> blocks differently, either pre-allocate or create holes. That can even be a restore option.

You are right, the behavior can be set at restore time.

I'm curious how much of a win the 4 KB zero detection is on random
Linux or Windows guest images.  Have you collected numbers?

Stefan

Re: [Qemu-devel] [RFC 3/8] backup: write to BlockDriverState instead of BackupDumpFunc

[Stefan Hajnoczi]

On Sun, Mar 10, 2013 at 11:05 AM, Dietmar Maurer <dietmar@proxmox.com> wrote:
>> Remove the BackupDumpFunc function pointer and write directly to a
>> BlockDriverState.
>
> My callback approach is the generic one. You can easily implement the
> BlockDriverState approach using the BackupDumpFunc?

Yes it can be implemented on top of BackupDumpFunc.  But the approach
I'm showing is that backup archive formats should not be implemented
inside QEMU so the BackupDumpFunc is not needed.

Stefan

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Dietmar Maurer]

> > We track zero blocks at 4KB level to get small backup files. Restore
> > can handle those blocks differently, either pre-allocate or create holes. That
> can even be a restore option.
> 
> You are right, the behavior can be set at restore time.
> 
> I'm curious how much of a win the 4 KB zero detection is on random Linux or
> Windows guest images.  Have you collected numbers?

Well, seems that it does not help much - it saves about 0.2% space. So maybe we
can remove that feature (and increase address space in the VMA format instead).

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Dietmar Maurer]

> > > We track zero blocks at 4KB level to get small backup files. Restore
> > > can handle those blocks differently, either pre-allocate or create
> > > holes. That
> > can even be a restore option.
> >
> > You are right, the behavior can be set at restore time.
> >
> > I'm curious how much of a win the 4 KB zero detection is on random
> > Linux or Windows guest images.  Have you collected numbers?
> 
> Well, seems that it does not help much - it saves about 0.2% space. So maybe
> we can remove that feature (and increase address space in the VMA format
> instead).

Sorry, I need to correct myself - I had a bug in the code. 

I can see space reduction up to 4% using this feature. Considering the fact that it
comes at no cost, it would be stupid to remove it.

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Stefan Hajnoczi]

On Mon, Mar 11, 2013 at 09:26:26AM +0000, Dietmar Maurer wrote:
> > > > We track zero blocks at 4KB level to get small backup files. Restore
> > > > can handle those blocks differently, either pre-allocate or create
> > > > holes. That
> > > can even be a restore option.
> > >
> > > You are right, the behavior can be set at restore time.
> > >
> > > I'm curious how much of a win the 4 KB zero detection is on random
> > > Linux or Windows guest images.  Have you collected numbers?
> > 
> > Well, seems that it does not help much - it saves about 0.2% space. So maybe
> > we can remove that feature (and increase address space in the VMA format
> > instead).
> 
> Sorry, I need to correct myself - I had a bug in the code. 
> 
> I can see space reduction up to 4% using this feature. Considering the fact that it
> comes at no cost, it would be stupid to remove it.

Okay, looks like it's useful but not a huge win.

In the NBD approach pipelining writes (or discards) ought to make 4 KB
zero blocks usable without overhead.

Stefan

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Dietmar Maurer]

> > I can see space reduction up to 4% using this feature. Considering the
> > fact that it comes at no cost, it would be stupid to remove it.
> 
> Okay, looks like it's useful but not a huge win.
> 
> In the NBD approach pipelining writes (or discards) ought to make 4 KB zero
> blocks usable without overhead.

But you need to re-asemble 64KB block from that - I guess that is the difficult part here?

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Stefan Hajnoczi]

On Mon, Mar 11, 2013 at 03:00:38PM +0000, Dietmar Maurer wrote:
> > > I can see space reduction up to 4% using this feature. Considering the
> > > fact that it comes at no cost, it would be stupid to remove it.
> > 
> > Okay, looks like it's useful but not a huge win.
> > 
> > In the NBD approach pipelining writes (or discards) ought to make 4 KB zero
> > blocks usable without overhead.
> 
> But you need to re-asemble 64KB block from that - I guess that is the difficult part here?

That's not hard.  It can be added to the RFC code I posted.  Here is
an outline.

Say there is a 64 KB block with 2 zero regions:

0123456789abcdef
XXXXzzXXXXXzXXXX

X - populated 4 KB block
z - zero 4 KB block

Send 3 writes:
NBD_CMD_WRITE from=0 length=16 KB
...
NBD_CMD_WRITE from=24 KB length=20 KB
...
NBD_CMD_WRITE from=48 KB length=16 KB
...

The NBD server can pack this back into an extent's blockinfo:

def add_cluster(cluster, blockinfo, blocks):
    ...the code to add a cluster to the current extent...

def prepare_for_blockinfo_update(cluster):
    # We're still filling in the same cluster, all is fine
    if cur_cluster == cluster:
        return

    # Flush out last cluster, which is now complete
    add_cluster(cur_cluster, cur_blockinfo, cur_blocks)

    # Prepare to work on a new cluster
    cur_cluster = cluster
    cur_blockinfo = dev_id << 32 | cluster
    cur_blocks = []

def write(from, size, data):
    cluster = from / VMA_CLUSTER_SIZE
    prepare_for_blockinfo_update(cluster)

    cur_blockinfo |= build_blockinfo_mask(from, size) # bit manipulation
    cur_blocks.append(data)

In other words, the NBD server assembles the writes into a blockinfo and
list of blocks.

Stefan

Re: [Qemu-devel] [RFC 1/8] block: add virtual_size to query-block QMP output

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2351 bytes --]

On 03/09/2013 03:22 PM, Stefan Hajnoczi wrote:
> There is currently no way to query the size of a drive.  Add a
> 'virtual_size' field to the 'query-block' QMP output.
> 
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  block.c          | 1 +
>  qapi-schema.json | 5 ++++-
>  2 files changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/block.c b/block.c
> index 124a9eb..0128e27 100644
> --- a/block.c
> +++ b/block.c
> @@ -2908,6 +2908,7 @@ BlockInfo *bdrv_query_info(BlockDriverState *bs)
>          info->has_inserted = true;
>          info->inserted = g_malloc0(sizeof(*info->inserted));
>          info->inserted->file = g_strdup(bs->filename);
> +        info->inserted->virtual_size = bdrv_getlength(bs);
>          info->inserted->ro = bs->read_only;
>          info->inserted->drv = g_strdup(bs->drv->format_name);
>          info->inserted->encrypted = bs->encrypted;
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 28b070f..6b64aec 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -751,6 +751,8 @@
>  #
>  # @iops_wr: write I/O operations per second is specified
>  #
> +# @virtual_size: size of block device, in bytes

It is traditional to list 'since 1.5' when adding a field that did not
appear in earlier qemu releases.

> +#
>  # Since: 0.14.0
>  #
>  # Notes: This interface is only found in @BlockInfo.
> @@ -760,7 +762,8 @@
>              '*backing_file': 'str', 'backing_file_depth': 'int',
>              'encrypted': 'bool', 'encryption_key_missing': 'bool',
>              'bps': 'int', 'bps_rd': 'int', 'bps_wr': 'int',
> -            'iops': 'int', 'iops_rd': 'int', 'iops_wr': 'int'} }
> +            'iops': 'int', 'iops_rd': 'int', 'iops_wr': 'int',
> +            'virtual_size': 'int' } }

I was about to suggest 'virtual-size' instead of 'virtual_size', since
we prefer '-' over '_' in QMP; but since this command has pre-existing
uses of '_', keeping consistency within the struct trumps consistency
with the rest of QMP.

The idea makes sense, and my only complaint was a trivial documentation
matter; so feel free to add this when re-posting without the RFC:

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Kevin Wolf]

Am 10.03.2013 um 10:57 hat Dietmar Maurer geschrieben:
> > The difference between this approach and Dietmar's series is that the backup
> > archive format is implemented outside QEMU and runs as a separate program.
> > 
> > This way, management tools like proxmox, oVirt, OpenStack, and others can
> > provide their preferred backup archive formats without modifying QEMU.  
> 
> So you propose that everyone should use another backup format - this is a bad thing because
> it lead to interoperability problems (vendor lock-in?)

Maybe we could consider having the backup tool inside the qemu.git tree
and thus provide a common format that management applications can choose
to use, but still have it implemented outside the qemu process running
the VM.

I'm still not totally against having the VMA writer even inside the qemu
process, but only as an option. The important part for me is being able
to backup into _any_ BlockDriverState.

Kevin

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Stefan Hajnoczi]

On Tue, Mar 12, 2013 at 10:18:12AM +0100, Kevin Wolf wrote:
> Am 10.03.2013 um 10:57 hat Dietmar Maurer geschrieben:
> > > The difference between this approach and Dietmar's series is that the backup
> > > archive format is implemented outside QEMU and runs as a separate program.
> > > 
> > > This way, management tools like proxmox, oVirt, OpenStack, and others can
> > > provide their preferred backup archive formats without modifying QEMU.  
> > 
> > So you propose that everyone should use another backup format - this is a bad thing because
> > it lead to interoperability problems (vendor lock-in?)
> 
> Maybe we could consider having the backup tool inside the qemu.git tree
> and thus provide a common format that management applications can choose
> to use, but still have it implemented outside the qemu process running
> the VM.
> 
> I'm still not totally against having the VMA writer even inside the qemu
> process, but only as an option. The important part for me is being able
> to backup into _any_ BlockDriverState.

If VMA was the common backup archive format I'd be more enthusiastic
about having a writer in QEMU and letting that be the "interface" for
backups.

Writing into BlockDriverState is important is because we have the
opportunity here to provide an interface that can be used in a number of
ways beyond just writing backup archives.

Either VMA needs to have the prerequisites[1] for becoming a common
backup archive format or else we'll just be adding a new zoo of obscure
formats to QEMU.  Using the BlockDriverState interface instead is a
clear step to prevent proliferation of formats inside QEMU.

Since I'm not convinced that VMA is the going to get traction, I'd
rather we focussed on a clean interface that allowed any backup archive
format.

It's a slippery slope to put VMA into QEMU.  What will you say when
patches for OVF or some other format turn up?

[1] Prerequisites for becoming widely adopted:
 * Open specification
 * Review of VMA limitations, extensibility, and missing features
 * Packaged VMA read/write tools
 * Look into which formats existing backup solutions support (maybe
   there is already something universal, I certainly don't know)

Stefan

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Dietmar Maurer]

> It's a slippery slope to put VMA into QEMU.  What will you say when patches for
> OVF or some other format turn up?

I agree, we should concentrate to have the basic framework. Time will show what archive 
format people prefer.

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Kevin Wolf]

Am 12.03.2013 um 11:50 hat Stefan Hajnoczi geschrieben:
> On Tue, Mar 12, 2013 at 10:18:12AM +0100, Kevin Wolf wrote:
> > Maybe we could consider having the backup tool inside the qemu.git tree
> > and thus provide a common format that management applications can choose
> > to use, but still have it implemented outside the qemu process running
> > the VM.
> > 
> > I'm still not totally against having the VMA writer even inside the qemu
> > process, but only as an option. The important part for me is being able
> > to backup into _any_ BlockDriverState.
> 
> If VMA was the common backup archive format I'd be more enthusiastic
> about having a writer in QEMU and letting that be the "interface" for
> backups.
> 
> Writing into BlockDriverState is important is because we have the
> opportunity here to provide an interface that can be used in a number of
> ways beyond just writing backup archives.

Yes, that's why I would want to have VMA implemented as a write-only
BlockDriver. The part about saving multiple images into one VMA is a bit
tricky; it's basically the same thing as exposing several snapshots of a
qcow2 image as separate BlockDriverStates at the same time.

> Either VMA needs to have the prerequisites[1] for becoming a common
> backup archive format or else we'll just be adding a new zoo of obscure
> formats to QEMU.  Using the BlockDriverState interface instead is a
> clear step to prevent proliferation of formats inside QEMU.
> 
> Since I'm not convinced that VMA is the going to get traction, I'd
> rather we focussed on a clean interface that allowed any backup archive
> format.

We don't have a native streamable image format, so I think this is a
legitimate topic for discussion. I'm not sure if the latest proposal for
VMA is what I'd like to make qemu's native streamable format, though,
and this is something that needs to be discussed. Some comments in the
email threads made me think that it can be improved.

> It's a slippery slope to put VMA into QEMU.  What will you say when
> patches for OVF or some other format turn up?

OVF isn't an image format. I would really like to produce OVA archives,
with a OVF description passed by the management tool and images in one
of our native formats. If this isn't streamable though (as I expect),
tbat would be more for storing an archive of qcow2s instead of VMAs.

Kevin

> [1] Prerequisites for becoming widely adopted:
>  * Open specification
>  * Review of VMA limitations, extensibility, and missing features
>  * Packaged VMA read/write tools
>  * Look into which formats existing backup solutions support (maybe
>    there is already something universal, I certainly don't know)
> 
> Stefan

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Dietmar Maurer]

> OVF isn't an image format. I would really like to produce OVA archives, with a
> OVF description passed by the management tool and images in one of our native
> formats. If this isn't streamable though (as I expect), tbat would be more for
> storing an archive of qcow2s instead of VMAs.

OVF does not even specify the disk image format. They simply use tar to archive
disk images with unspecified format.

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Dietmar Maurer]

> OVF isn't an image format. I would really like to produce OVA archives, with a
> OVF description passed by the management tool 

Another interesting part is how you translate a libvirt configuration into ovf description, and back - without losing any information?

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Stefan Hajnoczi]

On Tue, Mar 12, 2013 at 12:22 PM, Kevin Wolf <kwolf@redhat.com> wrote:
> Am 12.03.2013 um 11:50 hat Stefan Hajnoczi geschrieben:
>> On Tue, Mar 12, 2013 at 10:18:12AM +0100, Kevin Wolf wrote:
>> > Maybe we could consider having the backup tool inside the qemu.git tree
>> > and thus provide a common format that management applications can choose
>> > to use, but still have it implemented outside the qemu process running
>> > the VM.
>> >
>> > I'm still not totally against having the VMA writer even inside the qemu
>> > process, but only as an option. The important part for me is being able
>> > to backup into _any_ BlockDriverState.
>>
>> If VMA was the common backup archive format I'd be more enthusiastic
>> about having a writer in QEMU and letting that be the "interface" for
>> backups.
>>
>> Writing into BlockDriverState is important is because we have the
>> opportunity here to provide an interface that can be used in a number of
>> ways beyond just writing backup archives.
>
> Yes, that's why I would want to have VMA implemented as a write-only
> BlockDriver. The part about saving multiple images into one VMA is a bit
> tricky; it's basically the same thing as exposing several snapshots of a
> qcow2 image as separate BlockDriverStates at the same time.
>
>> Either VMA needs to have the prerequisites[1] for becoming a common
>> backup archive format or else we'll just be adding a new zoo of obscure
>> formats to QEMU.  Using the BlockDriverState interface instead is a
>> clear step to prevent proliferation of formats inside QEMU.
>>
>> Since I'm not convinced that VMA is the going to get traction, I'd
>> rather we focussed on a clean interface that allowed any backup archive
>> format.
>
> We don't have a native streamable image format, so I think this is a
> legitimate topic for discussion. I'm not sure if the latest proposal for
> VMA is what I'd like to make qemu's native streamable format, though,
> and this is something that needs to be discussed. Some comments in the
> email threads made me think that it can be improved.
>
>> It's a slippery slope to put VMA into QEMU.  What will you say when
>> patches for OVF or some other format turn up?
>
> OVF isn't an image format. I would really like to produce OVA archives,
> with a OVF description passed by the management tool and images in one
> of our native formats. If this isn't streamable though (as I expect),
> tbat would be more for storing an archive of qcow2s instead of VMAs.

Your reply sounds like there are two separate issues here:

1. A streamable image format native to QEMU would be nice.  The killer
feature would be streamable qcow2 with the ability to run the image
and perform normal writes to it so that no conversion is necessary
when restoring the image.

I support such a feature being in QEMU.

2. Archive formats (zip/tar/OVA/VMA).  Here I'm against putting them
into QEMU since the management tool needs to put them together anyway.

Stefan

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Stefan Hajnoczi]

On Tue, Mar 12, 2013 at 12:15 PM, Dietmar Maurer <dietmar@proxmox.com> wrote:
>> It's a slippery slope to put VMA into QEMU.  What will you say when patches for
>> OVF or some other format turn up?
>
> I agree, we should concentrate to have the basic framework. Time will show what archive
> format people prefer.

I'm happy with block-backup being in QEMU for now and if things
gravitate to one common archive format in the future, then it could be
added to eliminate IPC.

Stefan

Re: [Qemu-devel] [RFC 4/8] block: add block_backup QMP command

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]

On 03/09/2013 03:22 PM, Stefan Hajnoczi wrote:
> @block-backup
> 
> Start a point-in-time copy of a block device to a new destination.
> 

Is a BLOCK_JOB_COMPLETED event emitted when the copy is completed?  If
not, it should be.

> +# Since 1.5
> +##
> +{ 'command': 'block-backup',
> +  'data': { 'device': 'str', 'target': 'str', '*format': 'str',
> +            '*mode': 'NewImageMode', '*speed': 'int' } }

This creates a new job type - do you need to update other locations
(such as BlockJobInfo's @type, and in QMP/qmp-events.txt under
BLOCK_JOB_CANCELLED/BLOCK_JOB_COMPLETED @type) to call out what string
is used for the new job type?  For that matter, is it finally time to
introduce a new enum type for all valid block job types, and use that
enum type instead of 'str' anywhere QMP data structures distinguish
based on job type?

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]

Re: [Qemu-devel] [RFC 4/8] block: add block_backup QMP command

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1452 bytes --]

On 03/09/2013 03:22 PM, Stefan Hajnoczi wrote:
> @block-backup
> 
> Start a point-in-time copy of a block device to a new destination.
> 

I'm trying to figure out how this is different from drive-mirror.  If I
understand correctly:

After starting drive-mirror, a write to the block device is also written
to the mirror, so that the destination sees the new data

After starting block-backup, a write to the block device flushes the old
data to the destination, so that the destination sees the old data

Timing-wise, I can accomplish a backup through either command, with the
following differences:

With drive-mirror, I start a job, wait for it to hit sync'd state, then
cancel the job. The copy is tied to the point where I cancel, and the
moment I cancel, I no longer have to worry about keeping the destination
writable (that is, the bulk of the copying is done prior to the point in
time).

With block-backup, I start a job, then wait for it to complete.  The
copy is tied to the point where I started the job, but as that may take
some time, I have to keep the destination writable until the job
completes (that is, the bulk of the work is done after the point in time).

The concept is indeed useful; more so if we can wire this into
'transaction' to capture multiple disks at the same point in time.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]

Re: [Qemu-devel] [RFC 4/8] block: add block_backup QMP command

[Stefan Hajnoczi]

On Thu, Mar 14, 2013 at 03:52:16PM -0600, Eric Blake wrote:
> On 03/09/2013 03:22 PM, Stefan Hajnoczi wrote:
> > @block-backup
> > 
> > Start a point-in-time copy of a block device to a new destination.
> > 
> 
> I'm trying to figure out how this is different from drive-mirror.  If I
> understand correctly:
> 
> After starting drive-mirror, a write to the block device is also written
> to the mirror, so that the destination sees the new data
> 
> After starting block-backup, a write to the block device flushes the old
> data to the destination, so that the destination sees the old data
> 
> Timing-wise, I can accomplish a backup through either command, with the
> following differences:
> 
> With drive-mirror, I start a job, wait for it to hit sync'd state, then
> cancel the job. The copy is tied to the point where I cancel, and the
> moment I cancel, I no longer have to worry about keeping the destination
> writable (that is, the bulk of the copying is done prior to the point in
> time).
> 
> With block-backup, I start a job, then wait for it to complete.  The
> copy is tied to the point where I started the job, but as that may take
> some time, I have to keep the destination writable until the job
> completes (that is, the bulk of the work is done after the point in time).
> 
> The concept is indeed useful; more so if we can wire this into
> 'transaction' to capture multiple disks at the same point in time.

Although drive-mirror can be used in a similar way, the problem is that
you have to reach sync state before you can make the "point-in-time"
copy.  In other words, you need to plan ahead.

With block-backup the point-in-time copy is made the instant you issue
the command - this is much more practical.  Nevertheless, at one point I
think I asked Paolo if the block-backup code could be a sub-mode of
drive-mirror.  It seems that keeping them as separate block jobs is
reasonable since they work differently.

Regarding 'transaction', I agree.  It is handy to support 'block-backup'
as a 'transaction' action.  It can simplify QMP client error handling
since it is not necessary to manually cancel block jobs that have been
started when the last one fails to start.  Another consideration is that
'transaction' eliminates QMP round-trips and can therefore reduce guest
downtime when 'block-backup' is used after 'migrate' (guest is paused).

Stefan

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[zhangleiqiang]

Hi, Stefan, Dietmar,

  Is there already a plan to support the differential or incremental backups in this Live backup feature?  

----------
leiqzhang

Best regards

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Stefan Hajnoczi]

On Tue, Apr 09, 2013 at 07:00:23PM +0800, zhangleiqiang@gmail.com wrote:
>   Is there already a plan to support the differential or incremental backups in this Live backup feature?  

No, it has been mentioned but no design or patches have been proposed.

It seems like a persistent dirty bitmap would need to be implemented.
Today QEMU has code for dirty bitmaps but they are in-memory and not
persistent.

Stefan

Re: [Qemu-devel] [RFC 0/8] block: Live backup prototype

[Dietmar Maurer]

> >   Is there already a plan to support the differential or incremental backups in
> this Live backup feature?
> 
> No, it has been mentioned but no design or patches have been proposed.
> 
> It seems like a persistent dirty bitmap would need to be implemented.
> Today QEMU has code for dirty bitmaps but they are in-memory and not
> persistent.

Another possibility is to use some kind of content addressable storage. That way
we can have full backups with space consumption of incremental backups.

Re: [Qemu-devel] [RFC 4/8] block: add block_backup QMP command

[Paolo Bonzini]

Il 09/03/2013 23:22, Stefan Hajnoczi ha scritto:
> @block-backup
> 
> Start a point-in-time copy of a block device to a new destination.
> 
> @device:  the name of the device whose writes should be mirrored.
> 
> @target: the target of the new image. If the file exists, or if it
>          is a device, the existing file/device will be used as the new
>          destination.  If it does not exist, a new file will be created.
> 
> @format: #optional the format of the new destination, default is to
>          probe if @mode is 'existing', else the format of the source
> 
> @mode: #optional whether and how QEMU should create a new image, default is
>        'absolute-paths'.
> 
> @speed:  #optional the maximum speed, in bytes per second
> 
> Returns: nothing on success
>          If @device is not a valid block device, DeviceNotFound

Two features that would be nice to have:

1) a sync mode (full/top/none) like drive-mirror;

2) transaction support to start multiple backups live without stopping
the VM (for use cases that do not include migration)

Thanks,

Paolo

> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  blockdev.c       | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  qapi-schema.json | 28 +++++++++++++++++
>  qmp-commands.hx  |  6 ++++
>  3 files changed, 126 insertions(+)
> 
> diff --git a/blockdev.c b/blockdev.c
> index 0e67d06..95a72de 100644
> --- a/blockdev.c
> +++ b/blockdev.c
> @@ -1208,6 +1208,98 @@ void qmp_block_commit(const char *device,
>      drive_get_ref(drive_get_by_blockdev(bs));
>  }
>  
> +void qmp_block_backup(const char *device, const char *target,
> +                      bool has_format, const char *format,
> +                      bool has_mode, enum NewImageMode mode,
> +                      bool has_speed, int64_t speed,
> +                      Error **errp)
> +{
> +    BlockDriverState *bs;
> +    BlockDriverState *target_bs;
> +    BlockDriver *proto_drv;
> +    BlockDriver *drv = NULL;
> +    Error *local_err = NULL;
> +    int flags;
> +    uint64_t size;
> +    int ret;
> +
> +    if (!has_speed) {
> +        speed = 0;
> +    }
> +    if (!has_mode) {
> +        mode = NEW_IMAGE_MODE_ABSOLUTE_PATHS;
> +    }
> +
> +    bs = bdrv_find(device);
> +    if (!bs) {
> +        error_set(errp, QERR_DEVICE_NOT_FOUND, device);
> +        return;
> +    }
> +
> +    if (!bdrv_is_inserted(bs)) {
> +        error_set(errp, QERR_DEVICE_HAS_NO_MEDIUM, device);
> +        return;
> +    }
> +
> +    if (!has_format) {
> +        format = mode == NEW_IMAGE_MODE_EXISTING ? NULL : bs->drv->format_name;
> +    }
> +    if (format) {
> +        drv = bdrv_find_format(format);
> +        if (!drv) {
> +            error_set(errp, QERR_INVALID_BLOCK_FORMAT, format);
> +            return;
> +        }
> +    }
> +
> +    if (bdrv_in_use(bs)) {
> +        error_set(errp, QERR_DEVICE_IN_USE, device);
> +        return;
> +    }
> +
> +    flags = bs->open_flags | BDRV_O_RDWR;
> +
> +    proto_drv = bdrv_find_protocol(target);
> +    if (!proto_drv) {
> +        error_set(errp, QERR_INVALID_BLOCK_FORMAT, format);
> +        return;
> +    }
> +
> +    bdrv_get_geometry(bs, &size);
> +    size *= 512;
> +    if (mode != NEW_IMAGE_MODE_EXISTING) {
> +        assert(format && drv);
> +        bdrv_img_create(target, format,
> +                        NULL, NULL, NULL, size, flags, &local_err, false);
> +    }
> +
> +    if (error_is_set(&local_err)) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    target_bs = bdrv_new("");
> +    ret = bdrv_open(target_bs, target, flags, drv);
> +
> +    if (ret < 0) {
> +        bdrv_delete(target_bs);
> +        error_set(errp, QERR_OPEN_FILE_FAILED, target);
> +        return;
> +    }
> +
> +    backup_start(bs, target_bs, speed, block_job_cb, bs, &local_err);
> +    if (local_err != NULL) {
> +        bdrv_delete(target_bs);
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
> +    /* Grab a reference so hotplug does not delete the BlockDriverState from
> +     * underneath us.
> +     */
> +    drive_get_ref(drive_get_by_blockdev(bs));
> +}
> +
>  #define DEFAULT_MIRROR_BUF_SIZE   (10 << 20)
>  
>  void qmp_drive_mirror(const char *device, const char *target,
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 6b64aec..1dbf7b5 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -1716,6 +1716,34 @@
>              '*speed': 'int' } }
>  
>  ##
> +# @block-backup
> +#
> +# Start a point-in-time copy of a block device to a new destination.
> +#
> +# @device:  the name of the device whose writes should be mirrored.
> +#
> +# @target: the target of the new image. If the file exists, or if it
> +#          is a device, the existing file/device will be used as the new
> +#          destination.  If it does not exist, a new file will be created.
> +#
> +# @format: #optional the format of the new destination, default is to
> +#          probe if @mode is 'existing', else the format of the source
> +#
> +# @mode: #optional whether and how QEMU should create a new image, default is
> +#        'absolute-paths'.
> +#
> +# @speed:  #optional the maximum speed, in bytes per second
> +#
> +# Returns: nothing on success
> +#          If @device is not a valid block device, DeviceNotFound
> +#
> +# Since 1.5
> +##
> +{ 'command': 'block-backup',
> +  'data': { 'device': 'str', 'target': 'str', '*format': 'str',
> +            '*mode': 'NewImageMode', '*speed': 'int' } }
> +
> +##
>  # @drive-mirror
>  #
>  # Start mirroring a block device's writes to a new destination.
> diff --git a/qmp-commands.hx b/qmp-commands.hx
> index 95022e2..bda8eb4 100644
> --- a/qmp-commands.hx
> +++ b/qmp-commands.hx
> @@ -889,6 +889,12 @@ EQMP
>      },
>  
>      {
> +        .name       = "block-backup",
> +        .args_type  = "device:B,target:s,speed:i?,mode:s?,format:s?",
> +        .mhandler.cmd_new = qmp_marshal_input_block_backup,
> +    },
> +
> +    {
>          .name       = "block-job-set-speed",
>          .args_type  = "device:B,speed:o",
>          .mhandler.cmd_new = qmp_marshal_input_block_job_set_speed,
>