[Qemu-devel] [Patch v12 00/10] Block replication for continuous checkpoints

[Qemu-devel] [Patch v12 00/10] Block replication for continuous checkpoints

[Wen Congyang]

Block replication is a very important feature which is used for
continuous checkpoints(for example: COLO).

You can get the detailed information about block replication from here:
http://wiki.qemu.org/Features/BlockReplication

Usage:
Please refer to docs/block-replication.txt

This patch series is based on the following patch series:
1. http://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg04949.html
2. http://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg06043.html

You can get the patch here:
https://github.com/coloft/qemu/tree/wency/block-replication-v12

You can get the patch with framework here:
https://github.com/coloft/qemu/tree/wency/colo_framework_v11.2

TODO:
1. Continuous block replication. It will be started after basic functions
   are accepted.

Changs Log:
V12:
1. Rebase to the newest codes
2. Use backing reference to replcace 'allow-write-backing-file'
V11:
1. Reopen the backing file when starting blcok replication if it is not
   opened in R/W mode
2. Unblock BLOCK_OP_TYPE_BACKUP_SOURCE and BLOCK_OP_TYPE_BACKUP_TARGET
   when opening backing file
3. Block the top BDS so there is only one block job for the top BDS and
   its backing chain.
V10:
1. Use blockdev-remove-medium and blockdev-insert-medium to replace backing
   reference.
2. Address the comments from Eric Blake
V9:
1. Update the error messages
2. Rebase to the newest qemu
3. Split child add/delete support. These patches are sent in another patchset.
V8:
1. Address Alberto Garcia's comments
V7:
1. Implement adding/removing quorum child. Remove the option non-connect.
2. Simplify the backing refrence option according to Stefan Hajnoczi's suggestion
V6:
1. Rebase to the newest qemu.
V5:
1. Address the comments from Gong Lei
2. Speed the failover up. The secondary vm can take over very quickly even
   if there are too many I/O requests.
V4:
1. Introduce a new driver replication to avoid touch nbd and qcow2.
V3:
1: use error_setg() instead of error_set()
2. Add a new block job API
3. Active disk, hidden disk and nbd target uses the same AioContext
4. Add a testcase to test new hbitmap API
V2:
1. Redesign the secondary qemu(use image-fleecing)
2. Use Error objects to return error message
3. Address the comments from Max Reitz and Eric Blake

Wen Congyang (10):
  unblock backup operations in backing file
  Store parent BDS in BdrvChild
  Backup: clear all bitmap when doing block checkpoint
  Allow creating backup jobs when opening BDS
  docs: block replication's description
  Add new block driver interfaces to control block replication
  quorum: implement block driver interfaces for block replication
  Implement new driver for block replication
  support replication driver in blockdev-add
  Add a new API to start/stop replication, do checkpoint to all BDSes

 block.c                    | 145 ++++++++++++
 block/Makefile.objs        |   3 +-
 block/backup.c             |  14 ++
 block/quorum.c             |  78 +++++++
 block/replication.c        | 549 +++++++++++++++++++++++++++++++++++++++++++++
 blockjob.c                 |  11 +
 docs/block-replication.txt | 227 +++++++++++++++++++
 include/block/block.h      |   9 +
 include/block/block_int.h  |  15 ++
 include/block/blockjob.h   |  12 +
 qapi/block-core.json       |  34 ++-
 11 files changed, 1093 insertions(+), 4 deletions(-)
 create mode 100644 block/replication.c
 create mode 100644 docs/block-replication.txt

-- 
2.5.0

[Qemu-devel] [Patch v12 01/10] unblock backup operations in backing file

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
 block.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/block.c b/block.c
index bfc2be8..eaf479a 100644
--- a/block.c
+++ b/block.c
@@ -1275,6 +1275,24 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd)
     /* Otherwise we won't be able to commit due to check in bdrv_commit */
     bdrv_op_unblock(backing_hd, BLOCK_OP_TYPE_COMMIT_TARGET,
                     bs->backing_blocker);
+    /*
+     * We do backup in 3 ways:
+     * 1. drive backup
+     *    The target bs is new opened, and the source is top BDS
+     * 2. blockdev backup
+     *    Both the source and the target are top BDSes.
+     * 3. internal backup(used for block replication)
+     *    Both the source and the target are backing file
+     *
+     * In case 1, and 2, the backing file is neither the source nor
+     * the target.
+     * In case 3, we will block the top BDS, so there is only one block
+     * job for the top BDS and its backing chain.
+     */
+    bdrv_op_unblock(backing_hd, BLOCK_OP_TYPE_BACKUP_SOURCE,
+                    bs->backing_blocker);
+    bdrv_op_unblock(backing_hd, BLOCK_OP_TYPE_BACKUP_TARGET,
+                    bs->backing_blocker);
 out:
     bdrv_refresh_limits(bs, NULL);
 }
-- 
2.5.0

[Qemu-devel] [Patch v12 02/10] Store parent BDS in BdrvChild

[Wen Congyang]

We need to access the parent BDS to get the root BDS.

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
 block.c                   | 1 +
 include/block/block_int.h | 1 +
 2 files changed, 2 insertions(+)

diff --git a/block.c b/block.c
index eaf479a..0a0468f 100644
--- a/block.c
+++ b/block.c
@@ -1204,6 +1204,7 @@ BdrvChild *bdrv_attach_child(BlockDriverState *parent_bs,
     BdrvChild *child = g_new(BdrvChild, 1);
     *child = (BdrvChild) {
         .bs     = child_bs,
+        .parent = parent_bs,
         .name   = g_strdup(child_name),
         .role   = child_role,
     };
diff --git a/include/block/block_int.h b/include/block/block_int.h
index ea20d12..1f56046 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -357,6 +357,7 @@ extern const BdrvChildRole child_format;
 
 struct BdrvChild {
     BlockDriverState *bs;
+    BlockDriverState *parent;
     char *name;
     const BdrvChildRole *role;
     QLIST_ENTRY(BdrvChild) next;
-- 
2.5.0

[Qemu-devel] [Patch v12 03/10] Backup: clear all bitmap when doing block checkpoint

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Reviewed-by: Jeff Cody <jcody@redhat.com>
---
 block/backup.c           | 14 ++++++++++++++
 blockjob.c               | 11 +++++++++++
 include/block/blockjob.h | 12 ++++++++++++
 3 files changed, 37 insertions(+)

diff --git a/block/backup.c b/block/backup.c
index 3b39119..1ca102d 100644
--- a/block/backup.c
+++ b/block/backup.c
@@ -253,11 +253,25 @@ static void backup_abort(BlockJob *job)
     }
 }
 
+static void backup_do_checkpoint(BlockJob *job, Error **errp)
+{
+    BackupBlockJob *backup_job = container_of(job, BackupBlockJob, common);
+
+    if (backup_job->sync_mode != MIRROR_SYNC_MODE_NONE) {
+        error_setg(errp, "The backup job only supports block checkpoint in"
+                   " sync=none mode");
+        return;
+    }
+
+    hbitmap_reset_all(backup_job->bitmap);
+}
+
 static const BlockJobDriver backup_job_driver = {
     .instance_size  = sizeof(BackupBlockJob),
     .job_type       = BLOCK_JOB_TYPE_BACKUP,
     .set_speed      = backup_set_speed,
     .iostatus_reset = backup_iostatus_reset,
+    .do_checkpoint  = backup_do_checkpoint,
     .commit         = backup_commit,
     .abort          = backup_abort,
 };
diff --git a/blockjob.c b/blockjob.c
index 80adb9d..0c8edfe 100644
--- a/blockjob.c
+++ b/blockjob.c
@@ -533,3 +533,14 @@ void block_job_txn_add_job(BlockJobTxn *txn, BlockJob *job)
     QLIST_INSERT_HEAD(&txn->jobs, job, txn_list);
     block_job_txn_ref(txn);
 }
+
+void block_job_do_checkpoint(BlockJob *job, Error **errp)
+{
+    if (!job->driver->do_checkpoint) {
+        error_setg(errp, "The job %s doesn't support block checkpoint",
+                   BlockJobType_lookup[job->driver->job_type]);
+        return;
+    }
+
+    job->driver->do_checkpoint(job, errp);
+}
diff --git a/include/block/blockjob.h b/include/block/blockjob.h
index d84ccd8..abdba7c 100644
--- a/include/block/blockjob.h
+++ b/include/block/blockjob.h
@@ -70,6 +70,9 @@ typedef struct BlockJobDriver {
      * never both.
      */
     void (*abort)(BlockJob *job);
+
+    /** Optional callback for job types that support checkpoint. */
+    void (*do_checkpoint)(BlockJob *job, Error **errp);
 } BlockJobDriver;
 
 /**
@@ -443,4 +446,13 @@ void block_job_txn_unref(BlockJobTxn *txn);
  */
 void block_job_txn_add_job(BlockJobTxn *txn, BlockJob *job);
 
+/**
+ * block_job_do_checkpoint:
+ * @job: The job.
+ * @errp: Error object.
+ *
+ * Do block checkpoint on the specified job.
+ */
+void block_job_do_checkpoint(BlockJob *job, Error **errp);
+
 #endif
-- 
2.5.0

[Qemu-devel] [Patch v12 04/10] Allow creating backup jobs when opening BDS

[Wen Congyang]

When opening BDS, we need to create backup jobs for
image-fleecing.

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Jeff Cody <jcody@redhat.com>
---
 block/Makefile.objs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block/Makefile.objs b/block/Makefile.objs
index 58ef2ef..fa05f37 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -22,10 +22,10 @@ block-obj-$(CONFIG_ARCHIPELAGO) += archipelago.o
 block-obj-$(CONFIG_LIBSSH2) += ssh.o
 block-obj-y += accounting.o
 block-obj-y += write-threshold.o
+block-obj-y += backup.o
 
 common-obj-y += stream.o
 common-obj-y += commit.o
-common-obj-y += backup.o
 
 iscsi.o-cflags     := $(LIBISCSI_CFLAGS)
 iscsi.o-libs       := $(LIBISCSI_LIBS)
-- 
2.5.0

[Qemu-devel] [Patch v12 05/10] docs: block replication's description

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
 docs/block-replication.txt | 227 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 227 insertions(+)
 create mode 100644 docs/block-replication.txt

diff --git a/docs/block-replication.txt b/docs/block-replication.txt
new file mode 100644
index 0000000..c7bad0e
--- /dev/null
+++ b/docs/block-replication.txt
@@ -0,0 +1,227 @@
+Block replication
+----------------------------------------
+Copyright Fujitsu, Corp. 2015
+Copyright (c) 2015 Intel Corporation
+Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+
+This work is licensed under the terms of the GNU GPL, version 2 or later.
+See the COPYING file in the top-level directory.
+
+Block replication is used for continuous checkpoints. It is designed
+for COLO (COurse-grain LOck-stepping) where the Secondary VM is running.
+It can also be applied for FT/HA (Fault-tolerance/High Assurance) scenario,
+where the Secondary VM is not running.
+
+This document gives an overview of block replication's design.
+
+== Background ==
+High availability solutions such as micro checkpoint and COLO will do
+consecutive checkpoints. The VM state of Primary VM and Secondary VM is
+identical right after a VM checkpoint, but becomes different as the VM
+executes till the next checkpoint. To support disk contents checkpoint,
+the modified disk contents in the Secondary VM must be buffered, and are
+only dropped at next checkpoint time. To reduce the network transportation
+effort at the time of checkpoint, the disk modification operations of
+Primary disk are asynchronously forwarded to the Secondary node.
+
+== Workflow ==
+The following is the image of block replication workflow:
+
+        +----------------------+            +------------------------+
+        |Primary Write Requests|            |Secondary Write Requests|
+        +----------------------+            +------------------------+
+                  |                                       |
+                  |                                      (4)
+                  |                                       V
+                  |                              /-------------\
+                  |      Copy and Forward        |             |
+                  |---------(1)----------+       | Disk Buffer |
+                  |                      |       |             |
+                  |                     (3)      \-------------/
+                  |                 speculative      ^
+                  |                write through    (2)
+                  |                      |           |
+                  V                      V           |
+           +--------------+           +----------------+
+           | Primary Disk |           | Secondary Disk |
+           +--------------+           +----------------+
+
+    1) Primary write requests will be copied and forwarded to Secondary
+       QEMU.
+    2) Before Primary write requests are written to Secondary disk, the
+       original sector content will be read from Secondary disk and
+       buffered in the Disk buffer, but it will not overwrite the existing
+       sector content (it could be from either "Secondary Write Requests" or
+       previous COW of "Primary Write Requests") in the Disk buffer.
+    3) Primary write requests will be written to Secondary disk.
+    4) Secondary write requests will be buffered in the Disk buffer and it
+       will overwrite the existing sector content in the buffer.
+
+== Architecture ==
+We are going to implement block replication from many basic
+blocks that are already in QEMU.
+
+         virtio-blk       ||
+             ^            ||                            .----------
+             |            ||                            | Secondary
+        1 Quorum          ||                            '----------
+         /      \         ||
+        /        \        ||
+   Primary    2 filter
+     disk         ^                                                             virtio-blk
+                  |                                                                  ^
+                3 NBD  ------->  3 NBD                                               |
+                client    ||     server                                          2 filter
+                          ||        ^                                                ^
+--------.                 ||        |                                                |
+Primary |                 ||  Secondary disk <--------- hidden-disk 5 <--------- active-disk 4
+--------'                 ||        |          backing        ^       backing
+                          ||        |                         |
+                          ||        |                         |
+                          ||        '-------------------------'
+                          ||           drive-backup sync=none 6
+
+1) The disk on the primary is represented by a block device with two
+children, providing replication between a primary disk and the host that
+runs the secondary VM. The read pattern for quorum can be extended to
+make the primary always read from the local disk instead of going through
+NBD.
+
+2) The new block filter (the name is replication) will control the block
+replication.
+
+3) The secondary disk receives writes from the primary VM through QEMU's
+embedded NBD server (speculative write-through).
+
+4) The disk on the secondary is represented by a custom block device
+(called active-disk). It should start as an empty disk, and the format
+should support bdrv_make_empty() and backing file.
+
+5) The hidden-disk is created automatically. It buffers the original content
+that is modified by the primary VM. It should also start as an empty disk,
+and the driver supports bdrv_make_empty() and backing file.
+
+6) The drive-backup job(sync=none) is run to allow hidden-disk to buffer
+any state that would otherwise be lost by the speculative write-through
+of the NBD server into the secondary disk. So before block replication,
+the primary disk and secondary disk should contain the same data.
+
+== Failure Handling ==
+There are 6 internal errors when block replication is running:
+1. I/O error on primary disk
+2. Forwarding primary write requests failed
+3. Backup failed
+4. I/O error on secondary disk
+5. I/O error on active disk
+6. Making active disk or hidden disk empty failed
+In case 1 and 5, we just report the error to the disk layer. In case 2, 3,
+4 and 6, we just report block replication's error to FT/HA manager (which
+decides when to do a new checkpoint, when to do failover).
+There is no internal error when doing failover.
+
+== New block driver interface ==
+We add three block driver interfaces to control block replication:
+a. bdrv_start_replication()
+   Start block replication, called in migration/checkpoint thread.
+   We must call bdrv_start_replication() in secondary QEMU before
+   calling bdrv_start_replication() in primary QEMU. The caller
+   must hold the I/O mutex lock if it is in migration/checkpoint
+   thread.
+b. bdrv_do_checkpoint()
+   This interface is called after all VM state is transferred to
+   Secondary QEMU. The Disk buffer will be dropped in this interface.
+   The caller must hold the I/O mutex lock if it is in migration/checkpoint
+   thread.
+c. bdrv_stop_replication()
+   It is called on failover. We will flush the Disk buffer into
+   Secondary Disk and stop block replication. The vm should be stopped
+   before calling it if you use this API to shutdown the guest, or other
+   things except failover. The caller must hold the I/O mutex lock if it is
+   in migration/checkpoint thread.
+
+== Usage ==
+Primary:
+  -drive if=xxx,driver=quorum,read-pattern=fifo,id=colo1,vote-threshold=1\
+         children.0.file.filename=1.raw,\
+         children.0.driver=raw,\
+
+  Run qmp command in primary qemu:
+    { 'execute': 'human-monitor-command',
+      'arguments': {
+          'command-line': 'drive_add buddy driver=replication,mode=primary,file.driver=nbd,file.host=xxxx,file.port=xxxx,file.export=colo1,node-name=nbd_client1,if=none'
+      }
+    }
+    { 'execute': 'x-blockdev-change',
+      'arguments': {
+          'parent': 'colo1',
+          'node': 'nbd_client1'
+      }
+    }
+  Note:
+  1. There should be only one NBD Client for each primary disk.
+  2. host is the secondary physical machine's hostname or IP
+  3. Each disk must have its own export name.
+  4. It is all a single argument to -drive and you should ignore the
+     leading whitespace.
+  5. The qmp command line must be run after running qmp command line in
+     secondary qemu.
+
+Secondary:
+  -drive if=none,driver=raw,file.filename=1.raw,id=colo1 \
+  -drive if=xxx,driver=replication,mode=secondary,\
+         file.file.filename=active_disk.qcow2,\
+         file.driver=qcow2,\
+         file.backing.file.filename=hidden_disk.qcow2,\
+         file.backing.driver=qcow2,\
+         file.backing.backing=colo1
+
+  Then run qmp command in secondary qemu:
+    { 'execute': 'nbd-server-start',
+      'arguments': {
+          'addr': {
+              'type': inet',
+              'data': {
+                  'host': 'xxx',
+                  'port': 'xxx'
+              }
+          }
+      }
+    }
+    { 'execute': 'nbd-server-add',
+      'arguments': {
+          'device': 'colo1',
+          'writable': true
+      }
+    }
+
+  Note:
+  1. The export name in secondary QEMU command line is the secondary
+     disk's id.
+  2. The export name for the same disk must be the same
+  3. The qmp command nbd-server-start and nbd-server-add must be run
+     before running the qmp command migrate on primary QEMU
+  4. Active disk, hidden disk and nbd target's length should be the
+     same.
+  5. It is better to put active disk and hidden disk in ramdisk.
+  6. It is all a single argument to -drive, and you should ignore
+     the leading whitespace.
+
+After Failover:
+Primary:
+  The secondary host is down, so we should run the following qmp command
+  to remove the nbd child from the quorum:
+  { 'execute': 'x-blockdev-change',
+    'arguments': {
+        'parent': 'colo1',
+        'child': 'children.1'
+    }
+  }
+  Note: there is no qmp command to remove the blockdev now
+
+Secondary:
+  The primary host is down, so we should do the following thing:
+  { 'execute': 'nbd-server-stop' }
+
+TODO:
+1. Continuous block replication
+2. Shared disk
-- 
2.5.0

[Qemu-devel] [Patch v12 06/10] Add new block driver interfaces to control block replication

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Cc: Luiz Capitulino <lcapitulino@redhat.com>
Cc: Michael Roth <mdroth@linux.vnet.ibm.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 block.c                   | 43 +++++++++++++++++++++++++++++++++++++++++++
 include/block/block.h     |  5 +++++
 include/block/block_int.h | 14 ++++++++++++++
 qapi/block-core.json      | 13 +++++++++++++
 4 files changed, 75 insertions(+)

diff --git a/block.c b/block.c
index 0a0468f..213bee8 100644
--- a/block.c
+++ b/block.c
@@ -4390,3 +4390,46 @@ void bdrv_del_child(BlockDriverState *parent_bs, BlockDriverState *child_bs,
 
     parent_bs->drv->bdrv_del_child(parent_bs, child_bs, errp);
 }
+
+void bdrv_start_replication(BlockDriverState *bs, ReplicationMode mode,
+                            Error **errp)
+{
+    BlockDriver *drv = bs->drv;
+
+    if (drv && drv->bdrv_start_replication) {
+        drv->bdrv_start_replication(bs, mode, errp);
+    } else if (bs->file) {
+        bdrv_start_replication(bs->file->bs, mode, errp);
+    } else {
+        error_setg(errp, "The BDS %s doesn't support starting block"
+                   " replication", bs->filename);
+    }
+}
+
+void bdrv_do_checkpoint(BlockDriverState *bs, Error **errp)
+{
+    BlockDriver *drv = bs->drv;
+
+    if (drv && drv->bdrv_do_checkpoint) {
+        drv->bdrv_do_checkpoint(bs, errp);
+    } else if (bs->file) {
+        bdrv_do_checkpoint(bs->file->bs, errp);
+    } else {
+        error_setg(errp, "The BDS %s doesn't support block checkpoint",
+                   bs->filename);
+    }
+}
+
+void bdrv_stop_replication(BlockDriverState *bs, bool failover, Error **errp)
+{
+    BlockDriver *drv = bs->drv;
+
+    if (drv && drv->bdrv_stop_replication) {
+        drv->bdrv_stop_replication(bs, failover, errp);
+    } else if (bs->file) {
+        bdrv_stop_replication(bs->file->bs, failover, errp);
+    } else {
+        error_setg(errp, "The BDS %s doesn't support stopping block"
+                   " replication", bs->filename);
+    }
+}
diff --git a/include/block/block.h b/include/block/block.h
index 1d3b9c6..cd39d50 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -648,4 +648,9 @@ void bdrv_add_child(BlockDriverState *parent, BlockDriverState *child,
 void bdrv_del_child(BlockDriverState *parent, BlockDriverState *child,
                     Error **errp);
 
+void bdrv_start_replication(BlockDriverState *bs, ReplicationMode mode,
+                            Error **errp);
+void bdrv_do_checkpoint(BlockDriverState *bs, Error **errp);
+void bdrv_stop_replication(BlockDriverState *bs, bool failover, Error **errp);
+
 #endif
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 1f56046..a6aba8b 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -307,6 +307,20 @@ struct BlockDriver {
     void (*bdrv_del_child)(BlockDriverState *parent, BlockDriverState *child,
                            Error **errp);
 
+    void (*bdrv_start_replication)(BlockDriverState *bs, ReplicationMode mode,
+                                   Error **errp);
+    /* Drop Disk buffer when doing checkpoint. */
+    void (*bdrv_do_checkpoint)(BlockDriverState *bs, Error **errp);
+    /*
+     * After failover, we should flush Disk buffer into secondary disk
+     * and stop block replication.
+     *
+     * If the guest is shutdown, we should drop Disk buffer and stop
+     * block representation.
+     */
+    void (*bdrv_stop_replication)(BlockDriverState *bs, bool failover,
+                                  Error **errp);
+
     QLIST_ENTRY(BlockDriver) list;
 };
 
diff --git a/qapi/block-core.json b/qapi/block-core.json
index feb8da2..2c6bd3f 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1925,6 +1925,19 @@
             '*read-pattern': 'QuorumReadPattern' } }
 
 ##
+# @ReplicationMode
+#
+# An enumeration of replication modes.
+#
+# @primary: Primary mode, the vm's state will be sent to secondary QEMU.
+#
+# @secondary: Secondary mode, receive the vm's state from primary QEMU.
+#
+# Since: 2.5
+##
+{ 'enum' : 'ReplicationMode', 'data' : [ 'primary', 'secondary' ] }
+
+##
 # @BlockdevOptions
 #
 # Options for creating a block device.
-- 
2.5.0

[Qemu-devel] [Patch v12 07/10] quorum: implement block driver interfaces for block replication

[Wen Congyang]

Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
---
 block/quorum.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)

diff --git a/block/quorum.c b/block/quorum.c
index b7df14b..6fa54f3 100644
--- a/block/quorum.c
+++ b/block/quorum.c
@@ -85,6 +85,8 @@ typedef struct BDRVQuorumState {
     int bsize;
 
     QuorumReadPattern read_pattern;
+
+    int replication_index; /* store which child supports block replication */
 } BDRVQuorumState;
 
 typedef struct QuorumAIOCB QuorumAIOCB;
@@ -949,6 +951,7 @@ static int quorum_open(BlockDriverState *bs, QDict *options, int flags,
     s->bsize = s->num_children;
 
     g_free(opened);
+    s->replication_index = -1;
     goto exit;
 
 close_exit:
@@ -1148,6 +1151,77 @@ static void quorum_refresh_filename(BlockDriverState *bs, QDict *options)
     bs->full_open_options = opts;
 }
 
+static void quorum_start_replication(BlockDriverState *bs, ReplicationMode mode,
+                                     Error **errp)
+{
+    BDRVQuorumState *s = bs->opaque;
+    int count = 0, i, index;
+    Error *local_err = NULL;
+
+    /*
+     * TODO: support REPLICATION_MODE_SECONDARY if we allow secondary
+     * QEMU becoming primary QEMU.
+     */
+    if (mode != REPLICATION_MODE_PRIMARY) {
+        error_setg(errp, "The replication mode for quorum should be 'primary'");
+        return;
+    }
+
+    if (s->read_pattern != QUORUM_READ_PATTERN_FIFO) {
+        error_setg(errp, "Block replication needs read pattern 'fifo'");
+        return;
+    }
+
+    for (i = 0; i < s->num_children; i++) {
+        bdrv_start_replication(s->children[i]->bs, mode, &local_err);
+        if (local_err) {
+            error_free(local_err);
+            local_err = NULL;
+        } else {
+            count++;
+            index = i;
+        }
+    }
+
+    if (count == 0) {
+        error_setg(errp, "No child supports block replication");
+    } else if (count > 1) {
+        for (i = 0; i < s->num_children; i++) {
+            bdrv_stop_replication(s->children[i]->bs, false, NULL);
+        }
+        error_setg(errp, "Too many children support block replication");
+    } else {
+        s->replication_index = index;
+    }
+}
+
+static void quorum_do_checkpoint(BlockDriverState *bs, Error **errp)
+{
+    BDRVQuorumState *s = bs->opaque;
+
+    if (s->replication_index < 0) {
+        error_setg(errp, "Block replication is not running");
+        return;
+    }
+
+    bdrv_do_checkpoint(s->children[s->replication_index]->bs, errp);
+}
+
+static void quorum_stop_replication(BlockDriverState *bs, bool failover,
+                                    Error **errp)
+{
+    BDRVQuorumState *s = bs->opaque;
+
+    if (s->replication_index < 0) {
+        error_setg(errp, "Block replication is not running");
+        return;
+    }
+
+    bdrv_stop_replication(s->children[s->replication_index]->bs, failover,
+                          errp);
+    s->replication_index = -1;
+}
+
 static BlockDriver bdrv_quorum = {
     .format_name                        = "quorum",
     .protocol_name                      = "quorum",
@@ -1174,6 +1248,10 @@ static BlockDriver bdrv_quorum = {
 
     .is_filter                          = true,
     .bdrv_recurse_is_first_non_filter   = quorum_recurse_is_first_non_filter,
+
+    .bdrv_start_replication             = quorum_start_replication,
+    .bdrv_do_checkpoint                 = quorum_do_checkpoint,
+    .bdrv_stop_replication              = quorum_stop_replication,
 };
 
 static void bdrv_quorum_init(void)
-- 
2.5.0

Re: [Qemu-devel] [Patch v12 00/10] Block replication for continuous checkpoints

[Dr. David Alan Gilbert]

* Wen Congyang (wency@cn.fujitsu.com) wrote:
> Block replication is a very important feature which is used for
> continuous checkpoints(for example: COLO).
> 
> You can get the detailed information about block replication from here:
> http://wiki.qemu.org/Features/BlockReplication
> 
> Usage:
> Please refer to docs/block-replication.txt
> 
> This patch series is based on the following patch series:
> 1. http://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg04949.html
> 2. http://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg06043.html
> 
> You can get the patch here:
> https://github.com/coloft/qemu/tree/wency/block-replication-v12
> 
> You can get the patch with framework here:
> https://github.com/coloft/qemu/tree/wency/colo_framework_v11.2

Neither of these links work for me, and I see that  only messages 0..7 in the
series hit the list.

Dave

> 
> TODO:
> 1. Continuous block replication. It will be started after basic functions
>    are accepted.
> 
> Changs Log:
> V12:
> 1. Rebase to the newest codes
> 2. Use backing reference to replcace 'allow-write-backing-file'
> V11:
> 1. Reopen the backing file when starting blcok replication if it is not
>    opened in R/W mode
> 2. Unblock BLOCK_OP_TYPE_BACKUP_SOURCE and BLOCK_OP_TYPE_BACKUP_TARGET
>    when opening backing file
> 3. Block the top BDS so there is only one block job for the top BDS and
>    its backing chain.
> V10:
> 1. Use blockdev-remove-medium and blockdev-insert-medium to replace backing
>    reference.
> 2. Address the comments from Eric Blake
> V9:
> 1. Update the error messages
> 2. Rebase to the newest qemu
> 3. Split child add/delete support. These patches are sent in another patchset.
> V8:
> 1. Address Alberto Garcia's comments
> V7:
> 1. Implement adding/removing quorum child. Remove the option non-connect.
> 2. Simplify the backing refrence option according to Stefan Hajnoczi's suggestion
> V6:
> 1. Rebase to the newest qemu.
> V5:
> 1. Address the comments from Gong Lei
> 2. Speed the failover up. The secondary vm can take over very quickly even
>    if there are too many I/O requests.
> V4:
> 1. Introduce a new driver replication to avoid touch nbd and qcow2.
> V3:
> 1: use error_setg() instead of error_set()
> 2. Add a new block job API
> 3. Active disk, hidden disk and nbd target uses the same AioContext
> 4. Add a testcase to test new hbitmap API
> V2:
> 1. Redesign the secondary qemu(use image-fleecing)
> 2. Use Error objects to return error message
> 3. Address the comments from Max Reitz and Eric Blake
> 
> Wen Congyang (10):
>   unblock backup operations in backing file
>   Store parent BDS in BdrvChild
>   Backup: clear all bitmap when doing block checkpoint
>   Allow creating backup jobs when opening BDS
>   docs: block replication's description
>   Add new block driver interfaces to control block replication
>   quorum: implement block driver interfaces for block replication
>   Implement new driver for block replication
>   support replication driver in blockdev-add
>   Add a new API to start/stop replication, do checkpoint to all BDSes
> 
>  block.c                    | 145 ++++++++++++
>  block/Makefile.objs        |   3 +-
>  block/backup.c             |  14 ++
>  block/quorum.c             |  78 +++++++
>  block/replication.c        | 549 +++++++++++++++++++++++++++++++++++++++++++++
>  blockjob.c                 |  11 +
>  docs/block-replication.txt | 227 +++++++++++++++++++
>  include/block/block.h      |   9 +
>  include/block/block_int.h  |  15 ++
>  include/block/blockjob.h   |  12 +
>  qapi/block-core.json       |  34 ++-
>  11 files changed, 1093 insertions(+), 4 deletions(-)
>  create mode 100644 block/replication.c
>  create mode 100644 docs/block-replication.txt
> 
> -- 
> 2.5.0
> 
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [Patch v12 00/10] Block replication for continuous checkpoints

[Hailiang Zhang]

On 2015/12/1 18:40, Dr. David Alan Gilbert wrote:
> * Wen Congyang (wency@cn.fujitsu.com) wrote:
>> Block replication is a very important feature which is used for
>> continuous checkpoints(for example: COLO).
>>
>> You can get the detailed information about block replication from here:
>> http://wiki.qemu.org/Features/BlockReplication
>>
>> Usage:
>> Please refer to docs/block-replication.txt
>>
>> This patch series is based on the following patch series:
>> 1. http://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg04949.html
>> 2. http://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg06043.html
>>
>> You can get the patch here:
>> https://github.com/coloft/qemu/tree/wency/block-replication-v12
>>
>> You can get the patch with framework here:
>> https://github.com/coloft/qemu/tree/wency/colo_framework_v11.2
>
> Neither of these links work for me, and I see that  only messages 0..7 in the
> series hit the list.
>

Hi Dave,

You can refer to https://github.com/coloft/qemu/tree/colo-v2.2-periodic-mode,
The block replication part in this link is also the newest version.

Congyang has deleted this confused branch, we will pay attention to this later in next version.

Thanks,
Hailiang

>
>>
>> TODO:
>> 1. Continuous block replication. It will be started after basic functions
>>     are accepted.
>>
>> Changs Log:
>> V12:
>> 1. Rebase to the newest codes
>> 2. Use backing reference to replcace 'allow-write-backing-file'
>> V11:
>> 1. Reopen the backing file when starting blcok replication if it is not
>>     opened in R/W mode
>> 2. Unblock BLOCK_OP_TYPE_BACKUP_SOURCE and BLOCK_OP_TYPE_BACKUP_TARGET
>>     when opening backing file
>> 3. Block the top BDS so there is only one block job for the top BDS and
>>     its backing chain.
>> V10:
>> 1. Use blockdev-remove-medium and blockdev-insert-medium to replace backing
>>     reference.
>> 2. Address the comments from Eric Blake
>> V9:
>> 1. Update the error messages
>> 2. Rebase to the newest qemu
>> 3. Split child add/delete support. These patches are sent in another patchset.
>> V8:
>> 1. Address Alberto Garcia's comments
>> V7:
>> 1. Implement adding/removing quorum child. Remove the option non-connect.
>> 2. Simplify the backing refrence option according to Stefan Hajnoczi's suggestion
>> V6:
>> 1. Rebase to the newest qemu.
>> V5:
>> 1. Address the comments from Gong Lei
>> 2. Speed the failover up. The secondary vm can take over very quickly even
>>     if there are too many I/O requests.
>> V4:
>> 1. Introduce a new driver replication to avoid touch nbd and qcow2.
>> V3:
>> 1: use error_setg() instead of error_set()
>> 2. Add a new block job API
>> 3. Active disk, hidden disk and nbd target uses the same AioContext
>> 4. Add a testcase to test new hbitmap API
>> V2:
>> 1. Redesign the secondary qemu(use image-fleecing)
>> 2. Use Error objects to return error message
>> 3. Address the comments from Max Reitz and Eric Blake
>>
>> Wen Congyang (10):
>>    unblock backup operations in backing file
>>    Store parent BDS in BdrvChild
>>    Backup: clear all bitmap when doing block checkpoint
>>    Allow creating backup jobs when opening BDS
>>    docs: block replication's description
>>    Add new block driver interfaces to control block replication
>>    quorum: implement block driver interfaces for block replication
>>    Implement new driver for block replication
>>    support replication driver in blockdev-add
>>    Add a new API to start/stop replication, do checkpoint to all BDSes
>>
>>   block.c                    | 145 ++++++++++++
>>   block/Makefile.objs        |   3 +-
>>   block/backup.c             |  14 ++
>>   block/quorum.c             |  78 +++++++
>>   block/replication.c        | 549 +++++++++++++++++++++++++++++++++++++++++++++
>>   blockjob.c                 |  11 +
>>   docs/block-replication.txt | 227 +++++++++++++++++++
>>   include/block/block.h      |   9 +
>>   include/block/block_int.h  |  15 ++
>>   include/block/blockjob.h   |  12 +
>>   qapi/block-core.json       |  34 ++-
>>   11 files changed, 1093 insertions(+), 4 deletions(-)
>>   create mode 100644 block/replication.c
>>   create mode 100644 docs/block-replication.txt
>>
>> --
>> 2.5.0
>>
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>
> .
>

Re: [Qemu-devel] [Patch v12 00/10] Block replication for continuous checkpoints

[Wen Congyang]

On 12/01/2015 06:40 PM, Dr. David Alan Gilbert wrote:
> * Wen Congyang (wency@cn.fujitsu.com) wrote:
>> Block replication is a very important feature which is used for
>> continuous checkpoints(for example: COLO).
>>
>> You can get the detailed information about block replication from here:
>> http://wiki.qemu.org/Features/BlockReplication
>>
>> Usage:
>> Please refer to docs/block-replication.txt
>>
>> This patch series is based on the following patch series:
>> 1. http://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg04949.html
>> 2. http://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg06043.html
>>
>> You can get the patch here:
>> https://github.com/coloft/qemu/tree/wency/block-replication-v12
>>
>> You can get the patch with framework here:
>> https://github.com/coloft/qemu/tree/wency/colo_framework_v11.2
> 
> Neither of these links work for me, and I see that  only messages 0..7 in the
> series hit the list.

I forgot to push it to github...
And I also received the messages 0..7, and I don't know what's wrong...

I will push it to github, and resend them.

Thanks
Wen Congyang

> 
> Dave
> 
>>
>> TODO:
>> 1. Continuous block replication. It will be started after basic functions
>>    are accepted.
>>
>> Changs Log:
>> V12:
>> 1. Rebase to the newest codes
>> 2. Use backing reference to replcace 'allow-write-backing-file'
>> V11:
>> 1. Reopen the backing file when starting blcok replication if it is not
>>    opened in R/W mode
>> 2. Unblock BLOCK_OP_TYPE_BACKUP_SOURCE and BLOCK_OP_TYPE_BACKUP_TARGET
>>    when opening backing file
>> 3. Block the top BDS so there is only one block job for the top BDS and
>>    its backing chain.
>> V10:
>> 1. Use blockdev-remove-medium and blockdev-insert-medium to replace backing
>>    reference.
>> 2. Address the comments from Eric Blake
>> V9:
>> 1. Update the error messages
>> 2. Rebase to the newest qemu
>> 3. Split child add/delete support. These patches are sent in another patchset.
>> V8:
>> 1. Address Alberto Garcia's comments
>> V7:
>> 1. Implement adding/removing quorum child. Remove the option non-connect.
>> 2. Simplify the backing refrence option according to Stefan Hajnoczi's suggestion
>> V6:
>> 1. Rebase to the newest qemu.
>> V5:
>> 1. Address the comments from Gong Lei
>> 2. Speed the failover up. The secondary vm can take over very quickly even
>>    if there are too many I/O requests.
>> V4:
>> 1. Introduce a new driver replication to avoid touch nbd and qcow2.
>> V3:
>> 1: use error_setg() instead of error_set()
>> 2. Add a new block job API
>> 3. Active disk, hidden disk and nbd target uses the same AioContext
>> 4. Add a testcase to test new hbitmap API
>> V2:
>> 1. Redesign the secondary qemu(use image-fleecing)
>> 2. Use Error objects to return error message
>> 3. Address the comments from Max Reitz and Eric Blake
>>
>> Wen Congyang (10):
>>   unblock backup operations in backing file
>>   Store parent BDS in BdrvChild
>>   Backup: clear all bitmap when doing block checkpoint
>>   Allow creating backup jobs when opening BDS
>>   docs: block replication's description
>>   Add new block driver interfaces to control block replication
>>   quorum: implement block driver interfaces for block replication
>>   Implement new driver for block replication
>>   support replication driver in blockdev-add
>>   Add a new API to start/stop replication, do checkpoint to all BDSes
>>
>>  block.c                    | 145 ++++++++++++
>>  block/Makefile.objs        |   3 +-
>>  block/backup.c             |  14 ++
>>  block/quorum.c             |  78 +++++++
>>  block/replication.c        | 549 +++++++++++++++++++++++++++++++++++++++++++++
>>  blockjob.c                 |  11 +
>>  docs/block-replication.txt | 227 +++++++++++++++++++
>>  include/block/block.h      |   9 +
>>  include/block/block_int.h  |  15 ++
>>  include/block/blockjob.h   |  12 +
>>  qapi/block-core.json       |  34 ++-
>>  11 files changed, 1093 insertions(+), 4 deletions(-)
>>  create mode 100644 block/replication.c
>>  create mode 100644 docs/block-replication.txt
>>
>> -- 
>> 2.5.0
>>
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
> 
> .
> 

Re: [Qemu-devel] [Patch v12 00/10] Block replication for continuous checkpoints

[Wen Congyang]

On 12/01/2015 07:58 PM, Hailiang Zhang wrote:
> On 2015/12/1 18:40, Dr. David Alan Gilbert wrote:
>> * Wen Congyang (wency@cn.fujitsu.com) wrote:
>>> Block replication is a very important feature which is used for
>>> continuous checkpoints(for example: COLO).
>>>
>>> You can get the detailed information about block replication from here:
>>> http://wiki.qemu.org/Features/BlockReplication
>>>
>>> Usage:
>>> Please refer to docs/block-replication.txt
>>>
>>> This patch series is based on the following patch series:
>>> 1. http://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg04949.html
>>> 2. http://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg06043.html
>>>
>>> You can get the patch here:
>>> https://github.com/coloft/qemu/tree/wency/block-replication-v12
>>>
>>> You can get the patch with framework here:
>>> https://github.com/coloft/qemu/tree/wency/colo_framework_v11.2
>>
>> Neither of these links work for me, and I see that  only messages 0..7 in the
>> series hit the list.
>>
> 
> Hi Dave,
> 
> You can refer to https://github.com/coloft/qemu/tree/colo-v2.2-periodic-mode,
> The block replication part in this link is also the newest version.

No, I remove one patch, and the usage is changed.

Thanks
Wen Congyang

> 
> Congyang has deleted this confused branch, we will pay attention to this later in next version.
> 
> Thanks,
> Hailiang
> 
>>
>>>
>>> TODO:
>>> 1. Continuous block replication. It will be started after basic functions
>>>     are accepted.
>>>
>>> Changs Log:
>>> V12:
>>> 1. Rebase to the newest codes
>>> 2. Use backing reference to replcace 'allow-write-backing-file'
>>> V11:
>>> 1. Reopen the backing file when starting blcok replication if it is not
>>>     opened in R/W mode
>>> 2. Unblock BLOCK_OP_TYPE_BACKUP_SOURCE and BLOCK_OP_TYPE_BACKUP_TARGET
>>>     when opening backing file
>>> 3. Block the top BDS so there is only one block job for the top BDS and
>>>     its backing chain.
>>> V10:
>>> 1. Use blockdev-remove-medium and blockdev-insert-medium to replace backing
>>>     reference.
>>> 2. Address the comments from Eric Blake
>>> V9:
>>> 1. Update the error messages
>>> 2. Rebase to the newest qemu
>>> 3. Split child add/delete support. These patches are sent in another patchset.
>>> V8:
>>> 1. Address Alberto Garcia's comments
>>> V7:
>>> 1. Implement adding/removing quorum child. Remove the option non-connect.
>>> 2. Simplify the backing refrence option according to Stefan Hajnoczi's suggestion
>>> V6:
>>> 1. Rebase to the newest qemu.
>>> V5:
>>> 1. Address the comments from Gong Lei
>>> 2. Speed the failover up. The secondary vm can take over very quickly even
>>>     if there are too many I/O requests.
>>> V4:
>>> 1. Introduce a new driver replication to avoid touch nbd and qcow2.
>>> V3:
>>> 1: use error_setg() instead of error_set()
>>> 2. Add a new block job API
>>> 3. Active disk, hidden disk and nbd target uses the same AioContext
>>> 4. Add a testcase to test new hbitmap API
>>> V2:
>>> 1. Redesign the secondary qemu(use image-fleecing)
>>> 2. Use Error objects to return error message
>>> 3. Address the comments from Max Reitz and Eric Blake
>>>
>>> Wen Congyang (10):
>>>    unblock backup operations in backing file
>>>    Store parent BDS in BdrvChild
>>>    Backup: clear all bitmap when doing block checkpoint
>>>    Allow creating backup jobs when opening BDS
>>>    docs: block replication's description
>>>    Add new block driver interfaces to control block replication
>>>    quorum: implement block driver interfaces for block replication
>>>    Implement new driver for block replication
>>>    support replication driver in blockdev-add
>>>    Add a new API to start/stop replication, do checkpoint to all BDSes
>>>
>>>   block.c                    | 145 ++++++++++++
>>>   block/Makefile.objs        |   3 +-
>>>   block/backup.c             |  14 ++
>>>   block/quorum.c             |  78 +++++++
>>>   block/replication.c        | 549 +++++++++++++++++++++++++++++++++++++++++++++
>>>   blockjob.c                 |  11 +
>>>   docs/block-replication.txt | 227 +++++++++++++++++++
>>>   include/block/block.h      |   9 +
>>>   include/block/block_int.h  |  15 ++
>>>   include/block/blockjob.h   |  12 +
>>>   qapi/block-core.json       |  34 ++-
>>>   11 files changed, 1093 insertions(+), 4 deletions(-)
>>>   create mode 100644 block/replication.c
>>>   create mode 100644 docs/block-replication.txt
>>>
>>> -- 
>>> 2.5.0
>>>
>>>
>>>
>> -- 
>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>
>>
>> .
>>
> 
> 
> 
> 
> .
> 

Re: [Qemu-devel] [Patch v12 00/10] Block replication for continuous checkpoints

[Wen Congyang]

On 12/02/2015 09:00 AM, Wen Congyang wrote:
> On 12/01/2015 06:40 PM, Dr. David Alan Gilbert wrote:
>> * Wen Congyang (wency@cn.fujitsu.com) wrote:
>>> Block replication is a very important feature which is used for
>>> continuous checkpoints(for example: COLO).
>>>
>>> You can get the detailed information about block replication from here:
>>> http://wiki.qemu.org/Features/BlockReplication
>>>
>>> Usage:
>>> Please refer to docs/block-replication.txt
>>>
>>> This patch series is based on the following patch series:
>>> 1. http://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg04949.html
>>> 2. http://lists.nongnu.org/archive/html/qemu-devel/2015-11/msg06043.html
>>>
>>> You can get the patch here:
>>> https://github.com/coloft/qemu/tree/wency/block-replication-v12
>>>
>>> You can get the patch with framework here:
>>> https://github.com/coloft/qemu/tree/wency/colo_framework_v11.2
>>
>> Neither of these links work for me, and I see that  only messages 0..7 in the
>> series hit the list.
> 
> I forgot to push it to github...
> And I also received the messages 0..7, and I don't know what's wrong...

The reason is that: git send-email has a bug:
http://permalink.gmane.org/gmane.comp.version-control.git/274569

Thanks
Wen Congyang

> 
> I will push it to github, and resend them.
> 
> Thanks
> Wen Congyang
> 
>>
>> Dave
>>
>>>
>>> TODO:
>>> 1. Continuous block replication. It will be started after basic functions
>>>    are accepted.
>>>
>>> Changs Log:
>>> V12:
>>> 1. Rebase to the newest codes
>>> 2. Use backing reference to replcace 'allow-write-backing-file'
>>> V11:
>>> 1. Reopen the backing file when starting blcok replication if it is not
>>>    opened in R/W mode
>>> 2. Unblock BLOCK_OP_TYPE_BACKUP_SOURCE and BLOCK_OP_TYPE_BACKUP_TARGET
>>>    when opening backing file
>>> 3. Block the top BDS so there is only one block job for the top BDS and
>>>    its backing chain.
>>> V10:
>>> 1. Use blockdev-remove-medium and blockdev-insert-medium to replace backing
>>>    reference.
>>> 2. Address the comments from Eric Blake
>>> V9:
>>> 1. Update the error messages
>>> 2. Rebase to the newest qemu
>>> 3. Split child add/delete support. These patches are sent in another patchset.
>>> V8:
>>> 1. Address Alberto Garcia's comments
>>> V7:
>>> 1. Implement adding/removing quorum child. Remove the option non-connect.
>>> 2. Simplify the backing refrence option according to Stefan Hajnoczi's suggestion
>>> V6:
>>> 1. Rebase to the newest qemu.
>>> V5:
>>> 1. Address the comments from Gong Lei
>>> 2. Speed the failover up. The secondary vm can take over very quickly even
>>>    if there are too many I/O requests.
>>> V4:
>>> 1. Introduce a new driver replication to avoid touch nbd and qcow2.
>>> V3:
>>> 1: use error_setg() instead of error_set()
>>> 2. Add a new block job API
>>> 3. Active disk, hidden disk and nbd target uses the same AioContext
>>> 4. Add a testcase to test new hbitmap API
>>> V2:
>>> 1. Redesign the secondary qemu(use image-fleecing)
>>> 2. Use Error objects to return error message
>>> 3. Address the comments from Max Reitz and Eric Blake
>>>
>>> Wen Congyang (10):
>>>   unblock backup operations in backing file
>>>   Store parent BDS in BdrvChild
>>>   Backup: clear all bitmap when doing block checkpoint
>>>   Allow creating backup jobs when opening BDS
>>>   docs: block replication's description
>>>   Add new block driver interfaces to control block replication
>>>   quorum: implement block driver interfaces for block replication
>>>   Implement new driver for block replication
>>>   support replication driver in blockdev-add
>>>   Add a new API to start/stop replication, do checkpoint to all BDSes
>>>
>>>  block.c                    | 145 ++++++++++++
>>>  block/Makefile.objs        |   3 +-
>>>  block/backup.c             |  14 ++
>>>  block/quorum.c             |  78 +++++++
>>>  block/replication.c        | 549 +++++++++++++++++++++++++++++++++++++++++++++
>>>  blockjob.c                 |  11 +
>>>  docs/block-replication.txt | 227 +++++++++++++++++++
>>>  include/block/block.h      |   9 +
>>>  include/block/block_int.h  |  15 ++
>>>  include/block/blockjob.h   |  12 +
>>>  qapi/block-core.json       |  34 ++-
>>>  11 files changed, 1093 insertions(+), 4 deletions(-)
>>>  create mode 100644 block/replication.c
>>>  create mode 100644 docs/block-replication.txt
>>>
>>> -- 
>>> 2.5.0
>>>
>>>
>>>
>> --
>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>
>>
>> .
>>
> 
> 
> 
> 
> .
>