Re: [Qemu-devel] [PATCH v2 3/3] Update Documentation

Re: [Qemu-devel] [PATCH v2 3/3] Update Documentation

[Jing-Wei Su]

Hi, Lukas

If  PVM died, to let old  SVM be the new PVM, the old SVM's
filter-redirectors(f1 and f2) are removed in your document.
Why don't you remove the filter-rewriter from the old SVM?
The filter-rewrite is kept in the new PVM. What is its role in the new PVM?

Thanks!

Sincerely,
Jing-Wei

Re: [Qemu-devel] [PATCH v2 3/3] Update Documentation

[Lukas Straub]

On Wed, 26 Feb 2020 23:59:22 +0800
Jing-Wei Su <jwsu1986@gmail.com> wrote:

> Hi, Lukas
>
> If  PVM died, to let old  SVM be the new PVM, the old SVM's
> filter-redirectors(f1 and f2) are removed in your document.
> Why don't you remove the filter-rewriter from the old SVM?
> The filter-rewrite is kept in the new PVM. What is its role in the new PVM?
>
> Thanks!
>
> Sincerely,
> Jing-Wei

Hi,
The filter-rewriter rewrites the TCP-Packets to increase similarity of PVM and SVM, after the SVM takes over it needs to continue the rewriting, otherwise it would break the TCP connections.

Regards,
Lukas Straub

[Qemu-devel] [PATCH v2 0/3] colo: Add support for continious replication

[Lukas Straub]

Hello Everyone,
These Patches add support for continious replication to colo.
Please review.

Regards,
Lukas Straub

v2:
 - fix email formating
 - fix checkpatch.pl warnings
 - fix patchew error
 - clearer commit messages

Lukas Straub (3):
  Replication: Ignore requests after failover
  net/filter.c: Add Options to insert filters anywhere in the filter
    list
  Update Documentation

 block/replication.c  |  38 ++++++++-
 docs/COLO-FT.txt     | 185 ++++++++++++++++++++++++++++++++-----------
 include/net/filter.h |   2 +
 net/filter.c         |  71 ++++++++++++++++-
 qemu-options.hx      |  10 +--
 5 files changed, 250 insertions(+), 56 deletions(-)

--
2.20.1

[Qemu-devel] [PATCH v2 3/3] Update Documentation

[Lukas Straub]

Document the qemu command-line and qmp commands for continious replication

Signed-off-by: Lukas Straub <lukasstraub2@web.de>
---
 docs/COLO-FT.txt | 185 +++++++++++++++++++++++++++++++++++------------
 1 file changed, 138 insertions(+), 47 deletions(-)

diff --git a/docs/COLO-FT.txt b/docs/COLO-FT.txt
index ad24680d13..c08bfbd3a8 100644
--- a/docs/COLO-FT.txt
+++ b/docs/COLO-FT.txt
@@ -145,35 +145,64 @@ The diagram just shows the main qmp command, you can get the detail
 in test procedure.
 
 == Test procedure ==
+Note: Here we are running both instances on the same Machine for testing, 
+change the IP Addresses if you want to run it on two Hosts
+
 1. Startup qemu
 Primary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name primary \
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=virtio,id=primary-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
-         children.0.file.filename=1.raw,\
-         children.0.driver=raw -S
+# imagefolder="/mnt/vms/colo-test"
+
+# cp --reflink=auto $imagefolder/primary.qcow2 $imagefolder/primary-copy.qcow2
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp 1 -qmp stdio \
+   -vnc :0 -k de -device piix3-usb-uhci -device usb-tablet -name primary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper \
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=mirror0,host=127.0.0.1,port=9003,server,nowait \
+   -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \
+   -chardev socket,id=compare0,host=127.0.0.1,port=9001,server,nowait \
+   -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
+   -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server,nowait \
+   -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
+   -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
+   -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
+   -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
+   -object iothread,id=iothread1 \
+   -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,\
+outdev=compare_out0,iothread=iothread1 \
+   -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0.file.filename=$imagefolder/primary.qcow2,children.0.driver=qcow2 -S
+
 Secondary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name secondary \
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=none,id=secondary-disk0,file.filename=1.raw,driver=raw,node-name=node0 \
-  -drive if=virtio,id=active-disk0,driver=replication,mode=secondary,\
-         file.driver=qcow2,top-id=active-disk0,\
-         file.file.filename=/mnt/ramfs/active_disk.img,\
-         file.backing.driver=qcow2,\
-         file.backing.file.filename=/mnt/ramfs/hidden_disk.img,\
-         file.backing.backing=secondary-disk0 \
-  -incoming tcp:0:8888
+# imagefolder="/mnt/vms/colo-test"
+
+# qemu-img create -f qcow2 $imagefolder/secondary-active.qcow2 10G
+
+# qemu-img create -f qcow2 $imagefolder/secondary-hidden.qcow2 10G
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp 1 -qmp stdio \
+   -vnc :1 -k de -device piix3-usb-uhci -device usb-tablet -name secondary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper \
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=red0,host=127.0.0.1,port=9003,reconnect=1 \
+   -chardev socket,id=red1,host=127.0.0.1,port=9004,reconnect=1 \
+   -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
+   -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
+   -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
+   -drive if=none,id=parent0,file.filename=$imagefolder/primary-copy.qcow2,driver=qcow2 \
+   -drive if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,\
+top-id=childs0,file.file.filename=$imagefolder/secondary-active.qcow2,\
+file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/secondary-hidden.qcow2,\
+file.backing.backing=parent0 \
+   -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0.file=childs0,children.0.driver=raw \
+   -incoming tcp:0:9998
+
 
 2. On Secondary VM's QEMU monitor, issue command
 {'execute':'qmp_capabilities'}
-{ 'execute': 'nbd-server-start',
-  'arguments': {'addr': {'type': 'inet', 'data': {'host': 'xx.xx.xx.xx', 'port': '8889'} } }
-}
-{'execute': 'nbd-server-add', 'arguments': {'device': 'secondary-disk0', 'writable': true } }
+{'execute': 'nbd-server-start', 'arguments': {'addr': {'type': 'inet', 'data': {'host': '127.0.0.1', 'port': '9999'} } } }
+{'execute': 'nbd-server-add', 'arguments': {'device': 'parent0', 'writable': true } }
 
 Note:
   a. The qmp command nbd-server-start and nbd-server-add must be run
@@ -184,12 +213,10 @@ Note:
 
 3. On Primary VM's QEMU monitor, issue command:
 {'execute':'qmp_capabilities'}
-{ 'execute': 'human-monitor-command',
-  'arguments': {'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=xx.xx.xx.xx,file.port=8889,file.export=secondary-disk0,node-name=nbd_client0'}}
-{ 'execute':'x-blockdev-change', 'arguments':{'parent': 'primary-disk0', 'node': 'nbd_client0' } }
-{ 'execute': 'migrate-set-capabilities',
-      'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
-{ 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888' } }
+{'execute': 'human-monitor-command', 'arguments': {'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,file.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{'parent': 'colo-disk0', 'node': 'replication0' } }
+{'execute': 'migrate-set-capabilities', 'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments': {'uri': 'tcp:127.0.0.1:9998' } }
 
   Note:
   a. There should be only one NBD Client for each primary disk.
@@ -199,27 +226,91 @@ Note:
 
 4. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
 You can issue command '{ "execute": "migrate-set-parameters" , "arguments":{ "x-checkpoint-delay": 2000 } }'
-to change the checkpoint period time
+to change the idle checkpoint period time
 
 5. Failover test
-You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM's
-monitor at the same time, then SVM will failover and client will not detect this
-change.
+You can kill one of the VMs and Failover on the surviving VM: 
 
-Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have to
-issue block related command to stop block replication.
-Primary:
-  Remove the nbd child from the quorum:
-  { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0', 'child': 'children.1'}}
-  { 'execute': 'human-monitor-command','arguments': {'command-line': 'drive_del blk-buddy0'}}
-  Note: there is no qmp command to remove the blockdev now
+== Primary Failover ==
+The Secondary died, resume on the Primary
 
-Secondary:
-  The primary host is down, so we should do the following thing:
-  { 'execute': 'nbd-server-stop' }
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'child': 'children.1'} }
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_del replication0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'comp0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'iothread1' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'm0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire1' } }
+{'execute': 'x-colo-lost-heartbeat' }
+
+== Secondary Failover ==
+The Primary died, resume on the Secondary and prepare to become the new Primary
+
+{'execute': 'nbd-server-stop'}
+{'execute': 'x-colo-lost-heartbeat'}
+
+{'execute': 'object-del', 'arguments':{ 'id': 'f2' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'f1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red0' } }
+
+{'execute': 'chardev-add', 'arguments':{ 'id': 'mirror0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9003' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare1', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9004' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9001' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0-0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9001' } }, 'server': false } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': false } } } }
+
+== Primary resume replication ==
+Resume replication after new Secondary is up.
+
+Start the new Secondary, then:
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0', 'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode': 'existing', 'format': 'raw', 'sync': 'full'} }
+
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync'} }
+
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,file.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'node': 'replication0' } }
+
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror', 'id': 'm0', 'props': { 'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire0', 'props': { 'netdev': 'hn0', 'queue': 'rx', 'indev': 'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire1', 'props': { 'netdev': 'hn0', 'queue': 'rx', 'outdev': 'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id': 'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare', 'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in': 'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
+
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
+
+Note:
+If this Primary previously was a Secondary, then we need to insert the 
+filters before the filter-rewriter by using the 
+"'insert': 'before', 'position': 'rew0'" Options. See below.
+
+== Secondary resume replication ==
+Become Primary and resume replication after new Secondary is up.
+
+Start the new Secondary, then:
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0', 'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode': 'existing', 'format': 'raw', 'sync': 'full'} }
+
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync' } }
+
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,file.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'node': 'replication0' } }
+
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror', 'id': 'm0', 'props': { 'insert': 'before', 'position': 'rew0', 'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire0', 'props': { 'insert': 'before', 'position': 'rew0', 'netdev': 'hn0', 'queue': 'rx', 'indev': 'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire1', 'props': { 'insert': 'before', 'position': 'rew0', 'netdev': 'hn0', 'queue': 'rx', 'outdev': 'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id': 'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare', 'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in': 'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
+
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
 
 == TODO ==
-1. Support continuous VM replication.
-2. Support shared storage.
-3. Develop the heartbeat part.
-4. Reduce checkpoint VM’s downtime while doing checkpoint.
+1. Support shared storage.
+2. Develop the heartbeat part.
+3. Reduce checkpoint VM’s downtime while doing checkpoint.
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 3/3] Update Documentation

[Markus Armbruster]

Subject "Update Documentation" is too vague, because (1) it lacks a
subsystem prefix, and (2) it fails to hint at what it is updated for.
Suggest something like "colo: Update documentation for FIXME", or maybe
"docs/COLO-FT.txt: Update for FIXME", with a suitable replacement for
FIXME.

Re: [Qemu-devel] [PATCH v2 3/3] Update Documentation

[Zhang, Chen]

Hi Lukas,

Please address Markus's comments change this patch's name in next version.
And I wrote some comments behind, please check it.
Firstly, Please remove all the trailing whitespace in this patch.


> -----Original Message-----
> From: Lukas Straub <lukasstraub2@web.de>
> Sent: Friday, August 16, 2019 2:49 AM
> To: qemu-devel <qemu-devel@nongnu.org>
> Cc: Zhang, Chen <chen.zhang@intel.com>; Jason Wang
> <jasowang@redhat.com>; Wen Congyang <wencongyang2@huawei.com>;
> Xie Changlong <xiechanglong.d@gmail.com>
> Subject: [PATCH v2 3/3] Update Documentation
> 
> Document the qemu command-line and qmp commands for continious
> replication
> 
> Signed-off-by: Lukas Straub <lukasstraub2@web.de>
> ---
>  docs/COLO-FT.txt | 185 +++++++++++++++++++++++++++++++++++--------
> ----
>  1 file changed, 138 insertions(+), 47 deletions(-)
> 
> diff --git a/docs/COLO-FT.txt b/docs/COLO-FT.txt index
> ad24680d13..c08bfbd3a8 100644
> --- a/docs/COLO-FT.txt
> +++ b/docs/COLO-FT.txt
> @@ -145,35 +145,64 @@ The diagram just shows the main qmp command,
> you can get the detail  in test procedure.
> 
>  == Test procedure ==
> +Note: Here we are running both instances on the same Machine for
> +testing, change the IP Addresses if you want to run it on two Hosts
> +
>  1. Startup qemu
>  Primary:
> -# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name
> primary \
> -  -device piix3-usb-uhci -vnc :7 \
> -  -device usb-tablet -netdev tap,id=hn0,vhost=off \
> -  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
> -  -drive if=virtio,id=primary-disk0,driver=quorum,read-pattern=fifo,vote-
> threshold=1,\
> -         children.0.file.filename=1.raw,\
> -         children.0.driver=raw -S
> +# imagefolder="/mnt/vms/colo-test"
> +
> +# cp --reflink=auto $imagefolder/primary.qcow2
> +$imagefolder/primary-copy.qcow2

I think you can tell other people here that we need two same disk image before COLO startup.
The name "primary and primary-copy" will make reader very confused.

> +
> +# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp
> 1 -qmp stdio \
> +   -vnc :0 -k de -device piix3-usb-uhci -device usb-tablet -name primary \

What's mean of the "-k de" ?

> +   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper
> \
> +   -device rtl8139,id=e0,netdev=hn0 \
> +   -chardev socket,id=mirror0,host=127.0.0.1,port=9003,server,nowait \
> +   -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \
> +   -chardev socket,id=compare0,host=127.0.0.1,port=9001,server,nowait \
> +   -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
> +   -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server,nowait
> \
> +   -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
> +   -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
> +   -object filter-
> redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
> +   -object filter-
> redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
> +   -object iothread,id=iothread1 \
> +   -object
> +colo-compare,id=comp0,primary_in=compare0-
> 0,secondary_in=compare1,\
> +outdev=compare_out0,iothread=iothread1 \

Please keep the space style.

> +   -drive
> +if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
> +children.0.file.filename=$imagefolder/primary.qcow2,children.0.driver=q
> +cow2 -S
> +
>  Secondary:
> -# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name
> secondary \
> -  -device piix3-usb-uhci -vnc :7 \
> -  -device usb-tablet -netdev tap,id=hn0,vhost=off \
> -  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
> -  -drive if=none,id=secondary-disk0,file.filename=1.raw,driver=raw,node-
> name=node0 \
> -  -drive if=virtio,id=active-disk0,driver=replication,mode=secondary,\
> -         file.driver=qcow2,top-id=active-disk0,\
> -         file.file.filename=/mnt/ramfs/active_disk.img,\
> -         file.backing.driver=qcow2,\
> -         file.backing.file.filename=/mnt/ramfs/hidden_disk.img,\
> -         file.backing.backing=secondary-disk0 \
> -  -incoming tcp:0:8888
> +# imagefolder="/mnt/vms/colo-test"
> +
> +# qemu-img create -f qcow2 $imagefolder/secondary-active.qcow2 10G
> +
> +# qemu-img create -f qcow2 $imagefolder/secondary-hidden.qcow2 10G
> +
> +# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp
> 1 -qmp stdio \
> +   -vnc :1 -k de -device piix3-usb-uhci -device usb-tablet -name secondary \
> +   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper
> \
> +   -device rtl8139,id=e0,netdev=hn0 \
> +   -chardev socket,id=red0,host=127.0.0.1,port=9003,reconnect=1 \
> +   -chardev socket,id=red1,host=127.0.0.1,port=9004,reconnect=1 \
> +   -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
> +   -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
> +   -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
> +   -drive if=none,id=parent0,file.filename=$imagefolder/primary-
> copy.qcow2,driver=qcow2 \
> +   -drive
> +if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,
> +\
> +top-id=childs0,file.file.filename=$imagefolder/secondary-active.qcow2,\
> +file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/secon
> +dary-hidden.qcow2,\
> +file.backing.backing=parent0 \

Please keep the space style.

> +   -drive
> +if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
> +children.0.file=childs0,children.0.driver=raw \
> +   -incoming tcp:0:9998
> +

Here, you changed the original block design, can you explain the reasons why we need to do this?
Please draw a ASCII diagram to describe COLO block detail.

> 
>  2. On Secondary VM's QEMU monitor, issue command
> {'execute':'qmp_capabilities'} -{ 'execute': 'nbd-server-start',
> -  'arguments': {'addr': {'type': 'inet', 'data': {'host': 'xx.xx.xx.xx', 'port':
> '8889'} } } -}
> -{'execute': 'nbd-server-add', 'arguments': {'device': 'secondary-disk0',
> 'writable': true } }
> +{'execute': 'nbd-server-start', 'arguments': {'addr': {'type': 'inet',
> +'data': {'host': '127.0.0.1', 'port': '9999'} } } }
> +{'execute': 'nbd-server-add', 'arguments': {'device': 'parent0',
> +'writable': true } }
> 
>  Note:
>    a. The qmp command nbd-server-start and nbd-server-add must be run
> @@ -184,12 +213,10 @@ Note:
> 
>  3. On Primary VM's QEMU monitor, issue command:
>  {'execute':'qmp_capabilities'}
> -{ 'execute': 'human-monitor-command',
> -  'arguments': {'command-line': 'drive_add -n buddy
> driver=replication,mode=primary,file.driver=nbd,file.host=xx.xx.xx.xx,file.p
> ort=8889,file.export=secondary-disk0,node-name=nbd_client0'}}
> -{ 'execute':'x-blockdev-change', 'arguments':{'parent': 'primary-disk0',
> 'node': 'nbd_client0' } } -{ 'execute': 'migrate-set-capabilities',
> -      'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
> -{ 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888' } }
> +{'execute': 'human-monitor-command', 'arguments': {'command-line':
> +'drive_add -n buddy
> +driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,fil
> +e.port=9999,file.export=parent0,node-name=replication0'}}
> +{'execute': 'x-blockdev-change', 'arguments':{'parent': 'colo-disk0',
> +'node': 'replication0' } }
> +{'execute': 'migrate-set-capabilities', 'arguments': {'capabilities': [
> +{'capability': 'x-colo', 'state': true } ] } }
> +{'execute': 'migrate', 'arguments': {'uri': 'tcp:127.0.0.1:9998' } }
> 
>    Note:
>    a. There should be only one NBD Client for each primary disk.
> @@ -199,27 +226,91 @@ Note:
> 
>  4. After the above steps, you will see, whenever you make changes to PVM,
> SVM will be synced.
>  You can issue command '{ "execute": "migrate-set-parameters" ,
> "arguments":{ "x-checkpoint-delay": 2000 } }'
> -to change the checkpoint period time
> +to change the idle checkpoint period time
> 
>  5. Failover test
> -You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM's -
> monitor at the same time, then SVM will failover and client will not detect
> this -change.
> +You can kill one of the VMs and Failover on the surviving VM:
> 
> -Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have to
> -issue block related command to stop block replication.
> -Primary:
> -  Remove the nbd child from the quorum:
> -  { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0', 'child':
> 'children.1'}}
> -  { 'execute': 'human-monitor-command','arguments': {'command-line':
> 'drive_del blk-buddy0'}}
> -  Note: there is no qmp command to remove the blockdev now
> +== Primary Failover ==
> +The Secondary died, resume on the Primary
> 
> -Secondary:
> -  The primary host is down, so we should do the following thing:
> -  { 'execute': 'nbd-server-stop' }
> +{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
> +'child': 'children.1'} }
> +{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
> +'drive_del replication0' } }
> +{'execute': 'object-del', 'arguments':{ 'id': 'comp0' } }
> +{'execute': 'object-del', 'arguments':{ 'id': 'iothread1' } }
> +{'execute': 'object-del', 'arguments':{ 'id': 'm0' } }
> +{'execute': 'object-del', 'arguments':{ 'id': 'redire0' } }
> +{'execute': 'object-del', 'arguments':{ 'id': 'redire1' } }
> +{'execute': 'x-colo-lost-heartbeat' }
> +
> +== Secondary Failover ==
> +The Primary died, resume on the Secondary and prepare to become the
> new
> +Primary
> +
> +{'execute': 'nbd-server-stop'}
> +{'execute': 'x-colo-lost-heartbeat'}
> +
> +{'execute': 'object-del', 'arguments':{ 'id': 'f2' } }
> +{'execute': 'object-del', 'arguments':{ 'id': 'f1' } }
> +{'execute': 'chardev-remove', 'arguments':{ 'id': 'red1' } }
> +{'execute': 'chardev-remove', 'arguments':{ 'id': 'red0' } }
> +
> +{'execute': 'chardev-add', 'arguments':{ 'id': 'mirror0', 'backend':
> +{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
> +'127.0.0.1', 'port': '9003' } }, 'server': true } } } }
> +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare1', 'backend':
> +{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
> +'127.0.0.1', 'port': '9004' } }, 'server': true } } } }
> +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0', 'backend':
> +{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
> +'127.0.0.1', 'port': '9001' } }, 'server': true } } } }
> +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0-0', 'backend':
> +{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
> +'127.0.0.1', 'port': '9001' } }, 'server': false } } } }
> +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out',
> +'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
> +'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': true } } }
> +}
> +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out0',
> +'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
> +'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': false } }
> +} }
> +
> +== Primary resume replication ==
> +Resume replication after new Secondary is up.
> +
> +Start the new Secondary, then:
> +{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0',
> +'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode':
> +'existing', 'format': 'raw', 'sync': 'full'} }

I think the original secondary node has stopped the nbd server in failover:
> +{'execute': 'nbd-server-stop'}
> +{'execute': 'x-colo-lost-heartbeat'}

Why you can connect to it?

And please add new Secondary node start up script here,
I think it is different with original secondary node script.
same issue in resume secondary.

> +
> +Wait until disk is synced, then:
> +{'execute': 'stop'}
> +{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync'} }
> +
> +{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
> +'drive_add -n buddy
> +driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,fil
> +e.port=9999,file.export=parent0,node-name=replication0'}}
> +{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
> +'node': 'replication0' } }
> +
> +{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror',
> +'id': 'm0', 'props': { 'netdev': 'hn0', 'queue': 'tx', 'outdev':
> +'mirror0' } } }
> +{'execute': 'object-add', 'arguments':{ 'qom-type':
> +'filter-redirector', 'id': 'redire0', 'props': { 'netdev': 'hn0',
> +'queue': 'rx', 'indev': 'compare_out' } } }
> +{'execute': 'object-add', 'arguments':{ 'qom-type':
> +'filter-redirector', 'id': 'redire1', 'props': { 'netdev': 'hn0',
> +'queue': 'rx', 'outdev': 'compare0' } } }
> +{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id':
> +'iothread1' } }
> +{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare',
> +'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in':
> +'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
> +
> +{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [
> +{'capability': 'x-colo', 'state': true } ] } }
> +{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
> +
> +Note:
> +If this Primary previously was a Secondary, then we need to insert the
> +filters before the filter-rewriter by using the
> +"'insert': 'before', 'position': 'rew0'" Options. See below.
> +
> +== Secondary resume replication ==
> +Become Primary and resume replication after new Secondary is up.
> +
> +Start the new Secondary, then:
> +{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0',
> +'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode':
> +'existing', 'format': 'raw', 'sync': 'full'} }

Same issue as primary resume.

I want to re-tested the resume function but got stuck here.

Thanks
Zhang Chen

> +
> +Wait until disk is synced, then:
> +{'execute': 'stop'}
> +{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync' } }
> +
> +{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
> +'drive_add -n buddy
> +driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,fil
> +e.port=9999,file.export=parent0,node-name=replication0'}}
> +{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
> +'node': 'replication0' } }
> +
> +{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror',
> +'id': 'm0', 'props': { 'insert': 'before', 'position': 'rew0',
> +'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
> +{'execute': 'object-add', 'arguments':{ 'qom-type':
> +'filter-redirector', 'id': 'redire0', 'props': { 'insert': 'before',
> +'position': 'rew0', 'netdev': 'hn0', 'queue': 'rx', 'indev':
> +'compare_out' } } }
> +{'execute': 'object-add', 'arguments':{ 'qom-type':
> +'filter-redirector', 'id': 'redire1', 'props': { 'insert': 'before',
> +'position': 'rew0', 'netdev': 'hn0', 'queue': 'rx', 'outdev':
> +'compare0' } } }
> +{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id':
> +'iothread1' } }
> +{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare',
> +'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in':
> +'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
> +
> +{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [
> +{'capability': 'x-colo', 'state': true } ] } }
> +{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
> 
>  == TODO ==
> -1. Support continuous VM replication.
> -2. Support shared storage.
> -3. Develop the heartbeat part.
> -4. Reduce checkpoint VM’s downtime while doing checkpoint.
> +1. Support shared storage.
> +2. Develop the heartbeat part.
> +3. Reduce checkpoint VM’s downtime while doing checkpoint.
> --
> 2.20.1

Re: [Qemu-devel] [PATCH v2 3/3] Update Documentation

[Lukas Straub]

On Mon, 2 Sep 2019 12:17:43 +0000
"Zhang, Chen" <chen.zhang@intel.com> wrote:

> Hi Lukas,
> 
> Please address Markus's comments change this patch's name in next version.
> And I wrote some comments behind, please check it.
> Firstly, Please remove all the trailing whitespace in this patch.
> 
> 
> > -----Original Message-----
> > From: Lukas Straub <lukasstraub2@web.de>
> > Sent: Friday, August 16, 2019 2:49 AM
> > To: qemu-devel <qemu-devel@nongnu.org>
> > Cc: Zhang, Chen <chen.zhang@intel.com>; Jason Wang
> > <jasowang@redhat.com>; Wen Congyang <wencongyang2@huawei.com>;
> > Xie Changlong <xiechanglong.d@gmail.com>
> > Subject: [PATCH v2 3/3] Update Documentation
> > 
> > Document the qemu command-line and qmp commands for continious
> > replication
> > 
> > Signed-off-by: Lukas Straub <lukasstraub2@web.de>
> > ---
> >  docs/COLO-FT.txt | 185 +++++++++++++++++++++++++++++++++++--------
> > ----
> >  1 file changed, 138 insertions(+), 47 deletions(-)
> > 
> > diff --git a/docs/COLO-FT.txt b/docs/COLO-FT.txt index
> > ad24680d13..c08bfbd3a8 100644
> > --- a/docs/COLO-FT.txt
> > +++ b/docs/COLO-FT.txt
> > @@ -145,35 +145,64 @@ The diagram just shows the main qmp command,
> > you can get the detail  in test procedure.
> > 
> >  == Test procedure ==
> > +Note: Here we are running both instances on the same Machine for
> > +testing, change the IP Addresses if you want to run it on two Hosts
> > +
> >  1. Startup qemu
> >  Primary:
> > -# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name
> > primary \
> > -  -device piix3-usb-uhci -vnc :7 \
> > -  -device usb-tablet -netdev tap,id=hn0,vhost=off \
> > -  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
> > -  -drive if=virtio,id=primary-disk0,driver=quorum,read-pattern=fifo,vote-
> > threshold=1,\
> > -         children.0.file.filename=1.raw,\
> > -         children.0.driver=raw -S
> > +# imagefolder="/mnt/vms/colo-test"
> > +
> > +# cp --reflink=auto $imagefolder/primary.qcow2
> > +$imagefolder/primary-copy.qcow2  
> 
> I think you can tell other people here that we need two same disk image before COLO startup.
> The name "primary and primary-copy" will make reader very confused.
> 
> > +
> > +# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp
> > 1 -qmp stdio \
> > +   -vnc :0 -k de -device piix3-usb-uhci -device usb-tablet -name primary \  
> 
> What's mean of the "-k de" ?
Hi,

Oops, thats for German Keyboard layout, will remove.

> > +   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper
> > \
> > +   -device rtl8139,id=e0,netdev=hn0 \
> > +   -chardev socket,id=mirror0,host=127.0.0.1,port=9003,server,nowait \
> > +   -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \
> > +   -chardev socket,id=compare0,host=127.0.0.1,port=9001,server,nowait \
> > +   -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
> > +   -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server,nowait
> > \
> > +   -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
> > +   -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
> > +   -object filter-
> > redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
> > +   -object filter-
> > redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
> > +   -object iothread,id=iothread1 \
> > +   -object
> > +colo-compare,id=comp0,primary_in=compare0-
> > 0,secondary_in=compare1,\
> > +outdev=compare_out0,iothread=iothread1 \  
> 
> Please keep the space style.
> 
> > +   -drive
> > +if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
> > +children.0.file.filename=$imagefolder/primary.qcow2,children.0.driver=q
> > +cow2 -S
> > +
> >  Secondary:
> > -# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name
> > secondary \
> > -  -device piix3-usb-uhci -vnc :7 \
> > -  -device usb-tablet -netdev tap,id=hn0,vhost=off \
> > -  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
> > -  -drive if=none,id=secondary-disk0,file.filename=1.raw,driver=raw,node-
> > name=node0 \
> > -  -drive if=virtio,id=active-disk0,driver=replication,mode=secondary,\
> > -         file.driver=qcow2,top-id=active-disk0,\
> > -         file.file.filename=/mnt/ramfs/active_disk.img,\
> > -         file.backing.driver=qcow2,\
> > -         file.backing.file.filename=/mnt/ramfs/hidden_disk.img,\
> > -         file.backing.backing=secondary-disk0 \
> > -  -incoming tcp:0:8888
> > +# imagefolder="/mnt/vms/colo-test"
> > +
> > +# qemu-img create -f qcow2 $imagefolder/secondary-active.qcow2 10G
> > +
> > +# qemu-img create -f qcow2 $imagefolder/secondary-hidden.qcow2 10G
> > +
> > +# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp
> > 1 -qmp stdio \
> > +   -vnc :1 -k de -device piix3-usb-uhci -device usb-tablet -name secondary \
> > +   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper
> > \
> > +   -device rtl8139,id=e0,netdev=hn0 \
> > +   -chardev socket,id=red0,host=127.0.0.1,port=9003,reconnect=1 \
> > +   -chardev socket,id=red1,host=127.0.0.1,port=9004,reconnect=1 \
> > +   -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
> > +   -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
> > +   -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
> > +   -drive if=none,id=parent0,file.filename=$imagefolder/primary-
> > copy.qcow2,driver=qcow2 \
> > +   -drive
> > +if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,
> > +\
> > +top-id=childs0,file.file.filename=$imagefolder/secondary-active.qcow2,\
> > +file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/secon
> > +dary-hidden.qcow2,\
> > +file.backing.backing=parent0 \  
> 
> Please keep the space style.

The Idea here is that this can simply be copy-pasted into a Terminal and it will just 
work without having to edit it manually.

> > +   -drive
> > +if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
> > +children.0.file=childs0,children.0.driver=raw \
> > +   -incoming tcp:0:9998
> > +  
> 
> Here, you changed the original block design, can you explain the reasons why we need to do this?
> Please draw a ASCII diagram to describe COLO block detail.

will do.

> > 
> >  2. On Secondary VM's QEMU monitor, issue command
> > {'execute':'qmp_capabilities'} -{ 'execute': 'nbd-server-start',
> > -  'arguments': {'addr': {'type': 'inet', 'data': {'host': 'xx.xx.xx.xx', 'port':
> > '8889'} } } -}
> > -{'execute': 'nbd-server-add', 'arguments': {'device': 'secondary-disk0',
> > 'writable': true } }
> > +{'execute': 'nbd-server-start', 'arguments': {'addr': {'type': 'inet',
> > +'data': {'host': '127.0.0.1', 'port': '9999'} } } }
> > +{'execute': 'nbd-server-add', 'arguments': {'device': 'parent0',
> > +'writable': true } }
> > 
> >  Note:
> >    a. The qmp command nbd-server-start and nbd-server-add must be run
> > @@ -184,12 +213,10 @@ Note:
> > 
> >  3. On Primary VM's QEMU monitor, issue command:
> >  {'execute':'qmp_capabilities'}
> > -{ 'execute': 'human-monitor-command',
> > -  'arguments': {'command-line': 'drive_add -n buddy
> > driver=replication,mode=primary,file.driver=nbd,file.host=xx.xx.xx.xx,file.p
> > ort=8889,file.export=secondary-disk0,node-name=nbd_client0'}}
> > -{ 'execute':'x-blockdev-change', 'arguments':{'parent': 'primary-disk0',
> > 'node': 'nbd_client0' } } -{ 'execute': 'migrate-set-capabilities',
> > -      'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
> > -{ 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888' } }
> > +{'execute': 'human-monitor-command', 'arguments': {'command-line':
> > +'drive_add -n buddy
> > +driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,fil
> > +e.port=9999,file.export=parent0,node-name=replication0'}}
> > +{'execute': 'x-blockdev-change', 'arguments':{'parent': 'colo-disk0',
> > +'node': 'replication0' } }
> > +{'execute': 'migrate-set-capabilities', 'arguments': {'capabilities': [
> > +{'capability': 'x-colo', 'state': true } ] } }
> > +{'execute': 'migrate', 'arguments': {'uri': 'tcp:127.0.0.1:9998' } }
> > 
> >    Note:
> >    a. There should be only one NBD Client for each primary disk.
> > @@ -199,27 +226,91 @@ Note:
> > 
> >  4. After the above steps, you will see, whenever you make changes to PVM,
> > SVM will be synced.
> >  You can issue command '{ "execute": "migrate-set-parameters" ,
> > "arguments":{ "x-checkpoint-delay": 2000 } }'
> > -to change the checkpoint period time
> > +to change the idle checkpoint period time
> > 
> >  5. Failover test
> > -You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM's -
> > monitor at the same time, then SVM will failover and client will not detect
> > this -change.
> > +You can kill one of the VMs and Failover on the surviving VM:
> > 
> > -Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have to
> > -issue block related command to stop block replication.
> > -Primary:
> > -  Remove the nbd child from the quorum:
> > -  { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0', 'child':
> > 'children.1'}}
> > -  { 'execute': 'human-monitor-command','arguments': {'command-line':
> > 'drive_del blk-buddy0'}}
> > -  Note: there is no qmp command to remove the blockdev now
> > +== Primary Failover ==
> > +The Secondary died, resume on the Primary
> > 
> > -Secondary:
> > -  The primary host is down, so we should do the following thing:
> > -  { 'execute': 'nbd-server-stop' }
> > +{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
> > +'child': 'children.1'} }
> > +{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
> > +'drive_del replication0' } }
> > +{'execute': 'object-del', 'arguments':{ 'id': 'comp0' } }
> > +{'execute': 'object-del', 'arguments':{ 'id': 'iothread1' } }
> > +{'execute': 'object-del', 'arguments':{ 'id': 'm0' } }
> > +{'execute': 'object-del', 'arguments':{ 'id': 'redire0' } }
> > +{'execute': 'object-del', 'arguments':{ 'id': 'redire1' } }
> > +{'execute': 'x-colo-lost-heartbeat' }
> > +
> > +== Secondary Failover ==
> > +The Primary died, resume on the Secondary and prepare to become the
> > new
> > +Primary
> > +
> > +{'execute': 'nbd-server-stop'}
> > +{'execute': 'x-colo-lost-heartbeat'}
> > +
> > +{'execute': 'object-del', 'arguments':{ 'id': 'f2' } }
> > +{'execute': 'object-del', 'arguments':{ 'id': 'f1' } }
> > +{'execute': 'chardev-remove', 'arguments':{ 'id': 'red1' } }
> > +{'execute': 'chardev-remove', 'arguments':{ 'id': 'red0' } }
> > +
> > +{'execute': 'chardev-add', 'arguments':{ 'id': 'mirror0', 'backend':
> > +{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
> > +'127.0.0.1', 'port': '9003' } }, 'server': true } } } }
> > +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare1', 'backend':
> > +{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
> > +'127.0.0.1', 'port': '9004' } }, 'server': true } } } }
> > +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0', 'backend':
> > +{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
> > +'127.0.0.1', 'port': '9001' } }, 'server': true } } } }
> > +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0-0', 'backend':
> > +{'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host':
> > +'127.0.0.1', 'port': '9001' } }, 'server': false } } } }
> > +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out',
> > +'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
> > +'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': true } } }
> > +}
> > +{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out0',
> > +'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet',
> > +'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': false } }
> > +} }
> > +
> > +== Primary resume replication ==
> > +Resume replication after new Secondary is up.
> > +
> > +Start the new Secondary, then:
> > +{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0',
> > +'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode':
> > +'existing', 'format': 'raw', 'sync': 'full'} }  
> 
> I think the original secondary node has stopped the nbd server in failover:
> > +{'execute': 'nbd-server-stop'}
> > +{'execute': 'x-colo-lost-heartbeat'}  
> 
> Why you can connect to it?

Oh, I meant it differently. You execute this only if you killed the Secondary.

Thats what I meant with "You can kill one of the VMs and Failover on 
the surviving VM" above.

If you killed the Secondary, then you start a new Secondary (using the Secondary
commandline above, start nbd server and add the disk to the nbd server ) 
then execute the commands here under "Primary resume replication". After that, 
if you want to resume the replication, follow "Primary resume replication"

I will try to clarify this.

> And please add new Secondary node start up script here,
> I think it is different with original secondary node script.
> same issue in resume secondary.

No, Just the initial Primary cmdline is different. The Secondary cmdline is
always the same as above. Except if you run both on the same node, then you
need to choose a different Image than primary-copy.qcow2, I will clarify this.

Regards,
Lukas Straub

> > +
> > +Wait until disk is synced, then:
> > +{'execute': 'stop'}
> > +{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync'} }
> > +
> > +{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
> > +'drive_add -n buddy
> > +driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,fil
> > +e.port=9999,file.export=parent0,node-name=replication0'}}
> > +{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
> > +'node': 'replication0' } }
> > +
> > +{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror',
> > +'id': 'm0', 'props': { 'netdev': 'hn0', 'queue': 'tx', 'outdev':
> > +'mirror0' } } }
> > +{'execute': 'object-add', 'arguments':{ 'qom-type':
> > +'filter-redirector', 'id': 'redire0', 'props': { 'netdev': 'hn0',
> > +'queue': 'rx', 'indev': 'compare_out' } } }
> > +{'execute': 'object-add', 'arguments':{ 'qom-type':
> > +'filter-redirector', 'id': 'redire1', 'props': { 'netdev': 'hn0',
> > +'queue': 'rx', 'outdev': 'compare0' } } }
> > +{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id':
> > +'iothread1' } }
> > +{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare',
> > +'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in':
> > +'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
> > +
> > +{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [
> > +{'capability': 'x-colo', 'state': true } ] } }
> > +{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
> > +
> > +Note:
> > +If this Primary previously was a Secondary, then we need to insert the
> > +filters before the filter-rewriter by using the
> > +"'insert': 'before', 'position': 'rew0'" Options. See below.
> > +
> > +== Secondary resume replication ==
> > +Become Primary and resume replication after new Secondary is up.
> > +
> > +Start the new Secondary, then:
> > +{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0',
> > +'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode':
> > +'existing', 'format': 'raw', 'sync': 'full'} }  
> 
> Same issue as primary resume.
> 
> I want to re-tested the resume function but got stuck here.
> 
> Thanks
> Zhang Chen
> 
> > +
> > +Wait until disk is synced, then:
> > +{'execute': 'stop'}
> > +{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync' } }
> > +
> > +{'execute': 'human-monitor-command', 'arguments':{ 'command-line':
> > +'drive_add -n buddy
> > +driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,fil
> > +e.port=9999,file.export=parent0,node-name=replication0'}}
> > +{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0',
> > +'node': 'replication0' } }
> > +
> > +{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror',
> > +'id': 'm0', 'props': { 'insert': 'before', 'position': 'rew0',
> > +'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
> > +{'execute': 'object-add', 'arguments':{ 'qom-type':
> > +'filter-redirector', 'id': 'redire0', 'props': { 'insert': 'before',
> > +'position': 'rew0', 'netdev': 'hn0', 'queue': 'rx', 'indev':
> > +'compare_out' } } }
> > +{'execute': 'object-add', 'arguments':{ 'qom-type':
> > +'filter-redirector', 'id': 'redire1', 'props': { 'insert': 'before',
> > +'position': 'rew0', 'netdev': 'hn0', 'queue': 'rx', 'outdev':
> > +'compare0' } } }
> > +{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id':
> > +'iothread1' } }
> > +{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare',
> > +'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in':
> > +'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
> > +
> > +{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [
> > +{'capability': 'x-colo', 'state': true } ] } }
> > +{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
> > 
> >  == TODO ==
> > -1. Support continuous VM replication.
> > -2. Support shared storage.
> > -3. Develop the heartbeat part.
> > -4. Reduce checkpoint VM’s downtime while doing checkpoint.
> > +1. Support shared storage.
> > +2. Develop the heartbeat part.
> > +3. Reduce checkpoint VM’s downtime while doing checkpoint.
> > --
> > 2.20.1  

[Qemu-devel] [PATCH v2 3/3] Update Documentation

[Lukas Straub]

Document the qemu command-line and qmp commands for continious replication

Signed-off-by: Lukas Straub <lukasstraub2@web.de>
---
 docs/COLO-FT.txt | 185 +++++++++++++++++++++++++++++++++++------------
 1 file changed, 138 insertions(+), 47 deletions(-)

diff --git a/docs/COLO-FT.txt b/docs/COLO-FT.txt
index ad24680d13..c08bfbd3a8 100644
--- a/docs/COLO-FT.txt
+++ b/docs/COLO-FT.txt
@@ -145,35 +145,64 @@ The diagram just shows the main qmp command, you can get the detail
 in test procedure.
 
 == Test procedure ==
+Note: Here we are running both instances on the same Machine for testing, 
+change the IP Addresses if you want to run it on two Hosts
+
 1. Startup qemu
 Primary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name primary \
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=virtio,id=primary-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
-         children.0.file.filename=1.raw,\
-         children.0.driver=raw -S
+# imagefolder="/mnt/vms/colo-test"
+
+# cp --reflink=auto $imagefolder/primary.qcow2 $imagefolder/primary-copy.qcow2
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp 1 -qmp stdio \
+   -vnc :0 -k de -device piix3-usb-uhci -device usb-tablet -name primary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper \
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=mirror0,host=127.0.0.1,port=9003,server,nowait \
+   -chardev socket,id=compare1,host=127.0.0.1,port=9004,server,wait \
+   -chardev socket,id=compare0,host=127.0.0.1,port=9001,server,nowait \
+   -chardev socket,id=compare0-0,host=127.0.0.1,port=9001 \
+   -chardev socket,id=compare_out,host=127.0.0.1,port=9005,server,nowait \
+   -chardev socket,id=compare_out0,host=127.0.0.1,port=9005 \
+   -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 \
+   -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out \
+   -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 \
+   -object iothread,id=iothread1 \
+   -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,\
+outdev=compare_out0,iothread=iothread1 \
+   -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0.file.filename=$imagefolder/primary.qcow2,children.0.driver=qcow2 -S
+
 Secondary:
-# qemu-system-x86_64 -accel kvm -m 2048 -smp 2 -qmp stdio -name secondary \
-  -device piix3-usb-uhci -vnc :7 \
-  -device usb-tablet -netdev tap,id=hn0,vhost=off \
-  -device virtio-net-pci,id=net-pci0,netdev=hn0 \
-  -drive if=none,id=secondary-disk0,file.filename=1.raw,driver=raw,node-name=node0 \
-  -drive if=virtio,id=active-disk0,driver=replication,mode=secondary,\
-         file.driver=qcow2,top-id=active-disk0,\
-         file.file.filename=/mnt/ramfs/active_disk.img,\
-         file.backing.driver=qcow2,\
-         file.backing.file.filename=/mnt/ramfs/hidden_disk.img,\
-         file.backing.backing=secondary-disk0 \
-  -incoming tcp:0:8888
+# imagefolder="/mnt/vms/colo-test"
+
+# qemu-img create -f qcow2 $imagefolder/secondary-active.qcow2 10G
+
+# qemu-img create -f qcow2 $imagefolder/secondary-hidden.qcow2 10G
+
+# qemu-system-x86_64 -enable-kvm -cpu qemu64,+kvmclock -m 512 -smp 1 -qmp stdio \
+   -vnc :1 -k de -device piix3-usb-uhci -device usb-tablet -name secondary \
+   -netdev tap,id=hn0,vhost=off,helper=/usr/lib/qemu/qemu-bridge-helper \
+   -device rtl8139,id=e0,netdev=hn0 \
+   -chardev socket,id=red0,host=127.0.0.1,port=9003,reconnect=1 \
+   -chardev socket,id=red1,host=127.0.0.1,port=9004,reconnect=1 \
+   -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 \
+   -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 \
+   -object filter-rewriter,id=rew0,netdev=hn0,queue=all \
+   -drive if=none,id=parent0,file.filename=$imagefolder/primary-copy.qcow2,driver=qcow2 \
+   -drive if=none,id=childs0,driver=replication,mode=secondary,file.driver=qcow2,\
+top-id=childs0,file.file.filename=$imagefolder/secondary-active.qcow2,\
+file.backing.driver=qcow2,file.backing.file.filename=$imagefolder/secondary-hidden.qcow2,\
+file.backing.backing=parent0 \
+   -drive if=ide,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,\
+children.0.file=childs0,children.0.driver=raw \
+   -incoming tcp:0:9998
+
 
 2. On Secondary VM's QEMU monitor, issue command
 {'execute':'qmp_capabilities'}
-{ 'execute': 'nbd-server-start',
-  'arguments': {'addr': {'type': 'inet', 'data': {'host': 'xx.xx.xx.xx', 'port': '8889'} } }
-}
-{'execute': 'nbd-server-add', 'arguments': {'device': 'secondary-disk0', 'writable': true } }
+{'execute': 'nbd-server-start', 'arguments': {'addr': {'type': 'inet', 'data': {'host': '127.0.0.1', 'port': '9999'} } } }
+{'execute': 'nbd-server-add', 'arguments': {'device': 'parent0', 'writable': true } }
 
 Note:
   a. The qmp command nbd-server-start and nbd-server-add must be run
@@ -184,12 +213,10 @@ Note:
 
 3. On Primary VM's QEMU monitor, issue command:
 {'execute':'qmp_capabilities'}
-{ 'execute': 'human-monitor-command',
-  'arguments': {'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=xx.xx.xx.xx,file.port=8889,file.export=secondary-disk0,node-name=nbd_client0'}}
-{ 'execute':'x-blockdev-change', 'arguments':{'parent': 'primary-disk0', 'node': 'nbd_client0' } }
-{ 'execute': 'migrate-set-capabilities',
-      'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
-{ 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888' } }
+{'execute': 'human-monitor-command', 'arguments': {'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,file.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{'parent': 'colo-disk0', 'node': 'replication0' } }
+{'execute': 'migrate-set-capabilities', 'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments': {'uri': 'tcp:127.0.0.1:9998' } }
 
   Note:
   a. There should be only one NBD Client for each primary disk.
@@ -199,27 +226,91 @@ Note:
 
 4. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
 You can issue command '{ "execute": "migrate-set-parameters" , "arguments":{ "x-checkpoint-delay": 2000 } }'
-to change the checkpoint period time
+to change the idle checkpoint period time
 
 5. Failover test
-You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM's
-monitor at the same time, then SVM will failover and client will not detect this
-change.
+You can kill one of the VMs and Failover on the surviving VM: 
 
-Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have to
-issue block related command to stop block replication.
-Primary:
-  Remove the nbd child from the quorum:
-  { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0', 'child': 'children.1'}}
-  { 'execute': 'human-monitor-command','arguments': {'command-line': 'drive_del blk-buddy0'}}
-  Note: there is no qmp command to remove the blockdev now
+== Primary Failover ==
+The Secondary died, resume on the Primary
 
-Secondary:
-  The primary host is down, so we should do the following thing:
-  { 'execute': 'nbd-server-stop' }
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'child': 'children.1'} }
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_del replication0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'comp0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'iothread1' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'm0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire0' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'redire1' } }
+{'execute': 'x-colo-lost-heartbeat' }
+
+== Secondary Failover ==
+The Primary died, resume on the Secondary and prepare to become the new Primary
+
+{'execute': 'nbd-server-stop'}
+{'execute': 'x-colo-lost-heartbeat'}
+
+{'execute': 'object-del', 'arguments':{ 'id': 'f2' } }
+{'execute': 'object-del', 'arguments':{ 'id': 'f1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red1' } }
+{'execute': 'chardev-remove', 'arguments':{ 'id': 'red0' } }
+
+{'execute': 'chardev-add', 'arguments':{ 'id': 'mirror0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9003' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare1', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9004' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9001' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare0-0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9001' } }, 'server': false } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': true } } } }
+{'execute': 'chardev-add', 'arguments':{ 'id': 'compare_out0', 'backend': {'type': 'socket', 'data': {'addr': { 'type': 'inet', 'data': { 'host': '127.0.0.1', 'port': '9005' } }, 'server': false } } } }
+
+== Primary resume replication ==
+Resume replication after new Secondary is up.
+
+Start the new Secondary, then:
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0', 'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode': 'existing', 'format': 'raw', 'sync': 'full'} }
+
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync'} }
+
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,file.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'node': 'replication0' } }
+
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror', 'id': 'm0', 'props': { 'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire0', 'props': { 'netdev': 'hn0', 'queue': 'rx', 'indev': 'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire1', 'props': { 'netdev': 'hn0', 'queue': 'rx', 'outdev': 'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id': 'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare', 'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in': 'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
+
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
+
+Note:
+If this Primary previously was a Secondary, then we need to insert the 
+filters before the filter-rewriter by using the 
+"'insert': 'before', 'position': 'rew0'" Options. See below.
+
+== Secondary resume replication ==
+Become Primary and resume replication after new Secondary is up.
+
+Start the new Secondary, then:
+{'execute': 'drive-mirror', 'arguments':{ 'device': 'colo-disk0', 'job-id': 'resync', 'target': 'nbd://127.0.0.1:9999/parent0', 'mode': 'existing', 'format': 'raw', 'sync': 'full'} }
+
+Wait until disk is synced, then:
+{'execute': 'stop'}
+{'execute': 'block-job-cancel', 'arguments':{ 'device': 'resync' } }
+
+{'execute': 'human-monitor-command', 'arguments':{ 'command-line': 'drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=127.0.0.1,file.port=9999,file.export=parent0,node-name=replication0'}}
+{'execute': 'x-blockdev-change', 'arguments':{ 'parent': 'colo-disk0', 'node': 'replication0' } }
+
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-mirror', 'id': 'm0', 'props': { 'insert': 'before', 'position': 'rew0', 'netdev': 'hn0', 'queue': 'tx', 'outdev': 'mirror0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire0', 'props': { 'insert': 'before', 'position': 'rew0', 'netdev': 'hn0', 'queue': 'rx', 'indev': 'compare_out' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'filter-redirector', 'id': 'redire1', 'props': { 'insert': 'before', 'position': 'rew0', 'netdev': 'hn0', 'queue': 'rx', 'outdev': 'compare0' } } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'iothread', 'id': 'iothread1' } }
+{'execute': 'object-add', 'arguments':{ 'qom-type': 'colo-compare', 'id': 'comp0', 'props': { 'primary_in': 'compare0-0', 'secondary_in': 'compare1', 'outdev': 'compare_out0', 'iothread': 'iothread1' } } }
+
+{'execute': 'migrate-set-capabilities', 'arguments':{ 'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
+{'execute': 'migrate', 'arguments':{ 'uri': 'tcp:127.0.0.1:9998' } }
 
 == TODO ==
-1. Support continuous VM replication.
-2. Support shared storage.
-3. Develop the heartbeat part.
-4. Reduce checkpoint VM’s downtime while doing checkpoint.
+1. Support shared storage.
+2. Develop the heartbeat part.
+3. Reduce checkpoint VM’s downtime while doing checkpoint.
-- 
2.20.1