Re: [Qemu-devel] [PATCH RFC v3 00/27] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service

[Gao feng <gaofeng@cn.fujitsu.com>]

On 02/25/2015 04:13 AM, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> This is the 3th version of COLO, it is only COLO frame part, include: VM checkpoint,
>> failover, proxy API, block replication API, not include block replication.
>> The block part has been sent by wencongyang:
>> '[RFC PATCH 00/14] Block replication for continuous checkpoints'
>>
>> You can get the integrated qemu colo patches from github:
>> https://github.com/coloft/qemu/commits/colo-v1.0
>>
>> Compared with the previous version, we have realized all parts of COLO frame, 
>> and it is works now.
>>
>> The main change since last version is, we use colo proxy mode instead of
>> colo agent, they are all used for network packets compare, but proxy is more
>> efficient, it is based on netfilter.
>> Another modification is we implement new block replication scheme, 
>> you can get more info from wencongyang's block patch series 
>>
>> If you don't know about COLO, please refer to below link for detailed 
>> information.
>>
>> The idea is presented in Xen summit 2012, and 2013,
>> and academia paper in SOCC 2013. It's also presented in KVM forum in 2013:
>> http://www.linux-kvm.org/wiki/images/1/1d/Kvm-forum-2013-COLO.pdf
>>
>> Previous posted RFC proposal:
>> http://lists.nongnu.org/archive/html/qemu-devel/2014-06/msg05567.html
>> http://lists.nongnu.org/archive/html/qemu-devel/2014-09/msg04459.html
>>
>> The below is the detail about test COLO, you can also get this info
>> from http://wiki.qemu.org/Features/COLO.
>> * Hardware requirements
>> There is at least one directly connected nic to forward the network requests
>> from client to secondary VM. The directly connected nic must not be used by
>> any other purpose.
>>
>> * Network link topology
>> =================================normal ======================================
>>                                 +--------+
>>                                 |client  |
>>    master                       +----+---+                    slave
>> -------------------------+           |            + -------------------------+
>>    PVM                   |           +            |                          |
>> +-------+         +----[eth0]-----[switch]-----[eth0]---------+              |
>> |guest  |     +---+-+    |                        |       +---+-+            |
>> |     [tap0]--+ br0 |    |                        |       | br0 |            |
>> |       |     +-----+  [eth1]-----[forward]----[eth1]--+  +-----+      SVM   |
>> +-------+                |                        |    |            +-------+|
>>                          |                        |    |  +-----+   | guest ||
>>                        [eth2]---[checkpoint]---[eth2]  +--+br1  |-[tap0]    ||
>>                          |                        |       +-----+   |       ||
>>                          |                        |                 +-------+|
>> -------------------------+                        +--------------------------+
>> e.g.
>> master:
>> br0: 192.168.0.33
>> eth1: 192.168.1.33
>> eth2: 192.168.2.33
>>
>> slave:
>> br0: 192.168.0.88
>> br1: no ip address
>> eth1: 192.168.1.88
>> eth2: 192.168.2.88
>> (Actually, you can also use eth0 as checkpoint channel)
>> Note: in normal, SVM will always be linked to br1 like above until
>> failover.
>>
>> * Test environment prepare:
>> 1. Set Up the Bridge and network environment
>> You must setup you network environment like above picture,
>> In master, setup a bridge br0, using command brctl, like:
>> # ifconfig eth0 down
>> # ifconfig eth0 0.0.0.0
>> # brctl addbr br0
>> # brctl addif br0 eth0
>> # ifconfig br0 192.168.0.33 netmask 255.255.255.0
>> # ifconfig eth0 up
>> In slave, setup two bridge br0, br1, commands are same with above,
>> please note that br1 is linked to eth1(the forward nic).
>>
>> 2.Qemu-ifup
>> We need a script to bring up the TAP interface.
>> You can find this info from http://en.wikibooks.org/wiki/QEMU/Networking.
>> Master:
>> root@master# cat /etc/qemu-ifup
>> #!/bin/sh
>> switch=br0
>> if [ -n "$1" ]; then
>>         ip link set $1 up
>>         brctl addif ${switch} $1
>> fi
>> Slave:
>> root@slave # cat /etc/qemu-ifup
>> #!/bin/sh
>> switch=br1  #in primary, switch is br0. in secondary switch is br1
>> if [ -n "$1" ]; then
>>         ip link set $1 up
>>         brctl addif ${switch} $1
>> fi 
>>
>> 3. Prepare host kernel
>> colo-proxy kernel module need cooperate with linux kernel.
>> You should put a kernel patch 'colo-patch-for-kernel.patch'
>> (It's based on linux kernel-3.19) which you can get from 
>> https://github.com/gao-feng/colo-proxy.git
>> and then compile kernel and intall the new kernel.
>>
>> 4. Proxy module
>> proxy module is used for network packets compare, you can also get the lastest
>> version from: https://github.com/gao-feng/colo-proxy.git.
>> You can compile and install it by using command 'make' && 'make install'.
>>
>> 5. Modified iptables
>> We have add a new rule to iptables command, so please get the patch from
>> https://github.com/gao-feng/colo-proxy/blob/master/COLO-library_for_iptables-1.4.21.patch
>> It is based on version 1.4.21.
> 
> I'm getting closer but I don't think I'm getting packets from the secondary
> to the primary yet; it looks like the primary is holding onto it's packets
> until the end of the 10second checkpoint.
> Still debugging that but I'd take any tips.
> 
Hi David,

you can use tcpdump on secondary's tap device to see if secondary guest already
sent out the packet. and tcpdump on primary's forward device to see if the secondary
guest's packets are sent to primary node.

thanks,
Gao