Re: [Qemu-devel] [POC]colo-proxy in qemu

[Zhang Chen <zhangchen.fnst@cn.fujitsu.com>]

On 11/10/2015 06:54 PM, Dr. David Alan Gilbert wrote:
> * Tkid (zhangchen.fnst@cn.fujitsu.com) wrote:
>> Hi,all
>>
>> We are planning to reimplement colo proxy in userspace (Here is in qemu) to
>> cache and compare net packets.This module is one of the important components
>> of COLO project and now it is still in early stage, so any comments and
>> feedback are warmly welcomed,thanks in advance.
>>
>> ## Background
>> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop
>> Service)
>> project is a high availability solution. Both Primary VM (PVM) and Secondary
>> VM
>> (SVM) run in parallel. They receive the same request from client, and
>> generate
>> responses in parallel too. If the response packets from PVM and SVM are
>> identical, they are released immediately. Otherwise, a VM checkpoint (on
>> demand)
>> is conducted.
>> Paper:
>> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
>> COLO on Xen:
>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>> COLO on Qemu/KVM:
>> http://wiki.qemu.org/Features/COLO
>>
>> By the needs of capturing response packets from PVM and SVM and finding out
>> whether they are identical, we introduce a new module to qemu networking
>> called
>> colo-proxy.
>>
>> This document describes the design of the colo-proxy module
>>
>> ## Glossary
>>    PVM - Primary VM, which provides services to clients.
>>    SVM - Secondary VM, a hot standby and replication of PVM.
>>    PN - Primary Node, the host which PVM runs on
>>    SN - Secondary Node, the host which SVM runs on
>>
>> ## Our Idea ##
>>
>> COLO-Proxy
>> COLO-Proxy is a part of COLO,based on qemu net filter and it's a plugin for
>> qemu net filter.the function keep SVM connect normal to PVM and compare
>> PVM's packets to SVM's packets.if difference,notify COLO do checkpoint.
>>
>> == Workflow ==
>>
>>
>> +--+                                      +--+
>> |PN|                                      |SN|
>> +-----------------------+                 +-----------------------+
>> | +-------------------+ |                 | +-------------------+ |
>> | |                   | |                 | |                   | |
>> | |        PVM        | |                 | |        SVM        | |
>> | |                   | |                 | |                   | |
>> | +--+-^--------------+ |                 | +-------------^----++ |
>> |    | |                |                 |               |    |  |
>> |    | | +------------+ |                 | +-----------+ |    |  |
>> |    | | |    COLO    | |    (socket)     | |    COLO   | |    |  |
>> |    | | | CheckPoint +---------------------> CheckPoint| |    |  |
>> |    | | |            | |      (6)        | |           | |    |  |
>> |    | | +-----^------+ |                 | +-----------+ |    |  |
>> |    | |   (5) |        |                 |               |    |  |
>> |    | |       |        |                 |               |    |  |
>> | +--v-+--------------+ | Forward(socket) | +-------------+----v+ |
>> | |COLO Proxy  |      +-------+(1)+--------->seq&ack adjust(2)| | |
>> | |      +-----+------+ |                 | +-----------------+ | |
>> | |      | Compare(4) <-------+(3)+---------+     COLO Proxy    | |
>> | +-------------------+ | Forward(socket) | +-------------------+ |
>> ++Qemu+-----------------+                 ++Qemu+-----------------+
>>             | ^
>>             | |
>>             | |
>>    +--------v-+--------+
>>    |                   |
>>    |      Client       |
>>    |                   |
>>    +-------------------+
>>
>>
>> (1)When PN receive client packets,PN COLO-Proxy copy and forward packets to
>> SN COLO-Proxy.
>> (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send
>> adjusted packets to SVM
>> (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu
>> COLO-Proxy.
> What protocol are you using for the data carried over the Forward(socket)?
> I'm just wondering if there's an existing layer2 tunneling protocol that
> it would be best to use.
Currently,we will use raw tcp socket send a packet's length then send 
the packet.
>> (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then
>> compare PVM's packets data with SVM's packets data. If packets is different,
>> compare
>> module notify COLO CheckPoint module to do a checkpoint then send PVM's
>> packets to
>> client and drop SVM's packets, otherwise, just send PVM's packets to client
>> and
>> drop SVM's packets.
>> (5)notify COLO-Checkpoint module checkpoint is needed
>> (6)Do COLO-Checkpoint
>>
>> ### QEMU space TCP/IP stack(Based on SLIRP) ###
>> We need a QEMU space TCP/IP stack to help us to analysis packet. After
>> looking
>> into QEMU, we found that SLIRP
>>
>> http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29
>>
>> is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it
>> can
>> help use to handle the packet written to/read from backend(tap) device which
>> is
>> just like a link layer(L2) packet.
> I still think SLIRP might be painful; but it might be an easy one to start
> with.
>
>> ### Packet enqueue and compare ###
>> Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM
>> and
>> SVM on Primary QEMU, and then compare the packet payload for each
>> connection.
> Dave
>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> .
>