Re: [Qemu-devel] [POC]colo-proxy in qemu

[Jason Wang <jasowang@redhat.com>]

On 11/10/2015 04:30 PM, zhanghailiang wrote:
> On 2015/11/10 15:35, Jason Wang wrote:
>>
>>
>> On 11/10/2015 01:26 PM, Tkid wrote:
>>> Hi,all
>>>
>>> We are planning to reimplement colo proxy in userspace (Here is in
>>> qemu) to
>>> cache and compare net packets.This module is one of the important
>>> components
>>> of COLO project and now it is still in early stage, so any comments and
>>> feedback are warmly welcomed,thanks in advance.
>>>
>>> ## Background
>>> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop
>>> Service)
>>> project is a high availability solution. Both Primary VM (PVM) and
>>> Secondary VM
>>> (SVM) run in parallel. They receive the same request from client, and
>>> generate
>>> responses in parallel too. If the response packets from PVM and SVM are
>>> identical, they are released immediately. Otherwise, a VM checkpoint
>>> (on demand)
>>> is conducted.
>>> Paper:
>>> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
>>> COLO on Xen:
>>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>>> COLO on Qemu/KVM:
>>> http://wiki.qemu.org/Features/COLO
>>>
>>> By the needs of capturing response packets from PVM and SVM and
>>> finding out
>>> whether they are identical, we introduce a new module to qemu
>>> networking called
>>> colo-proxy.
>>>
>>> This document describes the design of the colo-proxy module
>>>
>>> ## Glossary
>>>    PVM - Primary VM, which provides services to clients.
>>>    SVM - Secondary VM, a hot standby and replication of PVM.
>>>    PN - Primary Node, the host which PVM runs on
>>>    SN - Secondary Node, the host which SVM runs on
>>>
>>> ## Our Idea ##
>>>
>>> COLO-Proxy
>>> COLO-Proxy is a part of COLO,based on qemu net filter and it's a
>>> plugin for
>>> qemu net filter.the function keep SVM connect normal to PVM and compare
>>> PVM's packets to SVM's packets.if difference,notify COLO do checkpoint.
>>>
>>> == Workflow ==
>>>
>>>
>>> +--+                                      +--+
>>> |PN|                                      |SN|
>>> +-----------------------+                 +-----------------------+
>>> | +-------------------+ |                 | +-------------------+ |
>>> | |                   | |                 | |                   | |
>>> | |        PVM        | |                 | |        SVM        | |
>>> | |                   | |                 | |                   | |
>>> | +--+-^--------------+ |                 | +-------------^----++ |
>>> |    | |                |                 |               |    |  |
>>> |    | | +------------+ |                 | +-----------+ |    |  |
>>> |    | | |    COLO    | |    (socket)     | |    COLO   | |    |  |
>>> |    | | | CheckPoint +---------------------> CheckPoint| |    |  |
>>> |    | | |            | |      (6)        | |           | |    |  |
>>> |    | | +-----^------+ |                 | +-----------+ |    |  |
>>> |    | |   (5) |        |                 |               |    |  |
>>> |    | |       |        |                 |               |    |  |
>>> | +--v-+--------------+ | Forward(socket) | +-------------+----v+ |
>>> | |COLO Proxy  |      +-------+(1)+--------->seq&ack adjust(2)| | |
>>> | |      +-----+------+ |                 | +-----------------+ | |
>>> | |      | Compare(4) <-------+(3)+---------+     COLO Proxy    | |
>>> | +-------------------+ | Forward(socket) | +-------------------+ |
>>> ++Qemu+-----------------+                 ++Qemu+-----------------+
>>>             | ^
>>>             | |
>>>             | |
>>>    +--------v-+--------+
>>>    |                   |
>>>    |      Client       |
>>>    |                   |
>>>    +-------------------+
>>>
>>>
>>>
>>>
>>> (1)When PN receive client packets,PN COLO-Proxy copy and forward
>>> packets to
>>> SN COLO-Proxy.
>>> (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's
>>> ack,send
>>> adjusted packets to SVM
>>> (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu
>>> COLO-Proxy.
>>> (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's
>>> packets,then
>>> compare PVM's packets data with SVM's packets data. If packets is
>>> different, compare
>>> module notify COLO CheckPoint module to do a checkpoint then send
>>> PVM's packets to
>>> client and drop SVM's packets, otherwise, just send PVM's packets to
>>> client and
>>> drop SVM's packets.
>>> (5)notify COLO-Checkpoint module checkpoint is needed
>>> (6)Do COLO-Checkpoint
>>>
>>> ### QEMU space TCP/IP stack(Based on SLIRP) ###
>>> We need a QEMU space TCP/IP stack to help us to analysis packet. After
>>> looking
>>> into QEMU, we found that SLIRP
>>>
>>> http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29
>>>
>>>
>>> is a good choice for us. SLIRP proivdes a full TCP/IP stack within
>>> QEMU, it can
>>> help use to handle the packet written to/read from backend(tap) device
>>> which is
>>> just like a link layer(L2) packet.
>>>
>>> ### Packet enqueue and compare ###
>>> Together with QEMU space TCP/IP stack, we enqueue all packets sent by
>>> PVM and
>>> SVM on Primary QEMU, and then compare the packet payload for each
>>> connection.
>>>
>>
>> Hi:
>>
>> Just have the following questions in my mind (some has been raised in
>> the previous rounds of discussion without a conclusion):
>>
>> - What's the plan for management layer? The setup seems complicated so
>> we could not simply depend on user to do each step. (And for security
>> reason, qemu was usually run as unprivileged user)
>
> We will do most of the setup works automatically in qemu as possible
> as we can.
> Compared with kernel proxy scheme, it is not a big deal. :)

Yes, but what I mean is for host. E.g setting up network like bridge or
others.
>
>> - What's the plan for vhost? Userspace network in qemu is rather slow,
>> most user will choose vhost.
>> - What if application generate packet based on hwrng device? This will
>> produce always different packets.
>
> Yes, that is really a big problem, actually, we have discussed it for
> many
> times, it seems that there is no perfect way to solve it. :(
> We have a compromise approach, when we find there are too many continuous
> checkpoint requests, we switch COLO from normal mode to periodic mode
> which SVM will stop running.
> (Dave have realized this before, which called Hybrid mode. The patches is
> "[RFC/COLO:  0/3] Hybrid mode and parameterisation")

Aha, I see. Thanks for the pointer.

>
>> - Not sure SLIRP is perfect matched for this task. As has been raised,
>> another method is to decouple the packet comparing from qemu. In this
>> way, lots of open source userspace stack could be used.
>
> Hmm, it seems to be a good idea, maybe we can add a checkpoint request
> command
> in COLO to support more packets comparing scheme ...
>

Yes, then when to synchronize could be determined by external program.
(Just an idea FYI).

> Thanks，
> zhanghailiang
>
>> - Haven't read the code of packet comparing, but if it needs to keep
>> track the state of each connection, it could be easily DOS from guest.
>>
>> Thanks
>>
>> .
>>
>
>
>