From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:35731) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zw1RI-0008OW-UR for qemu-devel@nongnu.org; Tue, 10 Nov 2015 00:25:54 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Zw1RE-0007Cw-Rl for qemu-devel@nongnu.org; Tue, 10 Nov 2015 00:25:52 -0500 Received: from [59.151.112.132] (port=1644 helo=heian.cn.fujitsu.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zw1RB-0007Bq-Tr for qemu-devel@nongnu.org; Tue, 10 Nov 2015 00:25:48 -0500 From: Tkid Message-ID: <56418003.9010208@cn.fujitsu.com> Date: Tue, 10 Nov 2015 13:26:27 +0800 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="------------060000070307080209070705" Subject: [Qemu-devel] [POC]colo-proxy in qemu List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org, stefanha@redhat.com, jasowang@redhat.com Cc: zhang.zhanghailiang@huawei.com, lizhijian@cn.fujitsu.com, jan.kiszka@siemens.com, eddie.dong@intel.com, dgilbert@redhat.com, peter.huangpeng@huawei.com, arei.gonglei@huawei.com, guijianfeng@cn.fujitsu.com, zhangchen.fnst@cn.fujitsu.com --------------060000070307080209070705 Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit Hi,all We are planning to reimplement colo proxy in userspace (Here is in qemu) to cache and compare net packets.This module is one of the important components of COLO project and now it is still in early stage, so any comments and feedback are warmly welcomed,thanks in advance. ## Background COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) project is a high availability solution. Both Primary VM (PVM) and Secondary VM (SVM) run in parallel. They receive the same request from client, and generate responses in parallel too. If the response packets from PVM and SVM are identical, they are released immediately. Otherwise, a VM checkpoint (on demand) is conducted. Paper: http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 COLO on Xen: http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping COLO on Qemu/KVM: http://wiki.qemu.org/Features/COLO By the needs of capturing response packets from PVM and SVM and finding out whether they are identical, we introduce a new module to qemu networking called colo-proxy. This document describes the design of the colo-proxy module ## Glossary PVM - Primary VM, which provides services to clients. SVM - Secondary VM, a hot standby and replication of PVM. PN - Primary Node, the host which PVM runs on SN - Secondary Node, the host which SVM runs on ## Our Idea ## COLO-Proxy COLO-Proxy is a part of COLO,based on qemu net filter and it's a plugin for qemu net filter.the function keep SVM connect normal to PVM and compare PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. == Workflow == +--+ +--+ |PN| |SN| +-----------------------+ +-----------------------+ | +-------------------+ | | +-------------------+ | | | | | | | | | | | PVM | | | | SVM | | | | | | | | | | | +--+-^--------------+ | | +-------------^----++ | | | | | | | | | | | | +------------+ | | +-----------+ | | | | | | | COLO | | (socket) | | COLO | | | | | | | | CheckPoint +---------------------> CheckPoint| | | | | | | | | | (6) | | | | | | | | | +-----^------+ | | +-----------+ | | | | | | (5) | | | | | | | | | | | | | | | | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | | | +-----+------+ | | +-----------------+ | | | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | | +-------------------+ | Forward(socket) | +-------------------+ | ++Qemu+-----------------+ ++Qemu+-----------------+ | ^ | | | | +--------v-+--------+ | | | Client | | | +-------------------+ (1)When PN receive client packets,PN COLO-Proxy copy and forward packets to SN COLO-Proxy. (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send adjusted packets to SVM (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu COLO-Proxy. (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then compare PVM's packets data with SVM's packets data. If packets is different, compare module notify COLO CheckPoint module to do a checkpoint then send PVM's packets to client and drop SVM's packets, otherwise, just send PVM's packets to client and drop SVM's packets. (5)notify COLO-Checkpoint module checkpoint is needed (6)Do COLO-Checkpoint ### QEMU space TCP/IP stack(Based on SLIRP) ### We need a QEMU space TCP/IP stack to help us to analysis packet. After looking into QEMU, we found that SLIRP http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29 is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it can help use to handle the packet written to/read from backend(tap) device which is just like a link layer(L2) packet. ### Packet enqueue and compare ### Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM and SVM on Primary QEMU, and then compare the packet payload for each connection. --------------060000070307080209070705 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: 8bit Hi,all

We are planning to reimplement colo proxy in userspace (Here is in qemu) to
cache and compare net packets.This module is one of the important components
of COLO project and now it is still in early stage, so any comments and
feedback are warmly welcomed,thanks in advance.

## Background
COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
project is a high availability solution. Both Primary VM (PVM) and Secondary VM
(SVM) run in parallel. They receive the same request from client, and generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint (on demand)
is conducted.
Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO

By the needs of capturing response packets from PVM and SVM and finding out
whether they are identical, we introduce a new module to qemu networking called
colo-proxy.

This document describes the design of the colo-proxy module

## Glossary
  PVM - Primary VM, which provides services to clients.
  SVM - Secondary VM, a hot standby and replication of PVM.
  PN - Primary Node, the host which PVM runs on
  SN - Secondary Node, the host which SVM runs on

## Our Idea ##

COLO-Proxy
COLO-Proxy is a part of COLO,based on qemu net filter and it's a plugin for
qemu net filter.the function keep SVM connect normal to PVM and compare
PVM's packets to SVM's packets.if difference,notify COLO do checkpoint.

== Workflow ==


+--+                                      +--+
|PN|                                      |SN|
+-----------------------+                 +-----------------------+
| +-------------------+ |                 | +-------------------+ |
| |                   | |                 | |                   | |
| |        PVM        | |                 | |        SVM        | |
| |                   | |                 | |                   | |
| +--+-^--------------+ |                 | +-------------^----++ |
|    | |                |                 |               |    |  |
|    | | +------------+ |                 | +-----------+ |    |  |
|    | | |    COLO    | |    (socket)     | |    COLO   | |    |  |
|    | | | CheckPoint +---------------------> CheckPoint| |    |  |
|    | | |            | |      (6)        | |           | |    |  |
|    | | +-----^------+ |                 | +-----------+ |    |  |
|    | |   (5) |        |                 |               |    |  |
|    | |       |        |                 |               |    |  |
| +--v-+--------------+ | Forward(socket) | +-------------+----v+ |
| |COLO Proxy  |      +-------+(1)+--------->seq&ack adjust(2)| | |
| |      +-----+------+ |                 | +-----------------+ | |
| |      | Compare(4) <-------+(3)+---------+     COLO Proxy    | |
| +-------------------+ | Forward(socket) | +-------------------+ |
++Qemu+-----------------+                 ++Qemu+-----------------+
           | ^
           | |
           | |
  +--------v-+--------+
  |                   |
  |      Client       |
  |                   |
  +-------------------+




(1)When PN receive client packets,PN COLO-Proxy copy and forward packets to
SN COLO-Proxy.
(2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send
adjusted packets to SVM
(3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu COLO-Proxy.
(4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then
compare PVM's packets data with SVM's packets data. If packets is different, compare
module notify COLO CheckPoint module to do a checkpoint then send PVM's packets to
client and drop SVM's packets, otherwise, just send PVM's packets to client and
drop SVM's packets.
(5)notify COLO-Checkpoint module checkpoint is needed
(6)Do COLO-Checkpoint

### QEMU space TCP/IP stack(Based on SLIRP) ###
We need a QEMU space TCP/IP stack to help us to analysis packet. After looking
into QEMU, we found that SLIRP

http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29

is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it can
help use to handle the packet written to/read from backend(tap) device which is
just like a link layer(L2) packet.

### Packet enqueue and compare ###
Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM and
SVM on Primary QEMU, and then compare the packet payload for each connection.

--------------060000070307080209070705--