From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59156) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zw6ZS-0004hD-Vm for qemu-devel@nongnu.org; Tue, 10 Nov 2015 05:54:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Zw6ZP-0001io-Nj for qemu-devel@nongnu.org; Tue, 10 Nov 2015 05:54:38 -0500 Received: from mx1.redhat.com ([209.132.183.28]:34239) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zw6ZP-0001iV-CP for qemu-devel@nongnu.org; Tue, 10 Nov 2015 05:54:35 -0500 Date: Tue, 10 Nov 2015 10:54:29 +0000 From: "Dr. David Alan Gilbert" Message-ID: <20151110105428.GC2456@work-vm> References: <56418003.9010208@cn.fujitsu.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <56418003.9010208@cn.fujitsu.com> Subject: Re: [Qemu-devel] [POC]colo-proxy in qemu List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Tkid Cc: zhang.zhanghailiang@huawei.com, lizhijian@cn.fujitsu.com, jan.kiszka@siemens.com, jasowang@redhat.com, eddie.dong@intel.com, qemu-devel@nongnu.org, peter.huangpeng@huawei.com, arei.gonglei@huawei.com, stefanha@redhat.com, guijianfeng@cn.fujitsu.com * Tkid (zhangchen.fnst@cn.fujitsu.com) wrote: > Hi,all > > We are planning to reimplement colo proxy in userspace (Here is in qemu) to > cache and compare net packets.This module is one of the important components > of COLO project and now it is still in early stage, so any comments and > feedback are warmly welcomed,thanks in advance. > > ## Background > COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop > Service) > project is a high availability solution. Both Primary VM (PVM) and Secondary > VM > (SVM) run in parallel. They receive the same request from client, and > generate > responses in parallel too. If the response packets from PVM and SVM are > identical, they are released immediately. Otherwise, a VM checkpoint (on > demand) > is conducted. > Paper: > http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 > COLO on Xen: > http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping > COLO on Qemu/KVM: > http://wiki.qemu.org/Features/COLO > > By the needs of capturing response packets from PVM and SVM and finding out > whether they are identical, we introduce a new module to qemu networking > called > colo-proxy. > > This document describes the design of the colo-proxy module > > ## Glossary > PVM - Primary VM, which provides services to clients. > SVM - Secondary VM, a hot standby and replication of PVM. > PN - Primary Node, the host which PVM runs on > SN - Secondary Node, the host which SVM runs on > > ## Our Idea ## > > COLO-Proxy > COLO-Proxy is a part of COLO,based on qemu net filter and it's a plugin for > qemu net filter.the function keep SVM connect normal to PVM and compare > PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. > > == Workflow == > > > +--+ +--+ > |PN| |SN| > +-----------------------+ +-----------------------+ > | +-------------------+ | | +-------------------+ | > | | | | | | | | > | | PVM | | | | SVM | | > | | | | | | | | > | +--+-^--------------+ | | +-------------^----++ | > | | | | | | | | > | | | +------------+ | | +-----------+ | | | > | | | | COLO | | (socket) | | COLO | | | | > | | | | CheckPoint +---------------------> CheckPoint| | | | > | | | | | | (6) | | | | | | > | | | +-----^------+ | | +-----------+ | | | > | | | (5) | | | | | | > | | | | | | | | | > | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | > | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | > | | +-----+------+ | | +-----------------+ | | > | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | > | +-------------------+ | Forward(socket) | +-------------------+ | > ++Qemu+-----------------+ ++Qemu+-----------------+ > | ^ > | | > | | > +--------v-+--------+ > | | > | Client | > | | > +-------------------+ > > > (1)When PN receive client packets,PN COLO-Proxy copy and forward packets to > SN COLO-Proxy. > (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send > adjusted packets to SVM > (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu > COLO-Proxy. What protocol are you using for the data carried over the Forward(socket)? I'm just wondering if there's an existing layer2 tunneling protocol that it would be best to use. > (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then > compare PVM's packets data with SVM's packets data. If packets is different, > compare > module notify COLO CheckPoint module to do a checkpoint then send PVM's > packets to > client and drop SVM's packets, otherwise, just send PVM's packets to client > and > drop SVM's packets. > (5)notify COLO-Checkpoint module checkpoint is needed > (6)Do COLO-Checkpoint > > ### QEMU space TCP/IP stack(Based on SLIRP) ### > We need a QEMU space TCP/IP stack to help us to analysis packet. After > looking > into QEMU, we found that SLIRP > > http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29 > > is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it > can > help use to handle the packet written to/read from backend(tap) device which > is > just like a link layer(L2) packet. I still think SLIRP might be painful; but it might be an easy one to start with. > ### Packet enqueue and compare ### > Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM > and > SVM on Primary QEMU, and then compare the packet payload for each > connection. Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK