From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49722) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZwLPW-0007Gp-Ac for qemu-devel@nongnu.org; Tue, 10 Nov 2015 21:45:23 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZwLPS-0005Ha-W5 for qemu-devel@nongnu.org; Tue, 10 Nov 2015 21:45:22 -0500 Received: from [59.151.112.132] (port=29548 helo=heian.cn.fujitsu.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZwLPR-0005GX-FO for qemu-devel@nongnu.org; Tue, 10 Nov 2015 21:45:18 -0500 References: <56418003.9010208@cn.fujitsu.com> <20151110105428.GC2456@work-vm> From: Zhang Chen Message-ID: <5642ABF4.1040904@cn.fujitsu.com> Date: Wed, 11 Nov 2015 10:46:12 +0800 MIME-Version: 1.0 In-Reply-To: <20151110105428.GC2456@work-vm> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [POC]colo-proxy in qemu List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Dr. David Alan Gilbert" Cc: zhang.zhanghailiang@huawei.com, lizhijian@cn.fujitsu.com, jan.kiszka@siemens.com, jasowang@redhat.com, eddie.dong@intel.com, qemu-devel@nongnu.org, peter.huangpeng@huawei.com, arei.gonglei@huawei.com, stefanha@redhat.com, guijianfeng@cn.fujitsu.com On 11/10/2015 06:54 PM, Dr. David Alan Gilbert wrote: > * Tkid (zhangchen.fnst@cn.fujitsu.com) wrote: >> Hi,all >> >> We are planning to reimplement colo proxy in userspace (Here is in qemu) to >> cache and compare net packets.This module is one of the important components >> of COLO project and now it is still in early stage, so any comments and >> feedback are warmly welcomed,thanks in advance. >> >> ## Background >> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop >> Service) >> project is a high availability solution. Both Primary VM (PVM) and Secondary >> VM >> (SVM) run in parallel. They receive the same request from client, and >> generate >> responses in parallel too. If the response packets from PVM and SVM are >> identical, they are released immediately. Otherwise, a VM checkpoint (on >> demand) >> is conducted. >> Paper: >> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 >> COLO on Xen: >> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping >> COLO on Qemu/KVM: >> http://wiki.qemu.org/Features/COLO >> >> By the needs of capturing response packets from PVM and SVM and finding out >> whether they are identical, we introduce a new module to qemu networking >> called >> colo-proxy. >> >> This document describes the design of the colo-proxy module >> >> ## Glossary >> PVM - Primary VM, which provides services to clients. >> SVM - Secondary VM, a hot standby and replication of PVM. >> PN - Primary Node, the host which PVM runs on >> SN - Secondary Node, the host which SVM runs on >> >> ## Our Idea ## >> >> COLO-Proxy >> COLO-Proxy is a part of COLO,based on qemu net filter and it's a plugin for >> qemu net filter.the function keep SVM connect normal to PVM and compare >> PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. >> >> == Workflow == >> >> >> +--+ +--+ >> |PN| |SN| >> +-----------------------+ +-----------------------+ >> | +-------------------+ | | +-------------------+ | >> | | | | | | | | >> | | PVM | | | | SVM | | >> | | | | | | | | >> | +--+-^--------------+ | | +-------------^----++ | >> | | | | | | | | >> | | | +------------+ | | +-----------+ | | | >> | | | | COLO | | (socket) | | COLO | | | | >> | | | | CheckPoint +---------------------> CheckPoint| | | | >> | | | | | | (6) | | | | | | >> | | | +-----^------+ | | +-----------+ | | | >> | | | (5) | | | | | | >> | | | | | | | | | >> | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | >> | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | >> | | +-----+------+ | | +-----------------+ | | >> | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | >> | +-------------------+ | Forward(socket) | +-------------------+ | >> ++Qemu+-----------------+ ++Qemu+-----------------+ >> | ^ >> | | >> | | >> +--------v-+--------+ >> | | >> | Client | >> | | >> +-------------------+ >> >> >> (1)When PN receive client packets,PN COLO-Proxy copy and forward packets to >> SN COLO-Proxy. >> (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send >> adjusted packets to SVM >> (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu >> COLO-Proxy. > What protocol are you using for the data carried over the Forward(socket)? > I'm just wondering if there's an existing layer2 tunneling protocol that > it would be best to use. Currently,we will use raw tcp socket send a packet's length then send the packet. >> (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then >> compare PVM's packets data with SVM's packets data. If packets is different, >> compare >> module notify COLO CheckPoint module to do a checkpoint then send PVM's >> packets to >> client and drop SVM's packets, otherwise, just send PVM's packets to client >> and >> drop SVM's packets. >> (5)notify COLO-Checkpoint module checkpoint is needed >> (6)Do COLO-Checkpoint >> >> ### QEMU space TCP/IP stack(Based on SLIRP) ### >> We need a QEMU space TCP/IP stack to help us to analysis packet. After >> looking >> into QEMU, we found that SLIRP >> >> http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29 >> >> is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it >> can >> help use to handle the packet written to/read from backend(tap) device which >> is >> just like a link layer(L2) packet. > I still think SLIRP might be painful; but it might be an easy one to start > with. > >> ### Packet enqueue and compare ### >> Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM >> and >> SVM on Primary QEMU, and then compare the packet payload for each >> connection. > Dave > > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > . >