From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39632) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zw5K4-0001PE-J8 for qemu-devel@nongnu.org; Tue, 10 Nov 2015 04:34:42 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Zw5K0-0007mc-FR for qemu-devel@nongnu.org; Tue, 10 Nov 2015 04:34:40 -0500 Received: from [59.151.112.132] (port=32413 helo=heian.cn.fujitsu.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zw5Jx-0007l2-VM for qemu-devel@nongnu.org; Tue, 10 Nov 2015 04:34:36 -0500 From: Tkid References: <56418003.9010208@cn.fujitsu.com> <56419E37.3010909@redhat.com> Message-ID: <5641BA61.8060202@cn.fujitsu.com> Date: Tue, 10 Nov 2015 17:35:29 +0800 MIME-Version: 1.0 In-Reply-To: <56419E37.3010909@redhat.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [POC]colo-proxy in qemu List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jason Wang , qemu-devel@nongnu.org, stefanha@redhat.com Cc: zhang.zhanghailiang@huawei.com, lizhijian@cn.fujitsu.com, jan.kiszka@siemens.com, eddie.dong@intel.com, dgilbert@redhat.com, peter.huangpeng@huawei.com, arei.gonglei@huawei.com, guijianfeng@cn.fujitsu.com On 11/10/2015 03:35 PM, Jason Wang wrote: > On 11/10/2015 01:26 PM, Tkid wrote: >> Hi,all >> >> We are planning to reimplement colo proxy in userspace (Here is in >> qemu) to >> cache and compare net packets.This module is one of the important >> components >> of COLO project and now it is still in early stage, so any comments and >> feedback are warmly welcomed,thanks in advance. >> >> ## Background >> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop >> Service) >> project is a high availability solution. Both Primary VM (PVM) and >> Secondary VM >> (SVM) run in parallel. They receive the same request from client, and >> generate >> responses in parallel too. If the response packets from PVM and SVM are >> identical, they are released immediately. Otherwise, a VM checkpoint >> (on demand) >> is conducted. >> Paper: >> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=3D0 >> COLO on Xen: >> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping >> COLO on Qemu/KVM: >> http://wiki.qemu.org/Features/COLO >> >> By the needs of capturing response packets from PVM and SVM and >> finding out >> whether they are identical, we introduce a new module to qemu >> networking called >> colo-proxy. >> >> This document describes the design of the colo-proxy module >> >> ## Glossary >> PVM - Primary VM, which provides services to clients. >> SVM - Secondary VM, a hot standby and replication of PVM. >> PN - Primary Node, the host which PVM runs on >> SN - Secondary Node, the host which SVM runs on >> >> ## Our Idea ## >> >> COLO-Proxy >> COLO-Proxy is a part of COLO,based on qemu net filter and it's a >> plugin for >> qemu net filter.the function keep SVM connect normal to PVM and compare >> PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. >> >> =3D=3D Workflow =3D=3D >> >> +--+ +--+ >> |PN| |SN| >> +-----------------------+ +-----------------------+ >> | +-------------------+ | | +-------------------+ | >> | | | | | | | | >> | | PVM | | | | SVM | | >> | | | | | | | | >> | +--+-^--------------+ | | +-------------^----++ | >> | | | | | | | | >> | | | +------------+ | | +-----------+ | | | >> | | | | COLO | | (socket) | | COLO | | | | >> | | | | CheckPoint +---------------------> CheckPoint| | | | >> | | | | | | (6) | | | | | | >> | | | +-----^------+ | | +-----------+ | | | >> | | | (5) | | | | | | >> | | | | | | | | | >> | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | >> | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | >> | | +-----+------+ | | +-----------------+ | | >> | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | >> | +-------------------+ | Forward(socket) | +-------------------+ | >> ++Qemu+-----------------+ ++Qemu+-----------------+ >> | ^ >> | | >> | | >> +--------v-+--------+ >> | | >> | Client | >> | | >> +-------------------+ >> >> >> (1)When PN receive client packets,PN COLO-Proxy copy and forward >> packets to >> SN COLO-Proxy. >> (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,se= nd >> adjusted packets to SVM >> (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu >> COLO-Proxy. >> (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,th= en >> compare PVM's packets data with SVM's packets data. If packets is >> different, compare >> module notify COLO CheckPoint module to do a checkpoint then send >> PVM's packets to >> client and drop SVM's packets, otherwise, just send PVM's packets to >> client and >> drop SVM's packets. >> (5)notify COLO-Checkpoint module checkpoint is needed >> (6)Do COLO-Checkpoint >> >> ### QEMU space TCP/IP stack(Based on SLIRP) ### >> We need a QEMU space TCP/IP stack to help us to analysis packet. After >> looking >> into QEMU, we found that SLIRP >> >> http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.2= 9 >> >> is a good choice for us. SLIRP proivdes a full TCP/IP stack within >> QEMU, it can >> help use to handle the packet written to/read from backend(tap) device >> which is >> just like a link layer(L2) packet. >> >> ### Packet enqueue and compare ### >> Together with QEMU space TCP/IP stack, we enqueue all packets sent by >> PVM and >> SVM on Primary QEMU, and then compare the packet payload for each >> connection. >> Thanks for review ~ > Hi: > > Just have the following questions in my mind (some has been raised in > the previous rounds of discussion without a conclusion): > > - What's the plan for management layer? The setup seems complicated so > we could not simply depend on user to do each step. (And for security > reason, qemu was usually run as unprivileged user) -We don't need to run as privileged user, colo-proxy just run like=20 filter-buffer. usage: primary: -netdev tap,id=3Dbn0 -device=20 e1000,netdev=3Dbn0 -object=20 colo-proxy,id=3Df0,netdev=3Dbn0,queue=3Dall,side=3Dprimary,host=3D3.3.3.8,p= ort=3Dxxx=20 secondary: -netdev tap,id=3Dbn0 -device e1000,netdev=3Dbn0 -object=20 colo-proxy,id=3Df0,netdev=3Dbn0,queue=3Dall,side=3Dsecondary,server=3Dtcp:x= xxx:port > - What's the plan for vhost? Userspace network in qemu is rather slow, > most user will choose vhost. colo-proxy in qemu space don't support vhost. people who want to use=20 colo must disable vhost, but virtio-net is another choice which is=20 enough in most case. > - What if application generate packet based on hwrng device? This will > produce always different packets. just like hailiang said. > - Not sure SLIRP is perfect matched for this task. As has been raised, > another method is to decouple the packet comparing from qemu. In this > way, lots of open source userspace stack could be used. -we just need the some capabilities(such as IP frag/defrag) of SLIRP=E3=80= =82=20 We have investigated some open source userspace stack,but not find one=20 better to SLIRP. if you know,please tell me. > - Haven't read the code of packet comparing, but if it needs to keep > track the state of each connection, it could be easily DOS from guest. -We think preventDOS from guest is out of our focus,it should be firewall t= o concerned. > Thanks > . >